Bonsai 27B brings giant models to phones and laptops
PrismML has shrunk Qwen3.6-27B to the point where it fits in a phone’s app memory. The new Bonsai 27B line ships two variants: a 1.71-bit ternary model at 5.9 GB and a 1.125-bit binary model at 3.9 GB. Neither requires retraining; both preserve the original architecture and Apache 2.0 license.
How the compression works
Each weight is encoded as a small integer, scaled by a shared FP16 factor every 128 elements. Ternary weights use –1, 0, +1; binary uses –1, +1. The shared scale adds only 16 bits per group, so the effective bitrate is 1.71 bits for ternary and 1.125 bits for binary—roughly 9× and 14× lighter than FP16. Vision, embeddings, attention projections, MLP layers and the LM head all use the same scheme; normalization tails remain higher precision.
Accuracy vs. footprint
On 15 benchmarks in “thinking mode,” ternary retains 94.6 % of the FP16 baseline, while binary keeps 89.5 %. Short-form scores can hide collapse: a conventional 2-bit build collapses on AIME26 (57.5) and LiveCodeBench (56.4), yet still scores 88.93 on MMLU-Redux. Bonsai avoids that pattern by keeping the original pretrain intact.
Why edge GPUs matter
Memory, not compute, is the binding constraint on phones. iOS caps a single app to about half of DRAM, so a 12 GB iPhone exposes ~6 GB. At 262 K tokens, the KV cache alone can balloon: FP16 KV cache is ~17.2 GB, 4-bit KV cache ~4.3 GB. Even then, ternary peaks at 14.7 GB and binary at 11.6 GB—still above the limit. PrismML’s low-bit weights plus 4-bit KV cache bring both variants under the ceiling, enabling local inference where it was previously impossible.
Why it matters
Bonsai 27B shows that extreme quantization no longer requires expensive retraining. By preserving the original model and compressing end-to-end, PrismML turns a 54 GB LLM into one that can run on a handset—opening on-device agents, private chatbots and real-time multimodal apps that were out of reach yesterday. The trade-off is measurable: binary drops 5–10 percentage points on complex tasks, but the gain in accessibility is immediate.
Source: MarkTechPost. AI-assisted editorial synthesis — TechnoExpress.