Ternarising a 1.85B-Parameter Language Model: What It Takes

Most of the talk about shrinking language models centers on 4-bit and 8-bit quantization. Ternarisation goes much further. It cuts every weight in a model down to one of three values: -1, 0, or +1.

That idea has been circulating in AI research for a while, most visibly through Microsoft Research's BitNet b1.58 work. It gets especially interesting at the scale of a model with around 1.85 billion parameters. A model that size is small enough to experiment with on modest hardware, but large enough to be useful for real tasks.

What ternarisation means

In a standard model, each weight is stored as a 16-bit or 32-bit floating-point number. Ternarisation replaces each weight with -1, 0, or +1, usually paired with a scaling factor that is shared across a group of weights or a whole layer.

Three possible values need about 1.58 bits of information per weight, since log2(3) is roughly 1.58. That figure is where the "b1.58" label in BitNet comes from.

The zero matters. A binary model, which only allows -1 and +1, has to give every weight some influence. A ternary model can switch individual connections off entirely, which gives it more room to represent what it has learned.

The math at 1.85 billion parameters

The storage savings are easy to work out. At 16-bit precision, 1.85 billion parameters take up about 3.7 GB. At a theoretical 1.58 bits per weight, the same parameters would need roughly 365 MB.

Real implementations don't usually hit that floor. Packing each weight into 2 bits is simpler for hardware to handle, and it puts the weights at about 460 MB. Many ternary designs also keep some parts of the model, such as the embedding layer, at higher precision. So the final file ends up somewhat larger.

Even so, that's a model that would normally need a few gigabytes of memory shrinking to well under one gigabyte. It fits comfortably on phones, older laptops, and small single-board computers.

There is one catch. Ternarisation shrinks the weights, not the memory a model uses while it is working. The key-value cache that grows with the size of the context window still takes up space, and long prompts can eat into the savings.

Why the compute side matters too

Memory is only half the story. When weights are limited to -1, 0, and +1, the matrix multiplications at the heart of a transformer no longer need real multiplication. Multiplying by +1 is just adding, multiplying by -1 is subtracting, and multiplying by 0 means skipping the value entirely.

That opens the door to simpler, more energy-efficient hardware. On ordinary CPUs, it can also mean faster inference, provided the software uses kernels built for ternary math. Generic code that unpacks ternary weights back into floating point gives up much of the advantage.

Tooling has started to catch up. The llama.cpp project, for example, has added support for ternary weight formats. That matters for anyone who wants to run these models locally instead of relying on a cloud API.

Training versus converting

There are two broad ways to get a ternary model, and they produce very different results.

The first is to take a finished full-precision model and round its weights to three values after training. This is cheap, but it tends to damage quality badly. A model that learned to rely on fine-grained weight values doesn't cope well when most of that detail disappears at once.

The second is to build ternary weights into training itself, often called quantization-aware training. The model keeps higher-precision weights in the background during training, but its forward pass uses the ternary versions, so it learns to work within that limit. BitNet b1.58 took this route, training its models with ternary weights from the start.

Middle-ground approaches exist as well. One is to start from an existing model and continue training it with ternary weights, sometimes with a larger full-precision model acting as a teacher through distillation. These methods try to capture most of the quality of from-scratch training at a fraction of the cost.

Where the limits are

Ternary models aren't a free lunch. At a given parameter count, they generally don't match a full-precision model of the same size on every task. The comparison that matters more is against models with a similar memory footprint. On that measure, a ternary model with 1.85 billion parameters is up against full-precision models that are several times smaller.

Hardware is the other open question. Most GPUs and AI accelerators on the market today are built around floating-point and low-bit integer math, not ternary operations. Until chips are designed with this format in mind, a lot of the theoretical efficiency stays on paper.

Why it matters

For developers building on open-weight models, ternarisation is another way to trade a bit of accuracy for a lot of portability. A model in the 1.85-billion-parameter range that fits in a few hundred megabytes could run on devices that have never been able to host a language model at all.

It also fits a wider push to run capable AI without a data center, the same trend behind projects that squeeze large models onto consumer graphics cards. Whether ternary models become mainstream will depend on training recipes getting cheaper and hardware learning to take advantage of them.