Most of the talk about shrinking language models centers on 4-bit and 8-bit quantization. Ternarisation goes much further. It cuts every weight in a model down to one of three values: -1, 0, or +1.

That idea has been circulating in AI research for a while, most visibly through Microsoft Research's BitNet b1.58 work. It gets especially interesting at the scale of a model with around 1.85 billion parameters. A model that size is small enough to experiment with on modest hardware, but large enough to be useful for real tasks.

What ternarisation means

In a standard model, each weight is stored as a 16-bit or 32-bit floating-point number. Ternarisation replaces each weight with -1, 0, or +1, usually paired with a scaling factor that is shared across a group of weights or a whole layer.

Read More