Quantization

Storing model weights at lower numeric precision to cut memory and cost, with a small quality trade-off.

Quantization shrinks each weight from, say, 16 bits to 8 or 4. The model gets dramatically smaller and faster with usually modest degradation, which is what allows capable models to run on a laptop or a phone. How much quality you lose depends on the method and how aggressive you get.

In practice: A 70B model compressed to run on a single consumer GPU.