6 November 2026 · how it works
How a local AI model works: why a chatbot fits in 8GB, what the four bits throw away, and why your laptop's speed is a plumbing number
The chatbot on your laptop came from the same download chart as the big one, at a third of the size. Nobody told you what got left out.
Here's the arithmetic that lets a model run at home, and the three facts that fall out of it.
Twelve billion dials, and the box they won't fit in
A language model consists of its weights: billions of learned numbers that pick the next word. Gemma 4 12B carries twelve billion of them1. Training stores each one in sixteen bits, two bytes. Twelve billion times two bytes makes 24GB. Your laptop holds 16GB, and the laptop wants most of that for itself.
So each number has to shrink. Quantisation stores every weight in fewer bits: eight, four, three, two2. At four bits a weight takes half a byte; twelve billion halves make 6GB, plus bookkeeping. The 4-bit file for Gemma 4 12B weighs 7 to 8GB3. So a film-sized download holds a 24GB brain with twelve of every sixteen bits cut from each number.
What four bits keeps
Picture a weight as a dial that points anywhere between minus one and one. Sixteen bits give the dial tens of thousands of positions; four bits give it sixteen2. How much of the model's behaviour ever lived in the fine positions?
Measured: surprisingly little, until suddenly all of it. Huang and colleagues took Llama 3 8B and tested it at every bit-width4. At sixteen bits it scored 68.6 on a set of common-sense questions. At eight bits, 68.6. At four bits, 67.3 to 68.2 depending on the method. At three bits, 61.7 to 64.4. At two bits it fell to 35 or 36, close to guessing4. The curve runs flat, then drops off a cliff, and the cliff sits between four bits and three.
Which explains why the Hugging Face guide calls Q4 "the best starting point"5 and the GGUF guide calls Q4_K_M "the most popular default"2. At four bits you've thrown away most of the file and almost none of the model.
How the throwing-away gets done
Not by rounding each number to the nearest of sixteen. That naive method, called RTN, collapsed to a score of 40 at three bits, where the clever methods held 62 to 644.
The clever methods work in blocks. A block of weights shares one scale factor stored at full precision, and each weight gets stored as a four-bit fraction of it2. The "K" in Q4_K_M groups the blocks into super-blocks with scales of their own, so the scales get quantised too. An importance matrix, built by running real text through the model first, tells the quantiser which weights carry the load and spends bits there2. Q4_K_M lands at about 4.5 bits per weight2: four for the numbers, half a bit for the bookkeeping that keeps them honest.
So the 8GB download doesn't hold a worse model. It holds the same model under a lossy compression scheme, tested against the original and tuned until the gap stopped showing on the benchmarks. Whether the gap shows on your task, the practitioner below will tell you.
Why the speed comes down to plumbing
Here's the fact that catches people who assume AI runs on computing power. To write one word, the model reads every weight once. All 8GB. Then it writes the next word and reads all 8GB again. Generation runs memory-bound: the processor spends most of its time waiting for weights to arrive, not calculating with them6.
So the ceiling on words per second comes from memory bandwidth divided by bytes read per word7. A MacBook Air M4 moves 120GB a second between memory and chip8. The arithmetic: 120GB per second divided by 8GB per word gives 15 words a second, flat out. Sustained bandwidth runs at 80 to 90% of peak7, so call it 12 or 13 in practice. Reading speed. Halve the file to 4GB and you double the speed. Double it to 16GB and it stops fitting, and a model that spills out of memory crawls3.
That's why quantisation buys speed as well as space. It's also why the spec that matters never appears in the advert: not the processor, not the GPU core count, but gigabytes per second between memory and chip. The 24GB model you couldn't fit was also the model you couldn't afford to read.
Why a model forty times bigger fits too
One more trick, and the labs in Hangzhou and Beijing lean on it hardest. A mixture-of-experts model splits its weights into specialist blocks and consults only a few per word. DeepSeek V4.1 Flash holds 552 billion weights and reads 16 billion of them for each word it writes1. Enormous on disk, light on the plumbing. Only the active experts get read, so a huge model with weights spilling into ordinary RAM stays usable where a dense model of the same size would not7. That's how a model that once needed a rack now runs on a Mac Studio.
What this rearranges
Three things follow.
First, the expensive part of a model never lived in the software. The value sits in the numbers, the numbers sit in a file, and files get copied. Everything the labs charge for by the month becomes, on your disk, a one-off download that doesn't expire.
Second, the labs no longer decide what you can run. The chip and memory makers who set bandwidth decide, and so do the quantisers, usually not the lab that trained the model, who choose which weights carry the load. The importance matrix that shaped your 8GB file was calibrated on text somebody picked2. Nobody tells you which text.
Third, you can see the floor. Four bits holds, three wobbles, two fails. Twelve billion weights at four bits, on a laptop with 16GB, read back at a dozen words a second. Not a demo: a working assistant with no account, for the price of the electricity.
What to do with this: when you download, take the Q4_K_M, not the smaller one that looks like a bargain. Check the file size against your free memory, not your total. And when a laptop salesman starts on the processor, ask him for the bandwidth. He'll have to look it up.
The number they never showed you sets the speed.