Local models in 2026: what fits on your laptop

The question "how many billions of parameters fit in my memory?" has an easy, wrong answer.

Wrong because three things compete for that memory, not one. The model weights are only the first. The attention cache is the second, and it grows with context size. The third is the rest: runtime, operating system, activations.

Almost every guide calculates the first line and stops. That is why so many people download a model that "fits", open a large file and watch the machine seize.

This post does all three sums. With the formula in the open, validated against a published example, and applied to real 2026 models. It is the last post in the cluster on harness engineering, and the only one where the constraint is something other than money. It is memory.

The error starts with a one-line sum: multiply parameters by bytes and declare that it fits.

That sum ignores the attention cache, which in 2026 contexts can exceed the size of the model. And it ignores that the system needs memory too.

A second problem exists, cruder, and naming it pays off because it affects anyone researching right now. A good share of the "best local models" guides published this year list models that do not exist. Combinations show up such as "Llama 3.3 8B" (the 3.3 family came out in 70B alone), "Mistral Small 3 7B" (Mistral Small 3 is 24B) and "Qwen 3 7B" (the family has 8B, not 7B).

Those are texts assembled with no spec sheet checked. Anyone following those lists picks hardware for an imaginary model.

The defense is simple and it holds for this whole post: check the model card. Parameters, layer count and attention heads are public. Those numbers are what makes the arithmetic close.

The budget has three lines, and only the first stays put.

The weights are the product of total parameters and the bytes per parameter of the chosen quantization. That line stays put during use.

The attention cache holds the keys and values already computed, so nothing gets recomputed on each new token. It grows in a straight line with the context. It is the line that surprises.

The overhead is what remains: the runtime, the system, the activation buffers. Reserving 2 GB is a reasonable floor on a general-purpose machine.

Anyone who manages what enters the context pays less on the last two lines. It is the same problem covered in why your agent forgets, seen now from the RAM side.

In 2026, the number deciding whether the model fits stopped being the number deciding how fast it runs.

Mixture-of-experts models activate only a fraction of the weights per token. Qwen3.6-35B-A3B has 35 billion parameters and activates 3 billion. It computes at the speed of a 3B model. But the whole 35B has to sit in memory, because any expert can get called on the next token.

Fast and fat. That is the combination that confuses.

The naming made the confusion worse. Gemma 4, launched by Google on 2 April 2026 under the Apache 2.0 license, has four sizes: E2B, E4B, 26B MoE and 31B dense. The "E" stands for effective, meaning parameters activated at inference rather than the size of the model (Google's blog, read 28 August 2026). The 26B MoE activates 3.8B of its 26B.

The name announces the speed and omits what has to fit in RAM.

The practical consequence for agents deserves recording: a small model with many tools declared spends a good part of the window before doing any work. The subject sits in tool design for agents, and it weighs more on a local model, where the window is tight.

The cache depends on the architecture, not on the size of the model. The formula:

cache = 2 × full_attention_layers × kv_heads × head_dim × tokens × bytes

The leading 2 exists because both key and value get stored. The bytes are 2 at 16-bit precision.

The formula checks out. Applied to an example published by an independent source, at 80 layers, 8 key-value heads, dimension 128 and 131,072 tokens, it returns 42.9 GB, matching the value the source reports to the decimal.

Now the real case. The Qwen3.6-35B-A3B model card publishes the architecture: 40 layers, in the arrangement 10 × (3 × Gated DeltaNet → 1 × Gated Attention). The full attention blocks have 16 query heads, 2 key-value heads and dimension 256.

The detail that changes everything: of the 40 layers, only 10 use full attention. The other 30 use DeltaNet, a form of linear attention that accumulates no cache proportional to the context.

Putting that in the formula, and comparing against a classic dense architecture:

A caveat of honesty: the right column is a constructed comparison, not a real measured model. It uses 40 full attention layers with 8 key-value heads and dimension 128, a typical configuration of earlier dense models. It serves to isolate the effect of the architecture, holding the layer count equal.

The result is large anyway: the same context costs eight times less memory.

The practical implication of long context on a small machine appears in what the measurements say about long context. Here the point comes earlier: without that architecture, the context would fail to fit at all.

And if it does not fit? Three ways out, in order of cost

The first way out is the cheapest and the most ignored: use less context. The cache grows in a straight line, so halving the context halves the cache. Most local tasks need nothing near 262 thousand tokens. Configure the runtime's limit to what the task uses in practice, instead of leaving the model's default.

The second is quantizing the cache, and not the weights alone. Storing keys and values at 8 bits instead of 16 halves that line. That 42.9 GB example falls to around 21.5 GB at FP8. Four-bit formats bring it near 10.7 GB, on hardware that supports them. It is a runtime option, and most people leave it off by default.

The third is switching models for the architecture, not for the size. Between two models of similar heft, the one with fewer full attention layers will cost far less memory at the context you intend to use. That criterion appears in no comparison table, and it decides more than the parameter count more often than not.

Note the order. The first two you apply today, on the machine you already have. The third requires downloading another model.

This is the reading that runs against the common sense of the subject.

Quantization shrinks the weights. It does close to nothing for the attention cache, which stays computed at 16 bits in the default configuration of most runtimes.

Anyone who bought memory believing Q4 would solve long context bought for the wrong reason. Going from FP16 to Q4 cuts the weights by four. The cache stays the same.

What made a 262 thousand token context viable on a laptop was swapping most of the full attention layers for linear attention. It is an architecture decision, taken by whoever trained the model. Applying it afterward is impossible.

The weights, for 35 billion total parameters:

The last two values are approximations. The k-quant schemes use different bit counts across layers, and vocabulary size plays a part, since this model's is large, at 248,320 entries. Check the file size before closing the sum.

The consensus says Q4 costs little. The perplexity ranges circulating in community guides hold that up: something around 0.1% to 0.3% increase at Q8, 0.5% to 1.5% at Q6, and 1.5% to 3% at Q4.

Except that perplexity is an instrument blind to what matters here.

A 2026 work measured that with statistical method. The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs, by Baha Rababah, Shahzeb Qamar, Lorenz Sparrenberg, Rafet Sifa, Murat Kantarcioglu, Cuneyt Gurcan Akcora and Carson K. Leung, tested widths from 8 to 2 bits across several models (arXiv 2607.08734, version 2 of August 2026).

The conclusion is direct: quantized variants are not equivalent to the base model. Per the authors, base and quantized variant tend to show a behavior change even where accuracy and perplexity hold steady.

That repositions the standard advice. Choosing quantization by a perplexity table is choosing by a metric that, by construction, misses the difference that will affect you.

The task itself is what remains to test on. If the local model will review code, write technical Portuguese or operate tools, that is what it needs comparing on, before and after quantizing. The method sits in evaluating an agent on the real task, and it holds the same for a local model.

Switching runtimes changes the speed more than switching quantization does.

On Apple Silicon two main options exist: MLX, Apple's own framework, and llama.cpp, which Ollama used as its base. Ollama migrated to MLX on 30 March 2026.

The measured numbers, and the reason they look contradictory:

Different sources cite "3× faster" and "5% to 15% faster" for the same comparison. Both are right, and they measure different things: MLX's advantage reaches 3× on MoE models and lands at 1.4× to 1.6× on dense ones. The table's last row shows that on older hardware the advantage can disappear outright.

One limitation to declare: apart from Ollama's migration date, these numbers come from technical material that publishes neither software versions nor measurement dates. Treat them as an order of magnitude, not as a reference. The conclusion that survives is a methodological one: measure your case, on your hardware, with both runtimes.

The year concentrated launches, and almost all of them under a permissive license.

Dates and licenses checked against the official announcements and the open weight release timeline, on 28 August 2026. That table is a snapshot and it ages fast. The formula from the earlier sections does not age.

Note the DeepSeek row. V4-Pro carries 1.6 trillion total parameters, ships open under the MIT license and runs on no laptop in any quantization. Open means something other than local. They are two independent properties, and conflating them is the origin of a good share of the frustration with open models.

Google itself records the upper limit in a useful way: the 31B Gemma 4, in bf16 and unquantized, fits on a single 80 GB H100. That is the size of machine in question when nobody quantizes anything.

Summing the three lines, with the context you use in practice and a reserve for the system:

Apple Silicon's structural advantage here is capacity rather than speed: the CPU and the GPU share the unified memory, so the model ceiling is the machine's RAM ceiling. On a dedicated card, the ceiling is the card's VRAM.

Three arguments tend to get used. Only two hold up.

Zero cost per token. True, and incomplete: you trade variable cost for a fixed hardware cost, plus your own operating time. The break-even depends on volume, and the method for calculating it sits in where the money leaks in production. For individual use on a subscription plan, the arithmetic changes again, and it sits in what each usage pattern costs.

Data that never leaves the machine. It is the most solid argument, and the only one no API price cut neutralizes. Proprietary code, customer data and confidential material carry a leak cost that appears in no token table. Worth remembering that running local reduces the data's exposure, and eliminates none of the manipulation risks of the model itself, covered in prompt injection.

Equivalent quality. This is the weak argument. The best open models of 2026 are good, and the best of them fail to fit on your laptop. The tier below is what fits, and for long agentic work the difference shows.

<!-- [PERSONAL EXPERIENCE] -->

These posts get written without a local model, and explaining why pays off, because the reason is not the one I expected.

I tested the hypothesis of running the cluster's research stage on a local model. The idea looked good: it is the stage that consumes the most context, and cutting that cost would have a real effect. Writing quality had nothing to do with why I gave up. It was the verification stage.

This cluster's work depends on reading a primary source and deciding whether a number checks out. That means long context with material published days earlier, often postdating any model's training, plus disciplined tool use for searching and checking. It is the precise combination where the difference between the top model and the one that fits on a laptop shows most.

The explicit judgment, then: for drafting, rewriting and transforming text you will review yourself, a local model in 2026 is good enough and the saving is real. For verification work, where a subtle error slips through and goes live, I do not use one.

And one useful irony sits in this post's arithmetic. The model I would run on my own machine fits in memory with room to spare. The context the task needs is what failed to fit alongside it.

How much RAM do I need to run a local LLM?

It depends on three lines, not one. Weights (total parameters × bytes of the quantization), attention cache (which grows with the context) and around 2 GB of system and runtime. With 8 GB you run models from 2B to 4B at Q4 with a short context. With 16 GB, a 9B at Q4 runs with room to spare.

Which model fits in 16 GB?

A 9B model at Q4 takes close to 5 GB of weights and leaves comfortable room for context. A 27B dense at Q4 lands around 15 GB and fits only under aggressive quantization and a short context, which brings a quality loss. The 16 GB tier is comfortable up to 9B.

Does Q4 lose quality for real?

It loses more than perplexity indicates. The usual ranges point to a 1.5% to 3% perplexity increase at Q4, but the paper The Illusion of Equivalency (arXiv 2607.08734, August 2026) shows quantized models presenting a behavior change even where accuracy and perplexity hold steady. Test on your task.

Does an MoE model run faster?

Yes, and it takes up the same space as before. A 35B-A3B activates 3 billion parameters per token, so it computes at the speed of a 3B model. But the 35 billion have to sit in memory, because any expert can get triggered on the next token.

Is a Mac with unified memory better than a PC with a dedicated GPU?

For fitting a large model, the Mac has the advantage: the CPU and the GPU share the unified memory, so the ceiling is the machine's RAM, which reaches 128 GB. On a consumer dedicated GPU, the ceiling is the card's VRAM. For raw speed on models that fit on both, the dedicated GPU tends to win.

Does an open model mean it runs on my computer?

No. They are independent properties. DeepSeek V4-Pro ships open under the MIT license and carries 1.6 trillion total parameters: it runs on no laptop, in no quantization. Open means the weights are available, not that your hardware can handle them.

• The memory has three lines. Calculating the weights alone is the origin of the most common error.
• Total parameters decide whether it fits. Active parameters decide the speed. On MoE, the two numbers are far apart.
• The attention cache depends on the architecture, not on the size. Only the full attention layers count.
• Long context on a laptop is the merit of the hybrid architecture, not of the quantization.
• Quantizing changes the behavior even with the metric preserved. Validate on your task.
• Open is not local. Open models exist that no laptop runs.

This post's catalog is a snapshot of 28 August 2026 and it will age in weeks. The attention cache formula will not. Keep the formula, and recalculate when the next model comes out.