Which LLMs and Quants Fit in 32 GB? The Numbers, Measured

The short answer: a 32B model at Q4_K_M with 32K of context, or a 14B at Q8_0 if you want no quantisation damage at all. A 70B technically loads, but only below three bits per weight, where it is measurably a worse model than the 32B you could have run instead.

The longer answer depends on something the question does not specify, and getting it wrong is the most common reason people’s plans fail before they start.

First: which kind of 32 GB do you have?

This is the question the search term leaves open, and the three answers are not interchangeable.

Three machines that all advertise 32 GB, with three different usable budgets.

If you have a 32 GB graphics card — a 5090, an A6000, an MI60 — all 32 GB is available to the model, minus about a gigabyte of runtime overhead. Everything in this guide applies directly.

If you have a 32 GB Mac, you do not have 32 GB. macOS reserves memory for itself and caps how much the GPU may claim. Plan around roughly 24 GB unless you deliberately raise the limit, and read every table below one row further down than you would like.

If you have 32 GB of system RAM and no GPU, everything still loads. It just runs on the CPU, where memory bandwidth is roughly an order of magnitude lower than a modern graphics card. The model that generates comfortably on a 5090 will feel like waiting for a fax. Capacity is not your constraint; speed is.

Why “Q4” does not mean four bits

Most guides size models by multiplying the parameter count by the nominal bit width. Q4 becomes 0.5 GB per billion parameters, so a 70B lands at 35 GB. That number is wrong, and it is wrong in the direction that breaks your setup.

We measured it instead. Taking published GGUF file sizes and dividing by the actual parameter count gives the true bits per weight for each quantisation level.

Nominal versus measured bits per weight, from real files across three model sizes.

Q4_K_M lands at 4.82 to 4.90 bits per weight across three different model sizes — consistently about 22% heavier than its name. Q3_K_M is nearly 4 bits. Only FP16 matches its label, at 16.01, which is the sanity check that tells you the method is sound.

The reason is that K-quants do not quantise everything to the same width. Token embeddings and certain attention layers are kept at higher precision because degrading them costs disproportionate quality. The name describes the dominant width, not the average.

The practical consequence: estimate Llama 3.3 70B at Q4 using the nominal four bits and you get 35 GB. The real file is 42.52 GB. That is the difference between planning for a 40 GB card and a failed load.

What actually fits in 32 GB

Weights are only the first number. The KV cache grows linearly with your context length and lives in the same memory, so the real question is not “does the model fit” but “does the model fit with enough room left to hold a conversation”.

What fits in 32 GB, with the maximum context each configuration leaves room for.

The 70B question, answered honestly

People search this term because they want to know whether 32 GB gets them a 70B. It does, technically, in a narrow band:

Below about three bits per weight, quantisation stops being a lossless-ish compression and starts removing capability. A 70B at IQ2_M is not a 70B with slightly rounded numbers; it is a model that has lost a meaningful amount of what made it worth running.

Meanwhile Qwen2.5 32B at Q4_K_M sits at 19.85 GB with 11 GB free, running at a quantisation level where damage is minimal. For most work that is the better machine, and it is not close.

How the KV cache eats your context

The cache formula is arithmetic, not a rule of thumb:

KV bytes = 2 × layers × kv_heads × head_dim × context × bytes_per_element

The leading 2 is because both Keys and Values are stored. Every other value is in the model’s config.json on Hugging Face. bytes_per_element is 2 for an FP16 cache and 1 for Q8.

For a 32B with 64 layers and 8 KV heads, that is 1.07 GB at 4K context and 8.59 GB at 32K. For a 70B with 80 layers, 1.34 GB and 10.74 GB. The cache is not a rounding error — at long context it rivals the weights.

The setting that doubles your context for free

Most runtimes will store the cache at eight bits instead of sixteen. That halves every cache figure at a quality cost most people cannot detect in normal use. In llama.cpp:

–cache-type-k q8_0 –cache-type-v q8_0

Look at the difference in the fits table. Qwen2.5 32B at Q4_K_M goes from 32K context to 64K. Llama 3.3 70B at IQ2_M goes from 16K to 32K. If you are short on context rather than short on weights, this is the first thing to change.

So what should you actually run?

Four recommendations for 32 GB, depending on what you are optimising for.

The uncomfortable recommendation is the last one. If you came here to run a 70B, the honest advice is to try Qwen2.5 32B at Q4_K_M first and see whether you can tell the difference. Many people find they cannot, and the 32B gives them 32K of context instead of 16K.

The rule you can apply to any model

Nothing above depends on these specific models. The arithmetic works for anything you download:

total GB ≈ (params_B × bpw ÷ 8)

          + (2 × layers × kv_heads × head_dim × ctx × bytes ÷ 1e9)

          + ~1 GB overhead

Use 0.61 GB per billion parameters for Q4_K_M, 0.71 for Q5_K_M, 0.82 for Q6_K and 1.06 for Q8_0 — the measured figures, not the nominal ones. Read layers, kv_heads and head_dim from config.json. That is the whole method, and it does not expire when the next model ships.

Frequently asked questions

Can I run a 70B on 32 GB?

Yes, at IQ2 or Q2 quantisation — between 19.10 GB and 26.38 GB of weights. Q3_K_M at 34.27 GB does not fit. Below three bits per weight the quality loss is real, and a 32B at Q4_K_M will usually serve you better.

What is the best model for 32 GB VRAM?

Qwen2.5 32B at Q4_K_M for general use: 19.85 GB of weights with 11 GB left for cache, giving 32K context. Qwen2.5 14B at Q8_0 if you want effectively no quantisation damage.

Is 32 GB of Mac unified memory the same as 32 GB of VRAM?

No. macOS reserves a portion for the system and caps GPU allocation, so plan around roughly 24 GB of usable model memory unless you raise the limit yourself.

How much VRAM does Q4_K_M actually use per billion parameters?

0.603 to 0.613 GB, measured from real GGUF files. The commonly quoted 0.5 GB assumes exactly four bits, which K-quants never hit.

Does the KV cache count toward VRAM?

Yes, and it grows linearly with context. At 32K a 32B model’s cache is 8.59 GB in FP16 — comparable to a whole small model.

Should I use Q4_K_M or Q5_K_M?

On 32 GB with a 32B model, Q4_K_M leaves room for 32K context and Q5_K_M leaves room for 16K. If you need long context, take Q4_K_M. If your prompts are short, Q5_K_M is the better model.

Can I run models on 32 GB of system RAM without a GPU?

Yes. Everything in the fits table still loads. Generation will be far slower because CPU memory bandwidth is roughly an order of magnitude below a modern GPU.

What is IQ2 and is it worth using?

IQ quantisations use importance-matrix weighting to squeeze below three bits. They make a 70B fit in 32 GB. Whether it is worth it depends on whether a damaged 70B beats an intact 32B for your work — usually it does not.

Related guides

• How to run LLMs locally
• Ollama troubleshooting
• The best Claude model for coding
• Claude Code alternatives
• Local coding models benchmark
• How to install Claude Code

Methodology and sources

Weights were measured by taking published GGUF file sizes from Hugging Face and dividing by each model’s parameter count, rather than assuming the nominal bit width. Architecture values — layer count, KV head count, head dimension — were read from each model’s config.json. Overhead is approximated at 1 GB and varies by runtime.

• bartowski/Llama-3.3-70B-Instruct-GGUF
• bartowski/Qwen2.5-32B-Instruct-GGUF
• Qwen2.5-32B-Instruct config.json
• Llama-3.3-70B-Instruct config.json

Figures verified 9 September 2026. File sizes change when repositories are re-quantised; if a number here disagrees with what you download, the file is the truth and we want to know. Vibe Coder Journal accepts no sponsorship or affiliate commission.