By Web Asset House · data reviewed 2026-10-22
“70B” describes a parameter scale, not a fixed memory requirement. Quantization changes weight storage, and attention architecture changes the KV-cache footprint. A complete estimate uses the selected repository files and the model’s configuration rather than multiplying the name by a single universal factor.
Recorded examples from this dataset
| Model | Quantization | Weights (GiB) | Record |
|---|---|---|---|
| Llama-3.3-70B-Instruct | Q4_K_M | 39.60 | File listing |
| Llama-3.3-70B-Instruct | Q5_K_M | 46.52 | File listing |
| Llama-3.1-70B-Instruct | Q4_K_M | 39.60 | File listing |
| Llama-3.1-70B-Instruct | Q5_K_M | 46.52 | File listing |
| DeepSeek-R1-Distill-Llama-70B | Q4_K_M | 39.60 | File listing |
| DeepSeek-R1-Distill-Llama-70B | Q5_K_M | 46.52 | File listing |
Context is a memory commitment
For a supported standard attention configuration, the calculator uses two caches, one for keys and one for values, multiplied by layer count, KV-head count, head dimension, context tokens, and bytes per cache element. One sequence and 16-bit cache elements are the defaults. Doubling context doubles this estimated cache allocation.
Required bytes = weight file bytes + (2 × layers × KV heads × head dimension × context × KV bytes) + runtime overhead
Why a nominal 48 GB setup may still be tight
Advertised capacity is not all available to inference. The operating system, driver, display, other applications, and runtime need memory too. Quantization metadata, model format, and allocator behavior also matter. Do not plan around a zero-headroom result. The calculator calls a result borderline above 90% of its usable budget.
Expert models are not smaller weight files
A mixture-of-experts model may activate only part of its parameter set for a token, but all required expert weights still need storage. We use active parameters only in the speed proxy, not to shrink the memory requirement.
Primary sources
- Ollama: memory, context, local and cloud behavior
- llama.cpp: supported build backends
- Hugging Face: GGUF documentation
Questions, answered.
Can 70B run on 32 GB?
It depends on the exact quantization, file size, context, and runtime. Use recorded files and a full memory estimate; a parameter-count shortcut is insufficient.
Do inactive experts take no memory?
No. Active parameter count is relevant to an approximate per-token compute and bandwidth model, but all required weights still count toward memory.