LLM VRAM Calculator
Estimate how much GPU or system memory a large language model needs to run locally. Pick a model preset or enter its size, choose a quantization and a context length, and see the weights, KV cache and overhead, with every assumption visible and editable.
Result
Estimate
Enter the number of parameters to see the estimate.
- Model weights
- KV cache
- Runtime overhead
- Estimated total
- Recommended (with headroom)
Weights and KV cache follow exact formulas applied to your numbers. The effective bits of a quantization format, the runtime overhead and the headroom are assumptions: real memory use depends on the runtime and its version.
Examples
Worked examples (runtime overhead 1 GiB, KV cache in FP16):
| Model | Quantization · context | Weights | KV cache | Total |
|---|---|---|---|---|
| Llama 3.1 8B | Q4_K_M · 8,192 | 4.56 GiB | 1 GiB | 6.56 GiB |
| Llama 3.1 70B | Q4_K_M · 8,192 | 39.88 GiB | 2.5 GiB | 43.38 GiB |
| Qwen2.5 7B Instruct | FP16 · 32,768 | 14.17 GiB | 1.75 GiB | 16.92 GiB |
Reference tables: LLM memory by model and format
Weight storage is estimated from parameters × bits per weight ÷ 8; KV cache and runtime overhead are additional.
Both tables assume: context 8,192 tokens; sequences 1; KV cache precision FP16; runtime overhead 1 GiB. They leave out headroom: the calculator’s default headroom is 10%, so add it if you want a recommended figure.
All values are in GiB (one GiB is 1,073,741,824 bytes), shown to two decimal places. The models listed are the calculator’s presets.
| Model | FP16 | Q8_0 | Q4_K_M |
|---|---|---|---|
| Llama 3.1 8B | 14.90 | 7.92 | 4.56 |
| Llama 3.1 70B | 130.39 | 69.27 | 39.88 |
| Llama 3.1 405B | 754.37 | 400.80 | 230.76 |
| Qwen2.5 7B Instruct | 14.17 | 7.53 | 4.34 |
| Qwen2.5 14B Instruct | 27.38 | 14.55 | 8.38 |
| Qwen2.5 32B Instruct | 60.54 | 32.16 | 18.52 |
| Qwen3 8B | 15.27 | 8.11 | 4.67 |
Excludes KV cache, runtime overhead, activations and headroom.
| Model | KV cache | Total FP16 | Total Q8_0 | Total Q4_K_M |
|---|---|---|---|---|
| Llama 3.1 8B | 1.00 | 16.90 | 9.92 | 6.56 |
| Llama 3.1 70B | 2.50 | 133.89 | 72.77 | 43.38 |
| Llama 3.1 405B | 3.94 | 759.31 | 405.73 | 235.70 |
| Qwen2.5 7B Instruct | 0.44 | 15.61 | 8.97 | 5.77 |
| Qwen2.5 14B Instruct | 1.50 | 29.88 | 17.05 | 10.88 |
| Qwen2.5 32B Instruct | 2.00 | 63.54 | 35.16 | 21.52 |
| Qwen3 8B | 1.13 | 17.40 | 10.24 | 6.80 |
Includes the KV cache and runtime overhead stated above. Excludes headroom, activation peaks and runtime allocations that are not modelled.
What these tables do not tell you
- These are formula-based estimates, not measured benchmarks. A configuration that fits on paper may still not run in a particular runtime.
- The Q8_0 and Q4_K_M sizes use the effective bits per weight published for llama.cpp’s GGUF formats, measured on one model (Llama-3.1-8B); actual files can vary.
- Totals are estimates before runtime-specific placement. They do not mean that all memory behaves as one pool, or that a listed model fits on one GPU; offloading and multi-GPU placement are not modelled.
- The tables list dense models. Mixture-of-experts models need the total parameter count and separate treatment of where the experts are stored.
- Models with sliding-window or latent attention can use much less KV cache than the standard formula used here.
Formula & Methodology
The calculator adds three parts:
weights = parameters × bits per weight ÷ 8KV cache = 2 × layers × KV heads × head dimension × tokens × bytes per value × sequencestotal = weights + KV cache + runtime overheadrecommended = total × (1 + headroom ÷ 100)What this estimate does not cover
- Mixture-of-experts models must load every expert, so use the total parameter count, not the “active” count.
- Models with multi-head latent attention, sliding-window or hybrid attention can use much less KV cache than the standard formula.
- Prompt processing can briefly need more memory than steady-state generation, and each runtime adds its own buffers.
- GB (10^9 bytes) and GiB (2^30 bytes) differ by about 7%: GPUs and RAM sold as “24 GB” are normally 24 GiB.
Method and references
These primary sources support the methods this tool relies on. Having read them does not verify every result the calculator produces, and figures listed as assumptions are ToolQuarter’s own.
Weights and KV cache follow arithmetic on the numbers you enter. The runtime overhead and headroom are ToolQuarter assumptions, and real memory use depends on the runtime.
- llama.cpp: quantize README: Source of the bits per weight of the GGUF quantization formats (for example Q4_K_M 4.8944), measured on Llama-3.1-8B. Other models and files vary. ()
- PagedAttention paper (arXiv:2309.06180v1): Gives the KV cache per token as 2 (keys and values) × hidden size × layers × bytes per value, with a worked example of 800 KB per token for a 13B model in FP16. This tool writes the same product with KV heads and head dimension, so it also covers models that share KV heads. ()
- Mixtral of Experts paper (arXiv:2401.04088v1): Defines total and active parameters for a mixture-of-experts model: each token has access to 47B parameters but uses 13B. The paper does not discuss memory. ()
- llama.cpp: server README: Documents --cpu-moe and --n-cpu-moe, which keep mixture-of-experts weights in CPU memory instead of VRAM, and --gpu-layers, the maximum number of layers stored in VRAM. ()
- NIST: Prefixes for binary multiples: Defines the gibibyte as 2^30 bytes (1,073,741,824) and the gigabyte as 10^9 bytes, which is why GB and GiB differ by about 7%. ()
ToolQuarter assumptions (no primary source):
- The runtime overhead (1 GiB by default) and the headroom (10%) are ToolQuarter heuristics, not figures from a source. Real overhead depends on the runtime and its version, and prompt processing can briefly need more memory than steady-state generation; activation memory is not modelled.
- Using the total parameter count for mixture-of-experts models is ToolQuarter’s inference: all experts have to be stored somewhere, but some runtimes can keep expert weights in system RAM instead of VRAM (llama.cpp’s --cpu-moe option, for example). That a card sold as “24 GB” has 24 GiB is a working assumption: check the manufacturer’s specification.
How to Use
- Choose a model preset, or enter the parameters in billions.
- Pick a quantization and the context length you plan to use.
- Open “Architecture” and enter layers, KV heads and head dimension (a preset fills them in) to include the KV cache.
- Adjust the overhead and headroom assumptions if you know your runtime, then read the total.
Frequently Asked Questions
How much VRAM do I need to run an 8B model?
It depends on quantization and context. For example, at Q4_K_M an 8B model needs about 4.6 GiB for weights; with an 8K context (about 1 GiB of KV cache) and roughly 1 GiB of runtime overhead, plan for about 6.6 GiB, or about 7.2 GiB with 10% headroom. FP16 weights alone are about 14.9 GiB.
Why does a longer context need more memory?
The KV cache stores keys and values for every token in the context in every layer, so it grows linearly with the context length and with the number of concurrent sequences. Long contexts can need more memory than the weights themselves.
Is this result exact?
The weight and KV-cache parts are exact arithmetic on the numbers you enter. The effective bits per weight of a quantization format, the runtime overhead and the headroom are estimates and assumptions, so treat the total as a planning figure and keep some margin.
What is the difference between GB and GiB?
1 GB is 1,000,000,000 bytes and 1 GiB is 1,073,741,824 bytes. Memory in GPUs and RAM is normally counted in GiB even when it is labelled GB, so this tool shows both and compares in GiB.
Does a mixture-of-experts model need less memory?
Not for the weights: every expert must be loaded, so memory follows the total parameter count. Only the computation per token is lower.
Last updated: