KV Cache Memory Calculator
The KV cache is the memory a language model uses to remember the tokens so far, and it grows with the context length. Enter the model’s layers, KV heads and head dimension (or pick a preset), the context length, precision and number of sequences to calculate it exactly, and see how much grouped-query attention saves.
Result
Exact for standard attention
Enter layers, KV heads, head dimension and context length to see the KV cache.
- Per token
- Total KV cache
- Attention heads ÷ KV heads
- Standard attention would need
Examples
Worked examples (one sequence):
| Model | Context (tokens) | KV precision | KV cache |
|---|---|---|---|
| Llama 3.1 8B | 8,192 | FP16 | 1 GiB |
| Llama 3.1 70B | 131,072 | FP16 | 40 GiB |
| Qwen2.5 7B Instruct | 32,768 | FP16 | 1.75 GiB |
Formula & Methodology
For standard full attention the cache is exact arithmetic:
KV cache = 2 × layers × KV heads × head dimension × tokens × bytes per value × sequencesbytes per token = 2 × layers × KV heads × head dimension × bytes per valuegrouping ratio = attention heads ÷ KV headsstandard attention cache = KV cache × grouping ratioGood to know
- The factor 2 is the keys plus the values; both are stored for every layer and every token.
- Bytes per value: FP32 is 4, FP16/BF16 is 2, FP8 is 1; llama.cpp’s Q8_0 and Q4_0 cache types use about 1.06 and 0.56.
- Models with multi-head latent attention, sliding-window or hybrid attention can use much less than this formula, and runtimes that page or share cache blocks can differ.
- Nothing is guessed: if a number is missing, find it in the model’s config.json or model card.
Method and references
These primary sources support the methods this tool relies on. Having read them does not verify every result the calculator produces, and figures listed as assumptions are ToolQuarter’s own.
The cache size is exact arithmetic for standard full attention on the numbers you enter. The sources support the formula, the head-sharing variants and the storage types; other attention designs can use less.
- PagedAttention paper (arXiv:2309.06180v1): Gives the KV cache per token as 2 (keys and values) × hidden size × layers × bytes per value, with a worked example of 800 KB per token for a 13B model in FP16. It also describes how paging reduces wasted cache memory, so real runtimes can differ from the plain formula. ()
- GQA paper (arXiv:2305.13245v3): Defines multi-query attention (a single key-value head) and grouped-query attention (more than one but fewer key-value heads than query heads). Fewer key-value heads means less to store, which the tool shows as attention heads ÷ KV heads. ()
- Mistral 7B paper (arXiv:2310.06825v1): Describes sliding window attention with a rolling buffer cache of fixed size, which stops the cache growing (8× less cache memory at 32k tokens in the paper), so the standard formula can overestimate for such models. ()
- DeepSeek-V2 paper (arXiv:2405.04434v5): Reports that Multi-head Latent Attention compresses the KV cache into a latent vector (93.3% smaller than DeepSeek 67B in that paper), so the standard formula does not describe models with this design. ()
- llama.cpp: server README: Lists the KV cache data types llama.cpp accepts for keys and values (f32, f16, bf16, q8_0, q4_0 and others; the default is f16). ()
- ggml: ggml-common.h: Shows the block layouts behind the byte counts: Q8_0 stores a 2-byte scale plus 32 one-byte values (34 bytes per 32 values, about 1.06) and Q4_0 a 2-byte scale plus 16 bytes (18 per 32, about 0.56). ()
ToolQuarter assumptions (no primary source):
- FP32 = 4, FP16 and BF16 = 2 and FP8 = 1 byte per value are the usual widths. FP8 cache support depends on the runtime (the llama.cpp list above does not include it) and no source was read for it here.
- That a lower-precision KV cache can slightly affect output quality, more at 4 bits, is ToolQuarter’s general guidance with no source read: test with your own prompts. The tool never guesses an architecture: layers, KV heads and head dimension come from you or from a preset.
How to Use
- Pick a model preset, or enter layers, KV heads and head dimension from the model’s config.
- Enter the context length in tokens and the number of concurrent sequences.
- Choose the precision the KV cache is stored in (FP16 is the usual default).
- Optionally enter the attention heads to see what grouped-query attention saves.
Frequently Asked Questions
What is the KV cache?
While generating text, a model keeps the attention keys and values of every token so far so it does not recompute them. That stored data is the KV cache. It grows with every token and every concurrent sequence.
What do GQA and MQA mean for memory?
In grouped-query attention several attention heads share one key/value head; in multi-query attention all of them share one. The cache shrinks by the ratio of attention heads to KV heads, for example 4× for a model with 32 attention heads and 8 KV heads.
Where do I find a model’s layers, KV heads and head dimension?
In its config.json on Hugging Face (num_hidden_layers, num_key_value_heads and head_dim, or hidden_size ÷ num_attention_heads) or in the model card or paper. This tool never guesses them.
Does storing the KV cache in fewer bits reduce quality?
It reduces memory (Q8_0 is roughly half of FP16) and can slightly affect output quality, more so at 4 bits. Support depends on the runtime, so test with your own prompts.
Is the result exact?
The formula is exact for standard full attention with the numbers you enter. Models with sliding-window, latent or hybrid attention, and runtimes that page or share cache blocks, can use less or behave differently.
Last updated: