Multi-GPU LLM Memory Planner

Plan whether a model fits across several GPUs. Enter the number of GPUs and the VRAM of each, the model size and quantization, the KV cache and the split mode, and see the pooled memory, what each GPU must hold and how many GPUs you would need.

Processed locally in your browser. Your data is not uploaded to our servers.

All GPUs are assumed to be identical. A “24 GB” card is 24 GiB.

Preset figures come from official model documentation, last verified October 3, 2026. You can edit every number.

Source: Llama 3 paper, Table 3

For example 8 for an 8B model. Mixture-of-experts models must load all of their parameters.

Used for a layer split: layers do not always divide evenly between GPUs, so the busiest GPU holds a little more.

Calculate it with the KV Cache Memory Calculator, or enter 0 to leave it out.

Assumption: every GPU pays for its own runtime buffers. About 0.5–1 GiB is typical.

Result

Estimate

Enter the number of GPUs, the VRAM per GPU and the model size to plan.

Pooled memory is not a guarantee

  • Pooled VRAM does not mean linear speed: more GPUs add memory, but communication between them (PCIe or NVLink) often limits speed.
  • Framework support differs: layer splitting is widely supported, while tensor parallelism depends on the engine and usually needs the GPU count to divide the number of attention heads.
  • The plan assumes identical GPUs and an even or whole-layer split. Mixed GPUs, pipeline bubbles and framework buffers are not modelled.
  • This is a memory plan only: it does not predict speed or whether your runtime supports the model.

Examples

Worked examples (Llama 3.1, Q4_K_M, 8K context, FP16 KV cache, 1 GiB overhead per GPU):

GPUsModelSplitBusiest GPUResult
1 × 24 GiBLlama 3.1 70BLayer43.38 GiB / 24 GiBDoes not fit
2 × 24 GiBLlama 3.1 70BLayer22.19 GiB / 24 GiBFits
4 × 80 GiBLlama 3.1 405BTensor59.67 GiB / 80 GiBFits

Formula & Methodology

The planner splits the model and adds each GPU’s own overhead:

weights = parameters × bits per weight ÷ 8per GPU = (weights + KV cache) × share of the model on that GPU + runtime overheadlayer split: the busiest GPU holds ⌈layers ÷ GPUs⌉ layers; tensor split: every GPU holds 1 ÷ GPUsfits if per GPU ≤ VRAM per GPU

For typical sizes by model and format, see the LLM VRAM reference tables.

Method and references

These primary sources support the methods this tool relies on. Having read them does not verify every result the calculator produces, and figures listed as assumptions are ToolQuarter’s own.

The planner divides weights and KV cache by the split and adds overhead per GPU. The sources show how runtimes split models across GPUs; pooled memory is not one shared pool, and speed is not estimated.

  • llama.cpp: server README: Documents llama.cpp’s split modes: layer (the default; splits layers and KV cache across GPUs, pipelined), row (splits weights by rows, with a main GPU holding intermediate results and KV) and tensor (experimental), plus --tensor-split for uneven proportions. Runtimes place the KV cache differently, so the even split used here is a simplification. ()
  • vLLM: Parallelism and Scaling: Describes tensor parallelism for a model that is too large for one GPU but fits on one node, and pipeline parallelism, which splits along layers and supports uneven splits. It advises pipeline parallelism where GPUs lack NVLink because of communication overhead. ()
  • Megatron-LM paper (arXiv:1909.08053v4): Describes intra-layer (tensor) model parallelism, where each layer is split across GPUs and every block needs an all-reduce between them. Its 76% scaling efficiency on 512 GPUs was measured for training, not inference: it shows that scaling is not free, not what your setup will do. ()
  • NIST: Prefixes for binary multiples: Defines the gibibyte as 2^30 bytes (1,073,741,824) and the gigabyte as 10^9 bytes, which is why GB and GiB differ by about 7%. ()

ToolQuarter assumptions (no primary source):

  • Identical GPUs, an even tensor split, whole-layer allocation for a layer split and one runtime overhead per GPU (about 0.5 to 1 GiB) are ToolQuarter’s simplifying assumptions. Mixed GPUs, pipeline bubbles, framework buffers and each runtime’s own placement of the KV cache are not modelled.
  • Pooled memory is only the sum of the GPUs’ memory: it is not one shared pool, and a plan that fits on paper does not show that a runtime supports the model or the split, and it promises no speed. That tensor parallelism usually needs the GPU count to divide the number of attention heads is stated on this page without a source read for it: check your engine’s documentation.

How to Use

  1. Enter the number of GPUs and the VRAM of each one in GiB.
  2. Pick a model preset or enter its parameters, and choose a quantization.
  3. Enter the KV cache (from the KV Cache Memory Calculator) and choose a split mode; add the layer count for a precise layer split.
  4. Read whether the busiest GPU fits, the allocation per GPU and how many GPUs you would need.

Frequently Asked Questions

Do two 24 GiB GPUs act like one 48 GiB GPU?

For fitting a model, only partly: if the runtime can split the model the memory adds up, but it is not one shared pool, each GPU pays its own overhead and the split is not always even. For speed, no: communication between the GPUs adds cost and the gain is rarely linear.

What is the difference between a layer split and a tensor split?

A layer split gives each GPU a block of whole layers and passes data from one GPU to the next; it is widely supported. A tensor split cuts every layer across all GPUs, which can be faster with a fast interconnect but depends on the engine and usually needs the GPU count to divide the number of attention heads.

Why does every GPU need its own overhead?

Each GPU has its own runtime context and compute buffers, so that overhead is paid once per GPU, not once overall.

Can I mix different GPUs?

Many runtimes allow it, but this planner assumes identical GPUs. With mixed cards the smallest card often limits an even split.

Where do I get the KV cache number?

Use the KV Cache Memory Calculator with the model’s architecture and your context length, then enter its total here.

Last updated: