AI Inference Time & Throughput Estimator

How long will a response take? Enter the prompt and output tokens together with the prefill speed (how fast the model reads the prompt) and the generation speed (how fast it writes), measured on your own setup or assumed, and get the prefill time, generation time, total time, effective tokens per second and requests per minute. Add a fixed overhead for network or queueing, or a concurrency for ideal parallel throughput. If you only know the generation speed, the simple mode estimates generation time alone.

Processed locally in your browser. Your data is not uploaded to our servers.

The tokens the model reads before it starts writing.

The tokens the model generates.

How fast the prompt is processed. Leave empty for the simple mode: the prompt is then not timed.

How fast tokens are written once generation starts. Measure it on your setup.

Network, queueing or tool latency added to every request.

Shows the throughput of this many identical requests, assuming none slows the others down.

Result

Estimate

Enter the tokens and the generation speed to estimate the time.

Exact arithmetic on the speeds you enter, which are measured or assumed: the result is an estimate of a real response time, not a benchmark of any model or hardware.

Examples

Worked examples using illustrative speeds (not benchmarks of any model or hardware), no overhead:

ScenarioPrompt / output tokensSpeeds (prefill / generation)Total timeTime to first token
Short chat reply500 / 3002,000 / 60 tokens/s5.25 s267 ms
Answer over retrieved documents6,000 / 4001,500 / 40 tokens/s14 s4.03 s
Long document summary100,000 / 1,0001,000 / 30 tokens/s2 min 13 s1 min 40 s

Formula & Methodology

The arithmetic is exact for the numbers you enter:

prefill time = prompt tokens ÷ prefill speedgeneration time = output tokens ÷ generation speedtotal time = fixed overhead + prefill time + generation timeeffective speed = output tokens ÷ total time; requests per minute = 60 ÷ total timetime to first token ≈ fixed overhead + prefill time + 1 ÷ generation speed

Good to know

  • The speeds are inputs, not facts about a model. This tool does not guess tokens per second from a GPU or model name: measure on your own setup, or use a figure you trust.
  • Real speeds vary with batching, queueing, other users, prompt length, the runtime, quantization, speculative decoding and the hardware. A single measured number is a snapshot.
  • Prefill is usually much faster per token than generation, which is why long prompts add less time than long outputs.
  • Streaming responses feel faster than the total time suggests, because the first tokens arrive after the time to first token.
  • Under heavy load each request slows down. The parallel figures are an ideal upper bound.

How to Use

  1. Enter the prompt and output tokens.
  2. Enter the prefill and generation speeds you measured or assume. Leave prefill empty for the simple mode.
  3. Optionally add a fixed overhead and a number of parallel requests.
  4. Read the prefill, generation and total times, the effective speed and the requests per minute.

Frequently Asked Questions

How do I estimate how long an LLM response will take?

Divide the prompt tokens by the prefill speed, divide the output tokens by the generation speed and add any fixed overhead. This calculator does that from the speeds you enter.

What is the difference between prefill and generation speed?

Prefill is how fast the model reads and processes your prompt, usually in large parallel chunks, so it is fast per token. Generation writes one token at a time and is slower. Both are tokens per second, measured separately.

Can you tell me the tokens per second of my GPU or model?

No. Speed depends on the runtime, quantization, batching, context length and much more, so this tool does not guess it from a name. Measure your own setup, or enter a figure from a source you trust.

What is time to first token?

The wait before the first generated token appears: any fixed overhead plus the time to process the prompt, plus the first token. It is only estimated when you give a prefill speed.

Why are the parallel numbers only an upper bound?

They multiply one request’s speed by the number of requests, as if each ran alone. In practice shared hardware, batching and queueing change each request’s speed, so measure real throughput under load.

Last updated: