AI Inference Time & Throughput Estimator
How long will a response take? Enter the prompt and output tokens together with the prefill speed (how fast the model reads the prompt) and the generation speed (how fast it writes), measured on your own setup or assumed, and get the prefill time, generation time, total time, effective tokens per second and requests per minute. Add a fixed overhead for network or queueing, or a concurrency for ideal parallel throughput. If you only know the generation speed, the simple mode estimates generation time alone.
Result
Estimate
Enter the tokens and the generation speed to estimate the time.
- Prefill time
- Generation time
- Fixed overhead
- Total response time
- Estimated time to first token
- Effective output speed
- Requests per minute (one at a time)
- Requests per minute in parallel
- Output speed in parallel
Simple mode: no prefill speed was given, so the prompt is not timed and the time to first token is not estimated. The total covers generation and any overhead only.
Time to first token is estimated as overhead + prefill time + the time to write one token.
Parallel figures assume every request keeps the same speed. Real systems usually slow down under load, so treat them as an upper bound.
Exact arithmetic on the speeds you enter, which are measured or assumed: the result is an estimate of a real response time, not a benchmark of any model or hardware.
Examples
Worked examples using illustrative speeds (not benchmarks of any model or hardware), no overhead:
| Scenario | Prompt / output tokens | Speeds (prefill / generation) | Total time | Time to first token |
|---|---|---|---|---|
| Short chat reply | 500 / 300 | 2,000 / 60 tokens/s | 5.25 s | 267 ms |
| Answer over retrieved documents | 6,000 / 400 | 1,500 / 40 tokens/s | 14 s | 4.03 s |
| Long document summary | 100,000 / 1,000 | 1,000 / 30 tokens/s | 2 min 13 s | 1 min 40 s |
Formula & Methodology
The arithmetic is exact for the numbers you enter:
prefill time = prompt tokens ÷ prefill speedgeneration time = output tokens ÷ generation speedtotal time = fixed overhead + prefill time + generation timeeffective speed = output tokens ÷ total time; requests per minute = 60 ÷ total timetime to first token ≈ fixed overhead + prefill time + 1 ÷ generation speedGood to know
- The speeds are inputs, not facts about a model. This tool does not guess tokens per second from a GPU or model name: measure on your own setup, or use a figure you trust.
- Real speeds vary with batching, queueing, other users, prompt length, the runtime, quantization, speculative decoding and the hardware. A single measured number is a snapshot.
- Prefill is usually much faster per token than generation, which is why long prompts add less time than long outputs.
- Streaming responses feel faster than the total time suggests, because the first tokens arrive after the time to first token.
- Under heavy load each request slows down. The parallel figures are an ideal upper bound.
How to Use
- Enter the prompt and output tokens.
- Enter the prefill and generation speeds you measured or assume. Leave prefill empty for the simple mode.
- Optionally add a fixed overhead and a number of parallel requests.
- Read the prefill, generation and total times, the effective speed and the requests per minute.
Frequently Asked Questions
How do I estimate how long an LLM response will take?
Divide the prompt tokens by the prefill speed, divide the output tokens by the generation speed and add any fixed overhead. This calculator does that from the speeds you enter.
What is the difference between prefill and generation speed?
Prefill is how fast the model reads and processes your prompt, usually in large parallel chunks, so it is fast per token. Generation writes one token at a time and is slower. Both are tokens per second, measured separately.
Can you tell me the tokens per second of my GPU or model?
No. Speed depends on the runtime, quantization, batching, context length and much more, so this tool does not guess it from a name. Measure your own setup, or enter a figure from a source you trust.
What is time to first token?
The wait before the first generated token appears: any fixed overhead plus the time to process the prompt, plus the first token. It is only estimated when you give a prefill speed.
Why are the parallel numbers only an upper bound?
They multiply one request’s speed by the number of requests, as if each ran alone. In practice shared hardware, batching and queueing change each request’s speed, so measure real throughput under load.
Last updated: