LLM Throughput & GPU Sizing Calculator (Inference Capacity)
Serving an LLM yourself starts with one capacity question: how many tokens per second must the cluster produce, and how many GPUs does that take? Enter the users generating at the same time and the streaming speed each should see (15–30 tok/s reads as fluent chat), pick a GPU, and the calculator prefills an order-of-magnitude aggregate throughput from published benchmarks — editable, because your real number depends on the model, engine, and workload. It plans at a utilization headroom rather than nameplate throughput, then reports the GPU count, the monthly on-demand cost at 720 hours, the cost per concurrent user, and how saturated the cluster actually runs at load — plus how the GPU count scales if traffic halves, doubles, or quadruples.
The load
Users generating at the same moment — not registered accounts. ~15–30 tok/s per user reads as fluent chat.
The hardware
Prefilled with the median of 5 benchmark rows for this GPU (per-GPU aggregate across batched requests — order-of-magnitude; real throughput depends on model, engine, batching, and sequence lengths).
Plan at this share of benchmark throughput, leaving slack for bursts, batching losses, and latency targets.
Prices as of 2026-07— on-demand list rates; reserved and spot capacity can cost far less. Verify against each provider's current pricing page.
GPUs needed
1
4,000 tok/s of demand at 6,623 effective tok/s per GPU
Monthly cost
$2,872.80
Per user / mo
$14.36
Utilization at load
42.27%
Sizing detail
GPUs needed as users scale
Compare scenarios
Run the same calculation with two or three input sets side by side. Differences are highlighted; every number comes from the same tested formula as the calculator above.
| Input | Scenario A | Scenario B |
|---|---|---|
| Concurrent Users | ||
| Target Tokens Per Sec Per User | ||
| Gpu Id | ||
| Gpu Hourly Usd | ||
| Aggregate Tokens Per Sec Per Gpu | ||
| Utilization Headroom Pct |
How it works
Required throughput = concurrent users × target tokens per second per user. Concurrent means generating at the same instant — not registered accounts, and not daily actives. A product with 10,000 signed-up users might see 50–200 generating simultaneously at peak. At the defaults, 200 concurrent users × 20 tok/s = 4,000 tok/s of aggregate demand on the cluster.
GPUs needed = required throughput ÷ (aggregate tokens per second per GPU × headroom), rounded up, minimum one. The per-GPU figure is what one GPU produces across ALL batched requests — vendor benchmarks put an 8B model on one H100 anywhere from ~9,500 tok/s (long context) to ~26,000 tok/s (short prompts), which is why the prefill is explicitly order-of-magnitude and editable. The 70% headroom default means you plan to run at 70% of that benchmark figure, leaving room for traffic bursts, batching inefficiency, and latency targets: 9,462 × 70% = 6,623 effective tok/s per GPU, so 4,000 tok/s of demand fits on a single GPU.
Monthly cost = GPUs × hourly price × 720 hours (an always-on cluster), and cost per concurrent user divides that by your user count — $2,872.80 and $14.36 at the defaults on an H100 at $3.99/h. Utilization at load = required throughput ÷ (GPUs × nameplate throughput): because GPUs come whole, a cluster sized for headroom often runs well below it (42% here), and the scaling bars show where the next GPU actually gets added as users go 0.5×, 1×, 2×, 4×.
Frequently asked questions
How accurate are the prefilled throughput numbers?+
They are order-of-magnitude anchors, not predictions — drawn from vendor and third-party benchmarks as of 2026-07, so verify before committing budget. Real aggregate throughput moves by multiples with the model size, quantization (FP8 vs BF16), serving engine (TensorRT-LLM, vLLM, SGLang), batch size, and above all sequence lengths: the same 8B model on the same H100 measures ~26,000 tok/s with 128-token prompts and ~9,500 tok/s at 2,048/2,048. The only number that deserves a purchase order is one you measured with your own model, engine, and traffic — this tool tells you whether the answer is 2 GPUs or 20, and the field is editable for exactly that reason.
What should I enter for concurrent users?+
The number of requests streaming tokens at the same moment — not registered users, not monthly actives. Chat traffic is bursty and each user spends most of a session reading or typing, so peak concurrency is typically a small fraction of your active user base; the ratio varies wildly by product, so measure yours rather than borrowing a rule of thumb. Size for your realistic peak, then use the 0.5×–4× scaling bars to see how sensitive the GPU count is to being wrong. If a batch or agent workload runs continuously rather than conversationally, one 'user' is simply one always-on stream at your target speed.
Is self-hosting cheaper than paying per token via an API?+
It depends almost entirely on utilization, and this calculator shows you why: GPUs bill by the hour whether or not tokens flow, so a cluster running at 42% of nameplate carries idle capacity you still pay for, while an API bills only for tokens. Self-hosting tends to win with sustained high volume, open-weight models, and strict data-locality needs; APIs win with spiky or low traffic and no ops team. Compare the monthly figure here (on-demand list prices as of 2026-07 — reserved and spot capacity can cost much less, so verify) against the same workload priced through our LLM API cost calculator before deciding. This is a sizing estimate, not procurement or financial advice.
Related tools
Sources
Embed this calculator
+
Drop this calculator into your own site — free. Keep the credit link below the frame: it’s how the free embed stays free.