LLM Inference Estimator

Model weights = params × bytes/param. Bytes come from the weight precision: FP16 = 2, FP8 = 1, FP4 ≈ 0.56 with its scales, INT4 = 0.5. As released uses the checkpoint's own mix; gpt-oss-120b, for example, ships MXFP4 experts with BF16 attention, 65 GB in all. Parameter counts include vision encoders and speculative-decoding heads, since those load too. This is fixed regardless of load.

KV cache is added up per kind of layer, from each model's config.json. A standard attention layer stores 2 × kv_heads × head_dim × bytes per token (the 2 is K and V). Sliding-window layers keep only their last N tokens. Latent attention (MLA) stores one compressed vector per token, and DeepSeek V4 keeps one entry per 4 or 128 tokens. Linear-attention and Mamba layers keep a fixed-size state per user (counted in fp32) that never grows with context, which is why hybrid models like Qwen 3.5+, Kimi K3 and Nemotron hold long contexts cheaply. Each user's cache is sized at prompt + reply, the most it holds, at the end of its reply; engines fill it as tokens arrive, so this is the peak. Total = one user's cache × users.

Activations = T × (hidden × 4 + ffn × 2) × 2 B plus users × vocab × 4 B of fp32 logits, where T is tokens per step — the prompt piece size set by Optimize for, or the number of users, whichever is larger. The MLP intermediate (ffn, 3.5–4× hidden for SwiGLU) dominates, which is why activations track the piece size rather than context length. Compute stays bf16 even when weights are quantized, and FlashAttention keeps the T² score matrix off HBM — so there is no quadratic term. Genuinely the smallest of the four on most configs; choose Fast first reply to see it grow.

Overhead = a fixed ~1.0 GB CUDA/driver context plus a slider % of the three above, standing in for allocator fragmentation, framework buffers, and CUDA graphs.

The page picks TP for you as part of the smallest layout, fixed in Advanced settings if you prefer. One copy of a model is split 1, 2, 4 or 8 ways: serving engines need the split to divide the model's attention heads and layer sizes evenly, and these are the splits production deployments run. This is a software rule that applies to every board, and it is separate from how many cards a server can hold. With TP > 1 the model is sharded across GPUs: weights and activations divide by TP, and the KV cache divides by min(TP, kv_heads). Sharding adds memory: every GPU carries its own ~1 GB CUDA context, plus NCCL all-reduce workspace and comm buffers (~0.3 GB + users × hidden × 8 B per GPU).

When TP exceeds a layer's KV heads, that part of the cache can't split further and copies to every GPU — watch it stop shrinking. Models with latent or compressed attention (DeepSeek, GLM, Kimi, Mistral Small 4) have a single cache head, so a plain split would copy their whole cache to every GPU. SGLang and vLLM serve them with data-parallel attention instead, where each GPU keeps only its own users' cache, and the page sizes them that way. Linear-attention state splits cleanly.

Every amount is computed in bytes and shown in the unit picked at the top: GB is 10⁹ bytes (spec sheets, download pages), GiB is 2³⁰ bytes (nvidia-smi, PyTorch and vLLM's logs). Switching changes the labels, never the answer. Board capacities are the size on the box, counted as GB, so an H200 is 141 GB or 131 GiB. Real boards expose that much or slightly more, so a fit here errs on the safe side.

Generating a token means reading this GPU's weight shard plus every active request's KV cache, so a decode step takes bytes read ÷ (memory bandwidth × ). At very large batches the compute cost users × 2 × active params ÷ (FLOPs × ) takes over, so the step uses whichever is slower. FLOPs are each board's dense BF16 figure from its datasheet; FP8 counts as × that, and FP8 weights realize of the bytes they save. Tensor parallelism adds two all-reduces per layer, each a fixed over NVLink or Infinity Fabric ( over PCIe) plus the batch's activations at the link's speed, and the engine adds per step. An NVSwitch gives any group of GPUs the full link speed; the MI355X's GPUs are wired to each other with no switch, one link per pair, so a copy on 2 of them uses 1 link, on 4 uses 3, and on 8 uses all 7.

With continuous batching every active request gets exactly one new token per step, so everyone streams at the same rate, 1 ÷ step time between prompt reads. Adding users doesn't slow the step much until their caches outweigh the weights.

Reading a prompt is compute-bound: prompt × (2 × active params + 2 × layers × prompt × hidden) ÷ (FLOPs × ). The prompt is read in pieces (512, 2,048 or 8,192 tokens, set by Optimize for), and each piece rides in a step with everyone else's next token: the step takes whichever is slower, the memory read or the decode compute plus the piece's compute, so a small piece fits in time the step would spend waiting on memory and a big one makes the step compute-bound. The pause other users feel is one such step.

Under load, new prompts keep arriving: each concurrent user sends a new request as soon as the last is answered, so with n requests in progress, each lasting D seconds, n ÷ D arrive every second. Reading their prompts keeps the engine busy a share of the time (busy = arrivals × read time), and every step writes one token for every user, so speed per user = steps a second = arrivals × pieces + (1 − busy) ÷ step time, and output for everyone = users × speed per user. Prompts are read one at a time, so a request that arrives while others are being read waits for them; arriving at random, the average wait is arrivals × read time² ÷ (2 × (1 − busy)) (the Pollaczek–Khinchine formula).

Time to first token = the the engine takes to take a request in (HTTP, tokenization, scheduling) + that wait + reading its own prompt, and a request lasts that plus its thinking and reply at the per-user speed. Each of these depends on the others, so the page solves for the point where they agree. More load means longer waits and slower replies, which space requests out, so an overloaded setup shows up as slow replies and long waits rather than a queue that grows forever. Smaller pieces keep replies flowing but leave prompts waiting longer; bigger pieces start prompts sooner but freeze replies for longer.

Requests are assumed to be halfway through their reply (prompt + half the reply in cache), and the first reply waits on the whole prompt. Quantized weights read fewer bytes but lose some of that gain to dequantization; FP8 also speeds up compute on boards with native FP8 cores. MoE models read more experts as the batch grows; this assumes tokens are routed evenly across experts, and real routing is skewed, so fewer distinct experts are read and real speed can be somewhat higher.

Bandwidth and link speeds are published specs; FLOPs are rough estimates. Every speed figure is an estimate at the server, averaged over steady traffic: it leaves out the network round trip, TLS and any gateway, and tokenization, which add little to the streaming rate but can outweigh a short first token. Real results land on both sides: routing skew, a warm prompt cache and users who pause between requests make things faster; bursts and long-tail prompts slower. The shares of bandwidth and compute, the all-reduce latency over NVLink and vLLM's fixed costs are fitted to 295 published benchmark points (NVIDIA NIM and SemiAnalysis InferenceX: Llama 3.1 8B, Llama 3.3 70B and gpt-oss-120b on H200, B200 and L40S); the PCIe latency and the other precisions are not, since no benchmark covers them.

Fitted with each group of benchmarks left out in turn, measured speed per user came to between the estimate for 80% of the points left out, and the first token with nothing queued to between it: that is the likely range shown beside each, and a speed goal counts as met only at its cautious end, so the figures shown are central estimates while the suggestion should hold in practice. Boards with no benchmark (L4, the RTX PRO cards, H200 NVL, MI350P, B300) rest on the same fitted shares applied to their datasheets. On AMD's software stack (ROCm) serving reaches less of the same hardware: the AMD boards' shares are scaled by , fitted to runs on the MI355X (the MI350P is half of its chip), and those runs predicted within the same range when the factor was fitted on another model's runs.

A speed goal is met on this estimate only, so confirm with a load test (for example vllm bench serve at your request rate) before committing to hardware. Method and every point: validation/README.md in the repository.

Chat and agent prompts mostly repeat: the same system prompt for every user, and each user's earlier turns. vLLM and SGLang cache that by default, so before the first reply only the new part of the prompt is read (it still attends to the cached part), and the shared system prompt's cache is stored once per GPU instead of once per user. Each preset sets the shape: Chat has a 512-token shared system prompt and about 512 new tokens a turn; RAG and Long-document summary read their documents fresh every time; Coding agent shares 4K tokens of system prompt and tool definitions and adds about 4K new tokens a step. Everything else in the prompt still sits in each user's cache, so memory changes only by the shared part. It can be turned off in Advanced settings.

A model that reasons step by step writes thinking tokens before its answer. They're generated like any other token, so a user's cache peaks at prompt + thinking + reply, speed is judged halfway through thinking and reply together, and cost per output token counts them. The First token stat is when the first token appears, which for a thinking model is the start of its thinking, and it also says when the answer starts: first token plus the thinking budget at the per-user speed. Users see the thinking says which wait the speed goal is judged on. On (the default), the UI streams the thinking, so the goal's first-reply limit is checked against the first token.

Off, users wait for the answer, so the tile becomes Answer starts and the limit is on that: within 15 s for Fast, 30 s for Smooth chat and 60 s for Readable, at the cautious end of the estimate (the first token at its slow end, the thinking at the low end of the speed range). Those limits are the estimator's own judgement; a first reply in 1 to 5 s is out of reach once thousands of tokens of thinking come first. To start answers sooner, shorten the thinking budget or pick a faster setup. Earlier turns' thinking is normally dropped by the model's chat template, so it doesn't build up in the prompt. The switch starts at each model's own default, taken from its model card and chat template.

One model often serves more than one kind of request at once: chat users alongside a RAG app, say. Add another workload gives each its own number of concurrent users and its own prompt and reply lengths. They share the same copies of the model, and each copy takes the workloads in the same proportions. Memory adds up: each workload's users hold a cache of that workload's length, and each workload is taken to have its own shared system prompt, stored once per GPU, so two rows of one preset cost one more shared prompt than a single row with their users combined. Speed per user is the same for everyone, because one decode step writes a token for the whole batch; the step reads every request's cache, so long-context users slow the others a little, and every workload's prompts take their share of the engine's time, so long-prompt users slow the others a lot.

First token differs, since each workload waits on its own new prompt tokens: the headline is the workload that waits longest, the speed goal's first-reply limit is checked against it (so a slow-starting workload can call for a bigger setup even when the others would be fine), and Speed on this setup lists each one. Every workload's prompts join one queue and are read one at a time, so each first token is the same average wait for prompts ahead plus the time to read that workload's own prompt. The pause is one piece's worth and is much the same for every workload; what a long prompt adds is how many pieces come back to back. One engine setting, Optimize for, reads every workload's prompts; it follows the first workload's preset and can be changed in Advanced settings.

The Thinking switch applies to every workload, since it is a property of the model. Because the cache grows in step with length on most models, memory and speed come out almost the same as one workload at the user-weighted average lengths; what a mix adds is each workload's own start time. Workloads that need a different model are not covered.

This estimates one server of up to 8 GPUs for each model. Every model's result is at most a single node; multi-node deployments and NVL rack-scale systems are not covered. With several models, their copies share servers (see Several models). If one node of a board can't serve every user, its row says how many it does serve. For production, plan at least one more node (or replica) than this, so the service keeps running if one fails or is being updated.

Every board is tried at 1, 2, 4 and 8 GPUs per copy (splitting a copy only across NVLink boards), with as many copies as the node's 8 GPUs allow. The suggestion is the node that serves every user within the speed goal at the cautious end of the estimate (speed per user at the low end of its likely range, first token at the slow end), with the least GPU memory in total (on any one board, the fewest GPUs). If no node serves everyone within the goal, it is the full node that serves the most users. GPU counts round up to what can be bought: H200 NVL servers take 1, 2, 4 or 8 cards, so 3 rounds up to 4 and 5 to 7 round up to 8; but an NVLink bridge joins at most 4 of them, so one copy of a model spans at most 4, and 8 cards run two copies.

The HGX H200, HGX B200 and HGX B300 ship only as 8-GPU systems, with all 8 GPUs joined through NVSwitch. RTX PRO 4500 servers hold any number of cards up to 5 (HPE's limit); because of the split rule above, one copy of a model uses at most 4 of them, and the fifth can run another copy. Spare GPUs run extra copies of the model. Room for (Handles up to in the summary) is the most users the node holds at the current prompt and reply lengths while fitting in memory and, with a goal, still reaching it on the estimate; when the goal is the limit, it also says how many memory alone would hold.

For more users, the page runs more copies of the model on the node's GPUs instead of splitting one copy wider. Each copy has its own GPUs and an even share of the users, so memory, the bar and the breakdown describe one copy, while GPU count, output and capacity add up across all of them. Copies don't exchange data while generating, so they need no direct GPU-to-GPU link between them; that's why PCIe boards run several copies well even though they split a single copy poorly. When two layouts use the same GPUs, the page takes more copies over splitting one wider, on every board: both reach the goal, and copies leave room for more users. A copy is split wider only when memory or the speed goal needs it. When a board needs more GPUs than memory alone calls for, the extra ones are there to reach the speed goal, and the page says so. This assumes requests are spread evenly across the copies.

Add up to four models, each with its own users, workloads, precision and speed goal; the controls above set the model whose tab is selected, while overhead, prompt caching and Server apply to all. Every model runs as copies of its own on GPUs of its own, sized as it would be alone but before rounding up to a server size. All the copies are then packed into servers of the board, the widest first, and a server that isn't full shrinks to the smallest size the board comes in; a copy of 4 on H200 NVL cards stays within one bridged group. Spare GPUs run extra copies of the models closest to missing their goals, and each model then uses the fewest GPUs per copy that still meet its goal. A model that one node can't serve keeps a node of its own.

The suggestion is the board where every model meets its goal with the least GPU memory in all, then the fewest servers; every model runs on the same kind of board. Models don't share GPUs, so each one's speed is its own; they share the servers' CPUs, memory and storage, and each server's storage holds the weights of every model it runs, once each.

The server sizing is in beta: its rules and figures are still being worked on and may change. Set Server to NVIDIA guidance in Advanced settings and the page follows the inference server recommendations in NVIDIA's Certified Systems Configuration Guide. GPU counts round up to 2, 4 or 8 per server, split evenly across two CPU sockets (2 or 4 for the RTX PRO 4500, whose servers hold 5), which can change which board is suggested; the HGX systems stay whole 8-GPU systems. Around the GPUs the guide asks for CPUs with PCIe Gen5 and a base clock of 2.0 GHz or more and at least 6 physical cores per GPU, but the page sizes cores as HPE builds the AI worker nodes of HPE Private Cloud AI, its inference platform with NVIDIA: 32 physical cores per GPU in a 2- or 4-GPU server and 16 per GPU in an 8-GPU one (2 × 32-core CPUs for 2 GPUs, 2 × 64-core for 4 or 8).

The guide also asks for system memory of at least 2 × the GPU memory spread over every memory channel of both sockets, each GPU in an x16 slot of its own PCIe generation, PCIe Gen5 switches with 8 GPUs (with one or two per socket the CPUs' own lanes may be enough), one NVMe drive of at least 1 TB per socket and, in NVIDIA's reference architectures, a 1 TB NVMe boot drive, with drives under the same switch or root complex as the GPUs. The guide also covers network cards, remote management and TPM; the page leaves those to the conversation with the customer. The storage figure is the page's own: the usable capacity the model needs, (2 × weights + 100 GB) ÷ 0.8, rounded up (to 10 GB, or a tenth of a TB above 1 TB).

That is two copies of the weights (the version serving and the next one downloading) and 100 GB for container images, compile caches and logs, kept under 80% full, beside a separate boot volume for the OS. How it is made from drives and RAID is the vendor's choice, so the page doesn't size drives. KV cache offloaded to disk, as NVIDIA Dynamo's KV block manager does, needs a disk tier at least as large as its CPU tier, which should be 100 GB or more and larger than the GPUs' KV cache; the page gives that figure in the storage tooltip only, since few single servers offload to disk. Server amounts are in GB as sold, whatever unit is picked at the top.

All of these are minimums for planning, and the guide adds that LLM servers should aim above them. An HGX system's CPUs, memory and storage come with it from its manufacturer, so the page leaves them to its spec sheet. The AMD boards are left out of the comparison while Server is set to NVIDIA guidance, since the guide is written for NVIDIA GPUs. A server is NVIDIA-Certified only if it is on NVIDIA's list of certified systems.

Requests in progress at once = requests per second × seconds each one takes. A request takes the wait before its first token plus its thinking and reply, streamed at your speed goal's rate; the goal is a minimum speed, so this errs on the high side. The busiest-hour multiple covers traffic that isn't spread evenly through the day; 2× suits a typical workday.

The estimate covers the LLM only. RAG also needs an embedding model and usually a reranker. Small ones (under ~1B parameters) add a few GB and can share a GPU or run on CPU at low traffic. A 4–8B reranker, or heavy traffic, needs its own GPU. The reranker is the larger of the two: it reads the question against every retrieved chunk, often tens of thousands of tokens per query.

On a hosted copy of this page, the address always holds the current settings, so Copy link sends the exact scenario, every model's settings included. Download summary saves a one-page report of the recommendation, the inputs and every board, which prints to PDF. Both record the estimator's version; a link made with an older version opens with what has changed in the figures since then, and the image of that version to run to see them as they were.