# DeepSeek-V4 Serving Parallelism And Blackwell Recipes Status: upstream-backed serving and routing snapshot; inspected through 2026-07-20 This appendix answers a narrower set of operational questions than [`04-inference-systems.md`](./04-inference-systems.md): how current vLLM and SGLang recipes place DeepSeek-V4 on Blackwell GPUs, what `TP`, `EP`, and DP-Attention mean in those recipes, and how to state token capacity without confusing model context length with the runtime KV-cache pool. The short answer is: - the official starting points are **4 B200/B300 GPUs for V4-Flash** and **8 B200/B300 GPUs for V4-Pro**; - an SGLang command containing `--tp 8` does not necessarily mean EP=8; low-latency TP-only and throughput-oriented DP-Attention+EP are different resolved layouts; - TP/EP is selected when the server starts. Token-to-expert routing and EPLB placement change at runtime, but the engine does not choose MoE TP versus EP independently for each request; - DP-Attention is especially natural for MLA and V4 because their one shared KV memory is otherwise replicated across query-head TP ranks, reducing the attention arithmetic intensity on each rank; - both Flash and Pro have a 1,048,576-token model context limit. That number is neither the aggregate resident-token capacity nor a promise that every software build can execute a 1M request correctly and within an acceptable latency; - V4-Pro does not fit the official mixed-precision checkpoint on 4 B200s. Its recipe declares about 960 GB minimum model VRAM, before cache and workspace, so a stock 4-B200 deployment has no usable token capacity because it cannot finish loading the model. ## 1. Source And Version Boundary The serving surface is moving quickly. The conclusions below use: - SGLang `main` commit [`a03ca46a284`](https://github.com/sgl-project/sglang/commit/a03ca46a2847eced368d38331991264b06852602), including its [V4 cookbook](https://github.com/sgl-project/sglang/blob/a03ca46a2847eced368d38331991264b06852602/docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx) and [generated recipe cells](https://github.com/sgl-project/sglang/blob/a03ca46a2847eced368d38331991264b06852602/docs_new/src/snippets/configs/deepseek-ai/deepseek-v4.jsx); - vLLM `main` commit [`e6d1310b2ac`](https://github.com/vllm-project/vllm/commit/e6d1310b2ac52e56e266450693af9de12a871a51), the official [V4 implementation note](https://vllm-project.github.io/2026/04/24/deepseek-v4.html), and the mutable [Flash](https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash) and [Pro](https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro) recipes inspected on the date above; - DeepSeek's Open Source Week [V3/R1 inference-system overview](https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md), [DeepEP](https://github.com/deepseek-ai/DeepEP), and [EPLB](https://github.com/deepseek-ai/EPLB). Recipe support means that the cited upstream publishes or marks the launch shape as verified. It does not imply that all combinations of 1M context, prefix cache, MTP, PD, CP, and cache offload have been stress-tested together. ## 2. One `tp` Flag, Several Resolved Parallelisms SGLang treats the launch `tp_size` as the width of the enclosing model-parallel group, then resolves smaller parallel groups for attention and MoE. At the inspected revision, the relevant sizes are $$ P_{attn,TP} =\frac{P_{TP}}{P_{attn,DP}P_{attn,CP}}, $$ and $$ P_{MoE,TP} =\frac{P_{TP}}{P_{MoE,DP}P_{EP}}. $$ The first equation is visible in SGLang's pinned [`engine.py`](https://github.com/sgl-project/sglang/blob/a03ca46a2847eced368d38331991264b06852602/python/sglang/srt/entrypoints/engine.py). The second is visible in [`parallel_state.py`](https://github.com/sgl-project/sglang/blob/a03ca46a2847eced368d38331991264b06852602/python/sglang/srt/distributed/parallel_state.py). DeepEP, MegaMoE, Mooncake, NIXL, FlashInfer, and the other registered MoE all-to-all backends set $P_{EP}=P_{TP}$ in the current [`overrides.py`](https://github.com/sgl-project/sglang/blob/a03ca46a2847eced368d38331991264b06852602/python/sglang/srt/arg_groups/overrides.py). For the two common `tp=8` shapes this resolves as follows: | Launch shape | Attention layout | Routed-expert layout | What `tp=8` means in practice | | --- | --- | --- | --- | | `--tp 8`, no DPA, no MoE A2A backend | attention TP=8 | MoE TP=8, EP=1 | all eight ranks shard query/projection and expert intermediate dimensions | | `--tp 8 --dp 8 --enable-dp-attention --moe-a2a-backend megamoe` | attention TP=1, DP=8 | MoE TP=1, EP=8 | each request's attention stays on one rank; routed experts are partitioned by expert identity | Thus the low-latency Pro recipe really does split an expert's hidden dimension. In the generic fused-MoE implementation, SGLang sets `intermediate_size_per_partition = intermediate_size / moe_tp_size`, shards the $W_1/W_3$ output dimension and corresponding $W_2$ input dimension, and reduces the partial expert result. TP does not require the engine to invent a new expert decomposition at runtime. ### 2.1 This choice is not per-request dynamic The parallel groups, weight shards, expert ownership, and collective objects are created during server initialization. They cannot be changed cheaply for one request while other requests keep running. The runtime does make several different dynamic decisions: - the router selects logical experts for each token; - an all-to-all backend dispatches tokens to the ranks owning those experts; - DeepEP may use a throughput-oriented or low-latency communication path; - EPLB may periodically change which rank holds a physical replica of a logical expert; - the scheduler changes batch composition and, in some engines, speculative width. None of these means “use TP for this request, but EP for the next request.” Elastic expert-parallel reconfiguration is a separate system problem. ## 3. Why The Low-Latency And Throughput Recipes Differ Neither TP nor EP is always better. Their break-even point depends primarily on how many routed tokens are available per step. ### 3.1 TP-only: useful at batch one With MoE TP, every rank participates in the selected experts. This is useful for low-concurrency latency because: - one token activates work on all ranks rather than sending a tiny number of tokens to a subset of expert owners; - there is no token all-to-all or expert-load imbalance; - one large expert is divided across ranks, so all GPUs can contribute to a single request. The costs are skinny expert GEMMs, a TP reduction after the expert, and the shared-KV attention inefficiency derived in Section 5. TP-only therefore stops being attractive once enough independent tokens exist to keep EP ranks busy. ### 3.2 DP-Attention plus EP: useful at sustained concurrency With attention DP and expert parallelism: - requests are distributed across attention replicas; - each attention rank owns the full attention weights and cache for its local requests; - tokens are sent to routed-expert owners; - every owner can form larger grouped expert GEMMs from many requests; - expert weights are partitioned by expert identity rather than replicated. This removes attention TP collectives and improves attention locality, but it introduces MoE all-to-all traffic and load imbalance. It needs a sufficiently large global batch, a good dispatch/combine implementation, and expert-load balancing. This explains the apparent tension between “TP is fastest” and “DeepSeek says to use EP.” The first usually refers to low-batch latency; the second refers to a production throughput regime with enough tokens to amortize EP. ## 4. DeepSeek's EP, DeepEP, And EPLB Lineage DeepSeek's published V3/R1 production system used much larger EP groups than the single-node recipes in this chapter: - prefill used routed-expert EP32 while MLA and the shared expert used DP32; - decode used routed-expert EP144 while MLA and the shared expert used DP144; - 32 redundant expert replicas were placed to smooth hot experts; - large global batches, dual-batch overlap, and a five-stage decode overlap hid expert communication and compute. These numbers describe V3/R1 production, not a claim about the private V4 service. Their value here is architectural: DeepSeek already separated request-parallel shared attention from expert-parallel routed MoE. [DeepEP](https://github.com/deepseek-ai/DeepEP) supplies specialized dispatch/combine collectives for high-throughput and low-latency regimes. [EPLB](https://github.com/deepseek-ai/EPLB) solves a different problem: given an expected logical-expert load, it duplicates hot experts and packs physical replicas onto ranks while trying to preserve group/node locality. Hierarchical placement is intended for smaller prefill EP groups; global placement is intended for larger decode EP groups. EPLB has a real cache tradeoff. Redundant experts consume HBM that could otherwise be used by KV/cache pools. More replicas can reduce tail latency and all-to-all imbalance but reduce resident-token capacity. The load estimator and rebalance cadence are runtime policy; the logical model router is unchanged. ## 5. Why DP-Attention Is Especially Natural For MLA And V4 ### 5.1 The MHA baseline Suppose an MHA layer is tensor-parallel across $P$ ranks and has enough KV heads to shard both Q and KV heads. To first order, each rank receives $$ \frac{1}{P}\text{ of the QK/AV FLOPs} \quad\text{and}\quad \frac{1}{P}\text{ of the KV-cache bytes}. $$ The arithmetic intensity of the core attention is therefore roughly preserved. TP may still pay projection and output collectives, but it does not necessarily make each rank reread the same KV history. ### 5.2 Shared KV changes the scaling MQA, MQA-mode MLA, and V4 instead have one shared latent/main KV memory reused by many query heads. If TP shards only the query heads, rank $p$ executes about $1/P$ of the QK/AV arithmetic but still needs the same shared history on its own device: $$ F_p\approx\frac{F}{P}, \qquad B_p\approx B, \qquad AI_p\approx\frac{AI}{P}. $$ The cache is small, but it is physically replicated across the TP ranks and read once by each rank's subset of query heads. For V4, the C4 main cache, C128 main cache, and C4 index-key history all exhibit this shared-memory property. CSA adds a particularly expensive replicated index scan. Using V4-Flash's $n_h=64$, shared width $c=512$, and 584-byte deployed main entry, a deliberately simplified QK+AV intensity per main entry is $$ AI_{TP=1} \approx\frac{4n_hc}{584} \approx224\ \text{FLOP/byte}. $$ Under TP8, eight query heads remain per rank and the same main entry is local to each device: $$ AI_{TP=8} \approx\frac{4(n_h/8)c}{584} \approx28\ \text{FLOP/byte}. $$ This is a trend model, not a kernel roofline: it omits softmax, scale reads, metadata, cache writes, tensor-core packing, and projection work. It still captures why sharding the query heads can turn shared-KV attention into a much more bandwidth-sensitive small kernel. ### 5.3 What attention DP buys At the same aggregate number of requests, DP-Attention gives each request to one attention rank. That rank uses all query heads against its one local cache copy. Compared with TP8: - the history is scanned/read once, not once on every query-head shard; - attention weights are replicated, but they are much smaller than the routed expert weights; - attention TP collectives disappear; - cache ownership and prefix affinity become request-local; - MoE can still use EP across the same physical ranks. The limitations are equally important: - one request does not receive an eight-GPU attention latency speedup; - one request cannot aggregate the KV capacity of eight GPUs; - dense attention weights are replicated; - an exceptionally long session is tied to one rank unless DCP, offload, or a different cache-transfer mechanism is added. This is why DP-Attention and decode CP solve different problems. DPA is the preferred throughput layout while each request fits one rank; DCP is a single-request history/cache sharding mechanism. ### 5.4 Request admission, load balance, and prefix ownership DP-Attention does not itself decide which request belongs to which attention rank. An admission controller or external router chooses one rank, after which prefill and decode normally remain on that rank for the lifetime of the request. Moving an in-flight request requires transferring its cache or recomputing the prefix, so request assignment is also a cache-ownership decision. There are two scheduling levels: 1. **cross-rank admission:** choose the attention-DP rank for a new request; 2. **rank-local continuous batching:** order and batch requests that already belong to that rank. A rank-local radix scheduler can exploit prefix structure among requests that have already arrived, but it cannot by itself discover that another rank owns a better prefix hit. Cross-rank cache awareness belongs in the admission router. The current public internal policies are simpler than a joint cache-and-cost model: - vLLM's internal DP client scores each engine as `4 * waiting_requests + running_requests`, chooses the minimum, locally increments the waiting estimate between coordinator updates, and rotates tie breaking. Its own deployment documentation says the internal policy is not yet KV-cache aware. An external router can specify `X-Data-Parallel-Rank`. - SGLang's in-process DP controller defaults to round robin outside PD mode. It also exposes `total_requests` and `total_tokens`; the latter chooses the rank with the smallest current token load and charges a new request by its input-token count. `X-Data-Parallel-Rank` / `routed_dp_rank` bypasses the internal choice. These policies do not perform a cross-rank prefix lookup. - SGLang Model Gateway has a cache-aware policy that maintains an approximate request-prefix tree and abandons affinity when absolute and relative load thresholds declare a serious imbalance. A newer experimental router can instead consume KV-cache events, match token-block hashes, and choose the least-loaded worker among the actual prefix holders. The pinned implementation evidence is [vLLM's DP client](https://github.com/vllm-project/vllm/blob/ace9fda495bc9a132584e054cc909d504164141f/vllm/v1/engine/core_client.py), [SGLang's DP controller](https://github.com/sgl-project/sglang/blob/bab1dd0d129531d25f23b756b6ebd423a0a978e2/python/sglang/srt/managers/data_parallel_controller.py), [Model Gateway's approximate cache-aware policy](https://github.com/sgl-project/sglang/blob/bab1dd0d129531d25f23b756b6ebd423a0a978e2/sgl-model-gateway/src/policies/cache_aware.rs), and the [KV-event-fed experimental policy](https://github.com/sgl-project/sglang/blob/bab1dd0d129531d25f23b756b6ebd423a0a978e2/experimental/sgl-router/src/policies/cache_aware_zmq.rs). These details are version-sensitive. An Agent-oriented router should compare saved prefill work with added queue delay rather than optimize either load or affinity alone. A useful conceptual cost for candidate rank $r$ is $$ C_r = C_{\text{queue},r} + C_{\text{uncached prefill},r} + C_{\text{expected decode},r} + C_{\text{cache pressure},r}. $$ This yields the following policy shape: 1. preserve hard affinity for an in-flight request; 2. for a new turn, find ranks holding a meaningful prefix and choose the least-loaded holder; 3. break affinity if its queue-delay penalty exceeds the prefill work saved; 4. use least-token, least-load, or power-of-two choice when no useful prefix exists; 5. replicate very hot shared prefixes across several ranks so one common system prompt does not create a permanent hotspot. Request count is only a rough load signal. A better V4 policy would account for uncached suffix length, current context length, expected output length, MTP width, and the context-dependent CSA index scan. Balanced attention token counts also matter to the shared MoE group: ranks with no useful local work still participate in aligned forward progress, while highly uneven token counts increase idle or padding exposure around expert collectives. For clarity, when TP is greater than one and DP-Attention is disabled, the same request executes across the attention TP group; it is not assigned to only one TP rank. V4's shared C4 main, C128 main, and index histories are normally present on every query-head TP rank, which is the replication and arithmetic-intensity cost derived above. Request-level ownership begins only when attention DP is enabled. ## 6. Current Blackwell Starting Points ### 6.1 SGLang B200 recipes The SGLang cookbook publishes three workload points for each model: | Model | Goal | Published B200 shape | MoE layout | MTP | | --- | --- | --- | --- | --- | | V4-Flash | low latency | TP4, FlashInfer MXFP4 MoE | MoE TP4 | EAGLE 3 steps, 4 draft tokens | | V4-Flash | balanced | TP4, DPA4, DeepEP | EP4 | 1 step, 2 draft tokens | | V4-Flash | high throughput | TP4, DPA4, MegaMoE | EP4 | disabled | | V4-Pro | low latency | TP8, FlashInfer MXFP4 MoE | MoE TP8 | EAGLE 3 steps, 4 draft tokens | | V4-Pro | balanced | TP8, DPA8, MegaMoE | EP8 | 1 step, 2 draft tokens | | V4-Pro | high throughput | TP8, DPA8, MegaMoE | EP8 | disabled | The generated cells also tune chunked prefill, CUDA Graph batch limits, static memory fraction, and SWA/full-token pool ratios. Those values should be copied from the pinned recipe rather than reconstructed from this summary. The MTP pattern is meaningful: wide speculation is a low-batch latency tool; the balanced point uses a smaller verify width; the saturated throughput point spends the compute budget on a larger ordinary batch and disables MTP. SGLang also requires $$ \text{max-running-requests}\times\text{draft-tokens} \leq \text{DeepEP dispatch capacity per rank}. $$ Changing MTP width without resizing the dispatch buffer can therefore fail even if the model and KV cache fit. Hardware minima in that cookbook are: - Flash: B200/B300/GB200/GB300/H200 use TP4; H100 uses TP8; - Pro: B200/B300 use TP8; GB300 can use TP4; GB200 uses two four-GPU trays; H200 uses TP8 for the mixed FP4 checkpoint and TP16 for converted FP8. ### 6.2 vLLM B200 recipes The vLLM implementation note gives a simple throughput-oriented starting point: | Model | Published Blackwell shape | Attention/MoE layout | | --- | --- | --- | | V4-Flash | 4 B200 or B300 | DP4 attention plus expert parallelism | | V4-Pro | 8 B200 or B300 | DP8 attention plus expert parallelism | Both use FP8 KV, 256-token logical blocks, full-and-piecewise CUDA Graphs, and an optional FP4 indexer cache. The current recipe adds a Blackwell MegaMoE backend. MTP is opt-in, normally with two speculative tokens; it should be benchmarked against the intended concurrency rather than enabled by habit. vLLM also publishes PD-disaggregated variants. Its current Flash recipe uses one four-GPU GB200 prefill pool and one four-GPU decode pool. Pro uses eight GB200 GPUs per pool or four GB300 GPUs per pool. The normal single-node recipes do not turn on PCP or DCP. ### 6.3 Practical starting rule For Blackwell, a reasonable initial choice is: - batch one or a few interactive requests: SGLang's TP-only low-latency cell; - ordinary multi-user serving: DP-Attention plus EP, with modest MTP; - saturated offline/high-QPS serving: DP-Attention plus EP/MegaMoE, initially with MTP disabled; - long cold prompts with strict TTFT: evaluate PD and PCP separately; - a single cache that cannot fit one attention rank: current public recipes do not solve this merely by setting TP; investigate real DCP or cache tiering. These are starting points, not universal winners. Prefix-hit rate, context distribution, request arrival process, and output length can change the result. ## 7. How Many Tokens Fit On 4 B200s? There are three different answers. ### 7.1 Model-visible maximum context Both V4-Flash and V4-Pro advertise $$ L_{max}=1{,}048{,}576 $$ tokens per request. The official vLLM Flash command is runnable on four B200s; SGLang likewise marks Flash TP4 on B200 as verified. This establishes a 1M single-request target, not the total resident pool. V4-Pro's current vLLM recipe declares approximately 960 GB minimum VRAM for the mixed FP4-expert/FP8-dense checkpoint. Four 192-GB B200s provide at most 768 GB before runtime overhead. Therefore: | Model on 4 B200s | Can the official checkpoint load? | Per-request architectural limit | Stock deployment resident-token answer | | --- | --- | ---: | --- | | V4-Flash | yes | 1,048,576 | at least one 1M request is the official target; aggregate pool is runtime-derived | | V4-Pro | no | 1,048,576 | zero, because the official model does not fit; use at least 8 B200s or a different unsupported/offloaded layout | It would be misleading to say that Pro “supports fewer context tokens on four B200s.” In the official layout, it supports no requests there because weight capacity fails before KV capacity is considered. ### 7.2 Attention-payload capacity model Chapter 04 derived a Flash attention-cache subtotal of about 3.43 GiB per 1M request at current serving entry formats. Applying the same entry formats to Pro's 30 CSA and 31 HCA layers gives about 4.92 GiB per 1M request: | Model | CSA/HCA schedule | Analytical 1M attention-cache payload | Payload bytes/original token | | --- | --- | ---: | ---: | | V4-Flash | 21 CSA, 20 HCA, plus local-only layers | 3.43 GiB | about 3,517 B | | V4-Pro | 30 CSA, 31 HCA | 4.92 GiB | about 5,036 B | This is a lower-level payload calculation, not a server admission number. It excludes compressor-state allocation, MTP state, allocator rounding, fragmentation, CUDA Graph and MoE workspaces, and engine-specific hybrid-pool rules. If an attention rank has a measured payload budget $M_{KV}$ after all other allocations, the idealized native-token equivalent is $$ T_{payload}\approx\frac{M_{KV}}{b_{token}}. $$ For intuition only: | Measured payload budget on one attention rank | Flash native-token equivalent | Pro native-token equivalent | | ---: | ---: | ---: | | 32 GiB | 9.77M | 6.82M | | 64 GiB | 19.54M | 13.65M | | 96 GiB | 29.31M | 20.47M | | 128 GiB | 39.07M | 27.29M | Do not use this table as a launch guarantee. In TP-only mode, the shared cache is replicated and the limiting value is the smallest per-rank pool. In DP-Attention mode, each rank owns different requests, so a balanced service can add the per-rank capacities across DP replicas, but one request still cannot exceed its owner's capacity. ### 7.3 The authoritative runtime number The authoritative capacity is the value reported after the engine profiles weights, activations, collectives, kernels, and graph workspaces. For SGLang: ```bash curl -s http://127.0.0.1:30000/server_info \ | jq '{max_total_num_tokens, max_running_requests, context_length}' ``` The startup log also prints `max_total_num_tokens`. Under DP-Attention it should be recorded per scheduler/DP rank together with the number of ranks and pool configuration; a single scalar without that topology is ambiguous. For vLLM, preserve these startup lines: ```text Available KV cache memory: ... GiB GPU KV cache size: ... tokens Maximum concurrency for 1,048,576 tokens per request: ...x ``` vLLM documents `GPU KV cache size` as the total tokens that its configured GPU KV cache can hold at once. Hybrid-cache accounting has changed across engine versions, so the version, command, and successful admission test are part of the result. One public 8-B200 V4-Pro report on vLLM 0.21.0 recorded 2,093,176 cache tokens and a reported 2.00x concurrency at a 1M maximum. It also completed coherent 1M-prose requests, but this is a user-submitted [performance report](https://github.com/vllm-project/vllm/issues/43753), not a vendor capacity guarantee. No equally controlled upstream 4-B200 Flash startup log was found in this pass. Capturing that exact number is therefore a profiling TODO rather than a guessed constant. ### 7.4 Required 4-B200 capacity experiment For Flash, launch both the low-latency TP4 and throughput DPA4+EP4 recipes and record: 1. engine commit/image and exact checkpoint revision; 2. per-GPU model, CUDA Graph, MoE workspace, and cache-pool bytes; 3. server-reported total token slots per DP scheduler; 4. successful admission at 1x, 2x, 4x, 8x, and 16x 1M-equivalent resident tokens, using coherent text rather than random token IDs for correctness; 5. prefix-sharing behavior, cache hit rate, TTFT, ITL, and eviction; 6. the same sweep with MTP off and with the recipe's MTP width; 7. HiSparse/HiCache or vLLM offload as separate configurations, not silently mixed with the device-only baseline. The useful answer is ultimately a curve of admitted tokens versus latency and cache tier, not one maximum integer. ## 8. Open Questions - Where is the TP-only versus DPA+EP break-even on B200 for 8K, 128K, and 1M histories? - How much HBM do redundant EPLB experts take away from V4 cache pools? - Does the FP4 indexer cache move the break-even mainly through index bandwidth or through capacity? - Can a scheduler use context length and prefix affinity to route long-lived Agent sessions without creating hot DP ranks? - How much larger can MTP verification batches become before V4's indexer and MoE paths become compute-bound? - Can future V4 DCP aggregate device capacity for one request without losing the DPA arithmetic-intensity advantage for ordinary requests? ## 9. Bottom Line `tp=8` is a launch-group size, not a complete description of execution. In the SGLang low-latency recipe it means attention TP8 and expert TP8; in the throughput recipe, DPA8 and the MoE A2A backend resolve it into attention TP1 plus EP8. That split is static for the server, while token routing and expert replica placement remain dynamic. The architectural reason for DPA is unusually strong in V4: TP divides query heads and arithmetic but replicates the shared KV/index history, so per-rank arithmetic intensity falls. DPA restores full query-head reuse on one cache copy and pairs naturally with EP for the large routed-expert weights. Finally, “maximum tokens” must retain its unit and scope. Flash targets one 1M request on 4 B200s, but total resident tokens are a measured runtime property. Pro requires 8 B200s in the official layout; on 4 B200s the weight-fit answer is zero before long-context cache capacity is even relevant.