# DeepSeek-V4 Attention Inference Systems Status: upstream-backed systems pass with cache-tier update; inspected through 2026-07-20 This chapter separates the serving problem into prefill and decode, then asks when Context Parallelism (CP) is useful in each phase. That distinction is essential for DeepSeek-V4: a prefill implementation can split the query-token axis while retaining or reconstructing a full cache on every rank, whereas a decode implementation must shard the persistent cache to obtain a memory benefit. The short answer is: - **Inference does use CP**, especially for very long prefills. Current SGLang has an explicit DeepSeek-V4 prefill-CP path. - **Prefill CP can encounter the same replication pressure as training CP.** In SGLang's current V4 path, rank-local BF16 KV and compressor inputs are all-gathered into global token order before cache insertion/compression. Query and activation work is distributed, but the completed cache is not thereby sequence-sharded for decode. - **Decode CP is a different algorithm.** A useful decode-CP design shards the persistent cache by compressed position, computes attention on each shard, and combines partial softmax results. It should not all-gather the full KV history every step. - **CSA makes decode CP harder than dense MLA.** Each rank first scans its local index-key shard, then the ranks need an exact global top-k and either owner-local sparse attention or selected-KV routing. HCA needs only the standard exact partial-softmax combine. - Public support is still uneven. At the inspected revisions, vLLM has generic PCP/DCP infrastructure but its V4 backend does not advertise PCP, and the V4 compressed indexer explicitly rejects DCP when `compress_ratio > 1`. SGLang's code clearly establishes V4 prefill CP; its generic DCP facilities do not, by themselves, prove an end-to-end V4 decode-CP path. Two companion notes keep deployment policy and workload-specific questions out of this already long mechanism chapter: - [`04a-serving-parallelism.md`](./04a-serving-parallelism.md) explains current vLLM/SGLang Blackwell recipes, resolves TP/EP/DP-Attention, and separates 1M model context from runtime resident-token capacity; - [`04b-long-context-agent-serving.md`](./04b-long-context-agent-serving.md) treats long-lived coding agents as a first-class inference workload and defines the 4-B200 capacity, cache-tier, MTP, and mixed-latency experiments. The implementation snapshots used below are: - vLLM `main` commit [`e6d1310b2ac`](https://github.com/vllm-project/vllm/commit/e6d1310b2ac52e56e266450693af9de12a871a51), dated 2026-07-19; - SGLang `main` commit [`a03ca46a284`](https://github.com/sgl-project/sglang/commit/a03ca46a2847eced368d38331991264b06852602), dated 2026-07-19; - the vLLM [DeepSeek-V4 implementation note](https://github.com/vllm-project/vllm-project.github.io/blob/main/_posts/2026-04-24-deepseek-v4.md); - DeepSeek's [FlashMLA repository](https://github.com/deepseek-ai/FlashMLA), which documents sparse prefill/decode kernels, FP8 decode cache, and the output plus log-sum-exp interface needed by split attention. “Supported” below means visible in the cited code snapshot. It is not a claim about an unreleased internal serving stack or every vendor backend. ## 1. Start With The Serving State Machine DeepSeek-V4 does not have one uniform KV cache. A request carries several objects with different append rates and precision: | Object | Layer use | Logical append cadence | Representative deployed entry | | --- | --- | ---: | ---: | | raw sliding-window KV | CSA, HCA, and window-only paths | every original token | 512 shared-KV dimensions | | C4 main compressed KV | CSA | every 4 original tokens | 448 FP8 NoPE + 64 BF16 RoPE + scales | | C4 index key | CSA | every 4 original tokens | 128-dimensional MXFP4 key + scales | | C128 main compressed KV | HCA | every 128 original tokens | same main-KV format | | compressor residual state | CSA/HCA | updated every token, bounded | C4 overlap or C128 partial group | For a newly accepted token at position $t$, the conceptual update is: 1. write the raw shared-KV entry into the 128-token local ring; 2. update the main compressor state; 3. on CSA layers, update the separate indexer compressor state; 4. when a C4 or C128 group completes, append the corresponding compressed entry; append the C4 index key at the same C4 boundary; 5. form the compressed-history candidate set and union it with the causal local window; 6. run CSA sparse or HCA dense-compressed MQA, inverse-RoPE the output, and apply the grouped output projection. An incomplete group is state, not a provisional globally visible entry. This is why prefix caching, speculative rollback, and prefill/decode transfer must preserve more than the append-only main cache. The public engines encode this heterogeneity explicitly. vLLM uses a common 256-native-token logical block, then maps C4, C128, SWA, indexer, and residual state into a few physical page-size buckets. SGLang similarly maintains separate SWA, C4, C4-indexer, C128, and compression-state pools. ## 2. Prefill And Decode Are Different Workloads ### 2.1 Cold and cached prefill For a cold prompt of $L$ tokens, all $L$ query rows exist at once. The system must: - project and compress the prompt into C4/C128 histories; - generate C4 index keys and top-k indices for CSA query rows; - execute causal sparse/dense-compressed attention and local attention; - persist the completed caches and the boundary state needed by later decode. Chunked prefill bounds temporary workspace and scheduler stalls but does not change total cache size. A chunk can attend to a cached prefix plus earlier tokens in the current chunk, so its metadata must distinguish cached pages, newly completed compressed entries, the causal local window, and the current incomplete tail. On a prefix hit, the engine ideally restores completed C4/C128 pages and the correct residual handoff state. vLLM avoids a separate residual snapshot protocol by representing the compressor residual like SWA state. A hit occurs on the common logical block boundary, at which the rolling state is already a valid handoff point. ### 2.2 Autoregressive and speculative decode Ordinary decode has one query token per request. MTP or another speculative method raises this to a small number of query rows, but it is still far from a long prefill. The history-dependent work is: | Path | Long-context read/compute behavior per query and layer | | --- | --- | | CSA indexer | scan all $L/4$ compressed index keys, then select 512 | | CSA main | read 512 selected main entries plus the local 128 | | HCA main | read all $L/128$ main entries plus the local 128 | | local-only | read at most the local 128 | At 1M context, the CSA index scan, rather than selected main attention, is the dominant sequence-dependent arithmetic derived in [`01-csa-hca-definition.md`](./01-csa-hca-definition.md). HCA scans only about 8K compressed entries, so it has no routing phase and is easier to distribute. Speculation adds state-management constraints. Draft tokens can cross a C4 or C128 completion boundary; rejected tokens must not leave accepted compressed entries or compressor state behind. A robust implementation needs tentative state slots plus commit/rollback, or must recompute bounded state after the accepted prefix is known. This is one reason that “the kernel accepts $s_q>1$” is not equivalent to full MTP integration. ## 3. What Does “Inference CP” Mean? The following mechanisms are often all called CP even though their memory contracts differ: | Mechanism | Split axis | Main purpose | Is persistent KV sharded? | | --- | --- | --- | --- | | training CP | training sequence/activation | fit long forward/backward | not necessarily; current V4 path all-gathers compressed KV | | prefill CP (PCP) | prompt query-token rows | reduce TTFT/activation per rank | not necessarily | | decode CP (DCP) | cached history positions | reduce per-rank cache and history scan | **yes, if implemented usefully** | | TP | heads/projection weights | fit weights and GEMMs | shared MLA KV is commonly replicated | | attention DP / DPA | requests | throughput and cache isolation | each request stays on one replica | | PD disaggregation | serving phase | specialize prefill/decode workers | cache is transferred between pools | This yields the practical decision rule: - if each request fits and aggregate throughput is the goal, request-level DP or DPA is usually preferable because it avoids per-layer CP collectives; - PCP becomes attractive when a single long prefill has unacceptable TTFT or activation/workspace size; - DCP becomes attractive when a single request's persistent cache or long-history decode scan is too large for one attention rank; - PCP and DCP may use the same physical ranks, but they should not be assumed to use the same cache layout or communication algorithm. ## 4. Prefill CP: Where The Training Problem Reappears With $P$ prefill ranks, each rank receives roughly $L/P$ query rows. Every local query still needs causal access to global history. There are two generic ways to provide it: 1. **partial Q, full KV:** all-gather or otherwise replicate the relevant KV, then compute local query rows against it; 2. **partial Q, partial KV:** retain sequence-sharded KV, circulate blocks or compute local attention fragments, then combine exact partial softmax. The first is easier and often good enough for bounded chunks, but it does not reduce per-rank persistent-cache capacity. It also resembles current training CP: sequence compute is divided, while the global attention memory is made available to every rank. ### 4.1 Current SGLang V4 evidence SGLang's V4 prefill path uses attention CP metadata and round-robin query-row reindexing. The model then: - all-gathers rank-local BF16 shared KV into global token order before the V4 cache/store path; - all-gathers compressor gate-score or input tensors before forming compressed entries; - all-gathers the final hidden states on the last pipeline rank. The relevant code is the pinned [`deepseek_v4.py`](https://github.com/sgl-project/sglang/blob/a03ca46a2847eced368d38331991264b06852602/python/sglang/srt/models/deepseek_v4.py) and [`dsv4/compressor.py`](https://github.com/sgl-project/sglang/blob/a03ca46a2847eced368d38331991264b06852602/python/sglang/srt/layers/attention/dsv4/compressor.py). [SGLang v0.5.12](https://github.com/sgl-project/sglang/releases/tag/v0.5.12) advertises DeepSeek-V4 Context Parallelism, PD disaggregation, and HiSparse, but the source is needed to determine which phase and layout a generic release bullet actually denotes. This is not identical to Megatron training CP: - inference has no backward, saved activations, or indexer auxiliary loss; - chunked prefill can bound the current all-gather workspace; - a cached prefix may already be resident; - the inference cache must survive into later scheduler steps. Nevertheless, the capacity concern is the same if every PCP rank finishes with a full persistent cache: adding ranks improves prefill parallelism but does not divide cache bytes per request. ### 4.2 Why the all-gather need not be 1M at once The current-chunk all-gather volume is proportional to the configured prefill chunk rather than requiring one monolithic 1M-token buffer. Conceptually, for a BF16 shared-KV width of 512 and a chunk of $C$ original tokens, a raw-KV all-gather moves on the order of $$ \frac{P-1}{P}\,C\times512\times2 $$ bytes per rank and layer, before compressor/indexer tensors and collective algorithm constants. Across all chunks, the total remains linear in prompt length. More importantly, completed pages can remain replicated after the temporary gather disappears. Thus chunking fixes peak workspace and scheduling granularity, not the fundamental cache-replication problem. ## 5. Decode CP: Shard Cache, Do Not Reconstruct It Splitting the query axis is unhelpful for a single-token decode. A useful DCP layout instead assigns compressed history position $j$ to an owner rank, for example $$ \operatorname{owner}(j)=j\bmod P. $$ Every rank receives or reconstructs the small query, attends to local cache entries, and contributes a partial result. For local score set $S_p$, define $$ m_p=\max S_p, \qquad \ell_p=\sum_{i\in p}\exp(s_i-m_p), \qquad u_p=\sum_{i\in p}\exp(s_i-m_p)v_i. $$ The exact global result is $$ m=\max_p m_p, $$ $$ Z=\sum_p e^{m_p-m}\ell_p, \qquad o=\frac{\sum_p e^{m_p-m}u_p}{Z}. $$ Therefore ranks exchange partial output plus LSE/max statistics, not the full KV history. vLLM's generic DCP offers all-gather plus reduce-scatter and an [all-to-all alternative](https://github.com/vllm-project/vllm/blob/e6d1310b2ac52e56e266450693af9de12a871a51/vllm/v1/attention/ops/dcp_alltoall.py) around precisely this LSE correction. SGLang's generic DCP also uses position ownership and partial-output correction for supported attention paths. For ordinary dense MLA this is already non-trivial because DCP ranks often reuse TP ranks: queries must be made visible to the relevant DCP ranks, local attention runs over fewer KV positions, and the combined output must return to the original head/TP layout. ### 5.1 HCA maps naturally onto DCP For HCA, rank $p$ owns roughly $L/(128P)$ completed entries. It executes dense attention over those entries, optionally handles a replicated 128-token local window, and participates in the exact partial-softmax combine above. This provides almost ideal sequence-axis scaling for persistent main-cache bytes and history-dependent attention until fixed projection work and collectives dominate. ### 5.2 CSA needs distributed top-k first CSA cannot run main attention until it has selected the global best 512 C4 entries. A sequence-sharded algorithm can be: 1. broadcast or all-gather the small index query as needed; 2. each rank scans its $L/(4P)$ local MXFP4 index keys; 3. each rank retains a local top-512 `(score, global_id)` list; 4. merge the at most $P\times512$ candidates into the exact global top-512; 5. route the selected IDs to their owner ranks; 6. each owner attends to its selected main-KV subset; 7. combine owner-local partial output and LSE exactly. The union of local top-$k$ lists contains the global top-$k$, so this merge is exact; no model approximation is required. The hard parts are systems details: - fused local scan plus top-k without materializing large score tensors; - global-ID and page-table translation; - load balance when selected entries cluster on one owner; - deterministic tie behavior; - CUDA Graph-compatible collective sizes; - overlap with compressor updates, MoE, and the other attention streams; - MTP batches in which several query rows select different owner sets. An alternative is to all-gather the selected main KV after global top-k and run attention on a designated rank. At $k=512$, this is fixed-size rather than $O(L)$, but it gives up some bandwidth advantage. Owner-local attention plus LSE combine communicates a partial output of order $n_hc$ per query; selected KV gather communicates order $kc$. Which is faster depends on TP layout, batch/spec width, topology, kernel fusion, and load balance. ### 5.3 SWA and incomplete compression state The local 128-token ring is small enough that replication is often preferable to another distributed irregular read. At the intended 584-byte runtime main format, one local ring is only about 73 KiB per layer and request. Replicated storage must not duplicate its probability mass: only a designated rank may include the local entries in its partial softmax, or the other ranks must mask their copies. The same rule applies to attention sinks and any other replicated candidate. The bounded compressor tail can similarly be replicated, or assigned to the rank that will own the next completed compressed entry. Replication simplifies speculative rollback; ownership reduces redundant update work. Neither state changes the $O(L/P)$ main-cache scaling, but both must participate in prefix restore and PD transfer. ## 6. Does Decode CP Encounter The Same Problem As Training CP? The answer depends on the algorithm: | Property | Megatron V4 training CP / full-KV PCP | Sequence-sharded DCP | | --- | --- | --- | | local query/activation rows | yes | decode query is shared or redistributed | | global compressed history present on every rank | yes after gather/replication | no | | per-rank persistent cache scales as $1/P$ | no | yes, approximately | | backward and activation checkpointing | yes | no | | global top-k over CSA shards | avoided by replicated index history | required | | attention communication | compressed/raw KV all-gather | query + top-k merge + output/LSE combine | | main risk at 1M | replicated KV/workspace | collective latency and irregular ownership | So a naive inference implementation can repeat the training problem, but it is not inevitable. Decode CP trades the replicated-cache problem for a distributed selection and reduction problem. For HCA this trade is straightforward; for CSA it is a major kernel/runtime project. ## 7. Cache Capacity At 1M The payload-only derivation in Chapter 01 uses 576 bytes per main entry and 64 bytes per FP4 index key. Current serving layouts make scale overhead concrete: - vLLM and SGLang represent one main entry as 584 bytes: 448 FP8 NoPE bytes, 128 BF16 RoPE bytes, and 8 scale/padding bytes; - SGLang's packed FP4 indexer layout uses 68 bytes: 64 FP4 payload bytes and 4 scale bytes. Its non-FP4 fallback is larger. For $L=2^{20}$ and the V4-Flash schedule already established in Chapter 01: | Cache component | Count per layer | Runtime bytes/entry | Approximate total | | --- | ---: | ---: | ---: | | 21 CSA main caches | 262,144 | 584 | 3.0 GiB | | 21 CSA FP4 index caches | 262,144 | 68 | 357 MiB | | 20 HCA main caches | 8,192 | 584 | 91 MiB | | 43 local rings | 128 | 584 | 3 MiB | | **attention cache subtotal** | | | **about 3.44 GiB/request** | This is an analytical subtotal, not an allocator measurement. It excludes page padding beyond the entry formats, compressor state, MTP state, temporary workspace, fragmentation, and non-attention model memory. It is slightly higher than Chapter 01's payload-only number because scales are now included. The same entry-format calculation gives about 4.92 GiB for V4-Pro's 30 CSA and 31 HCA layers at 1M. These values are cache payloads, not engine admission limits. Chapter 04a derives the distinction and records how to obtain the authoritative server-reported token pool. The CP consequence is immediate. With $P=8$: - replicated PCP leaves roughly 3.44 GiB per rank per 1M request and consumes about 27.5 GiB of aggregate device memory across the group; - ideal sequence-sharded DCP approaches 0.43 GiB per rank, plus replicated local/tail state and allocator overhead; - batch or concurrency multiplies both values by the resident request count. This is the strongest reason to pursue true V4 DCP. Faster attention is useful, but the larger opportunity is admitting more long requests, larger continuous batches, or wider speculative verification without cache OOM. ## 8. Prefix Caching, PD, And Tiered Cache ### 8.1 Prefix caching A valid V4 prefix-cache hit must be consistent across: - native-token logical length; - C4 main and indexer pages; - C128 main pages; - local/SWA state; - C4 overlapping and C128 incomplete compressor state. Using a common native-token block boundary greatly simplifies this contract. Hits inside an incomplete compressor group either need an exact residual snapshot or must roll back to a prior aligned boundary and recompute the tail. Under DCP, prefix pages also have an owner rank. Reusing a prefix on another DCP topology may require page redistribution; a cache key alone is not enough to restore the physical layout. ### 8.2 Prefill/decode disaggregation PD is attractive because prefill wants large token-parallel work while decode wants persistent cache capacity and stable small CUDA graphs. But the handoff now transfers three compressed cache families plus bounded state. If prefill ranks hold replicated cache and decode ranks use sequence-sharded cache, the transfer should scatter pages directly to final owners instead of first constructing another full decode-side copy. Compression reduces network bytes substantially relative to original-token MLA, but C4 main plus indexer state still dominates at 1M. ### 8.3 Tiering and HiSparse CSA has an unusual offload opportunity: the indexer can identify a small selected working set, but it must scan the index-key history first. A tiered design can keep the complete compact index resident and place colder main KV pages in host memory, then fetch selected pages. This is useful only if page fetch is prefetched/overlapped and selected IDs exhibit enough locality; one synchronous PCIe fetch round per layer can destroy decode latency. HCA lacks a selector and reads all C128 entries, but the cache is already small. Its natural optimization is sequential streaming, not sparse demand paging. Do not conflate this active sparse-fetch path with hierarchical prefix-cache tiering. SGLang currently has a V4 HiSparse path that keeps C4 main in host memory and swaps selected entries into a device buffer, and a separate V4 HiCache path that moves coherent prefix pages and sidecars across device, host, and optional storage. The paths are currently mutually exclusive because HiSparse disables radix caching. vLLM has generic CPU and secondary-tier block offload, but no audited CSA-selector-driven selected-main path. Chapter [`04b-long-context-agent-serving.md`](./04b-long-context-agent-serving.md) records the lifecycle, memory/traffic model, implementation evidence, and future integration requirements. ## 9. Kernel And Runtime Critical Path The V4 decode graph has three mostly independent branches after projection: ```text +--> indexer compress / scan / top-k --+ hidden projection --+--> main C4/C128 compression ---------+--> attention +--> raw SWA insertion -----------------+ ``` The public vLLM implementation overlaps the C4 indexer branch with main compression and SWA insertion; C128 overlaps main compression with SWA insertion. Its reported low-batch end-to-end improvement from this scheduling is 5--6%, while compressor/RMSNorm/RoPE/cache-insert fusion avoids several HBM round trips. Distributed execution adds synchronization points to this graph: - PCP all-gather must finish before a rank consumes global current-chunk data; - DCP CSA must finish global top-k before selected main attention; - DCP main attention must finish the LSE-correct combine before output projection; - speculative state cannot commit until acceptance is known. The main opportunity is therefore not merely a faster standalone FlashMLA kernel. It is scheduling local index scan, top-k merge, selected attention, cache updates, MoE communication, and MTP verification so the new collectives do not serialize the entire layer. ### 9.1 There are three different inference forward graphs Projection GEMMs are broadly shared, but cold/chunked prefill and decode do not use one interchangeable attention forward kernel: | Stage | Prefill | Decode | | --- | --- | --- | | input projections | large token-axis GEMMs; suitable for throughput-oriented GEMM shapes | small-M GEMMs; batching/MTP width is important for utilization | | compressor | process many C4/C128 boundaries and persist completed entries | update bounded state for every token, but emit an entry only at a boundary | | index search | causal top-k for many query rows, often chunked | scan the request's full C4 index history for one or a few query rows | | selected-KV preparation | current vLLM NVIDIA path dequantizes/gathers paged compressed and SWA entries into a contiguous BF16 workspace | keep the cache paged; supply physical selected ids and scheduler metadata directly to the decode kernel | | main attention | `flash_mla_sparse_fwd`-style sparse prefill | `flash_mla_with_kvcache`-style paged sparse decode | | cache behavior | create persistent heterogeneous cache and tail state | read and incrementally mutate that state | | output path | inverse RoPE plus grouped low-rank projection | same mathematics, but much smaller token dimension and stronger launch/fusion pressure | Training forward is closer to prefill in query shape, but still differs from it: training has no persistent paged cache, must retain or recompute state for backward, and includes the auxiliary indexer teacher/loss. Decode removes all backward state and therefore permits more aggressive in-place writes, fixed workspace reuse, CUDA Graph capture, and direct paged-cache consumption. ### 9.2 vLLM's concrete NVIDIA kernel pipeline At the inspected revision, [`DeepseekV4Attention.forward`](https://github.com/vllm-project/vllm/blob/e6d1310b2ac52e56e266450693af9de12a871a51/vllm/models/deepseek_v4/attention.py) separates metadata-independent projection work from metadata-dependent attention. The projection branch can remain inside the captured graph, while the custom attention implementation forms an eager boundary around dynamic request metadata. Its attention-side operator inventory includes: 1. fused Q/KV RMSNorm and RoPE, together with raw-KV cache insertion; 2. learned C4/C128 compression, RMSNorm, RoPE, quantization, and paged-cache insertion; 3. a separate C4 indexer-Q path and FP4 index-key compression; 4. index score scan, top-k, and logical-to-physical id transforms; 5. dequant/gather kernels for selected main/SWA cache entries on prefill; 6. FlashMLA sparse prefill or paged sparse decode; 7. inverse RoPE fused with FP8 quantization before grouped output GEMMs. The fused compressor is particularly different from a simple elementwise epilogue. One token program gathers the contributing state rows, evaluates a small learned-score softmax, reduces them, normalizes in FP32, applies RoPE, quantizes two cache regions with different formats, and writes the physical page. The C128 path also has a two-stage option with FP32 scratch. See the pinned [`fused_compress_quant_cache.py`](https://github.com/vllm-project/vllm/blob/e6d1310b2ac52e56e266450693af9de12a871a51/vllm/models/deepseek_v4/common/ops/fused_compress_quant_cache.py) and [`compressor.py`](https://github.com/vllm-project/vllm/blob/e6d1310b2ac52e56e266450693af9de12a871a51/vllm/models/deepseek_v4/compressor.py). The NVIDIA FlashMLA backend explicitly separates prefill and decode. Decode calls the paged-cache kernel after forming compressed/SWA indices and scheduler metadata. Prefill processes chunks, gathers/dequantizes the required cache into a BF16 workspace, combines sparse and window ids, and calls sparse prefill. This makes “remove the prefill gather workspace by consuming pages directly” a specific optimization opportunity rather than a generic request for a faster attention kernel. The current split is visible in [`nvidia/flashmla.py`](https://github.com/vllm-project/vllm/blob/e6d1310b2ac52e56e266450693af9de12a871a51/vllm/models/deepseek_v4/nvidia/flashmla.py). The engine also launches independent projection/compressor/indexer branches on auxiliary streams when their shapes make overlap worthwhile. The [vLLM implementation note](https://github.com/vllm-project/vllm-project.github.io/blob/main/_posts/2026-04-24-deepseek-v4.md) reports a 5--6% low-batch end-to-end latency improvement from this scheduling and larger local gains from several fusions; those are upstream reports, not measurements reproduced in this repository. ### 9.3 SGLang confirms that the non-FlashMLA surface is large SGLang reaches a similar decomposition with a different implementation mix. Its pinned V4 JIT directory contains dedicated CUDA/CUH kernels for: - C4 and C128 compression, including online and MTP-specialized C128 paths; - separate prefill/decode compression planners; - fused norm/RoPE and main norm/RoPE; - top-k implementations and transforms; - paged-MQA metadata construction and cache stores; - sparse prefill utilities and the final attention call. The Python wrappers expose separate `flash_compress4_decode`, `flash_compress4_prefill`, `flash_compress128_decode`, and `flash_compress128_prefill` paths rather than treating decode as prefill with sequence length one. See [`jit_kernel/dsv4`](https://github.com/sgl-project/sglang/tree/a03ca46a2847eced368d38331991264b06852602/python/sglang/jit_kernel/dsv4) and its [`csrc/deepseek_v4`](https://github.com/sgl-project/sglang/tree/a03ca46a2847eced368d38331991264b06852602/python/sglang/jit_kernel/csrc/deepseek_v4) sources. This is strong code evidence that the interesting implementation surface extends well beyond the final FlashMLA call. ### 9.4 Which parts are simple and which deserve study Individual metadata copies, concatenations, and elementwise transforms can be small kernels. They are not automatically unimportant: on low-batch decode, launch overhead and extra HBM round trips can dominate their arithmetic. The most research-worthy boundaries are: 1. **CSA FP4 scan plus top-k:** likely the largest new sequence-dependent decode component at long context; study scan tiling, dequantization, radix selection, query/spec reuse, and SM90 versus SM100 behavior. 2. **Stateful C4/C128 compression:** study one-stage versus two-stage/online reduction, boundary sparsity, mixed cache formats, and transactional MTP state. 3. **Sparse prefill cache preparation:** determine when BF16 gather workspace is necessary and whether paged gather/dequantization can be fused into the attention consumer. 4. **Inverse RoPE plus output projection:** quantify the value of fusing a mathematically small transform with quantization and low-rank GEMMs. 5. **Runtime scheduling:** characterize whether auxiliary-stream overlap hides useful work or merely moves synchronization, especially with CUDA Graphs, DP Attention, MoE communication, and changing batch composition. 6. **Distributed attention:** integrate local index scan, exact global top-k, owner-local attention or selected-KV gather, and exact LSE combination. FlashMLA remains a major kernel family, not a minor detail. Its public interface covers sparse prefill and sparse/paged decode and returns LSE for composition, but it does not decide compression boundaries, create index routes, update residual state, translate pages, or manage speculation. Those surrounding responsibilities explain why a complete V4 attention layer is a kernel-runtime pipeline rather than one kernel. ## 10. Current Public Implementation Matrix | Engine/path at inspected revision | V4 prefill CP | V4 decode CP | Important qualification | | --- | --- | --- | --- | | DeepSeek Python reference | no distributed serving path | no | functional reference and quantization simulation | | vLLM generic attention | PCP infrastructure exists | DCP with AG+RS or A2A/LSE exists | capability does not imply every backend supports it | | vLLM V4 | **not advertised by the custom V4 backend** | **rejected by the compressed C4 indexer** | `DeepseekV4FlashMLABackend` has no PCP impl class; `DeepseekV4IndexerBackend` raises for DCP with ratio $>1$ | | SGLang V4 | **yes** | not established by the inspected V4-specific path | prefill path reindexes queries and all-gathers current KV/compressor inputs | | SGLang generic MLA/MHA | prefill CP exists for supported models | DCP infrastructure exists | backend, platform, cache format, PD, and speculation restrictions apply | Two negative lessons matter: 1. an engine exposing `--decode-context-parallel-size` does not prove that V4's compressed indexer, special cache pools, and MTP path support it; 2. a release note saying “Context Parallelism” does not say whether cache is replicated after prefill or sequence-sharded during decode. The vLLM evidence is pinned in [`backend.py`](https://github.com/vllm-project/vllm/blob/e6d1310b2ac52e56e266450693af9de12a871a51/vllm/v1/attention/backend.py), [`sparse_mla.py`](https://github.com/vllm-project/vllm/blob/e6d1310b2ac52e56e266450693af9de12a871a51/vllm/models/deepseek_v4/sparse_mla.py), and [`mla/indexer.py`](https://github.com/vllm-project/vllm/blob/e6d1310b2ac52e56e266450693af9de12a871a51/vllm/v1/attention/backends/mla/indexer.py). These limitations may change quickly and should be rechecked before profiling. ## 11. TODO: GPU Profiling And Performance Validation No GPU profiling was performed for this pass. The current contribution is a source-backed map of existing vLLM, SGLang, and FlashMLA solutions. The matrix below records the future H100/B200 profiling plan and should not be read as measured behavior on this CPU-only development host. The useful experiment matrix should separate capacity from latency: | Axis | Suggested values | Why it matters | | --- | --- | --- | | context | 8K, 32K, 128K, 512K, 1M | index scan and C128 history scale linearly | | resident batch | 1, 2, 4, 8, 16+ | changes GEMM reuse and cache capacity pressure | | query/spec width | 1, 2, 4, 8+ | increases attention arithmetic intensity and rollback work | | phase | cold prefill, prefix-hit extend, decode, verify | different cache and collective paths | | CP | PCP 1/2/4/8; DCP 1/2/4/8 | distinguish TTFT scaling from cache sharding | | cache layout | replicated, sequence-sharded | the central memory question | | topology | intra-node NVLink, inter-node fabric | top-k and LSE collectives are latency sensitive | | precision | BF16/FP8 main, FP8/MXFP4 index | changes both scan roofline and capacity | Record at least: - per-rank and aggregate persistent cache bytes; - prefill all-gather bytes and overlap; - local index-scan time, global top-k merge time, and owner imbalance; - selected main-attention and HCA local-attention time; - output/LSE combine bytes and time; - cache-update, compression-boundary, and rollback time; - TTFT, inter-token latency, throughput, and maximum resident requests. The critical comparisons are: 1. PCP versus DPA at the same total GPU count and request mix; 2. replicated-cache PCP versus sharded-cache DCP at 128K--1M; 3. CSA distributed top-k plus owner-local attention versus selected-KV gather; 4. DCP with fixed batch/spec width versus the larger batch/spec width enabled by its memory savings. The fourth comparison prevents a common mistake. At fixed batch and spec width that already fit, DCP can be slower because it adds collectives. Its system win may appear only after spending the saved memory on more concurrency or more speculative candidates. For coding agents, this generic matrix is not enough: pause/resume, tool-output suffixes, exact-prefix forks, cache tiers, and mixed 1M+short-request tails must also be represented. Chapter 04b defines that workload and its measurements. ## 12. Opportunities And Open Work ### Near-term engineering - add an exact distributed C4 indexer top-k primitive returning global IDs; - add DCP-aware C4/C128 page ownership and compressed-boundary writes; - choose between selected-KV routing and owner-local attention dynamically; - replicate the tiny SWA/tail state while sharding large completed histories; - scatter PCP/PD output directly into decode-owner layout; - make MTP cache updates transactional across compression boundaries; - capture fixed-size top-k and LSE collectives in CUDA Graph-compatible paths. ### Algorithm/runtime co-design - hierarchical top-k within NVLink islands followed by inter-node merge; - page-aware tie-breaking or routing that improves locality without changing the selected set; - cache-layer splitting for prefill workers when sequence sharding is not yet available; - benchmark and extend SGLang's current device-index/host-main HiSparse path, especially prefix-cache integration and selected-page locality; - selectively reuse routing across nearby CSA layers as an explicit future-work experiment, while retaining HCA for dense compressed-global coverage. ### Open questions - Does a production V4 stack keep PCP cache replicated, scatter it for decode, or use a unified sharded layout throughout? - At what context and batch does global top-k latency erase the local index-scan speedup from DCP? - Is selected-owner imbalance significant for real Agent traces? - How much prefix-cache reuse survives when requests move across DCP group size or PD topology? - Can DCP and MTP be co-designed so additional speculative rows amortize the collectives instead of merely increasing them? - How should a DP-Attention scheduler preserve long-lived coding-agent cache affinity without creating a hot rank? - Can a paused Agent retain C4 index state on device while demoting cold main pages, then resume without a synchronous per-layer host fetch? ## 13. Bottom Line Inference CP is real, but “does V4 use CP?” is underspecified. - **Prefill CP** is already useful and publicly implemented in SGLang for V4. It distributes prompt computation, yet its current all-gather/full-cache semantics can retain the same cache-replication weakness seen in training. - **Decode CP** is the architecture that could divide V4's roughly 3.44 GiB 1M-context attention cache per request across ranks. HCA fits the standard partial-softmax pattern; CSA additionally requires exact distributed top-k and sparse owner routing. - That new algorithm changes the bottleneck from replicated $O(L)$ cache to local $O(L/P)$ index scans plus fixed-size, latency-sensitive collectives. Whether it is faster at a fixed batch is workload- and topology-dependent, but its capacity gain can enable larger batches and wider speculation. The next implementation milestone should therefore be judged not by a generic CP flag, but by four concrete properties: cache ownership, global-top-k correctness, exact LSE combination, and compression-state commit/rollback.