# DeepSeek-V4 For Long-Context Coding Agents Status: workload framing, source-audited cache/routing notes, and experiment TODO; 2026-07-20 Coding agents are an important inference target for DeepSeek-V4, but “supports 1M context” is not a sufficient systems claim. A useful service must keep a long-lived, frequently paused session responsive while it appends tool output, shares repository prefixes, forks attempts, rolls back speculative tokens, and coexists with many short interactive requests. This note defines that workload and turns it into a profiling plan. It builds on the cache mechanics in [`04-inference-systems.md`](./04-inference-systems.md) and the deployment and capacity distinctions in [`04a-serving-parallelism.md`](./04a-serving-parallelism.md). The main hypothesis is: > V4's compressed cache makes very long Agent sessions feasible, but the > dominant limit may move from main-KV capacity to repeated CSA index scans, > cache residency policy, and warm-session scheduling. That is a hypothesis to test, not a production claim. ## 1. Why A Coding Agent Is Not A Long Document Request A document-QA request is often a large cold prefill followed by one moderate decode. A coding agent is a stateful event stream: ```text repository/system prefix -> user task -> model reasoning + tool call -> tool output / code / logs -> model reasoning + edit -> test output -> model reasoning + next tool call -> ... pause, resume, fork, compact, or finish ``` Its distinctive properties are: - **append-heavy:** most turns reuse the exact token prefix and add a suffix; - **long-lived:** a session can remain valuable while the model waits seconds or minutes for a build, test, reviewer, or user; - **bursty:** short decode bursts alternate with idle tool time; - **structured:** source code, diffs, compiler output, JSON, and tool grammar create different MTP acceptance patterns from prose; - **branching:** an agent may try two fixes from the same prefix or roll back an unsuccessful path; - **multi-agent:** several workers may share a repository/system prefix but diverge into task-specific histories; - **state-coupled:** an exact token prefix may refer to a repository state that has since changed, so semantic validity is stricter than token equality; - **tail-sensitive:** the newest files, errors, and tool results often matter more than a uniformly sampled old transcript, but rare early constraints must remain recoverable. This combination rewards prefix reuse and cheap persistent cache, yet punishes an engine that keeps every idle session on HBM or rescans a 1M history at full cost for every small follow-up. ## 2. The Relevant V4 State A warm V4 Agent session contains more than one append-only cache: - C4 main compressed pages for CSA layers; - C4 FP4 index-key pages; - C128 main compressed pages for HCA layers; - local 128-token rings; - overlapping C4 and incomplete C128 compressor state; - prefix-tree metadata and physical page ownership; - tentative MTP state for an in-flight verify step. At an exact prefix hit, all of these objects must correspond to the same native token length and compression boundary. A token hash that restores main pages but not the correct compressor tail can produce a wrong next compressed entry. For an Agent service, the cache key should also include or refer to the external state on which the prefix depends. Examples include repository commit or workspace snapshot, tool schema version, system prompt version, environment variables that enter the transcript, and sandbox identity. Token-level reuse can be mechanically correct while the task context is semantically stale. ## 3. Capacity Is Necessary But Not Sufficient The Chapter 04 runtime-format subtotal is about 3.43 GiB for a 1M V4-Flash attention history and about 4.92 GiB for V4-Pro, before engine overhead. That is small enough that a Blackwell node can plausibly keep multiple long sessions, but there are three independent capacity questions: 1. **single-session capacity:** can one request fit on its attention owner? 2. **device resident capacity:** how many active/warm sessions fit in HBM? 3. **multi-tier logical capacity:** how many paused sessions can be retained in host memory or storage and restored within an acceptable latency? DP-Attention improves aggregate capacity because different ranks own different sessions, but it does not pool HBM for one session. TP-only replicates the same shared history across ranks. DCP could shard one history, but the current public V4 path is incomplete as documented in Chapter 04. A large token count can also be the wrong objective. Keeping 20 inactive 1M sessions on device may reduce cache misses while starving CUDA Graph, MegaMoE/EPLB, or active-batch workspaces and worsening tail latency. Capacity must be measured together with service-level objectives. ## 4. Where V4 Helps A Long Agent ### 4.1 Cheap durable global memory CSA stores one main and one index entry per four original tokens; HCA stores one main entry per 128. This reduces the HBM and transfer cost of retaining a long conversation, especially compared with saving every original-token MLA entry at every layer. The savings enable: - more warm Agent sessions per attention rank; - longer exact prefixes before compaction is mandatory; - more cache levels or replicas; - cheaper prefill/decode handoff; - room for a larger ordinary batch or speculative verify width. ### 4.2 Prefix trees match Agent branching Repository instructions, tool definitions, and the early task transcript are often shared. A radix/prefix tree can store that base once and attach multiple attempt suffixes. This is more valuable than treating every branch as an independent 1M cache. The V4-specific difficulty is that a legal branch boundary must restore the compressor state. Native logical blocks that align all cache families simplify this; a branch inside a C4/C128 partial group either needs an exact state snapshot or bounded tail recomputation. ### 4.3 Idle tool time creates offload opportunity While an agent waits on a compiler or test, its GPU cache is not producing tokens. The engine can demote cold completed pages to host memory or storage, retain a compact directory/index, and prefetch before the tool result returns. Two different mechanisms must not be collapsed into one diagram: - **HiCache-style prefix tiering** moves coherent prefix pages between device, host memory, and storage. It exploits reuse across requests or turns. - **HiSparse-style active retrieval** keeps the selector on device, leaves the large C4 main history in host memory, and fetches only selected entries while the request is decoding. It exploits query-dependent sparse access. They are both forms of KV virtual memory, but they operate at different times and granularities. #### 4.3.1 HiCache: prefix-page virtual memory HiCache is analogous to a multi-level page cache: ```text device / HBM: active prefix pages host DRAM: warm prefixes and temporarily paused sessions storage: cold prefixes, long-paused sessions, or shared persistent cache ``` Completed GPU pages can be written through asynchronously, written back only on eviction, or selected by a policy. A later request performs an exact-prefix lookup, promotes the hit from storage to host and then device, and computes only the new suffix. This changes a prefix miss/re-prefill problem into a resume-transfer problem: $$ T_{\text{resume}} \approx \frac{\text{restored bytes}}{\text{tier bandwidth}} + T_{\text{lookup and scheduling}}. $$ HiCache therefore expands the number of retained or warm sessions, but it does not normally reduce the GPU working set of a request after that request has been promoted and admitted to run. V4 requires coherent treatment of more than one tensor. A logical prefix hit must restore C4 main, C4 index, C128 main, the relevant SWA/local tail, and compressor state at one valid native-token boundary. The current SGLang V4 HiCache assembler creates host-side entries for these cache families and can attach file, Mooncake, HF3FS, NIXL, or other storage backends. This is an implemented V4-specific extension of generic hierarchical prefix caching, not merely the design sketch in this chapter. See the pinned [V4 HiCache assembler](https://github.com/sgl-project/sglang/blob/bab1dd0d129531d25f23b756b6ebd423a0a978e2/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py) and [storage pool definitions](https://github.com/sgl-project/sglang/blob/bab1dd0d129531d25f23b756b6ebd423a0a978e2/python/sglang/srt/mem_cache/hicache_storage.py). vLLM also has a generic `OffloadingConnector`: completed blocks can be copied to pinned CPU memory and an optional filesystem, object-store, or P2P secondary tier, then promoted on a future hit. That is block-level prefix tiering. It should not be described as a CSA selector-aware implementation without separately validating all V4 cache groups and state. See the pinned [vLLM offloading guide](https://github.com/vllm-project/vllm/blob/ace9fda495bc9a132584e054cc909d504164141f/docs/features/kv_offloading_usage.md). #### 4.3.2 HiSparse: active CSA working-set virtual memory CSA is unusually suitable for a different split because its index entry is much smaller than its main entry and the indexer chooses a fixed 512/1024-entry working set. For an active request, the natural exact layout is: ```text device: full C4 index + selected/hot C4 main buffer + C128 + SWA/tails host: full C4 main history ``` For each CSA layer and decode step: 1. the GPU scans the complete C4 index and produces logical top-k positions; 2. a per-layer device map checks which selected positions are already in the hot buffer; 3. misses are loaded from pinned host memory into empty or LRU device slots; 4. logical positions are rewritten into physical device locations; 5. selected main attention reads only those device slots; 6. newly completed C4 entries are backed up asynchronously to host memory. SGLang currently implements this path for DeepSeek V4 behind `--enable-hisparse`. Its V4 allocator keeps a full logical history, a host mirror of C4 main pages, and a bounded device buffer; the indexer calls the coordinator after top-k so a JIT kernel can load missing selected entries. The model's `index_topk` supplies 512/1024 while the configured device buffer can be larger, allowing temporal reuse across adjacent queries. See [HiSparseCoordinator](https://github.com/sgl-project/sglang/blob/bab1dd0d129531d25f23b756b6ebd423a0a978e2/python/sglang/srt/managers/hisparse_coordinator.py), [the V4 HiSparse allocator](https://github.com/sgl-project/sglang/blob/bab1dd0d129531d25f23b756b6ebd423a0a978e2/python/sglang/srt/mem_cache/allocator/hisparse.py), and the [V4 indexer handoff](https://github.com/sgl-project/sglang/blob/bab1dd0d129531d25f23b756b6ebd423a0a978e2/python/sglang/srt/layers/attention/dsv4/indexer.py). This is close to the active device/host split proposed here, but its current scope and constraints matter: - it primarily virtualizes C4 main; C128 and SWA remain ordinary device working sets; - storage is not part of the per-layer selected-entry critical path; - current SGLang requires `--disable-radix-cache`, so HiSparse and HiCache are mutually exclusive and cross-request prefix reuse is sacrificed; - the ROCm unified-KV V4 path is currently rejected; - it is an opt-in path whose throughput and latency need workload-specific profiling, not a default production guarantee. The radix-cache restriction reflects an ownership conflict. A radix node normally points to stable reusable device pages, while HiSparse continually recycles small per-request/per-layer device slots for different logical C4 entries. Without integrated versioning and tier-aware ownership, an old prefix page could point at a recycled slot. The current implementation avoids that correctness hazard by disabling radix reuse. #### 4.3.3 Capacity and traffic model Let $N_4=L/4$, let $D_m$ be bytes per C4 main entry, let $D_i$ be bytes per index entry, let $W$ be device-buffer entries per request, and let $N_{CSA}$ be the number of CSA layers. For one request, a fully device-resident layout costs $$ M_{\text{ordinary}} \approx N_{CSA}N_4(D_m+D_i), $$ whereas a HiSparse-like layout costs approximately $$ M_{\text{device}} \approx N_{CSA}(N_4D_i+WD_m), \qquad M_{\text{host}} \approx N_{CSA}N_4D_m. $$ HiSparse therefore removes most C4 main residency from HBM, but it does not make all CSA device state constant in sequence length: the exact full index is still $O(L/4)$. At 1M context with a 68-byte FP4 index entry and a 584-byte main entry, one CSA layer has roughly 17 MiB of index and 146 MiB of main history before allocator overhead. For 21 Flash CSA layers, the C4 main subtotal is about 3 GiB while the index subtotal is about 0.35 GiB. A 4,096-entry main buffer is about 48 MiB per Flash request across those layers. These are explanatory subtotals, not measured SGLang allocations. The worst-case Host-to-device traffic if every selection misses is roughly $$ 512\times584\times21\approx6\ \text{MiB/token/request} $$ for Flash and $$ 1024\times584\times30\approx17\ \text{MiB/token/request} $$ for Pro. Device-buffer hits reduce this traffic; batch multiplies it. The important empirical question is therefore the per-layer temporal locality of selected IDs, not only the nominal top-k. #### 4.3.4 Why the index and storage tiers have different rules An exact CSA query must inspect the complete C4 index. Scanning that history synchronously over PCIe every layer and token would replace an HBM scan with a much slower Host scan and erase the point of sparse main attention. For an active session, the complete index should remain device resident, be consumed through coarse overlapped chunks, or be replaced by a hierarchical/approximate index that changes the algorithmic contract. Merely retaining “hot index pages” is not sufficient for exact flat top-k. Storage is slower again. It is appropriate for a paused session whose complete state will be restored before scheduling, but not for demand-fetching each layer's selected entries. HCA also lacks a selector and reads all C128 entries; because that history is already compressed by 128, the natural policy is to keep it on device while active and stream/prefetch it when resuming. The corrected three-tier picture is consequently phase-dependent: ```text active session: device = full index + C128/SWA/tails + selected C4 main buffer host = full C4 main history short tool pause: retain the active working set when capacity permits long pause: demote every coherent cache family to host/storage restore index, C128, SWA, and tails before decode resumes ``` #### 4.3.5 A future integrated Agent cache Conceptually the two mechanisms compose well: HiCache can preserve a paused session and HiSparse can bound its active C4-main HBM working set. A production integration would need one logical page namespace, coherent V4 sidecars, tier-aware radix ownership, stable slot versions, transactional MTP/tail state, and no duplicate host copy owned independently by both systems. Tool completion signals could prefetch the index and dense compact state before the next request arrives. This integrated design is future work. Current public SGLang contains the two important pieces as separate, mutually exclusive paths; current public vLLM contains generic multi-tier block offload but no audited CSA top-k-driven selected-main fetch path. ## 5. Where V4 May Become The Bottleneck ### 5.1 CSA rescans the compressed history every step At 1M context, one Flash CSA layer scans 262,144 C4 index entries. The Chapter 01 cost model estimates about 4.295 billion low-precision index QK FLOPs per query token and layer; there are 21 Flash CSA layers and 30 Pro CSA layers. The selected main attention remains fixed at 512 entries for Flash or 1,024 for Pro, but the index scan grows with history. Consequently, compression fixes the main-KV capacity problem more aggressively than it fixes the decode arithmetic problem. A coding agent that emits many short decode bursts against a stable 1M history pays that scan repeatedly. Questions to measure include: - whether the FP4 indexer is compute-, HBM-, top-k-, or launch-bound on B200; - how batch and MTP width amortize index-key reads; - whether nearby query tokens reuse enough routing or index tiles to help; - whether session mixing destroys index/cache locality; - whether TP repeats the same scan across ranks, as the shared-KV analysis predicts. ### 5.2 The newest suffix changes quickly Tool output may add tens of thousands of tokens between decode bursts. Even on a prefix hit, the engine must compress and index the new suffix before decode. Chunked prefill prevents one Agent from blocking the server, but too-small chunks raise scheduling and compression-boundary overhead; too-large chunks hurt interactive requests. PD disaggregation can isolate this work, but cache transfer and ownership must include C4/C128 pages and residual state. A 1M warm session should not be round-tripped through another full replicated cache merely to add a 2K tool suffix. ### 5.3 MTP acceptance is phase-dependent Source code and repetitive syntax may yield high MTP acceptance. Tool-call boundaries, JSON fields, compiler messages, and new external observations may yield lower acceptance. A single average acceptance rate hides this mixture. MTP also consumes verify compute and temporary state. It can improve latency at small batch, yet reduce saturated throughput, which is why current SGLang recipes use 3/4 at low latency, 1/2 in the balanced cell, and disable it at high throughput. Coding-agent measurements should report acceptance separately for reasoning, code generation, tool-call serialization, and post-tool recovery. ### 5.4 Long sessions create scheduler affinity Under DP-Attention, a warm session belongs to one attention rank. Moving it requires cache transfer or a miss and re-prefill. A scheduler must balance: - current rank load; - cache/prefix affinity; - session length and remaining HBM; - expected tool-return time; - shared-prefix placement across agents; - MoE expert load, which is a different axis from attention ownership. Naive least-loaded routing can repeatedly move a session; strict affinity can create one hot rank. Length-aware rendezvous hashing or a cache-aware router is a useful starting point, but it needs trace validation. ## 6. Warm, Cold, And Paused Latency Agent serving needs more latency metrics than TTFT and inter-token latency: | Event | Metric | Why it matters | | --- | --- | --- | | first repository request | cold TTFT | includes initial tokenize/prefill and cache construction | | exact-prefix next turn | warm TTFT | measures append-only reuse | | tool returns after a short pause | resume TTFT | exposes scheduling and HBM residency | | cache demoted to host | host-restore TTFT | measures tier transfer and overlap | | cache demoted to storage | cold-tier restore TTFT | determines whether preserving the session beats re-prefill/compaction | | branch from an earlier point | fork latency and incremental bytes | tests prefix-tree and compressor-boundary behavior | | rejected speculative tokens | rollback latency/correctness | tests transactional C4/C128 state | | many long and short sessions mixed | P95/P99 TTFT and ITL | detects head-of-line blocking | For coding workflows, the user-visible objective should include end-to-end task time and cost per successful task. A high token/s system can still be worse if it evicts useful prefixes, delays tool resumption, or forces frequent context compaction. ## 7. A Representative Agent Workload Matrix ### 7.1 Context phases Use at least: | Phase | Suggested history | New suffix | Decode burst | | --- | ---: | ---: | ---: | | repository bootstrap | 32K, 128K, 384K | full cold prefill | 256--2K | | normal edit/test loop | same history | 1K--16K tool suffix | 64--1K | | mature long session | 512K, 768K, 1M | 1K--32K | 64--2K | | exact-prefix fork | shared 128K--768K | two divergent suffixes | 256--2K each | | pause/resume | 128K--1M | 0--4K | 64--1K | Synthetic random IDs are useful for stress but not sufficient for correctness or MTP acceptance. Include real repositories, code, diffs, build logs, tool JSON, and multi-turn chat encoding. ### 7.2 Concurrency and topology Sweep: - 1, 2, 4, 8, 16, and capacity-limited concurrent sessions; - TP-only versus DP-Attention+EP on the same GPU count; - MTP off, balanced, and low-latency widths; - device-only, host-offloaded, and storage-backed cache; - PCP/PD for cold long prefixes, while recording whether the decode cache is replicated or placed directly on its owner; - prefix-hit ratios from 0% to near 100%; - session-idle distributions from subsecond to tens of minutes; - one 1M session mixed with many 8K sessions, not only homogeneous batches. The mixed-length case is critical. A public V4-Pro 8-B200 report found that small requests stretched substantially beside a 1M request even though both were correctness-clean. That observation is workload evidence, not a general performance guarantee, but it motivates isolation or explicit admission control for 1M traffic. ## 8. Required Measurements ### 8.1 Capacity and cache behavior - server-reported token slots per DP rank and aggregate; - bytes in C4 main, C4 index, C128, SWA/local, and compressor-state pools; - prefix-tree physical versus logical bytes; - cache hit, partial-hit, eviction, promotion, and demotion rates; - bytes transferred per resume and PD handoff; - time an idle session remains on each tier; - fragmentation and unusable tail capacity; - EPLB redundant-expert memory and MTP/CUDA Graph workspace. ### 8.2 Attention attribution - CSA index projection, scan, top-k, selected gather, and main attention; - HCA compressed dense attention; - local-window attention and cache append; - compression-boundary and tail-state updates; - TP duplicate index/cache reads; - CP top-k/LSE collective time if a future DCP path is tested; - achieved bandwidth, tensor-core utilization, and arithmetic intensity. ### 8.3 Service and Agent metrics - cold, warm, and resume TTFT; - P50/P95/P99 ITL and output token/s; - tool-call-to-next-token latency; - MTP acceptance by output phase; - completed tasks/hour and GPU-seconds per completed task; - task success under exact history, compacted history, and retrieved history; - cache bytes per successful task rather than only bytes per token. ## 9. Correctness Checklist Long-lived state makes rare boundary bugs important. Test: - prefix hits immediately before and after every C4/C128 completion boundary; - a tool suffix that crosses several boundaries; - MTP acceptance and rejection across a boundary; - branch/rollback to an earlier logical block; - cache transfer between prefill and decode workers; - offload and restore of every cache family plus residual state; - DP-rank migration or deliberate cache miss; - changing repository snapshot or tool schema with the same textual prefix; - interleaving very long and short requests; - engine restart with intentionally persisted external cache metadata. The tiny reference model should compare a continuous run with the same token stream split into prefill, prefix-hit extension, pause/restore, speculative verify, and branch operations. The outputs and emitted compressed entries must match at every accepted token. ## 10. Important Engineering TODOs ### P0: establish the real 4-B200 envelope - capture V4-Flash TP4 and DPA4+EP4 startup memory and resident-token capacity; - validate coherent 1M prompts, then admit multiple 1M-equivalent sessions; - measure mixed 1M+8K tail latency and prefix-affinity scheduling; - separate device-only capacity from HiSparse/HiCache or vLLM offload; - compare MTP off, 1/2, and 3/4 on coding traces. ### P0: collect realistic coding-agent traces Record only safe, scrubbed metadata and tokenized summaries needed for profiling: - prompt/suffix/output token counts by turn; - idle/tool latency and session lifetime; - exact-prefix reuse and branch depth; - tool/result type; - MTP acceptance and cache-tier events; - repository snapshot identity without private source or credentials. Raw private prompts, source trees, logs, and credentials should remain outside the repository. Commit only small derived summaries and the script that produces them when they support a written conclusion. ### P1: cache-aware Agent scheduling - add session affinity informed by cache location and remaining rank capacity; - isolate or admission-control 1M traffic when it harms interactive tails; - prefetch paused sessions based on tool completion signals; - share immutable repository/system prefixes across Agent branches; - charge redundant EPLB experts and graph workspaces to the capacity planner. ### P1: V4-aware tiering - profile SGLang's current V4 HiSparse path rather than treating device-index/host-main placement as unimplemented; - measure per-layer selected-page hit rate, Host-to-device bytes, and ITL as context, batch, MTP width, and device-buffer size change; - integrate tier-aware radix ownership so HiSparse active retrieval can coexist with HiCache prefix reuse without stale recycled device slots or duplicate host copies; - stream HCA C128 pages rather than demand-fetch them randomly; - persist and restore compressor tails transactionally; - compare device-only, HiCache, HiSparse, and a future combined path against compacting/retrieving the old transcript. ### P2: architecture/runtime co-design - reuse CSA routing over nearby queries or small layer groups as an explicit experiment, with correctness/quality evaluation; - hierarchical index structures that avoid a full flat scan for every token; - DCP global top-k and owner-local selected attention for a single oversized session; - adaptive speculation using Agent phase and current batch size; - model-aware context compaction that retains HCA-style dense global anchors. These P2 items may change model behavior or require training. They must remain separate from exact serving optimizations. ## 11. Bottom Line DeepSeek-V4 looks promising for coding agents because its long-term cache is far smaller than token-level full attention or DSA-MLA. That buys more warm sessions, longer exact prefixes, and more practical cache tiering. The saved capacity does not make a 1M Agent cheap automatically. CSA still rescans a C4 index history at every generated token, long-lived DP-owned caches create scheduling affinity, and MTP benefits depend on output phase and batch. The important next step is therefore a trace-driven 4-B200 study that measures warm/resume latency, indexer attribution, cache tiers, mixed-length tails, and completed coding tasks—not just maximum context or peak token/s.