# CSA And HCA: Definition, State, And Cost Model

Status: first-pass definition; paper equations and public inference reference
cross-checked on 2026-07-19

This note defines DeepSeek-V4's Compressed Sparse Attention (CSA) and Heavily
Compressed Attention (HCA) precisely enough to reason about correctness,
training, and serving. Its main conclusion is:

> CSA and HCA do not merely read fewer old token KV entries. They replace old
> token-level KV with learned cross-token memory entries. CSA retrieves a
> sparse subset of moderately compressed entries; HCA densely reads a much
> shorter, heavily compressed sequence. Both mix that global memory with an
> uncompressed local window in one shared-KV MQA operation.

The primary sources are the
[DeepSeek-V4 report, Section 2.3](https://arxiv.org/html/2606.19348v1), the
[public reference model](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/inference/model.py),
and the public
[Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/inference/config.json)
and
[Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/inference/config.json)
configurations. Statements are labeled when the distinction matters:

- **paper-backed**: stated by the technical report;
- **code-backed**: visible in the public inference reference;
- **derived**: follows from those definitions but is not a reported
  measurement;
- **open**: needs training code, kernels, or measurement to settle.

This is the Phase 1 definition note. The detailed NSA/DSA lineage and the full
training and inference studies remain separate chapters.

## 1. Configuration And Notation

Let:

| Symbol | Meaning |
| --- | --- |
| $n$ | original sequence length |
| $d$ | model hidden dimension |
| $c$ | shared main KV entry and query-head dimension |
| $c^I$ | CSA indexer head/key dimension |
| $d_c$ | low-rank query latent dimension |
| $n_h$ | number of main query heads |
| $n_h^I$ | number of CSA indexer query heads |
| $m$ | CSA compression ratio and stride |
| $m'$ | HCA compression ratio and stride |
| $k$ | CSA top-k compressed entries |
| $w$ | local sliding-window length |
| $g$ | grouped-output projection groups |
| $d_g$ | intermediate output dimension per group |

The published configurations are:

| Parameter | V4-Flash | V4-Pro |
| --- | ---: | ---: |
| $d$ | 4096 | 7168 |
| $n_h$ | 64 | 128 |
| $c$ | 512 | 512 |
| RoPE dimensions | 64 | 64 |
| $d_c$ | 1024 | 1536 |
| $m$ | 4 | 4 |
| $m'$ | 128 | 128 |
| $n_h^I$ | 64 | 64 |
| $c^I$ | 128 | 128 |
| $k$ | 512 | 1024 |
| $w$ | 128 | 128 |
| $g$ | 8 | 16 |
| $d_g$ | 1024 | 1024 |

In this note, token and compressed-entry indices are zero-based. At query token
$t$, the completed entries for compression ratio $r$ are

$$
\mathcal{E}^{(r)}_t
= \left\{s \mid 0 \le s < \left\lfloor\frac{t+1}{r}\right\rfloor\right\}.
$$

This convention matches the public prefill mask and incremental decode path.
It means that the final token of a compression block may use the entry that
has just become complete; earlier tokens in that block cannot see it.

## 2. The Learned Compression Operator

CSA and HCA use feature-wise, learned weighted pooling rather than an average
or a single scalar weight per token. For a projected token entry
$C_j\in\mathbb{R}^{c}$ and gate logits $Z_j\in\mathbb{R}^{c}$, the softmax is
normalized over positions independently for every one of the $c$ feature
channels. A learnable positional bias also lets a channel prefer a particular
offset inside the compression block.

### 2.1 CSA: overlapping two-stream compression

Given hidden states $H\in\mathbb{R}^{n\times d}$, CSA first produces two KV
streams and two gate streams:

$$
C^a=H W^{aKV},\qquad C^b=H W^{bKV},
$$

$$
Z^a=H W^{aZ},\qquad Z^b=H W^{bZ},
$$

where each output has shape $n\times c$. For compressed entry $i$, the $a$
stream uses the current block and the $b$ stream uses the preceding block:

$$
\begin{aligned}
\left[S^a_{mi:m(i+1)-1};S^b_{m(i-1):mi-1}\right]
=\operatorname{Softmax}_{\text{position}}\Big(&[
Z^a_{mi:m(i+1)-1}+B^a;\\
&Z^b_{m(i-1):mi-1}+B^b]\Big),
\end{aligned}
$$

$$
C_i^{\mathrm{Comp}}
=\sum_{j=mi}^{m(i+1)-1}S^a_j\odot C^a_j
+\sum_{j=m(i-1)}^{mi-1}S^b_j\odot C^b_j.
$$

For $i=0$, the nonexistent preceding block is padded so its weight is zero.
Although one output can pool information from as many as $2m$ token
positions, consecutive outputs advance by $m$ positions. The sequence length
is therefore approximately $n/m$, not $n/(2m)$. This overlap gives adjacent
compressed entries a shared boundary block:

```text
entry 0:             a(tokens 0..3)
entry 1: b(tokens 0..3) + a(tokens 4..7)
entry 2: b(tokens 4..7) + a(tokens 8..11)
```

For V4, $m=4$. The public implementation realizes the two streams as one
$d\rightarrow 2c$ projection for entries and another for gate logits, then
rearranges the two halves into the overlapping windows.

#### Provenance and design rationale for the overlap

The V4 report presents Equations 9--12 as part of CSA and does not attach a
reference to an earlier overlapping-compression mechanism. It states how the
two streams are combined, but does not provide an ablation or an explicit
explanation for why overlap is necessary. Within the primary sources checked
for this note, this should therefore be treated as a V4 design rather than an
operation inherited from DSA, NSA, or a cited pooling paper.

The public
[reference implementation](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/inference/model.py)
adds one short code-backed motivation: overlap is used "for smoother
compression boundaries." The more detailed interpretation below is derived
from the operator, not claimed by the report:

- A non-overlapping ratio-4 compressor makes tokens 3 and 4 fall into separate
  memories even though they are adjacent. Including the previous block gives
  each new entry information across that hard boundary.
- The receptive field of one entry grows from $m$ to at most $2m$ token
  positions while the output stride remains $m$. It gains boundary context
  without doubling the number of persistent entries.
- Separate $a$ and $b$ projections let a token play a different role when it
  is in the newest block versus when it is carried into the following entry.
  The feature-wise softmax can choose between these roles independently in
  every output channel.
- The overlap is backward-looking rather than symmetric because an emitted
  entry may use completed past and current blocks, but not a future block,
  without violating autoregressive causality.

The cost is also concrete: CSA doubles both entry and gate projection width,
keeps one preceding block of bounded state, and represents every completed
block in two neighboring memories. HCA does not use overlap. The report does
not establish whether that difference is primarily about HCA's much larger
ratio, acceptable coarse redundancy, compute cost, or all three.

### 2.2 HCA: non-overlapping one-stream compression

HCA uses the same kind of feature-wise gated pooling, but has one stream,
non-overlapping blocks, and a much larger ratio $m'=128$:

$$
C=H W^{KV},\qquad Z=H W^Z,
$$

$$
S_{m'i:m'(i+1)-1}
=\operatorname{Softmax}_{\text{position}}
\left(Z_{m'i:m'(i+1)-1}+B\right),
$$

$$
C_i^{\mathrm{Comp}}
=\sum_{j=m'i}^{m'(i+1)-1}S_j\odot C_j.
$$

HCA therefore creates about $n/128$ long-term entries. It is not sparse after
compression: every completed HCA entry is a candidate for the global path.

### 2.3 What the compressor changes

The compressed entry is the persistent semantic memory object. It is not an
index that points back to the original token KV. Once the required block and
overlap state have produced a completed entry, old token-level compressor
state can be retired except for:

- the recent uncompressed SWA entries;
- the incomplete current block;
- for overlapping CSA, the preceding block state needed to form the next
  entry.

This is the key storage distinction from DSA over MLA and from NSA's selected
fine-grained branch.

## 3. CSA Forward Path

CSA is best read as `compression -> DSA-style indexing -> shared-KV MQA`.

### 3.1 A separate compressed indexer memory

CSA applies a second instance of the same overlapping compression pattern to
produce indexer keys
$K^{I\mathrm{Comp}}\in\mathbb{R}^{(n/m)\times c^I}$. Its projections, gates,
positional biases, and final normalized entries are separate from the main
$C^{\mathrm{Comp}}$ memory.

The query first forms one low-rank latent shared by main attention and the
indexer:

$$
\mathbf{c}_t^Q=\mathbf{h}_t W^{DQ},
\qquad \mathbf{c}_t^Q\in\mathbb{R}^{d_c}.
$$

Indexer query heads and their learned scalar combination weights are

$$
[\mathbf{q}_{t,1}^I;\ldots;\mathbf{q}_{t,n_h^I}^I]
=\mathbf{c}_t^QW^{IUQ},
$$

$$
[w_{t,1}^I;\ldots;w_{t,n_h^I}^I]=\mathbf{h}_tW^w.
$$

For eligible compressed entry $s\in\mathcal{E}^{(m)}_t$, the lightning-index
score is

$$
I_{t,s}=\sum_{h=1}^{n_h^I}w_{t,h}^I
\operatorname{ReLU}\left(
\mathbf{q}_{t,h}^I\cdot K_s^{I\mathrm{Comp}}
\right).
$$

CSA then selects compressed memory entries, not token positions:

$$
\mathcal{S}_t
=\operatorname{TopK}_{s\in\mathcal{E}^{(m)}_t}(I_{t,s}, k).
$$

The public reference applies partial RoPE to indexer Q/K, a randomized
Hadamard rotation, and FP4 quantization simulation. The report states that the
indexer attention computation uses FP4.

### 3.2 Main query and shared-KV MQA

Main query heads use the same low-rank latent:

$$
[\mathbf{q}_{t,1};\ldots;\mathbf{q}_{t,n_h}]
=\mathbf{c}_t^QW^{UQ},
\qquad \mathbf{q}_{t,h}\in\mathbb{R}^{c}.
$$

Each selected $C_s^{\mathrm{Comp}}\in\mathbb{R}^{c}$ is both the key and the
value for every query head. This is literal shared-KV MQA:

$$
\mathbf{o}_{t,h}
=\operatorname{CoreAttn}
\left(
\mathbf{q}_{t,h},
\operatorname{key}=\{C_s^{\mathrm{Comp}}:s\in\mathcal{S}_t\},
\operatorname{value}=\{C_s^{\mathrm{Comp}}:s\in\mathcal{S}_t\}
\right).
$$

It is not MLA reconstruction. The low-rank query path resembles MLA's query
compression, but the persistent entry is already the one shared key/value
vector consumed by attention.

## 4. HCA Forward Path

HCA removes the indexer and top-k. It forms the same low-rank main queries as
CSA and densely attends over every completed HCA entry:

$$
\mathbf{o}_{t,h}
=\operatorname{CoreAttn}
\left(
\mathbf{q}_{t,h},
\operatorname{key}=\{C_s^{\mathrm{Comp}}:s\in\mathcal{E}^{(m')}_t\},
\operatorname{value}=\{C_s^{\mathrm{Comp}}:s\in\mathcal{E}^{(m')}_t\}
\right).
$$

Thus `dense HCA` is dense in the compressed domain. At one million original
tokens it scans roughly $10^6/128=7{,}812$ global entries, not one million.

The complementary roles are:

```text
CSA: ratio 4   -> more detailed global memory -> top-k retrieval is required
HCA: ratio 128 -> very coarse global memory   -> dense mixing is affordable
```

## 5. Common Local Path, Positioning, And Output

The equations above omit four components that are part of both real operators.

### 5.1 One candidate list, not two independently normalized branches

Each layer also projects uncompressed, recent token entries
$C_j^{\mathrm{Local}}\in\mathbb{R}^{c}$. For query $t$, define

$$
\mathcal{L}_t
=\{j\mid \max(0,t-w+1)\le j\le t\}.
$$

The actual candidate list is the concatenation

$$
\mathcal{A}^{CSA}_t
=\{C_s^{\mathrm{Comp}}:s\in\mathcal{S}_t\}
\mathbin{\Vert}
\{C_j^{\mathrm{Local}}:j\in\mathcal{L}_t\},
$$

or

$$
\mathcal{A}^{HCA}_t
=\{C_s^{\mathrm{Comp}}:s\in\mathcal{E}^{(m')}_t\}
\mathbin{\Vert}
\{C_j^{\mathrm{Local}}:j\in\mathcal{L}_t\}.
$$

The public reference sends this combined list through one sparse-attention
kernel and one softmax per head. This is different from NSA's independently
computed and gated compression, selection, and sliding branches.

The local path has two jobs: it exposes tokens in the not-yet-complete block,
and it preserves fine-grained recency that compression may blur.

### 5.2 RMSNorm and partial/inverse RoPE

A step-by-step derivation, two-dimensional example, compressed-entry position
anchors, and comparison with MLA are in
[Partial And Inverse RoPE In CSA/HCA](./01a-partial-inverse-rope.md). This
section keeps only the forward-path contract needed by the main definition.

Immediately before core attention, every query head and the single shared KV
head are RMS-normalized. Only the last 64 dimensions use RoPE. If
$\rho_p(\cdot)$ rotates those dimensions at position $p$, the main-attention
logit uses

$$
z_{t,h,j}
=\frac{\rho_t(\mathbf{q}_{t,h})^T\rho_{p_j}(C_j)}{\sqrt{c}}.
$$

Compressed entry $s$ is assigned the start position of its newest block in
the public reference: $p_s=ms$ for CSA and $p_s=m's$ for HCA. Local entry $j$
uses $p_j=j$.

Because the same rotated entry is also the value, the weighted output carries
absolute rotation. The implementation therefore applies inverse RoPE at query
position $t$ to the last 64 output dimensions. This converts that component
back into a relative-position form before output projection.

### 5.3 Attention sink

Each query head has a learnable sink logit $z'_h$. For a real candidate $j$,

$$
a_{t,h,j}
=\frac{\exp(z_{t,h,j})}
{\sum_{u\in\mathcal{A}_t}\exp(z_{t,h,u})+\exp(z'_h)}.
$$

The sink contributes no value vector, so real attention weights may sum to
less than one. A head can suppress the whole attention update instead of being
forced to distribute unit mass over weak candidates.

### 5.4 Grouped output projection

Directly projecting $n_hc$ concatenated output dimensions to $d$ would be
expensive because $c=512$. V4 splits heads into $g$ groups. Each group first
projects from $c(n_h/g)$ to $d_g$, the $g$ intermediate vectors are
concatenated, and a final projection maps $gd_g$ to $d$:

$$
\widehat{\mathbf{o}}_t
=W^{O,B}
\left[
W^{O,A}_1\mathbf{o}_{t,1}^{G};\ldots;
W^{O,A}_g\mathbf{o}_{t,g}^{G}
\right].
$$

Ignoring bias and counting one multiply-add as two FLOPs, its per-token cost is

$$
F_{out}\approx 2n_hcd_g+2gd_gd,
$$

versus $2n_hcd$ for one direct dense projection.

## 6. Causality And A Small Worked Example

Use a toy CSA ratio $m=4$, HCA ratio $m'=8$, and local window $w=3$.

| Query $t$ | Completed CSA entries | Completed HCA entries | Local tokens |
| ---: | --- | --- | --- |
| 0 | none | none | 0 |
| 2 | none | none | 0, 1, 2 |
| 3 | 0 | none | 1, 2, 3 |
| 6 | 0 | none | 4, 5, 6 |
| 7 | 0, 1 | 0 | 5, 6, 7 |
| 8 | 0, 1 | 0 | 6, 7, 8 |

At $t=7$:

- CSA entry 1 has just completed and pools its current tokens 4--7 plus an
  overlapping stream from tokens 0--3;
- CSA's indexer chooses among entries 0 and 1, not among tokens 0--7;
- HCA entry 0 has just completed and represents tokens 0--7;
- the local path independently exposes tokens 5, 6, and 7.

No entry contains a token later than the query. The local path is necessary
for queries in the middle of a compression block because their global
compressed entry does not exist yet.

## 7. Incremental Decode State Machine

The serving state is heterogeneous:

| State | CSA | HCA | Update rule |
| --- | --- | --- | --- |
| local shared-KV ring | last $w$ entries | last $w$ entries | overwrite one slot per token |
| main compressed cache | about $L/m$ entries | about $L/m'$ entries | append at block completion |
| indexer compressed cache | about $L/m$ entries | none | append with CSA main block cadence |
| incomplete tail | current block plus CSA overlap state | current block | bounded state, not append-only history |

For every new token $t$, the conceptual order is:

1. Form and insert its local shared-KV entry into the $w$-slot ring.
2. Update main compressor projections and gate state.
3. For CSA, update the separate indexer compressor state.
4. If $(t+1)$ is divisible by the layer's compression ratio, emit and append a
   completed main entry; CSA also appends an indexer entry.
5. Build the eligible compressed set, run CSA top-k or HCA dense access, append
   the local-window indices, and execute shared-KV MQA.

The newly completed entry is eligible for the block's final token. If the
block is incomplete, no provisional global entry is exposed.

The report's production design separates an append-style classical cache for
completed CSA/HCA entries from a fixed-size state cache for SWA and incomplete
tails. This differs from ordinary PagedAttention, where layers usually share
one token-to-page mapping and one append cadence.

Prefix-cache implications follow directly:

- completed compressed entries can be persisted and reused;
- an incomplete prefix tail must be recomputed to restore compressor state;
- SWA can be fully stored, periodically checkpointed, or recomputed, trading
  storage bandwidth for prefix-hit compute.

## 8. Analytical Cost Model

This section separates persistent bytes, decode traffic, and arithmetic. It is
a derived model, not a wall-time prediction.

Let $L$ be the current original-token history, $b_C$ the stored bytes for one
main/local shared-KV entry, and $b_I$ the stored bytes for one CSA indexer key.
Ignoring scale metadata and bounded state:

$$
M_{CSA}(L)
\approx wb_C+\left\lfloor\frac{L}{m}\right\rfloor(b_C+b_I),
$$

$$
M_{HCA}(L)
\approx wb_C+\left\lfloor\frac{L}{m'}\right\rfloor b_C.
$$

The report specifies BF16 for the 64 RoPE dimensions and FP8 for the remaining
main KV dimensions, so an idealized $c=512$ entry is approximately

$$
b_C\approx 64\times2+(512-64)\times1=576\ \text{bytes},
$$

before scale and alignment overhead. The public reference retains BF16
buffers for simplicity while simulating the quantized paths, so its literal
buffer allocation should not be confused with the intended production cache
format.

The production precision statement for the CSA indexer is stronger: the V4
report says its QK activations are cached, loaded, and multiplied entirely in
MXFP4. Ignoring scale metadata, a $c^I=128$ indexer key therefore has payload

$$
b_I\approx128\times\frac{1}{2}=64\ \text{bytes}.
$$

The public Python reference allocates a BF16 tensor and applies FP4 fake
quantization for clarity; this is not the intended packed production layout.
With the reference's one FP8 scale per 32 FP4 elements, a packed index key
would additionally need four scale bytes before alignment. Likewise, the
448-dimensional FP8 portion of a main entry needs scale metadata. All byte
tables below report payload only so that unknown kernel padding and alignment
are not disguised as exact measurements.

The relevant precision paths are:

| Component | Published or reference precision | Qualification |
| --- | --- | --- |
| Main compressed/local KV cache | 448 FP8 dimensions + 64 BF16 RoPE dimensions | paper-backed production format; 576-byte payload per entry |
| CSA indexer cached Q/K operands | MXFP4 | paper-backed QAT/deployment path; 64-byte key payload at $c^I=128$ |
| Index scores passed toward top-k | BF16 instead of FP32 | paper-backed QAT optimization; accumulation details are kernel-dependent |
| Compressor projections, gates, and pooling | FP32 in the public reference | code-backed reference behavior; checkpoint comments say compressor weights are stored in BF16 |
| Main attention projections | predominantly quantized linear paths | public config defaults to FP8 weights; exact production accumulation precision is not fully specified |
| Main sparse attention softmax/accumulation | not fully specified publicly | do not infer it solely from the mixed cache format |

### 8.1 Per-token decode reads

At long context, the idealized history reads are

$$
B_{CSA,main}\approx(k+w)b_C,
\qquad
B_{CSA,index}\approx\frac{L}{m}b_I,
$$

$$
B_{HCA,main}\approx\left(\frac{L}{m'}+w\right)b_C.
$$

MQA lets all query heads reuse each fetched entry, but every query head still
performs its own dot product and weighted-value accumulation. Main attention
FLOPs are therefore approximately

$$
F_{main}(N)\approx4n_hcN,
$$

where $N=k+w$ for long-context CSA and $N=L/m'+w$ for HCA. CSA's indexer dot
products add roughly

$$
F_{index}\approx2n_h^Ic^I\frac{L}{m},
$$

plus ReLU, learned head reduction, top-k, and sparse-gather overhead.

The following tables instantiate V4-Flash with $m=4$, $k=512$, $w=128$,
$n_h=64$, $c=512$, $n_h^I=64$, and $c^I=128$. Context labels use powers of
two, so `1M` means $1{,}048{,}576$. Counts are for one query token, one CSA
layer, batch one, before TP/CP sharding. They exclude scale metadata, padding,
softmax, top-k comparisons, sparse-address processing, and cache writes.

First, the persistent state per request and CSA layer is:

| Original context | Compressed entries $L/4$ | Main compressed cache | FP4 index-key cache | Local 128 entries | Total payload |
| ---: | ---: | ---: | ---: | ---: | ---: |
| 8K | 2,048 | 1.125 MiB | 0.125 MiB | 0.070 MiB | 1.320 MiB |
| 32K | 8,192 | 4.500 MiB | 0.500 MiB | 0.070 MiB | 5.070 MiB |
| 128K | 32,768 | 18.000 MiB | 2.000 MiB | 0.070 MiB | 20.070 MiB |
| 1M | 262,144 | 144.000 MiB | 16.000 MiB | 0.070 MiB | 160.070 MiB |

V4-Flash has 21 CSA layers, so the corresponding CSA-only payload at 1M is
about 3.28 GiB per request. This is not the full-model KV cache: the 20 HCA
layers, two main SWA layers, MTP SWA state, scale metadata, alignment, and
compressor tail state are additional.

Incremental persistent writes are small compared with reads. Every token
writes one 576-byte local entry; every four tokens, a CSA layer emits one
576-byte main entry and one 64-byte index key. The payload-only amortized write
is therefore

$$
576+\frac{576+64}{4}=736\ \text{bytes/token/layer}.
$$

This excludes scales and bounded compressor-state traffic. The public
reference keeps FP32 tail buffers for straightforward incremental pooling;
production kernels may keep or fuse that state differently.

The per-step history reads and arithmetic are:

| Context | FP4 index-key read | Index QK FLOPs | Selected + local main-KV read | Main QK/AV FLOPs | Fixed projection/compressor/output FLOPs | Modeled total |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 8K | 128 KiB | 33.6M | 360 KiB | 83.9M | 252.2M | 369.6M |
| 32K | 512 KiB | 134.2M | 360 KiB | 83.9M | 252.2M | 470.3M |
| 128K | 2.0 MiB | 536.9M | 360 KiB | 83.9M | 252.2M | 872.9M |
| 1M | 16.0 MiB | 4.295B | 360 KiB | 83.9M | 252.2M | 4.631B |

If the reduced index-score vector is materialized in BF16, it adds about 4,
16, 64, and 512 KiB respectively before top-k. A fused or streaming selector
can avoid materializing larger per-head score tensors, but top-k remains an
irregular synchronization and selection stage.

Several crossover points make the trade-off easier to see:

- CSA reaches its full 512 compressed-entry budget at $L=mk=2048$; beyond
  that point, main sparse attention stays constant at 83.9M modeled FLOPs.
- FP4 index-key payload read exceeds the 360 KiB selected-main read at roughly
  $L=23{,}040$ tokens.
- Indexer dot-product FLOPs exceed all modeled fixed projection plus main
  attention FLOPs around $L=82$K. At 1M they account for about 93% of the
  modeled CSA-layer FLOPs.
- Across all 21 Flash CSA layers at 1M, the logical counts are about 336 MiB
  of index-key reads and 97.3B modeled FLOPs per generated token, before HCA,
  SWA, MoE, MTP, and communication.

Payload-only arithmetic intensity is about 256 FLOPs per FP4 index-key byte
for the index scan and 228 FLOPs per mixed-format main-KV byte for selected
MQA. These ratios assume perfect reuse across all 64 query heads. With Tensor
Parallel query-head sharding and replicated shared KV, each rank can read the
same keys for fewer local heads, lowering per-rank arithmetic intensity; CP or
sequence-sharded cache layouts introduce a different read/communication
trade-off.

For contrast, HCA at 1M reads about 8,192 completed compressed entries plus
128 local entries and performs approximately 1.09B main-attention FLOPs per
layer. CSA spends more low-precision work on a full compressed-index scan,
then keeps its wide main attention and main-KV read fixed.

### 8.2 Fixed per-token projection and compression work

History-dependent attention is only part of decode. Important fixed costs
include:

| Operation | Leading FLOPs | V4-Flash per token |
| --- | ---: | ---: |
| shared query down-projection | $2dd_c$ | 8.4M |
| main query up-projection | $2d_cn_hc$ | 67.1M |
| CSA indexer query up-projection | $2d_cn_h^Ic^I$ | 16.8M |
| CSA indexer head-weight projection | $2dn_h^I$ | 0.5M |
| local KV projection | $2dc$ | 4.2M |
| CSA main compressor projections | about $8dc$ per original token | 16.8M |
| CSA indexer compressor projections | about $8dc^I$ per original token | 4.2M |
| grouped output projection | $2n_hcd_g+2gd_gd$ | 134.2M |
| **Total shown for one CSA layer** |  | **252.2M** |
| HCA compressor projections, for comparison | about $4dc$ per original token | 8.4M, excluded above |

The compressor costs follow from entry and gate projections: CSA has two
streams for each; HCA has one. Pooling, normalization, RoPE, quantization,
softmax, and elementwise work are additional.

The main attention module in one Flash CSA layer contains about 126.1M weight
elements across the projections shown above. A naive batch-one lower bound is
roughly 240.5 MiB of weight payload if all were BF16 or 120.3 MiB if all were
FP8, before scales. The real checkpoint mixes formats, so these are endpoints,
not an asserted deployed byte count.

This fixed weight traffic has a different batching behavior from history
traffic. At batch one the projections are matrix-vector-like and can be
weight-bandwidth bound. Larger continuous batches or wider MTP verification
reuse the same weights across more query rows and turn them into better-shaped
GEMMs, while each request still brings its own history entries. Consequently,
the 82K arithmetic crossover above is not itself a wall-time crossover.

At short context these fixed projections and kernel-launch costs can dominate,
so lower asymptotic history work does not guarantee lower single-request
latency. Two useful regime boundaries are:

- CSA reaches its full top-k only once $L\gtrsim mk$: about 2K tokens for
  Flash and 4K for Pro.
- HCA's compressed-global candidate count exceeds its 128-token local window
  around $L\approx m'w=16{,}384$.

### 8.3 Training and prefill scaling

For a length-$n$ sequence, omitting causal factors of roughly one half:

$$
F_{CSA,index}=O\left(\frac{n^2}{m}n_h^Ic^I\right),
$$

$$
F_{CSA,main}=O(nn_hc(k+w)),
$$

$$
F_{HCA,main}=O\left(nn_hc\left(\frac{n}{m'}+w\right)\right).
$$

CSA's main sparse attention is linear in $n$ for fixed $k$ and $w$, but its
full index scan remains quadratic with a $1/m$ reduction and much smaller,
low-precision index dimensions. HCA remains quadratic with a $1/m'$ sequence
reduction. Calling either mechanism simply `linear attention` would therefore
be inaccurate.

## 9. Layer Schedule

The public Flash configuration has 43 main Transformer attention layers:

```text
layers 0, 1:  pure SWA
layers 2..42: CSA, HCA, CSA, HCA, ... (21 CSA, 20 HCA)
MTP layer 43: pure SWA
```

The `compress_ratios` array has 44 values because its final value configures
the one MTP block in addition to the 43 main layers. It must not be counted as
a final main-Transformer SWA layer.

The public Pro configuration has 61 main Transformer attention layers:

```text
layers 0, 1:  HCA
layers 2..60: CSA, HCA, CSA, HCA, ... (30 CSA, 29 HCA)
MTP layer 61: pure SWA
```

Thus the Pro main Transformer contains 30 CSA and 31 HCA layers. `Hybrid`
applies at two levels: compressed-global plus local candidates inside CSA/HCA,
and different attention layer types across the Transformer stack. MTP adds a
separate pure-SWA attention block in both public configurations.

## 10. Relationship To MLA, DSA, And NSA

| Mechanism | Persistent long-term entry | Global access | Does main attention return to old original-token KV? |
| --- | --- | --- | --- |
| dense MLA | one latent plus positional state per token | all token latents | yes, its memory is token-indexed |
| DSA over MLA | all token MLA entries plus indexer state | top-k token entries | yes |
| NSA | compressed branch plus selectable fine KV and local state | compressed + selected-original + local | yes, in selected branch |
| CSA | one shared-KV entry per 4 tokens plus indexer entry | top-k compressed entries + local | no |
| HCA | one shared-KV entry per 128 tokens | all compressed entries + local | no |

The direct lineage is `DSA indexer/top-k -> CSA indexer/top-k`, but the indexed
unit changes from an original-token MLA entry to a learned compressed entry.
The NSA relationship is structural: both combine coarse/global and local
information, but NSA's selected branch retrieves original fine-grained KV,
whereas CSA selects compressed memory and V4 distributes its moderately and
heavily compressed roles across different layer types.

## 11. Immediate Systems Challenges And Opportunities

The detailed systems chapters will expand these, but the definition already
implies the main contracts.

### Training

- Compression blocks can cross Context Parallel rank boundaries. The report
  uses neighbor exchange for boundary entries, local compression, compressed
  all-gather, and a fused select-and-pad step for variable packed-sequence
  lengths.
- Packed samples must compress independently. Padding, trailing incomplete
  blocks, and sample-level causal masks cannot leak information across samples.
- Sparse-attention backward scatters gradients into selected compressed
  entries. Atomic accumulation is nondeterministic; DeepSeek reports per-SM
  buffers followed by deterministic global reduction.
- Feature-wise gates and FP32 compression in the public reference introduce
  nontrivial activation and recomputation choices even though the resulting
  cache is small.
- The report and inference-only reference do not fully expose how the discrete
  CSA indexer is supervised or warmed up. Importing DSA's exact training recipe
  is a hypothesis until checked against V4 training evidence.
- Opportunity: compression reduces both the global attention sequence and the
  amount of compressed state exchanged across long-context ranks; HCA also
  avoids irregular top-k in half of the interleaved core.

### Inference

- One request owns append-only CSA/HCA entries, a CSA indexer cache, circular
  SWA state, and incomplete compressor state with different update cadences.
- Ordinary PagedAttention assumptions no longer suffice. Cache block alignment
  must satisfy both ratios and sparse-kernel constraints; for ratios 4 and 128,
  the report uses multiples of their least common multiple, 128 original
  tokens.
- CSA can become index-scan/top-k/gather bound; HCA can become dense-compressed
  attention bound; short contexts can remain projection or launch bound.
- MQA-style sharing makes main candidate fetches highly reusable across query
  heads, creating an opportunity for fused kernels that retain shared entries
  while processing many heads.
- Small completed caches make larger batches, more resident prefixes, and
  wider speculative verification feasible, but do not guarantee lower latency
  at fixed batch and speculation width.
- Prefix persistence is attractive for completed compressed entries, while
  incomplete tails and SWA require explicit checkpoint/recompute policy.

## 12. Claims Established And Questions Carried Forward

Established by paper plus reference implementation:

- CSA selects learned compressed entries, not original tokens or pointers to
  original KV.
- HCA is dense only over a sequence compressed by 128.
- Main compressed and local entries are shared keys and values for all query
  heads.
- The local and compressed candidates share one core attention normalization.
- CSA uses a separate compressed indexer cache.
- Causality is enforced by exposing only completed compression blocks, with a
  local window covering the current incomplete block.
- Main cache length, per-query read length, and attention arithmetic are three
  different quantities and must be reported separately.

Questions for the next chapters:

- What exact loss, target distribution, and training schedule teach the V4
  indexer around discrete top-k?
- How closely does the V4 indexer training contract match DSA's indexer
  distillation, and where did it change because the indexed unit is compressed?
- How are compressor softmax intermediates and sparse backward activations
  checkpointed in the training implementation?
- What are the measured stage times for compression, index scan, top-k,
  gather, main attention, local attention, and grouped output projection?
- Which packed indexer format and scale metadata does the production serving
  cache use?
- Where do Hopper and Blackwell cross from projection-bound to indexer-,
  attention-, or memory-bound as batch, context, and MTP width change?