# MLA Primer For DSA And DeepSeek-V4 Attention Status: first-pass reconstruction for discussion This note builds only the Multi-head Latent Attention (MLA) foundation needed to study DeepSeek Sparse Attention (DSA), Compressed Sparse Attention (CSA), and Heavily Compressed Attention (HCA). It is not a complete history of DeepSeek-V2/V3 or a FlashMLA kernel survey. The companion [MLA systems and roofline note](./00a-mla-systems-roofline.md) covers FlashMLA, Hopper/Blackwell arithmetic intensity, serving throughput, and the distinction between MTP-1 and target-model query width. Unless stated otherwise, cache sizes below are **elements per token per attention layer**. Bytes depend on the storage dtype and quantization metadata. ## 1. The One-Sentence Definition MLA learns one low-rank content latent for each source token and caches that latent together with a small, shared RoPE key. Per-head content keys and values can be reconstructed from the latent, or the same projections can be algebraically moved to the query and output sides during decode. The most important boundary is: - MLA compresses the **feature representation of every token**. It does not shorten the sequence and dense MLA attention is still quadratic in sequence length. - DSA adds **token selection** over the per-token MLA entries. - DeepSeek-V4 CSA/HCA create **cross-token compressed entries**. That is a different memory object from an MLA latent, even though some low-rank-query and shared-KV ideas remain useful. ## 2. Cache Baseline: MHA, GQA, MQA, And MLA Let: - $n_h$ be the number of query heads; - $n_{kv}$ be the number of KV heads or groups; - $d_k$ and $d_v$ be the key and value dimensions; - $r_{kv}$ be MLA's compressed KV rank; - $d_{rope}$ be MLA's decoupled RoPE-key dimension. The persistent autoregressive cache is: | Mechanism | Cached content per source token | Element count | | --- | --- | ---: | | MHA | one K and V for every query head | $n_h(d_k+d_v)$ | | GQA | one K and V for every KV group | $n_{kv}(d_k+d_v)$ | | MQA | one K and V shared by all query heads | $d_k+d_v$ | | MLA | one content latent plus one shared RoPE key | $r_{kv}+d_{rope}$ | With equal key/value head dimensions $d_h$, the first three simplify to $2n_h d_h$, $2n_{kv}d_h$, and $2d_h$. MLA is not merely GQA with a fractional number of heads: each query head has learned up-projections from the shared latent, and those projections may instead be absorbed into the surrounding operations. ### DeepSeek-V3 worked example The public DeepSeek-V3 configuration uses: | Symbol | Meaning | Value | | --- | --- | ---: | | $d_{model}$ | hidden dimension | 7168 | | $n_h$ | query heads | 128 | | $d_{nope}$ | non-positional Q/K dimension per head | 128 | | $d_{rope}$ | RoPE Q/K dimension | 64 | | $d_v$ | value dimension per head | 128 | | $r_q$ | query low-rank dimension | 1536 | | $r_{kv}$ | KV low-rank dimension | 512 | MLA therefore caches $512+64=576$ elements per token per layer. At BF16, that is 1,152 bytes before allocator, page-table, or quantization overhead. A fully materialized V3-shaped K/V tensor would contain $128\left((128+64)+128\right)=40{,}960$ elements. The compressed cache has about 71.1 times fewer elements. This is an implementation-shape comparison, not the exact comparison used in the V2 paper. The V2 paper's canonical equal-head-dimension comparison uses 128 heads of dimension 128, so MHA has $2\times128\times128=32{,}768$ elements. Its 576-element MLA cache is equal in size to $576/(2\times128)=2.25$ GQA groups. “2.25 groups” is a cache-equivalence calculation, not a realizable integer GQA configuration. ## 3. Forward Definition And Tensor Shapes For token hidden state $\mathbf{h}_t\in\mathbb{R}^{d_{model}}$, use the following implementation-oriented notation. RMS normalization between the down- and up-projections is shown explicitly because it exists in the public implementation; compact paper equations sometimes omit it. ### Query path $$ \begin{aligned} \mathbf{c}_t^Q &=\operatorname{RMSNorm}(W^{DQ}\mathbf{h}_t) &&\in\mathbb{R}^{r_q},\\ \mathbf{q}_{t,i}^C &=W_i^{UQ,C}\mathbf{c}_t^Q &&\in\mathbb{R}^{d_{nope}},\\ \mathbf{q}_{t,i}^R &=\operatorname{RoPE}_t(W_i^{UQ,R}\mathbf{c}_t^Q) &&\in\mathbb{R}^{d_{rope}},\\ \mathbf{q}_{t,i} &=\operatorname{Concat}(\mathbf{q}_{t,i}^C,\mathbf{q}_{t,i}^R) &&\in\mathbb{R}^{d_{nope}+d_{rope}}. \end{aligned} $$ There is one query vector per head $i$. Query compression reduces projection and training-activation costs, but $\mathbf{c}_t^Q$ is not persistent history and is not the source of decode KV-cache savings. ### KV path The first KV projection jointly produces a low-rank content latent and a small positional key: $$ \begin{aligned} [\mathbf{c}_{raw,t}^{KV};\mathbf{k}_{raw,t}^{R}] &=W^{DKV}\mathbf{h}_t,\\ \mathbf{c}_t^{KV} &=\operatorname{RMSNorm}(\mathbf{c}_{raw,t}^{KV}) &&\in\mathbb{R}^{r_{kv}},\\ \mathbf{k}_t^R &=\operatorname{RoPE}_t(\mathbf{k}_{raw,t}^R) &&\in\mathbb{R}^{d_{rope}},\\ \mathbf{k}_{t,i}^C &=W_i^{UK}\mathbf{c}_t^{KV} &&\in\mathbb{R}^{d_{nope}},\\ \mathbf{v}_{t,i} &=W_i^{UV}\mathbf{c}_t^{KV} &&\in\mathbb{R}^{d_v},\\ \mathbf{k}_{t,i} &=\operatorname{Concat}(\mathbf{k}_{t,i}^C,\mathbf{k}_t^R) &&\in\mathbb{R}^{d_{nope}+d_{rope}}. \end{aligned} $$ $\mathbf{k}_t^R$ is shared across heads. Only $\mathbf{c}_t^{KV}$ and $\mathbf{k}_t^R$ need to persist in the compressed cache. The per-head score and output are conventional once K and V are materialized: $$ \begin{aligned} s_{t,u,i} &=\frac{ \langle\mathbf{q}_{t,i}^C,\mathbf{k}_{u,i}^C\rangle +\langle\mathbf{q}_{t,i}^R,\mathbf{k}_u^R\rangle} {\sqrt{d_{nope}+d_{rope}}},\\ \mathbf{a}_{t,:,i} &=\operatorname{CausalSoftmax}(\mathbf{s}_{t,:,i}),\\ \mathbf{o}_{t,i} &=\sum_u a_{t,u,i}\mathbf{v}_{u,i},\\ \mathbf{y}_t &=W^O\operatorname{Concat}_i(\mathbf{o}_{t,i}). \end{aligned} $$ ## 4. Why RoPE Is Decoupled Without position encoding, the content-key up-projection can move across the dot product: $$ \left\langle\mathbf{q}_{t,i}^C,W_i^{UK}\mathbf{c}_u^{KV}\right\rangle =\left\langle(W_i^{UK})^T\mathbf{q}_{t,i}^C, \mathbf{c}_u^{KV}\right\rangle. $$ This lets decode score the cached latent directly. Applying RoPE to the whole reconstructed key would insert a position-dependent transformation between $W^{UK}$ and the dot product, preventing one fixed matrix from being absorbed into the query projection for all positions. MLA therefore separates: - non-positional content dimensions, where the up-projection can be moved to the query side; and - a small RoPE sub-vector that must be cached after applying the source position's rotation. The 64-dimensional RoPE key in V3 is not an incidental implementation detail: it is the residual state that preserves positional information while allowing the much larger content path to remain latent. ## 5. Two Algebraically Equivalent Execution Modes DeepSeek-V3.2 calls these MHA mode and MQA mode. They use the same MLA weights and compute the same mathematical attention, aside from finite-precision and quantization-order effects. The names describe execution and data sharing, not two separately trained architectures. ### MHA mode: expand first ```text cached/current latent -> per-head W_UK and W_UV -> materialized per-head K and V -> ordinary multi-head attention ``` This mode is attractive for training and prefill because large projections and dense attention can use regular batched GEMMs and FlashAttention-style kernels. Expanded K/V may be transient; using this mode does not require a server to persist the full per-head representation after prefill. ### MQA mode: keep shared latent state For every query head, first project the non-positional query into latent space: $$ \begin{aligned} \widetilde{\mathbf{q}}_{t,i} &=(W_i^{UK})^T\mathbf{q}_{t,i}^C &&\in\mathbb{R}^{r_{kv}},\\ s_{t,u,i} &=\frac{ \langle\widetilde{\mathbf{q}}_{t,i},\mathbf{c}_u^{KV}\rangle +\langle\mathbf{q}_{t,i}^R,\mathbf{k}_u^R\rangle} {\sqrt{d_{nope}+d_{rope}}},\\ \mathbf{z}_{t,i} &=\sum_u a_{t,u,i}\mathbf{c}_u^{KV} &&\in\mathbb{R}^{r_{kv}},\\ \mathbf{o}_{t,i} &=W_i^{UV}\mathbf{z}_{t,i} &&\in\mathbb{R}^{d_v}. \end{aligned} $$ The implementation may also compose $W^{UV}$ with the corresponding slice of $W^O$; the public reference path aggregates latents, applies the value up-projection, then applies $W^O$. Every historical position now exposes one shared latent entry to all query heads, while head-specific transformed queries still produce different attention weights. This is why the V3.2 report calls it MQA mode even though MLA is not identical to conventional MQA. ### System trade-off | Stage | Usually favorable mode | Main reason | | --- | --- | --- | | training forward/backward | MHA mode | regular dense kernels and simpler saved/recomputed tensors | | long prefill | MHA mode | projection and attention are compute-heavy and highly parallel | | one-token decode | MQA mode | avoids reading reconstructed per-head K/V from HBM | | sparse DSA decode | MQA mode | selected latent entries can be reused by all query heads | MQA mode saves persistent cache and bandwidth, but adds query-side latent projection and post-attention value reconstruction. Whether a concrete kernel is faster depends on sequence length, batch size, dtype, head/rank dimensions, cache layout, and fusion. ## 6. Training And Inference Consequences ### Training - MLA reduces the size of the persistent representation but training does not consist only of a decode-style KV cache. Q/K/V activations, attention scores or softmax statistics, normalization state, and projection inputs still participate in backward or recomputation. - MHA-mode materialization gives existing dense-attention kernels a friendly interface, at the cost of transient per-head K/V activations. - The low-rank query and KV projections introduce extra normalization and projection boundaries. Tensor-parallel sharding must respect the fact that the latent and RoPE key are shared while the up-projected heads are sharded. - MLA changes constants but not dense attention's $O(L^2)$ score work. It is not, by itself, a long-context training solution. ### Prefill - The whole prompt is available, so expanding K/V once and feeding a dense attention kernel can be more efficient than repeatedly operating in latent space. - A serving implementation should distinguish transient expanded K/V used by the prefill kernel from the compressed state committed to the decode cache. - Quantizing the latent cache changes both its byte count and the exact point at which dequantization or projection occurs; element-count comparisons alone are insufficient for bandwidth estimates. ### Decode - Each step appends only $r_{kv}+d_{rope}$ cache elements per layer. - The main opportunity is HBM traffic reduction: history stays compressed and is read once in a representation shared across heads. - Efficient kernels must fuse or schedule query absorption, latent-score computation, RoPE-score computation, softmax, latent aggregation, and value reconstruction without materializing large intermediates. - The shared latent may be duplicated across tensor-parallel ranks in a simple implementation. Whether production kernels replicate, partition, or communicate it is an implementation decision, not fixed by the paper equation. ## 7. Exact Relationship To DSA DSA keeps MLA as its main attention representation and adds a separate lightweight indexer that ranks historical token positions. For a query token: 1. the indexer computes a relevance score for source positions; 2. top-k chooses token indices; 3. the main MLA attention evaluates only the chosen per-token latent entries and their RoPE keys. The sparse unit is therefore an **original token position**, represented by one MLA cache entry. The indexer state is separate from the main MLA cache and must be counted separately in memory and system analysis. V3.2 requires each selected KV entry to be shared by multiple query heads for efficient sparse-kernel reuse. It therefore uses MLA's MQA execution mode for sparse decode, with one selected index set shared across the main query heads. For short-sequence prefill, the implementation can instead use MHA mode and apply the indexer's selection as a mask over a dense attention computation. This separation is useful: ```text MLA: how each token's main KV content is represented and executed DSA: which token positions the main attention is allowed to read ``` ## 8. Bridge To CSA And HCA The following is a routing hypothesis for the next chapter, not yet the full CSA/HCA definition. | Property | MLA / DSA | V4 CSA / HCA direction | | --- | --- | --- | | memory unit | one latent per original token | one learned entry summarizes multiple tokens | | sequence length | unchanged by MLA | shortened by sequence compression | | sparsity | DSA top-k selects token positions | CSA top-k selects compressed entries | | shared KV | latent entry shared in MQA-mode execution | compressed entry directly serves shared-KV attention | | dense global path | dense MLA scans all token latents | HCA scans a much shorter compressed sequence | | local detail | main dense/sparse token attention | separate sliding-window path in the hybrid stack | Concepts that remain useful are the low-rank query path, shared-KV kernel contract, absorbed-projection intuition, and separation between an indexer and main attention. What changes is the semantics of a cache entry: V4's learned sequence compressor merges information across positions before CSA/HCA reads it. The next chapter must reconstruct the compressor, causal visibility, incomplete-block state, and whether each compressed entry is directly a key and value rather than an MLA reconstruction latent. ## 9. Common Misreadings To Avoid - **“MLA is sparse attention.”** No. Vanilla MLA still attends to every source token; DSA supplies sparsity. - **“MLA compresses the sequence.”** No. It compresses each token's feature state independently. - **“MLA is just MQA.”** No. It retains head-specific learned up-projections; MQA mode is an algebraic execution of the shared latent. - **“Query compression is the KV-cache saving.”** No. Persistent cache saving comes from the KV latent and small RoPE key. - **“All reconstructed K can be discarded without qualification.”** The content key can be absorbed, but the source-position RoPE key must still be stored or reproducibly regenerated. - **“The paper's cache ratio is universal.”** Ratios depend on the chosen MHA baseline, actual K/V dimensions, dtype, quantization scales, and allocator overhead. ## 10. Questions To Carry Forward - In DeepSeek-V4, which parts of the low-rank query path are inherited exactly from MLA, and which dimensions or normalizations change? - Is a CSA/HCA compressed entry directly used as both shared K and V, or is there any remaining MLA-style latent reconstruction on the main path? - How are completed compressed blocks, the incomplete tail, the sliding-window cache, and indexer cache represented and quantized independently? - What is the exact selection-sharing granularity across query heads and grouped output projections in CSA? - During training, when does V4 materialize dense representations and when does it run a true sparse gather/scatter backward? ## 11. Primary Evidence Map Claims in this first pass are based on: - **Paper-backed:** DeepSeek-V2 §2.1 for the MLA definition, low-rank KV/query paths, decoupled RoPE, matrix absorption, and cache accounting: . - **Paper-backed:** DeepSeek-V3 §2.1 for the inherited architecture and V3 dimensions: . - **Code-backed:** the public DeepSeek-V3 reference `MLA` implementation for projection shapes, normalization, naive/full-cache mode, and absorbed latent cache mode: . - **Paper-backed:** DeepSeek-V3.2 §2.1 and Appendix A for DSA, the shared-index contract, and MHA-mode versus MQA-mode execution: . - **Code-backed:** the DeepSeek-V3.2-Exp reference implementation for dense masked prefill and absorbed sparse decode: . The CSA/HCA bridge is intentionally marked as a hypothesis until it is checked line by line against the DeepSeek-V4 report and reference implementation.