# Attention Design Space: A High-Level Map

Status: high-level synthesis and navigation map

This note places the attention mechanisms discussed in this reading kit into
one design space. It is intentionally less detailed than the prerequisite
notes:

- [MLA primer](./00-mla-primer.md): definitions and tensor paths;
- [MLA systems and roofline](./00a-mla-systems-roofline.md): hardware balance;
- [attention serving comparison](./00b-attention-serving-comparison.md): model
  configurations, cache accounting, and decode FLOPs.
- [YOCO and You Only Index Once](./00d-yoco-yoio.md): cross-layer KV and
  routing reuse.

The purpose here is to prevent mechanisms on different design axes from being
treated as mutually exclusive alternatives.

## 1. Four Mostly Orthogonal Design Axes

An attention stack can be described by four questions:

| Axis | Question | Representative choices |
| --- | --- | --- |
| Feature representation | What state is stored for each memory entry, and how do query heads share it? | MHA, GQA, MQA, MLA |
| Sequence representation and access | How many memory entries exist, and which entries can a query read? | full, SWA, DSA, NSA, CSA, HCA, KDA |
| Cross-layer reuse | Do layers own independent memory and routing, or reuse either across depth? | layer-wise KV, YOCO shared KV, CLSA shared KV and routing |
| Generation | How many candidate output positions does one target pass verify? | autoregressive decode, MTP/speculative decode |

These axes can be composed. Examples include:

```text
DeepSeek-V3:    MLA x full history x MTP
DeepSeek-V3.2:  MLA x DSA selection x MTP
GLM-5:          MLA-256 x DSA selection x MTP
MiMo-V2-Flash:  GQA x hybrid SWA/global x MTP
Kimi Linear:    MLA/KDA layer hybrid x ordinary autoregressive checkpoint
DeepSeek-V4:    MQA-style shared compressed KV
                x hybrid CSA/HCA/local-SWA x MTP

YOCO:           GQA x full shared cross-memory
                x cross-layer KV reuse x autoregressive decode

YOCO + CLSA:    GQA x token-level top-k
                x cross-layer KV and routing reuse x autoregressive decode
```

Consequently, *MLA versus hybrid attention* is not a strict either/or choice.
MLA describes how a token-level entry is represented. Hybrid attention
describes how different access or memory mechanisms are composed across
branches or layers. A model could use SWA-MLA in local layers and full MLA in
global layers, subject to quality, kernel, and cache-management constraints.

## 2. Feature-Representation Lineage

The MHA-to-MLA family primarily changes the feature state associated with one
historical position:

```text
MHA
  independent K/V for every query head
  -> expressive, large cache

GQA
  a small number of shared KV groups
  -> less cache and HBM traffic

MQA
  one shared KV head
  -> minimal direct KV sharing cost

MLA
  one shared latent plus positional state
  + head-specific transformations around that latent
  -> cache near very-low-group GQA/MQA
  -> stronger head-specific capacity
  -> more absorbed decode arithmetic
```

MLA is therefore an improved capability/cache Pareto point, not an unconditional
replacement that is always faster than cache-equivalent GQA or MQA. It trades
HBM capacity and bandwidth for matrix arithmetic.

This axis alone does not reduce the number of history positions. Dense MLA
still has one persistent latent entry for every token and scans all entries at
every full-attention layer.

## 3. Sequence-Representation And Access Lineage

The second axis asks two separate questions:

1. How many persistent memory entries represent a sequence of $L$ tokens?
2. How many of those entries are read by one query?

Those questions produce the following progression:

```text
full attention
  L token entries stored, L entries read
        |
        +--> SWA
        |      retain and read only recent W entries
        |
        +--> DSA
        |      retain all L token entries
        |      index all L, read top-k main entries
        |
        +--> NSA
        |      compressed coarse branch
        |      + selected original-token branch
        |      + local sliding branch
        |
        +--> learned cross-token compression
               |
               +--> CSA
               |      store about L/C compressed entries
               |      index compressed sequence
               |      read top-k compressed entries
               |
               +--> HCA
                      store about L/C_h compressed entries
                      densely read the heavily compressed sequence

KDA / recurrent linear attention
  replace token-indexed history in a layer with fixed-size recurrent state
```

DSA and NSA principally reduce per-query reads and attention computation. They
do not, by themselves, authorize deletion of all old fine-grained KV:

- DSA may select any original-token MLA entry for a future query;
- NSA's selected branch may retrieve fine-grained K/V from any old selected
  block, while its compressed and local branches add their own state.

CSA and HCA change the persistent memory object. Once a complete group of
original entries has been converted into a compressed entry, old uncompressed
entries can be discarded except for the local SWA state and the incomplete
compression tail. This is the step that makes sequence-dimension storage
compression explicit.

Primary definitions:

- [Native Sparse Attention](https://arxiv.org/html/2502.11089)
- [DeepSeek-V3.2 / DSA report](https://huggingface.co/deepseek-ai/DeepSeek-V3.2/resolve/main/assets/paper.pdf)
- [DeepSeek-V4 CSA/HCA report](https://arxiv.org/html/2606.19348v1)

## 4. Store, Read, And Compute Matrix

Use `store` for persistent resident history, `read` for history traffic in one
decode step, and `compute` for the main attention work. The table describes
the native mechanism, excluding optional quantization, cache offload, or
eviction policies:

| Mechanism | Persistent store | Per-query read | Main attention compute | Main qualification |
| --- | --- | --- | --- | --- |
| Full MHA/GQA | $O(L)$ | $O(L)$ | $O(L)$ | baseline |
| Dense MLA | much smaller bytes/entry, still $O(L)$ | much smaller bytes, still $O(L)$ | $O(L)$, with more latent arithmetic | feature compression |
| SWA | $O(W)$ per local layer | $O(W)$ | $O(W)$ | fixed structural forgetting |
| DSA over MLA | main + indexer are $O(L)$ | index scan $O(L)$ + main $O(K)$ | index scan + main $O(K)$ | content-dependent original-token retrieval |
| NSA | fine KV $O(L)$ + compressed/local state | compressed + selected + local | compressed + selected + local | three branches within a layer |
| CSA | compressed main/indexer $O(L/C)$ + fixed state | index $O(L/C)$ + main $O(K)$ + local $O(W)$ | same asymptotic components | DSA over compressed entries |
| HCA | $O(L/C_h)$ + fixed state | $O(L/C_h+W)$ | $O(L/C_h+W)$ | dense attention over a very short memory |
| KDA | fixed recurrent and convolution state | fixed state | sequence-independent recurrent decode step | finite-state memory rather than retrieval |
| Dense YOCO cross-decoder | one shared full-length KV sequence | the same full sequence is read by every cross layer | $O(L)$ per cross layer | storage sharing does not imply read sharing |
| YOCO + CLSA | one shared full KV plus lightweight index memory | one full index scan, then selected KV in every cross layer | $O(K)$ per cross layer after one shared routing pass | token-level routing shared across depth |

$O(\cdot)$ hides entry dimensions, head sharing, compression construction,
softmax, top-k, sparse gathers, and kernel efficiency. A lower asymptotic count
does not guarantee lower wall time at short sequence lengths.

## 5. NSA, DSA, CSA, And HCA Relationship

These names are related but are not successive versions of one identical
operator.

### DSA to CSA: direct operational lineage

DSA supplies the lightning-indexer and top-k idea. CSA first compresses the
sequence and then applies a DSA-style indexer to compressed entries:

```text
DSA: query -> index original-token MLA entries -> top-k original entries

CSA: original tokens -> learned compressed entries
                     -> index compressed entries
                     -> top-k compressed entries
```

CSA therefore combines two reductions:

- sequence compression reduces persistent entries and indexer scan length;
- sparse selection reduces the compressed entries read by core attention.

### NSA to V4: structural relationship

NSA puts three information scales in parallel within a layer:

```text
coarse compressed + selected fine-grained + recent local
```

V4 preserves a similar coarse/retrieval/local hierarchy but reorganizes it:

- CSA provides moderately compressed, content-selected global memory;
- HCA provides heavily compressed, dense global memory;
- both include a 128-token local SWA branch;
- CSA and HCA layer types are interleaved through the model.

The analogy is strong, but CSA is not simply NSA renamed. In particular, CSA
selects and consumes compressed entries rather than returning to old
fine-grained token KV.

## 6. Two Meanings Of Hybrid Attention

*Hybrid* can refer to composition at different levels.

### Within one layer

```text
NSA: compressed + selected-original + local, with gated aggregation
CSA: selected-compressed + local SWA
HCA: dense-heavily-compressed + local SWA
```

### Across layers

```text
MiMo:          SWA-GQA layers interleaved with global-GQA layers
Kimi Linear:   KDA layers interleaved with global-MLA layers
DeepSeek-V4:   CSA layers interleaved with HCA layers
```

The first kind combines information paths inside an attention layer. The
second assigns different memory/access roles to different Transformer layers.
Many architectures use both.

## 7. Why Hybrid Attention And MTP Can Be Complementary

Hybrid attention removes repeated full-history work from many layers. MTP asks
the target model to verify multiple candidate positions in one pass. These
operations can complement each other:

```text
SWA / sequence compression / sparsity
  -> less cache residency, history traffic, and attention arithmetic
  -> larger feasible batch or more serving headroom

MTP
  -> wider query verification and more compute per target step
  -> multiple accepted output tokens when prediction quality is high
```

The saved work is not automatically converted into speed. MTP still performs
projections, MoE, communication, and sampling for candidate positions. Its
benefit depends on acceptance, verification width, batch load, context length,
and kernel quality. The useful systems claim is that efficient hybrid
attention enlarges the feasible
$\text{batch}\times\text{speculation-width}$ region.

## 8. Model Coordinates In The Design Space

| Model | Feature representation | Sequence access / memory | Generation observation |
| --- | --- | --- | --- |
| DeepSeek-V3/R1 base | MLA | dense full history | MTP-trained architecture |
| Kimi K2/K2.5 text | MLA | dense full history | no bundled next-token layer in public checkpoints |
| DeepSeek-V3.2 | MLA | DSA over original-token entries | bundled next-token layer |
| GLM-5 | MLA-256 | DSA over original-token entries | parameter-shared multi-step MTP recipe |
| MiMo-V2-Flash | GQA | 39 SWA + 9 global layers | three MTP layers |
| Kimi Linear 48B-A3B | MLA in global layers | 20 KDA + 7 global MLA layers | no bundled next-token layer |
| DeepSeek-V4 | shared compressed MQA-style KV | CSA/HCA interleave, both with local SWA | MTP depth one; serving may schedule wider verification |
| YOCO | GQA shared across query heads | efficient self-decoder + dense cross-decoder over one shared full KV | ordinary autoregressive checkpoint |
| YOCO + CLSA | GQA shared across query heads | one token-level top-k set reused by all cross-decoder layers | ordinary autoregressive checkpoint |

This table describes public architecture, not unpublished production serving
policy. An engine can also add an external speculative drafter to a checkpoint
without bundled MTP weights.

## 9. Compact Mental Model

The shortest reusable summary is:

```text
MHA / GQA / MQA / MLA
  decide how wide one memory entry is and how heads share it.

Full / SWA / DSA
  decide which original-token entries remain and are read.

NSA
  combines compressed, selected-original, and local views inside a layer.

CSA / HCA
  replace old token entries with learned cross-token memory entries;
  CSA selects them, HCA densely reads a much shorter sequence.

Hybrid architecture
  composes different memory and access mechanisms across branches or layers.

YOCO / CLSA
  decide whether full-length KV and sparse routing are recomputed across
  depth; selected attention outputs remain layer-specific.

MTP
  changes how many future positions one target step tries to validate.
```

From this perspective, DeepSeek-V4's main change beyond MLA/DSA is not merely
stronger sparsity. It changes the unit of long-term memory from one entry per
original token to one learned entry per group of tokens, then uses CSA and HCA
to provide complementary retrieval and dense-global paths.

## 10. Questions To Carry Into The Detailed CSA/HCA Study

- What information is lost or preserved when original-token KV is retired
  after compression?
- Why does CSA use moderate overlapping compression while HCA uses much heavier
  non-overlapping compression?
- What exact CSA/HCA layer schedule supplies enough retrieval and dense-global
  mixing?
- How much runtime belongs to compression update, indexer scan, sparse gather,
  local SWA, and grouped output projection?
- How do incomplete compression blocks and SWA state interact with prefix
  caching, PagedAttention, and on-disk caches?
- How does MTP verification change the roofline of CSA indexer, selected MQA,
  HCA dense-compressed attention, and the local branch?

Those questions are routed to the later definition, lineage, and inference
chapters rather than resolved at this high-level layer.