# Attention Serving Comparison: MLA, DSA, Hybrid SWA, And KDA

Status: discussion synthesis; public-configuration-backed analytical model, not
an end-to-end benchmark

This note compares the serving implications of the attention stacks in
DeepSeek-V3/V3.2/V4, Kimi K2/K2.5/K3 and Kimi Linear, GLM-5/5.2,
MiMo-V2-Flash, and gpt-oss-120b. It records the conclusions needed to
interpret claims that one architecture is more suitable for agent workloads
or leaves more room for speculative decoding.

The central distinction is:

```text
persistent cache capacity != bytes read per decode step != attention FLOPs
```

MLA compresses the feature state stored for every original token. DSA retains
that complete history but dynamically reads only selected entries for the main
attention. Sliding-window attention can discard old entries in most layers.
KDA replaces token-by-token history in most layers with fixed-size recurrent
state. These mechanisms therefore optimize different resources.

## 1. Scope And Accounting Conventions

Unless otherwise stated, the calculations use:

- one text request with context length $L$;
- one target query token, $T=1$;
- BF16 logical cache elements at two bytes per element;
- all query heads local, without attention Tensor Parallelism;
- the optimized absorbed MLA decode formulation;
- QK and attention-value matmuls only for core-attention FLOPs.

The estimates exclude projections, softmax and top-k implementation overhead,
PagedAttention fragmentation, allocator metadata, MoE, communication,
sampling, MTP draft layers, temporary activations, and multimodal encoder
state. Public FP8 weight checkpoints do not imply FP8 KV cache, so BF16 is used
as a common logical-storage baseline.

For dense absorbed MLA with $H$ query heads, cached score dimension $D_k$,
latent accumulation dimension $D_v$, and $N$ layers:

$$
\begin{aligned}
M_{cache}&\approx NLD_kb_e,\\
F_{attn}&\approx2NTHL(D_k+D_v),\\
AI_{BF16}&\approx\frac{HT(D_k+D_v)}{D_k}
\quad\text{FLOP/byte},
\end{aligned}
$$

where $b_e$ is the number of bytes per stored cache element.

For the DeepSeek-style shape, $D_k=512+64=576$ and $D_v=512$ in the
absorbed path. The $D_v=512$ accumulation is the latent value-side dimension,
not the model configuration's reconstructed per-head `v_head_dim`.

## 2. Do Not Conflate The Kimi Models

Kimi K2 and Kimi K2.5 use a 61-layer dense MLA text decoder. K2.5 adds a vision
encoder and extends the public text context limit, but its text attention
configuration remains the K2-style MLA stack. Neither public text checkpoint
contains built-in next-token prediction layers.

Kimi Linear is a separate 48B-A3B research model. Its 27-layer stack uses a
roughly 3:1 ratio of KDA to global MLA: 20 KDA layers and seven MLA layers. It
is the model described by the phrase "linear attention plus MLA"; that phrase
does not describe Kimi K2 or K2.5.

Kimi K3 is a newer production model, but it is not yet possible to reduce it
to the same public-configuration-backed cost model. As of 2026-07-20, the
official launch post establishes a 2.8T-parameter model using KDA and
Attention Residuals, with a 1M-token context, but says that the full weights
and technical report will follow by 2026-07-27. It does not yet publish the
complete layer schedule, KDA recurrent-state shapes, or Gated-MLA head/cache
configuration needed for resident bytes, history reads, and attention FLOPs.
The comparison table therefore retains an explicit K3 row marked pending
rather than filling it from third-party guesses.

Primary sources:

- [Kimi K2 repository and model summary](https://github.com/MoonshotAI/Kimi-K2)
- [Kimi K2.5 text configuration](https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/config.json)
- [Kimi Linear paper](https://arxiv.org/abs/2510.26692)
- [Kimi Linear configuration](https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct/blob/main/config.json)
- [Kimi K3 official launch post](https://www.kimi.com/blog/kimi-k3)

## 3. Public Attention Configurations

| Model | Layers and attention | Main cache entry | Selection / window | Public MTP state |
| --- | --- | ---: | --- | --- |
| DeepSeek-V3 | 61 dense MLA layers, 128 heads | 576 elements/token/layer | full history | one next-token layer |
| Kimi K2/K2.5 | 61 dense MLA layers, 64 heads | 576 elements/token/layer | full history | none in checkpoint |
| Kimi K3 | KDA + Gated MLA; exact stack pending | not yet published | not yet published | not yet published |
| DeepSeek-V3.2 | 61 DSA-over-MLA layers, 128 heads | 576 main + 128 indexer | top-2048 main attention | one next-token layer |
| GLM-5 | 78 DSA-over-MLA-256 layers, 64 heads | 576 main + 128 indexer | top-2048 main attention | shared multi-step MTP recipe; one layer stored |
| GLM-5.2 | 78 DSA-over-MLA-256 layers, 64 heads | 576 main per layer; 128 indexer in 21 layers | top-2048 in every layer; 21 layers compute routes and 57 reuse recent token IDs | one next-token layer; IndexShare enabled for MTP iteration |
| MiMo-V2-Flash | 9 global GQA + 39 SWA-GQA layers | global: 4 x 320; SWA: 8 x 320 | SWA window 128 | three MTP layers |
| gpt-oss-120b | 18 global GQA + 18 SWA-GQA layers | 8 x (64 K + 64 V) | SWA window 128 | none in checkpoint |
| Kimi Linear | 7 global MLA + 20 KDA layers | MLA: 576/token; KDA: recurrent state | 3:1 KDA:MLA | none in checkpoint |
| DeepSeek-V4-Flash | 2 SWA + 21 CSA + 20 HCA main layers | C4/C128 512-d shared K=V; CSA 128-d index | CSA top-512; HCA dense compressed; local 128 | one MTP layer |
| DeepSeek-V4-Pro | 30 CSA + 31 HCA main layers | C4/C128 512-d shared K=V; CSA 128-d index | CSA top-1024; HCA dense compressed; local 128 | one MTP layer |

GLM-5 increases the reconstructed MLA head dimension from 192 to 256 and
reduces the head count relative to DeepSeek-V3. Its report calls this
$\text{MLA-256}$: training/prefill work and parameter count remain approximately
constant while absorbed decode uses fewer query heads. Its DSA indexer has 32
heads of dimension 128 and selects 2,048 original-token MLA entries.

MiMo-V2-Flash is not hybrid MLA. Both its global and sliding-window layers use
GQA. The global layers use four KV heads, while SWA layers use eight KV heads
and retain a 128-token window. The released 48-layer pattern contains nine
global and 39 SWA layers; the paper summarizes it as an approximately 5:1
hybrid ratio.

Primary sources:

- [DeepSeek-V3 configuration](https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json)
- [DeepSeek-V3.2 configuration](https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/config.json)
- [GLM-5 report](https://arxiv.org/html/2602.15763)
- [GLM-5 configuration](https://huggingface.co/zai-org/GLM-5/blob/main/config.json)
- [GLM-5.2 configuration](https://huggingface.co/zai-org/GLM-5.2/blob/f6142f127a14b58dc602592e996cd7d8ff139351/config.json)
- [GLM-5.2 official launch post](https://z.ai/blog/glm-5.2)
- [MiMo-V2-Flash report](https://arxiv.org/abs/2601.02780)
- [MiMo-V2-Flash configuration](https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash/blob/main/config.json)
- [gpt-oss-120b configuration](https://huggingface.co/openai/gpt-oss-120b/blob/main/config.json)
- [DeepSeek-V4 report](https://arxiv.org/html/2606.19348v1)

## 4. Persistent Cache Capacity

### Dense MLA

DeepSeek-V3 and Kimi K2/K2.5 have the same logical cache capacity despite
their different query-head counts:

$$
61\times576\times2
=70{,}272\ \text{bytes/token}
=68.625\ \text{KiB/token}.
$$

Query heads affect absorbed attention computation and reuse, not the size of
the single shared latent cache entry. Kimi's 64 heads therefore halve the core
attention arithmetic relative to DeepSeek-V3's 128 heads without halving the
cache.

### DSA over MLA

DSA must retain the main MLA cache for every original token because an
arbitrary future query may select any old entry. It also stores a smaller
indexer key for every original token:

$$
\begin{aligned}
M_{V3.2}/L
  &=61(576+128)\times2
    =83.875\ \text{KiB/token},\\
M_{GLM5}/L
  &=78(576+128)\times2
    =107.25\ \text{KiB/token}.
\end{aligned}
$$

DSA is sparse in main-attention reads and arithmetic, not in persistent
history capacity. It can use cache quantization, tiering, or cross-layer
index sharing as separate system optimizations, but top-k selection alone does
not authorize eviction of unselected main KV entries.

### MiMo hybrid SWA/GQA

MiMo's resident logical cache is:

$$
E_{MiMo}(L)
=9L\times4(192+128)
+39\min(L,128)\times8(192+128)
$$

logical cache elements per request.

After the SWA windows fill, only the nine global layers cause resident cache to
grow with $L$:

$$
\frac{\Delta M_{MiMo}}{\Delta L}
=9\times4\times320\times2
=23{,}040\ \text{bytes/token}
=22.5\ \text{KiB/token}.
$$

The saturated SWA cache is a fixed approximately 24.4 MiB per BF16 request.
At 128K, this produces about 2.84 GiB total, or 22.69 KiB averaged over the
sequence. At short contexts the comparison reverses: before its windows
saturate, MiMo stores more GQA KV elements per layer. Against the 61-layer MLA
baseline, the BF16 resident-capacity break-even is around 541 tokens.

SWA also writes the new window entry and evicts the old entry in every local
layer. Its cache write traffic per step is therefore larger than its net
resident-capacity growth. Capacity and update bandwidth should not be
conflated.

### Kimi Linear

Only seven of 27 layers retain token-indexed MLA history; the 20 KDA layers use
fixed-size recurrent and short-convolution state. The long-context MLA slope
is therefore approximately:

$$
\frac{\Delta M_{\text{Kimi Linear}}}{\Delta L}
=7\times576\times2
=7.875\ \text{KiB/token}.
$$

The KDA state adds a per-request constant rather than an $O(L)$ term. Its exact
deployed bytes depend on state precision and kernel layout. The paper reports
up to 75% lower KV use than the matched full-MLA architecture, consistent with
retaining global MLA in roughly one quarter of the layers.

## 5. A Common 128K Analytical Baseline

The following table uses $L=131{,}072$, BF16 logical cache, $T=1$, and the
scope in Section 1. `Resident attention state` includes an index-key cache
when the architecture requires one; `history read` counts one ideal read of
the logical KV/index entries consumed by one generated query, not allocator
traffic or cache-line replay:

| Model | Public attention stack | Resident attention state/request | History read/generated query | Core attention FLOPs/generated query |
| --- | --- | ---: | ---: | ---: |
| DeepSeek-V3 | 61 dense MLA | 8.58 GiB | 8.58 GiB | 2.227 TF |
| DeepSeek-V3.2 | 61 DSA-MLA, top-2048 | 10.48 GiB | 2.04 GiB | 166.8 GF |
| Kimi K2/K2.5 | 61 dense MLA, 64 query heads | 8.58 GiB | 8.58 GiB | 1.113 TF |
| Kimi K3 | KDA + Gated MLA; full configuration pending | **TBD** | **TBD** | **TBD** |
| GLM-5.2 | 78 DSA-MLA-256; top-2048 main in every layer, index scan in 21 | 11.63 GiB | 0.83 GiB | about 44.8 GF |
| MiMo-V2-Flash | 9 global + 39 SWA-128 GQA | 2.84 GiB | 2.84 GiB | 48.5 GF |
| gpt-oss-120b | 18 global + 18 SWA-128 GQA | 4.50 GiB | 4.50 GiB | 38.7 GF |
| DeepSeek-V4-Flash | 2 SWA + 21 CSA-C4 + 20 HCA-C128 | 0.85 GiB | 0.20 GiB | about 16.1 GF |
| DeepSeek-V4-Pro | 30 CSA-C4 + 31 HCA-C128 | 1.21 GiB | 0.30 GiB | about 34.5 GF |

These values are useful for structural comparison, not as measured latency.
In particular, DSA's indexer top-k, irregular gather, and sparse kernel
efficiency make realized speedup much smaller than the raw main-attention FLOP
reduction. V4's rows are BF16-equivalent logical comparisons; their published
mixed cache formats reduce the corresponding 128K payloads to about 0.43 GiB
for Flash and 0.62 GiB for Pro before allocator/workspace overhead. None of the FLOP values includes
projection, compressor, softmax/top-k, MoE, communication, or MTP work.

The new rows use the following auditable formulas.

GLM-5.2 is still DSA-over-MLA in all 78 attention layers. `full` and `shared`
in its `indexer_types` array describe the **indexer**, not whether the layer's
main attention is dense or sparse. The 21 `full` indexer layers scan their own
128-dimensional index-key histories and produce top-2048 original-token IDs.
Each following `shared` layer skips that index scan and reuses the recent ID
set, but still executes its own top-2048 main MLA attention with layer-specific
queries, main KV cache, and output weights. GLM-5.2 therefore retains 78
per-token main MLA histories while storing/scanning index-key history in only
21 layers:

$$
\begin{aligned}
M_{GLM5.2}
  &=2L(78\times576+21\times128),\\
R_{GLM5.2}
  &=2(21L\times128+78\times2048\times576),\\
F_{GLM5.2}
  &\approx21(2\times32\times L\times128)
    +78(2\times64\times2048\times(576+512)).
\end{aligned}
$$

The official GLM-5.2 configuration contains 21 `full` and 57 `shared`
indexer types. The launch post describes IndexShare as reusing one indexer
across every four sparse layers and explicitly cautions that the FLOP reduction
does not proportionally reduce KV-cache size. Sources:
[configuration](https://huggingface.co/zai-org/GLM-5.2/blob/f6142f127a14b58dc602592e996cd7d8ff139351/config.json)
and [official launch post](https://z.ai/blog/glm-5.2).

gpt-oss-120b has 36 layers alternating between 18 global and 18 SWA-128
layers, with 64 query heads, eight KV heads, and 64-dimensional keys and
values:

$$
\begin{aligned}
M_{gpt\text{-}oss}=R_{gpt\text{-}oss}
  &=2\times8\times(64+64)(18L+18\times128),\\
F_{gpt\text{-}oss}
  &\approx2\times64\times(64+64)(18L+18\times128).
\end{aligned}
$$

Source: [official gpt-oss-120b configuration](https://huggingface.co/openai/gpt-oss-120b/blob/main/config.json).

For V4-Flash, the BF16-equivalent row treats the 512-dimensional main entry
and 128-dimensional CSA index key as two-byte logical elements. With 21 CSA,
20 HCA, two pure-SWA layers, and a 128-token local path in every main layer:

$$
\begin{aligned}
M_{V4F}/2
  &=21\frac{L}{4}(512+128)
    +20\frac{L}{128}512
    +43\times128\times512,\\
R_{V4F}/2
  &=21\left(\frac{L}{4}128+(512+128)512\right)\\
  &\quad+20\left(\left(\frac{L}{128}+128\right)512\right)
    +2\times128\times512.
\end{aligned}
$$

Its core-FLOP estimate counts the 64-head, 128-dimensional CSA index scan and
the 64-head, 512-dimensional shared-KV QK/AV work. See
[CSA/HCA definition, Section 8](./01-csa-hca-definition.md) for the component
model and [inference systems, Section 7](./04-inference-systems.md) for deployed
mixed-format payload accounting.

V4-Pro uses 30 CSA layers, 31 HCA layers, 128 main query heads, and CSA
$k=1024$. At 128K, both its CSA main candidate list and HCA dense-compressed
candidate list happen to contain $1024+128=1152$ entries:

$$
\begin{aligned}
M_{V4P}/2
  &=30\frac{L}{4}(512+128)
    +31\frac{L}{128}512
    +61\times128\times512,\\
R_{V4P}/2
  &=30\left(\frac{L}{4}128+(1024+128)512\right)\\
  &\quad+31\left(\left(\frac{L}{128}+128\right)512\right),\\
F_{V4P}
  &\approx30\left(2\times64\times128\times\frac{L}{4}
        +4\times128\times1152\times512\right)\\
  &\quad+31(4\times128\times1152\times512).
\end{aligned}
$$

## 6. Why Dense MLA FLOPs Are Large

MLA reduces cache bytes by moving work into the latent-space attention path.
For every generated token, every dense layer still scans all $L$ positions:

$$
\begin{aligned}
F_{V3}(L)&\approx61\times2\times128\times L\times(576+512),\\
F_{K2}(L)&\approx61\times2\times64\times L\times(576+512).
\end{aligned}
$$

At 128K these are about 2.23 and 1.11 TFLOPs per generated token. At 4K they
fall to about 69.6 and 34.8 GFLOPs. The large 128K numbers are therefore the
product of dense history scan, layer count, head count, and latent dot-product
width; they are not a constant cost at all context lengths.

Compared with an unabsorbed MHA-style core using per-head key/value dimensions
192 and 128, the V3 absorbed core performs roughly:

$$
\frac{576+512}{192+128}=3.4
$$

as much QK/AV arithmetic per head-history pair, while replacing a
head-specific cache with one shared 576-element latent entry. This is MLA's
fundamental exchange of Tensor Core arithmetic for persistent capacity and HBM
traffic.

## 7. DSA And SWA Remove Different Costs

The decode paths can be summarized as:

```text
DSA:
  scan a small index key for every historical token
    -> choose content-dependent top-k
    -> gather top-k entries from the complete main MLA cache

SWA:
  keep only the most recent W entries in local layers
    -> read and attend to W entries
    -> use periodic global layers for full-history communication
```

Consequences:

| Resource | DSA over MLA | Hybrid SWA/global |
| --- | --- | --- |
| Complete main history retained | yes | only in global layers |
| Extra selection cache | yes | no |
| Main attention length | top-$k$ | $W$ locally, $L$ globally |
| Selection pattern | content-dependent | fixed by position and layer |
| Can evict old local KV | no | yes |
| Long-range recall path | dynamic in every sparse layer | periodic global layers |

MiMo therefore chooses stronger structural forgetting to minimize serving
cost. DSA pays more capacity to keep every old token dynamically retrievable.
Neither point dominates the other without a model-quality and workload
comparison.

## 8. Batch Capacity And Speculation Headroom

Smaller resident state allows more active requests, retained prefixes, or
longer contexts before cache eviction. In a purely KV-capacity-limited 128K
comparison, MiMo's roughly 2.84 GiB/request leaves substantially more admission
room than the roughly 8.58 GiB dense-MLA baselines. Real admission gains are
smaller because weights, workspaces, graph pools, page fragmentation, and MTP
state also consume memory.

Speculative verification creates a separate opportunity. If $T$ target query
tokens reuse one cache read, ideal arithmetic intensity grows linearly with
$T$:

| Core attention | $T=1$ BF16 AI | $T=4$ BF16 AI |
| --- | ---: | ---: |
| MiMo global GQA, 4 KV heads | about 16 FLOP/B | about 64 FLOP/B |
| MiMo SWA-GQA, 8 KV heads | about 8 FLOP/B | about 32 FLOP/B |
| Kimi/GLM 64-head MLA | about 121 FLOP/B | about 484 FLOP/B |
| DeepSeek-V3 128-head MLA | about 242 FLOP/B | about 967 FLOP/B |

Against a practical Hopper ridge point around 250--300 FLOP/byte, MiMo
attention can remain bandwidth-oriented at useful verification widths, while
V3 MLA is close to the ridge at one query and becomes compute-oriented as soon
as verification widens. This is the narrow systems basis for saying hybrid
SWA/GQA leaves more MTP headroom.

It does not mean MTP is free. Every verified token still incurs projections,
MoE expert work, communication, and draft overhead. MiMo-V2-Flash reports up
to 3.6 accepted length and 2.6 times decode speedup with three MTP layers; this
is an empirical property of its model, kernels, acceptance distribution, and
test workload, not a guarantee of the attention pattern alone.

Batch and verification width also compete for the same compute and temporary
capacity:

```text
low load / small batch -> spend idle compute on wider speculation
high load / large batch -> narrow speculation to protect aggregate throughput
```

The architecture expands the feasible
$\text{batch}\times\text{speculative-width}$ scheduling region. It does not
imply that both axes should always be maximized.

## 9. Fixed Batch And Fixed Speculation

If batch size, context length, verification width, accepted length, hardware,
and model-side non-attention work were all held fixed, MiMo's attention path
would usually be faster at long context. Fitting in memory removes its admission
advantage but does not remove the lower per-step KV reads and arithmetic:

$$
t_{step}\gtrsim
\max\left(
\frac{F_{attn}}{P_{compute}},
\frac{B_{attn}}{BW_{HBM}}
\right)
+Tt_{nonattn}+t_{overhead}.
$$

At 128K and $T=1$, the common baseline estimates approximately 3.05 GB of
main KV reads and 48.5 GFLOPs for MiMo attention, versus about 9.21 GB and 1.11
TFLOPs for Kimi K2, or 9.21 GB and 2.23 TFLOPs for DeepSeek-V3. A simplified
H100-class roofline gives attention-only lower bounds near 0.9 ms for MiMo and
roughly 2.5--2.8 ms for the dense-MLA cases.

These lower bounds are not predicted end-to-end token latency. Sequential
layers, kernel launches, page tables, softmax, sparse gathers, MoE, collectives,
and scheduling all add time. Once attention is reduced, MoE or communication
may become the new bottleneck. Actual released-model comparisons are further
confounded by different activated parameter counts and training choices.

Final output rate depends on acceptance as well as verification step time:

$$
\text{generated tokens/s}
\approx\frac{\mathbb{E}[\text{accepted tokens}]}
{t_{\text{verification step}}}.
$$

Holding speculative width fixed is therefore insufficient for a strict speed
comparison unless the accepted length is also held fixed or measured.

## 10. Reusable Conclusions

1. MiMo is more KV-capacity-efficient than the dense-MLA and DSA examples at
   long context because 39 of 48 layers stop retaining history beyond 128
   tokens. Its per-layer GQA cache is not intrinsically smaller than MLA.
2. DSA reduces main-attention reads and arithmetic but retains all main MLA
   entries and adds an indexer cache. It is a compute/read optimization, not a
   persistent-capacity compression method.
3. DeepSeek-V3 and Kimi K2 have equal per-token MLA capacity, but Kimi's 64
   query heads halve ideal absorbed core-attention FLOPs relative to V3's 128.
4. Kimi K2.5 remains a dense-MLA text model. Kimi Linear is the separate
   KDA-plus-MLA hybrid.
5. Hybrid SWA/GQA can support a larger resident batch and can leave more
   attention-side room for MTP. These are two uses of shared system headroom,
   not independent gains that can always be maximized together.
6. With batch and speculation fixed, hybrid attention can still be faster
   because it reads and computes less. End-to-end gain is bounded by the
   fraction of decode time that attention originally consumed.

## 11. Measurement Follow-Up

The analytical table should eventually be paired with measurements that hold
constant:

- hardware and cache precision;
- batch and context-length distribution;
- target verification width and accepted length;
- attention-only versus full-layer versus end-to-end decode time;
- active model parameters and parallel layout;
- prefix-cache hit rate and cache admission policy.

Report resident cache, bytes read per step, core FLOPs, achieved bandwidth,
achieved Tensor Core throughput, and generated tokens per second separately.
Without that decomposition, a faster model cannot be attributed cleanly to
hybrid attention, MTP, smaller MoE activation, or serving policy.