# MLA Systems And Roofline Notes Status: discussion synthesis; analytical predictions are not yet measured here This note extends the [MLA primer](./00-mla-primer.md) with the systems conclusions needed for later DSA, CSA, and HCA analysis. Its central claim is that MLA is best understood as an architecture-hardware co-design point: it trades abundant matrix arithmetic for scarce persistent cache capacity and HBM bandwidth while retaining head-specific expressivity. The note separates four questions that are easy to conflate: 1. Is an MLA attention kernel faster than MHA? 2. Is it faster than a cache-equivalent GQA or MQA kernel? 3. Does it reduce one request's token-to-token latency? 4. Does it increase maximum serving throughput and concurrency? Answers to one do not automatically answer the others. ## 1. MLA As A Hardware-Aware Efficiency Frontier The high-level resource exchange is: ```text MHA head-specific K/V, strong capacity -> large persistent cache and HBM traffic MQA / low-group GQA directly shared K/V -> small cache and relatively little arithmetic -> less head-specific KV capacity MLA one shared latent + small RoPE key + head-specific query/key/value transformations -> cache near very-low-group GQA -> capacity empirically near or above MHA in DeepSeek's experiments -> more matrix arithmetic than cache-equivalent MQA/GQA ``` MLA therefore seeks a better Pareto point across: - model capability; - persistent KV capacity; - HBM traffic; - Tensor Core arithmetic; - practical kernel utilization. It is not simply an algorithm that performs fewer operations. It moves work from a resource that commonly constrains decode—HBM capacity and bandwidth—to a resource modern accelerators supply in abundance—low-precision matrix throughput. This interpretation is hardware-aware but not hardware-universal. Its benefit depends on the device's compute/bandwidth balance, how many query heads remain local to a device, cache precision, query width, and kernel quality. ## 2. What FlashMLA Optimizes The original FlashMLA work focused on the core absorbed MLA decode operation. Its input query has already been transformed into latent space, and its cache already contains the compressed latent plus RoPE key. For the V3 shape: | Tensor | Shape | | --- | --- | | query | $[B,s_q,128,576]$ | | paged KV cache | $[L,1,576]$ | | output | $[B,s_q,128,512]$ | The kernel fuses or coordinates: ```text paged / variable-length KV reads -> latent-space QK -> online softmax -> attention-weighted latent accumulation -> split-KV partial results -> combine / log-sum-exp reduction ``` The surrounding MLA layer still performs work outside this core kernel, including the query-side absorbed projection and the value/output-side projection. FlashMLA is therefore not the complete MLA layer. Important implementation mechanisms include: - a tile scheduler across requests, heads, and sequence splits; - split-KV to expose parallel work for a small number of long requests; - paged and ragged KV-cache access; - online softmax without materializing the score matrix; - Hopper TMA and overlap between Tensor Core, CUDA Core, and memory work; - FP8 latent-cache dequantization; - sparse index gathers for DSA. The current repository is broader than the original decode kernel. Its support matrix includes dense and sparse decode, sparse MLA prefill, and dense MHA prefill, with different coverage on SM90 and SM100. In particular, as checked on 2026-07-17, dense MLA decode is still listed as SM90-only, while sparse FP8 decode supports SM90 and SM100 but is explicitly described as not yet well optimized on B200. Primary implementation sources: - [FlashMLA README and support matrix](https://github.com/deepseek-ai/FlashMLA) - [dense decode test and FLOP/byte accounting](https://github.com/deepseek-ai/FlashMLA/blob/main/tests/test_flash_mla_dense_decoding.py) - [Hopper dense decode kernel deep dive](https://github.com/deepseek-ai/FlashMLA/blob/main/docs/20250422-new-kernel-deep-dive.md) - [Hopper FP8 sparse decode deep dive](https://github.com/deepseek-ai/FlashMLA/blob/main/docs/20250929-hopper-fp8-sparse-deep-dive.md) ## 3. Dense Decode Arithmetic Intensity Use the absorbed V3 MLA dimensions: | Symbol | Meaning | V3 value | | --- | --- | ---: | | $B$ | independent requests in the batch | variable | | $T=s_q$ | target query tokens processed per request | 1 or more | | $H$ | local query heads | 128 without head TP | | $L$ | attended KV length | variable | | $D_k$ | latent plus RoPE score dimension | 576 | | $D_v$ | latent accumulation dimension | 512 | Ignoring softmax and lower-order input/output traffic, the main attention work is: $$ F_{attn}\approx2BTHL(D_k+D_v). $$ For a BF16 latent cache, the dominant HBM traffic is: $$ B_{HBM}\approx BLD_k\operatorname{sizeof}(\mathrm{BF16})=2BLD_k. $$ The steady-state arithmetic intensity is therefore: $$ \begin{aligned} AI_{BF16} &\approx\frac{HT(D_k+D_v)}{D_k}\\ &=\frac{128T(576+512)}{576}\\ &\approx241.8T\ \text{FLOP/byte}. \end{aligned} $$ This yields the first important result: - independent-request batch size $B$ cancels; - KV length $L$ cancels once it dominates fixed overhead; - the number of query tokens sharing a cache read, $T$, does not cancel; - the number of query heads local to the device, $H$, does not cancel. Batch size and sequence length strongly affect whether the GPU is filled, but they do not by themselves change the ideal steady-state compute/memory ratio. Query-token reuse and local-head reuse do. ### Lower-order traffic at short sequence lengths The query read and output write add approximately: $$ B_{fixed}\approx2TH(D_k+D_v)\ \text{bytes per request}. $$ For BF16 V3 dimensions, that fixed traffic is comparable to about 242 cached tokens for $T=1$. Consequently, the long-sequence approximation is poor at a few hundred tokens or less. Launch, metadata, page-table, softmax, and combine costs can dominate even beyond that point. ## 4. Hopper And Blackwell Roofline Representative dense-BF16 hardware ratios are: | Device | Dense BF16 peak | HBM bandwidth | Theoretical ridge point | | --- | ---: | ---: | ---: | | H100/H800 SXM class | about 1.0 PFLOP/s | about 3.35 TB/s | about 299 FLOP/byte | | B200 | about 2.25 PFLOP/s | about 8 TB/s | about 281 FLOP/byte | Sources: - [NVIDIA Hopper architecture specifications](https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/) - [NVIDIA HGX B200 specifications](https://images.nvidia.com/aem-dam/Solutions/documents/HGX-B200-PCF-Summary.pdf) - [NVIDIA Blackwell tuning guide](https://docs.nvidia.com/cuda/blackwell-tuning-guide/) For ordinary one-token decode, $T=1$ gives about 242 FLOP/byte. Against theoretical peaks, this lies near the compute/bandwidth boundary and slightly on the bandwidth side for both device classes. The practical boundary is lower. DeepSeek's Hopper analysis uses an H800 practical compute peak near 865 TFLOP/s under its operating clock and 3.35 TB/s bandwidth, or about 258 FLOP/byte. The optimized dense FlashMLA kernel reports up to 660 TFLOP/s in compute-bound configurations and 3 TB/s in memory-bound configurations. V3 MLA with 128 local heads is therefore deliberately close to the practical Hopper ridge point: modest changes in clock, tiling, sequence shape, or overhead can move it between the two regimes. This is the strongest basis for viewing MLA as hardware-conscious co-design. The dimensions do not merely minimize cache; their shared reuse creates enough arithmetic intensity to make Tensor Core scheduling a first-order decode problem. ## 5. Batch, Sequence Length, Query Width, And TP The expected regimes are: | Workload region | Likely limiter | Expected behavior | | --- | --- | --- | | short $L$, small $B$ | launch, latency, occupancy | a larger GPU can be harder to fill | | long $L$, small $B$ | split-KV and combine | sequence splitting exposes SM parallelism but creates partial-output overhead | | short $L$, large $B$ | request parallelism and fixed overhead | batch improves utilization without changing ideal AI | | long $L$, large $B$, $T=1$ | roofline boundary | optimized BF16 MLA can be bandwidth- or compute-bound | | long $L$, large $B$, $T\ge2$ | Tensor Core compute | cache reuse across query tokens raises AI linearly | | fewer local query heads | HBM bandwidth | head TP reduces reuse and moves the kernel toward memory-bound | For example: $$ \begin{array}{c|c|c} H&T&AI_{BF16}\ (\text{FLOP/byte})\\ \hline 128&1&\approx242\\ 128&2&\approx484\\ 64&1&\approx121\\ 32&1&\approx60 \end{array} $$ This explains why the FlashMLA deep dive explicitly notes that DeepSeek's decode instances did not use Tensor Parallelism: keeping all 128 query heads local maximizes reuse of the single latent KV head. This is a system choice, not merely a model equation. ## 6. Does MLA Make Decode Faster? The answer depends on the comparison. ### Versus MHA MLA usually has a strong long-context decode advantage. It performs more latent-space arithmetic, but reads tens of times less persistent KV state. It can convert an extremely bandwidth-bound MHA operation into a balanced or compute-bound MLA operation. This does not imply that end-to-end token latency improves by the same factor. MoE execution, collective communication, layer projections, scheduling, and sampling remain. At short context, attention may be a small fraction of the decode step. ### Versus cache-equivalent GQA or MQA MLA is not guaranteed to be faster. A two- or three-group GQA cache has a similar byte count but a smaller attention dimension and fewer latent transformations. MQA can be smaller still. MLA spends extra arithmetic to retain head-specific transformations and empirical model quality. The correct claim is therefore: ```text MLA improves the capability / cache Pareto frontier. It does not dominate cache-equivalent GQA/MQA in raw kernel latency. ``` ### Serving throughput and concurrency Smaller KV state lets a serving system admit more requests, retain longer prefixes, or avoid cache eviction. Larger feasible batches can move the engine from underfilled to compute-saturating. This capacity effect can improve total throughput even when one request's core attention kernel is not much faster. DeepSeek-V2 reports a 93.3% KV-cache reduction and 5.76 times the maximum generation throughput of DeepSeek 67B on an 8-H800 node. That is an end-to-end model and deployment comparison, not an isolated MLA speedup: it also includes MoE architecture, activated-parameter differences, FP8 weights, and KV-cache quantization. The safe conclusion is that MLA enabled a much larger batch and was one major contributor. Source: [DeepSeek-V2 inference-efficiency discussion](https://arxiv.org/html/2405.04434). ## 7. MTP-1 Is Not Necessarily $s_q=1$ The DSpark paper identifies MTP-1 as the former DeepSeek-V4 production baseline. MTP-1 means that the drafter proposes one token. The target can still verify the drafted token and compute a bonus position in one pass, so the target-model verification budget is statically two tokens. DSpark explicitly describes its scheduler as expanding that two-token MTP-1 budget to roughly four to six target tokens under moderate load. This distinction matters for an MLA-style roofline model: $$ \begin{aligned} \text{ordinary autoregressive target width:}\quad&s_q=1,\\ \text{MTP-1 target verification budget:}\quad&s_q\approx2. \end{aligned} $$ At $H=128$, moving from $s_q=1$ to $s_q=2$ raises the ideal BF16 MLA intensity from about 242 to 484 FLOP/byte, placing it clearly on the compute side of Hopper's roofline. The DSpark production result should not be used as a direct MLA benchmark: it evaluates DeepSeek-V4, whose main attention is the CSA/HCA hybrid rather than V3's per-token MLA cache. It is evidence for the serving relationship between verification width, batch capacity, and useful compute—not a clean measurement of MLA latency. DSpark also explains why static MTP-3/5 was not the former production default: under high concurrency, verifying low-probability draft suffixes consumed critical batch capacity and reduced aggregate throughput. Its contribution is to vary verification width with confidence and engine load. Source: [DSpark production deployment and MTP-1 baseline](https://arxiv.org/html/2607.05147). ## 8. FP8 Cache Creates A Third Bottleneck DeepSeek-V3.2's FlashMLA FP8 cache stores each token in 656 bytes: ```text 512 bytes FP8 latent 16 bytes four FP32 scales 128 bytes 64 BF16 RoPE values --------- 656 bytes ``` The attention MMAs still operate in BF16 after dequantization. With the same 278,528 attention FLOPs per cached token at $H=128,T=1$, ideal intensity rises to: $$ AI_{\text{FP8 cache}}\approx\frac{278{,}528}{656} \approx424.6\ \text{FLOP/byte}. $$ This removes HBM bandwidth as the obvious limiter but does not make the work free. On Hopper, DeepSeek estimates roughly 34 cycles of MMA work versus at least 50 cycles to dequantize one token for a 64-head CTA. The kernel can become CUDA-Core dequantization-bound. FlashMLA uses paired CTAs and Hopper Distributed Shared Memory to split and exchange dequantized state. The appropriate model is therefore not a two-resource roofline alone: ```text HBM bandwidth Tensor Core MMA CUDA Core dequantization / softmax / address work ``` Blackwell creates an opportunity rather than an automatic win. It has higher HBM bandwidth, faster BF16/FP8 Tensor Cores, larger L2, and native block-scaled MMA support. A kernel that still dequantizes to BF16 before MMA may fail to use those advantages. The current FlashMLA README reports only about 350 TFLOP/s for B200 sparse decode and labels it not yet optimized, while sparse prefill is reported near 1450 TFLOP/s. Relevant hardware source: [Blackwell SM100 narrow-precision and block-scaled MMA](https://docs.nvidia.com/cutlass/latest/media/docs/cpp/blackwell_functionality.html). ## 9. Connection To DSA, CSA, And HCA The progression can be framed as successive bottleneck removal: ```text MLA compress each token's feature state -> feature-dimension efficiency DSA select fewer original-token MLA entries -> token-selection efficiency CSA / HCA build fewer entries by compressing across tokens -> sequence-dimension efficiency ``` MLA does not reduce the number of history positions. At sufficiently long context, even a 576-element cache entry is expensive to scan. DSA reduces the scan to top-$k$, while V4 changes the memory unit itself. These mechanisms must be evaluated with the same separation used above: cache capacity, bytes read, arithmetic, sparse/index overhead, and end-to-end serving throughput. ## 10. Measurement Plan A useful kernel sweep is: ```text GPU: H100/H800, B200 KV format: BF16, FP8 batch B: 1, 4, 16, 64, 128 query T: 1, 2, 4 KV length L: 256, 1K, 4K, 16K, 64K, 128K local heads: 128, 64, 32 attention: dense, DSA top-k ``` Record at least: - end-to-end kernel time, split kernel time, and combine time; - achieved TFLOP/s and effective HBM GB/s; - `num_splits`, active CTAs, and occupancy; - Tensor Core utilization and DRAM throughput; - L2 hit rate and page/block-table overhead; - CUDA Core time attributable to dequantization and softmax; - kernel-only results separately from the full MLA layer and full decode step. The primary hypotheses are: 1. small-batch, short-sequence cases are utilization- and overhead-bound; 2. $T=1$ dense BF16 MLA sits near the Hopper/Blackwell ridge; 3. $T=2$ moves dense MLA decisively toward compute-bound; 4. reducing local heads with TP makes MLA increasingly bandwidth-bound; 5. FP8 cache moves Hopper from HBM-bound toward dequantization-bound; 6. B200 gains will depend more on an SM100-specific decode schedule and native low-precision path than on nominal peak hardware ratios. These are analytical predictions. They should remain labeled inferred until a reproducible GPU sweep is preserved under the later experiments phase.