# Partial And Inverse RoPE In CSA/HCA Status: paper- and code-backed derivation; cross-checked on 2026-07-19 This note expands one easily missed part of DeepSeek-V4 CSA/HCA: it uses **partial RoPE**, and then applies **inverse RoPE to the attention output**. These are two separate operations: - partial RoPE restricts positional rotation to 64 of the 512 head dimensions; - inverse RoPE changes the rotated value output from an absolute-position coordinate system into one relative to the current query. Primary sources are the [DeepSeek-V4 report, Section 2.3.3](https://arxiv.org/html/2606.19348v1) and the public [DeepSeek-V4-Flash inference reference](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/inference/model.py). The reference code disambiguates the final operation with `apply_rotary_emb(..., inverse=True)`. For production-serving evidence, this note also cross-checks the [vLLM DeepSeek-V4 engineering post](https://vllm-project.github.io/2026/04/24/deepseek-v4.html) and SGLang's [DeepSeek-V4 model path](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/deepseek_v4.py). This note assumes the shared-KV MQA and candidate construction defined in [CSA/HCA definition](./01-csa-hca-definition.md). ## 1. What Is Partial RoPE Here? The V4 main-attention head dimension is $c=512$, split into $$ c=c_N+c_R=448+64. $$ For a query at original-token position $t$ and a candidate memory entry $j$, write $$ q_t=[q_t^N;q_t^R], \qquad C_j=[C_j^N;C_j^R]. $$ The superscripts mean: - $N$: the 448-dimensional non-RoPE content subspace; - $R$: the 64-dimensional RoPE subspace. Let $R_p$ denote the block-diagonal 2D rotations that RoPE applies at position $p$. V4 sends the following vectors into core attention: $$ \widetilde q_t =[q_t^N;R_tq_t^R], $$ $$ \widetilde C_j =[C_j^N;R_{p_j}C_j^R], $$ where $p_j$ is the exact token position for a local entry or the representative position assigned to a compressed entry. Only the final 64 dimensions rotate. This is why the mechanism is correctly described as partial RoPE rather than full-head RoPE. ## 2. The Attention Score Already Uses Relative Position Ignoring normalization and scale, one attention logit is $$ \begin{aligned} z_{t,j} &=\widetilde q_t^T\widetilde C_j\\ &=(q_t^N)^TC_j^N +(R_tq_t^R)^T(R_{p_j}C_j^R). \end{aligned} $$ RoPE rotations satisfy $$ R_t^TR_{p_j}=R_{p_j-t}, $$ so $$ z_{t,j} =(q_t^N)^TC_j^N +(q_t^R)^TR_{p_j-t}C_j^R. $$ The two subspaces therefore have different roles: ```text 448 NoPE dimensions content matching without an explicit RoPE phase 64 RoPE dimensions content matching modulated by relative position p_j - t ``` The score side needs no further correction: ordinary RoPE algebra has already converted the two absolute rotations into a relative displacement. ## 3. Shared K=V Creates A Value-Side Problem In conventional attention, it is common to rotate query and key but leave value unrotated: $$ K_j=R_{p_j}k_j, \qquad V_j=v_j. $$ V4 CSA/HCA instead uses one shared memory vector as both key and value: $$ K_j=V_j=C_j. $$ This shared-KV MQA representation is central to its small cache and reuse across query heads. Once the last 64 dimensions of the cached entry are rotated for key matching, those same rotated dimensions are also consumed as the value. For attention weights $a_{t,j}$, the raw core-attention output is $$ o_t =\left[ \sum_j a_{t,j}C_j^N; \sum_j a_{t,j}R_{p_j}C_j^R \right]. $$ The first component is an ordinary weighted content sum. The second is a sum of values expressed in absolute-position rotation frames. Attention logits are relative-position-aware, but this output is not yet expressed relative to the query. The attention sink does not change the derivation: it adds denominator mass with a zero value contribution. ## 4. Inverse RoPE Converts The Output To Query-Relative Coordinates V4 applies the inverse query-position rotation to the last 64 output dimensions: $$ \bar o_t^R=R_t^{-1}o_t^R. $$ Substituting the raw value sum gives $$ \begin{aligned} \bar o_t^R &=R_t^{-1}\sum_j a_{t,j}R_{p_j}C_j^R\\ &=\sum_j a_{t,j}R_{p_j-t}C_j^R. \end{aligned} $$ The complete output after inverse RoPE is therefore $$ \bar o_t =\left[ \sum_j a_{t,j}C_j^N; \sum_j a_{t,j}R_{p_j-t}C_j^R \right]. $$ Inverse RoPE does **not** erase positional information. It replaces the absolute phase $p_j$ carried by a rotated shared value with the relative phase $p_j-t$. A useful mental model is: ```text partial RoPE on Q and shared K=V -> relative-position attention scores -> absolute-position phase remains inside the value sum inverse RoPE on the output at query position t -> value contribution is also represented relative to t ``` ### Is the output numerically invalid without inverse RoPE? No. The model can still execute and could in principle be trained with the raw rotated value output. The issue is a coordinate-frame mismatch, not NaN, overflow, or an undefined attention operation: ```text without inverse RoPE attention score: depends on relative position p_j - t returned value: carries absolute-position phase p_j with inverse RoPE attention score: depends on relative position p_j - t returned value: also carries relative-position phase p_j - t ``` Without the correction, two identical content-and-relative-position patterns shifted to different absolute positions produce output vectors related by an extra rotation $R_\Delta$. A fixed grouped output projection and the residual stream then see different feature directions for otherwise equivalent local geometry. A sufficiently trained model might compensate for some of this behavior, but it would no longer preserve the relative-coordinate property that motivates RoPE here. ## 5. Two-Dimensional Example Consider one 2D RoPE pair. Let the query be at $t=10$, the memory entry at $p=6$, and the per-position angle be $\theta$. Before the dot product: $$ q\rightarrow R_{10\theta}q, \qquad C\rightarrow R_{6\theta}C. $$ The logit depends on $$ R_{10\theta}^TR_{6\theta} =R_{-4\theta}, $$ so it sees that the memory is four positions before the query. The raw value contribution is still $R_{6\theta}C$. Applying inverse RoPE at the query position gives $$ R_{-10\theta}R_{6\theta}C =R_{-4\theta}C. $$ The score and returned value now use the same relative displacement. ## 6. Global-Shift Sanity Check Shift both query and memory positions by the same $\Delta$: $$ t'=t+\Delta, \qquad p_j'=p_j+\Delta. $$ The attention score is unchanged because $$ p_j'-t'=p_j-t. $$ Without output inverse RoPE, the rotated value sum would acquire an extra global rotation $R_\Delta$. With inverse RoPE, $$ R_{-(t+\Delta)}R_{p_j+\Delta} =R_{p_j-t}, $$ so the final representation is invariant to this common shift. This is the cleanest algebraic reason to view the inverse operation as a change from an absolute to a query-relative coordinate frame. ## 7. Positions Of Local And Compressed Entries Compressed memory does not have one exact original-token position, so V4 must assign a representative RoPE anchor. For local-window entries, the position is exact: $$ p_j=j. $$ For completed compressed entry $s$, the public reference uses $$ p_s^{CSA}=ms, \qquad p_s^{HCA}=m's. $$ With the published ratios: $$ p_s^{CSA}=4s, \qquad p_s^{HCA}=128s. $$ For example, CSA entry 1 can pool the previous block at positions 0--3 and the newest block at positions 4--7, but its external RoPE anchor is $p_1=4$. HCA entry 1 represents positions 128--255 and is anchored at $p_1=128$. This creates two levels of positional representation: 1. feature-wise compression gates and learned positional biases distinguish offsets inside the pooling window; 2. partial/inverse RoPE expresses the compressed entry's coarse anchor relative to the current query. The anchor cannot reconstruct every original position inside a compressed entry. It is a coarse external coordinate attached to a learned memory object, not a claim that all pooled tokens occurred at the block start. ## 8. Why Not Keep Separate Rotated K And Unrotated V? An alternative design could store or construct $$ K_j=[C_j^N;R_{p_j}C_j^R], \qquad V_j=[C_j^N;C_j^R]. $$ That would look more like conventional RoPE attention, but it weakens the literal shared-KV contract. A system would need to do at least one of: - store distinct key and value representations; - retain unrotated entries and rotate every selected key on read; - fuse position-dependent key construction into the sparse gather kernel. V4 instead stores one partially rotated memory representation, consumes the same vector as key and value, and applies one inverse rotation after value aggregation. The interpretation that this choice preserves cache and kernel simplicity is derived from the implementation; the report does not provide an ablation against the alternatives above. ## 9. Relationship To MLA's Decoupled RoPE MLA and V4 both reserve a smaller subspace for positional information, but the cache objects and repair mechanisms differ. | Mechanism | Content memory | Positional path | Value treatment | | --- | --- | --- | --- | | MLA | compressed latent reconstructed through absorbed projections | separate decoupled RoPE query/key component | values are not the same rotated positional key object | | V4 CSA/HCA | one learned compressed entry directly consumed as shared K=V | last 64 of 512 entry dimensions use partial RoPE | rotated shared values are inverse-rotated after aggregation | The reusable distinction is: ```text MLA decouples a positional key component from its compressed content latent. V4 keeps one shared K=V entry, partially rotates it, and repairs the output coordinate frame with inverse RoPE. ``` The relationship is conceptual rather than an identical inherited operator. ## 10. Precision And Systems Consequences Partial RoPE aligns with V4's mixed cache format: ```text 448 non-RoPE dimensions: FP8 payload 64 RoPE dimensions: BF16 payload ``` The positional subspace keeps higher precision while most of the 512-dimensional logical entry receives the FP8 capacity and bandwidth saving. The public Python reference performs the complex rotation through FP32 temporaries and copies back to the activation dtype; this describes the reference path, not a published production-kernel accumulation contract. Inverse RoPE itself is small. For V4-Flash, it rotates $$ 64\ \text{heads}\times64\ \text{dimensions} =4096\ \text{scalar dimensions} $$ per query token. Treating every pair as one complex multiply gives only about 12K elementary multiply/add FLOPs, versus approximately 83.9M QK/AV FLOPs for one saturated CSA main-attention layer. Its importance is representational, not its arithmetic cost. ### Where the inverse rotation runs The public reference places inverse RoPE outside its custom `sparse_attn` call, after AV aggregation and before grouped output projection: ```text partial RoPE on query and newly cached KV -> sparse_attn: QK, softmax, AV -> inverse RoPE on output[..., -64:] -> reshape output into groups -> first grouped output projection -> final output projection -> residual stream ``` This is an implementation fact about the public correctness-oriented reference, not evidence that the production kernel launches a separate inverse-RoPE kernel. The public `sparse_attn` interface receives no position or frequency argument, so it cannot perform the inverse internally. An optimized implementation has two natural fusion points: 1. **Sparse-attention epilogue.** After the final softmax/AV reduction, rotate the final accumulator's last 64 dimensions while it is still in registers, then store it once. This is the most direct implementation. 2. **Grouped-output-projection prologue.** Rotate the incoming last 64 dimensions immediately before the first grouped projection. This avoids a standalone launch but requires a position-dependent input transform. The two serving engines provide a useful concrete comparison as of 2026-07-19: | Implementation | Observed output path | Interpretation | | --- | --- | --- | | DeepSeek public reference | sparse attention $\rightarrow$ inverse RoPE $\rightarrow$ grouped output projection | correctness-oriented, explicit operator boundary | | vLLM | attention $\rightarrow$ fused inverse RoPE + FP8 quantization $\rightarrow$ FP8 `o_lora` batched matmul | realizes the grouped-projection-prologue fusion point; vLLM reports roughly 2--3x speedup for this fused segment | | SGLang | attention backend $\rightarrow$ `fused_rope_inplace(..., inverse=True)` $\rightarrow$ optional FP8 quantization $\rightarrow$ `wo_a` | inverse RoPE is outside the attention backend in the current model path, using a dedicated fused rotation kernel | Both optimized serving paths therefore keep the **semantic** inverse-RoPE boundary after attention. They differ in physical fusion: vLLM documents fusion across inverse RoPE, activation quantization, and the first output projection, whereas the inspected SGLang path exposes inverse RoPE before its optional quantization and DeepGEMM call. These are current implementation choices, not architectural requirements; they can change as kernels evolve. The inverse rotation cannot be statically absorbed into output-projection weights because $R_t^{-1}$ changes with query position. If attention uses split-KV or multi-SM partial outputs, the clean placement is after the final normalized output merge. Linearity permits rotating every partial output before reduction, but that repeats work and can complicate deterministic accumulation order. Applying the rotation after AV is also what keeps it cheap. Moving it to the value side before attention would make every selected value query-dependent: $$ C_j^{(t),R}=R_t^{-1}R_{p_j}C_j^R. $$ That requires rotating roughly $(k+w)\times n_h\times64$ scalar dimensions for every query, rather than rotating only $n_h\times64$ output dimensions once. It would also weaken direct reuse of the cached shared K=V representation. At decode batch one, the 64 heads by 64 rotated output dimensions occupy an 8 KiB BF16 payload. A separate kernel therefore does little arithmetic but adds a launch and at least one extra read/write of that slice. At prefill, the same traffic scales with the number of query positions. Fusing the operation is primarily about avoiding launch and intermediate-memory traffic, not about reducing its roughly 12K FLOPs. Serving code also clarifies why one should not assign a single dtype to "attention." vLLM documents FP4 indexer cache and FP8 main-attention cache in deployment, while noting BF16 KV use during prefill and a partially token-wise-FP8 decode path. In the public FlashMLA FP8-cache contract, the FP8 payload is dequantized to BF16 and the main QK/AV attention calculation and output are BF16. FP8 is therefore primarily the persistent-cache and HBM transport format in that path, not the main-attention MMA operand precision. Online-softmax statistics and matrix accumulators normally retain higher precision inside the kernel, but their exact contract is backend-specific. SGLang exposes an FP8-E4M3 cache mode, but retains BF16 staging or fallback paths where cross-rank prefill communication or a backend requires it; its optional FP8 `wo_a` path quantizes the post-inverse output before the grouped projection. Thus the useful precision map is: ```text persistent index/main cache format != attention MMA and accumulation dtype != temporary prefill/staging dtype != output-projection activation/weight dtype ``` The exact answer depends on phase, backend, GPU generation, quantization configuration, and fallback selection. The paper's FP4/FP8/BF16 cache contract is the architectural target; vLLM and SGLang show how a serving system maps that contract onto real kernels. The larger systems costs remain: - loading the mixed-format selected or HCA compressed memory; - the CSA FP4 index scan and top-k; - wide 64-head, 512-dimensional main attention; - grouped output projection. ## 11. Compact Mental Model The shortest correct summary is: > V4 uses partial RoPE because only 64 of 512 shared-KV dimensions rotate. It > needs inverse RoPE because the same rotated entry is also the value. The > inverse output rotation does not remove position; it changes absolute memory > phase into position relative to the current query.