kimi delta attn from first principles

Oct 10, 2026

+ linear attention, delta net, and gated delta net

some reference material:

https://arxiv.org/pdf/2510.26692

https://haileyschoelkopf.github.io/blog/2024/linear-attn/

The idea is to have a small memory matrix to continuously learn and forget information as it arrives. KDA builds upon linear attention, DeltaNet and gated DeltaNet and adds fine-grained gating mechanism

Let’s go from first principles.

Self attention

What’s wrong with regular softmax attention?

• in self attention the query is compared against ALL previous keys
• but this is quadratic - scales to $O(n^2)$
• the KV cache also grows at $O(N)$ with every new token
• this is not very efficient for fast inference and takes up a lot of memory especially for long context tasks
• it would be ideal if we can compress the KV cache to a fixed size memory and perform attention at linear time (instead of quadratic)

Linear Attention

To linearize attention, we can replace softmax with a similarity function

• we can use the multiplicative associative property and rearrange the multiplication from $(QK)V$ to $Q(KV)$

• this makes it linear
• because changing the order of matrix multiplication means we never have to construct the $N\times N$ attention matrix

• this gives us $O(N^2d)$ - quadratic

• this gives us $O(Nd^2)$ - linear
• We can’t do $Q^\top K$ of course because then this gives us feature correlations instead of token scores

Why can’t we just reorder this in self attn?

• We can’t simply change this multiplicative order in softmax attention though because softmax is non-linear
• and in causal attention we can’t compute $K^\top V$ because then tokens can see all future tokens too
• so for causal linear attention we keep a running state of the linearized attention
• $S_t$ is introduced as a memory matrix that stores
• recurrent state

• this is way faster to compute than regular self attention
• does not require softmax

Associative memory

• $S_t$ is an associative memory between keys and values.
• associative memory is a mapping between two things (Alex → blue) etc.
• but if two key vectors are the same, this causes memory interference

So what’s wrong with Linear attention?

• the recurrent formula $S_t=S_{t-1}+k_tv_t^\top$ only learns to update the $S_t$ memory matrix with new information
• it does not remove old or outdated associations
• this brings us to the delta rule

DeltaNet

• delta net uses the delta rule to update the memory matrix $S_t$
• instead of continually updating KV pairs into memory, it first checks what the current memory predicts for a given key, and computes a delta between the prediction and the actual value.
• from memory with previous tokens, $S_{t-1}$, we find the predicted value from current key

• we also have the current value $v_t$, using this find the error

• now we update the memory using this error. $\beta_t$ is a learnable param and it controls how strongly the memory is updated

Online gradient descent on a reconstruction objective:

• so say we define the reconstruction loss as: (which is the mean square error)

• in gradient descent we differentiate the loss function wrt to $S$

• the update rule becomes:

• you can substitute the loss term ($\nabla_S\mathcal{L}_t$) to get

• which is equal to

• so the memory update rule can be interpreted as an online gradient descent on the reconstructed loss- during inference, the DeltaNet memory $S$ weights are dynamically updated for each token, called fast weights.

Gated DeltaNet

• so far delta net attention learns to update/overwrite memory, but it can’t forget redundant memory yet - gated deltanet solves this
• gated DeltaNet adds a scalar forget gate to the update rule

• where $\alpha_t$ is a form of weight decay on the fast weights

Problems with GDN

• the problem here is that GDN uses a single scalar $\alpha$ for the entire memory matrix of a head.
• which means that every row of the matrix (each row corresponds to a key feature vector) gets forced to the same forget gate
• some rows may contain memory that need to persist longer than the others - we need fine-grained gating
• fixing this was the motivation for kimi delta attention

KDA

• improves the delta rule with fine-grained gating
• it replaces the single scalar forget gate $\alpha$ with a vector of forget gates, one forget gate per row
• KDA applies $\operatorname{diag}(\alpha)$

Breaking down the KDA equation from the paper:

1. Each row of the memory matrix $S$ gets its own decay factor to forget memory
   $$\bar{S}_t=\operatorname{Diag}(\alpha_t)S_{t-1}$$
2. starting with the delta net update rule
   $$S_t=S_{t-1}+\beta_t k_t(v_t-S_{t-1}^\top k_t)^\top$$
   • can be expanded to
     $$S_t=S_{t-1}+\beta_t k_t v_t^\top-\beta_t k_t k_t^\top S_{t-1}$$
   • factor out $S_{t-1}$ using an identity matrix $I$
     $$S_t=(I-\beta_t k_t k_t^\top)S_{t-1}+\beta_t k_t v_t^\top$$
   • putting this together we get the KDA equation

KDA has three gates actually, forget gate $\alpha_t$, update gate $\beta_t$ and output gate $g_t$:

• while computing attention, just like how $Q$, $K$, $V$ matrices are projected from the input hidden state, $\alpha$ and $\beta$ gates are also projected from the input hidden state
• the $\alpha$ and $\beta$ weights are learned during the model’s pretraining

• to compute $\alpha$ projection we have an up projection and a down projection
• to produce the decay vector $\alpha$:
  • a low-rank projection is used
• and for the update vector $\beta$
  • we apply sigmoid (since $\beta$ is a scalar value) to normalize it to be between $[0,1]$
• only the memory matrix $S_t$ (the fast weights) are learnt during inference through online gd
  • this does not need backprop or an optimizer
• and then the resulting memory matrix is multiplied by the queries to get the output

• KDA also introduces an output gate
  • RMSNorm applied to $o_t=S_t^\top q_t$
  • then perform elementwise multiplication between output gate and the output $y_t$
  • the output gate is to control how much of the attention result is passed on to the next layers

What is low rank parameterization:

• the forget gate $\alpha_t$ and output gate $g_t$ both use low rank parameterization
• typically we’d have one large learnable matrix, but in this case if $W_\alpha$ was $2048\times1024$, that’d be ~2 million params to compute just for a forget gate
• to make it more efficient we can split that into two smaller matrices
• compress the input 1024 to 128 (low rank) using the down proj and expand to 2048 using up proj
• this uses far fewer parameters

KDA Architecture:

• To further improve the expressiveness of Kimi linear they add several changes to the attention architecture
• instead of computing $q=W_qx_t$, $k=W_kx_t$ and so on, they add:
  • short conv - to include nearby tokens
  • swish - for nonlinearity. without nonlinearity consecutive projections can collapse into a single projection
  • L2 norm - $QK$ vectors are normalized so that magnitudes don’t grow too large

• kimi linear uses hybrid attention with a 3:1 ratio - repeating 3 KDA layers and 1 MLA layer
• NoPE for MLA, so KDA is responsible for all positional encodings
• the paper says that global NoPE attention with dedicated position aware attention yields better long context performance