Attention

Attention mixes information across positions of a sequence. Each output position pulls a weighted combination of all other positions, with weights that depend on the content at each position rather than its index. For one query, $\operatorname{Attn}(q, K, V) = \sum_i \alpha_i(q)\, v_i$, with weights $\alpha_i(q) \propto \exp(q \cdot k_i / \tau)$.

1. Fixed and query-dependent weights

A sufficient statistic compresses data with fixed weights. Kernel regression makes the weights depend on a query and a chosen similarity. Attention makes the weights depend on a query and learned dot-product similarities.

2. Attention as soft dictionary lookup

The default reading is a content-addressed lookup. Each position projects its residual vector into three roles: a query $q = W^Q x$, a key $k = W^K x$, and a value $v = W^V x$. A query is scored against every key by dot product, the scores are normalized with a softmax, and the output is the resulting blend of values, $\operatorname{Attn}(q, K, V) = \sum_i \alpha_i(q)\, v_i$ with $\alpha_i(q) = \operatorname{softmax}_i(q \cdot k_i / \tau)$. A key advertises what its position offers, a value carries what that position contributes, and a query states what the current position is looking for.

A hard dictionary returns the one value whose key matches the query. Attention returns a weighted average of every value, each weighted by how well its key matches. This produces a differentiable lookup over an associative memory. The temperature $\tau$ sets how peaked the match is: small $\tau$ approaches a hard argmax, large $\tau$ averages over everything. The QK and OV circuits page separates the matching half (which positions a query reads) from the value-write half (what gets copied back).

3. Attention as Nadaraya–Watson kernel regression

The Nadaraya–Watson estimator $\hat f(q) = \sum_i K(q, x_i)\, y_i \;/\; \sum_j K(q, x_j)$ is a weighted average of $y$-values, with weights given by a kernel of distance from the query. Soft attention with a Gaussian-shaped softmax ($\alpha_i \propto \exp(-\|q - k_i\|^2/2\sigma^2)$, up to a query-only normalization) has the same normalized weighted-average form.

At small $\tau$ the kernel is sharp and the prediction at $q$ is essentially the $y$-value of the nearest data point. Attention puts all its mass on one token, the entropy of the weight distribution is near zero, and the smoothed curve becomes a jagged step function. At large $\tau$ the kernel spans the whole interval, the weight distribution is nearly uniform, and the prediction flattens toward the global mean. Intermediate values trade fidelity against stability, the usual bias-variance tradeoff in non-parametric regression.

4. Softmax is the unique entropy-regularized retrieval rule

Among all retrieval distributions $a$ over the memory items, softmax maximizes $$\mathbb{E}_a[s] \;+\; \tau\, H(a) \;=\; \sum_i a_i s_i \;-\; \tau \sum_i a_i \log a_i$$ subject to $\sum a_i = 1$. The objective trades score against entropy: higher expected score prefers high-score items, while higher entropy spreads mass across items. The temperature $\tau$ sets that tradeoff.

The temperature parameter controls how concentrated the retrieval distribution is.

• $\tau \to 0$ (argmax). The total objective collapses to $\max_i s_i$; the retrieval distribution puts all mass on the highest-scoring item; entropy is zero. The result is brittle but maximally relevant.
• $\tau \to \infty$ (uniform). The objective is dominated by the entropy term; the retrieval distribution is uniform; effective $k$ is $N$. The distribution is maximally spread, and scores don't matter.
• Intermediate $\tau$. The softmax interpolates. Multiple items contribute; the "effective $k$" (i.e., $\exp H(a)$) tells you roughly how many memory items you're actually pulling from.

5. Kernel variants

Changing the kernel changes how retrieval weight falls off with similarity or position. Dot-product attention, linear attention, local windows, and positional-bias schemes make different choices here. The slider sets the bandwidth / temperature; the curves show the resulting weight profile in each coordinate system.

Each kernel encodes a prior about which positions count as neighbors. The dot-product softmax permits global retrieval at $O(N^2)$ cost. Linear attention swaps the kernel for one with an explicit feature map $\phi$, which permits the rearrangement $\sum_i \phi(q)^\top \phi(k_i) v_i = \phi(q)^\top \big(\sum_i \phi(k_i) v_i^\top\big)$ so the bracketed sum can be accumulated once, bringing cost to $O(N)$. Local-window attention restricts the kernel to a sliding interval. Position-bias schemes like ALiBi and T5 relative bias add a position-only term $b_i$ to the scores before the softmax. That multiplies the kernel by a position prior $r_i = e^{b_i/\tau}$, giving $a_i \propto r_i \exp(s_i/\tau)$: the maximizer of the §4 objective with the entropy term replaced by $-\tau\,\mathrm{KL}(a \,\|\, r)$.

6. Each head computes its own query-dependent statistic

A multi-head layer computes several query-dependent statistics in parallel. Each head has its own learned $W_Q, W_K, W_V$, so one head can implement a previous-token pattern while another routes by syntax, delimiter position, or a copied token if training finds such a solution. The four patterns here are stylized examples, not measurements from a particular model.

In the stylized example, H1 implements a "look at the previous token" rule, a building block of induction circuits for in-context copying. H2 routes verbs to subjects and prepositions to head verbs. H3 is positional: weights decay with distance regardless of content. H4 attends to determiners and modifiers. Each head computes a different summary statistic of the prefix; their outputs concatenate and pass through the next layer's matrices, where they can be combined into higher-level statistics.

7. Encoder and decoder masks

An encoder such as a masked-language model can use bidirectional context: each token can attend left and right, and a special masked position asks for a missing word. A causal decoder can only attend left when predicting the next token. The same attention operation therefore supports different evidence patterns because the mask changes which keys are visible.

This difference matters for probe comparisons. A bidirectional encoder can represent a token using future context that a causal decoder has not seen. A masked-language-model objective also trains direct reconstruction of hidden tokens, while a causal-language-model objective trains next-token continuation. Layerwise syntax or binding readouts can therefore fragment differently across encoder, decoder, and encoder-decoder models even when the attention equation is the same.

8. Causal masking and the KV cache

During autoregressive generation, attention at step $t$ depends on the previous keys and values through $\sum_i \alpha_i(q_t) v_i$. An efficient decoder caches those $(k_i, v_i)$ pairs as it goes. The cache stores raw keys and values rather than an accumulated summary, because each new query reweights them differently.

The "broadcast / uniform" pattern is what an unconditional summary statistic would look like: the same weights regardless of the query, so the cache could be collapsed to one accumulated $\sum v_i$. The non-uniform patterns (subject-pointer, previous-token) reweight differently for each query, which is why a real KV cache stores the raw vectors. For memory growth, paging, and attention sinks, see KV cache memory.