Attention as Learned Routing

This is an AI-generated placeholder article.

Attention is often introduced as a metaphor: a model “looks at” the relevant tokens. A more useful engineering picture is learned routing. Each token produces a query, a key, and a value; the query-key match determines where information flows.

For a sequence represented by $X$, one attention head computes

The matrix inside the softmax is a table of routing scores. Dividing by $\sqrt{d_k}$ keeps those scores from growing too sharp as the key dimension increases.

Why this picture helps #

“Routing” makes three practical facts easier to remember:

1. the weights depend on the input;
2. the destination can be far away in the sequence; and
3. multiple heads can learn different routing policies.

That is not the whole story, but it is a sturdy first handle.