← All blog topics
Attention

Attention Mechanism

How Query, Key, and Value vectors let a model decide which tokens matter most for the current context.

Why attention matters in a sequence
Image 1: Attention lets a model focus more strongly on the relevant context.

Background

In sequence tasks like translation, summarization, or language understanding, not every word is equally important at every moment.

A model may need to focus strongly on one word while mostly ignoring another. Without attention, important information can be diluted, especially when the sequence is long.

Attention gives the model a way to dynamically decide which pieces of information matter most for the current token or current step. Instead of treating all inputs equally, it computes relevance scores, turns them into weights, and uses those weights to gather useful context.

Idea

Attention answers one simple question: what should I focus on right now?

The current token creates a query, which represents what it is looking for. Every token has a key, which represents what kind of information it can offer.

The query is compared with the keys to produce attention scores. These scores are turned into attention weights, and the weights are used to combine the values, which contain the information being passed forward.

The result is a context-aware representation. It does not just represent the token alone; it represents the token plus the most relevant surrounding information.

Query, key, value and attention weights
Image 2: Query compares with keys, scores become weights, and weights combine values.

Formula

\[s_{ij}=q_i\cdot k_j\]

A simple dot product measures how similar a query is to a key.

\[a_{ij}=\frac{q_i\cdot k_j}{\sqrt{d_k}}\]

Scaled dot-product attention divides by the key dimension scale to keep scores stable.

\[\alpha_{ij}=\frac{\exp(a_{ij})}{\sum_m \exp(a_{im})}\]

Softmax converts scaled scores into attention weights.

\[c_i=\sum_j \alpha_{ij}v_j\]

The final context vector is a weighted combination of value vectors.

\[\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V\]

The matrix form applies the same process to all tokens at once.

Symbols

  1. q_i: query vector for token i.
  2. k_j: key vector for token j.
  3. v_j: value vector for token j.
  4. s_ij: raw dot-product score from token i to token j.
  5. a_ij: scaled attention score from token i to token j.
  6. alpha_ij: normalized attention weight from token i to token j.
  7. c_i: output or context vector for token i.
  8. Q, K, V: matrices collecting all queries, keys, and values.
  9. d_k: key dimension used for scaling.
  10. softmax: turns raw scores into weights that sum to 1.

Example

Consider the sentence: The animal did not cross the street because it was tired.

The token it needs context. It should connect more strongly to animal than to less relevant words like the. Attention helps the model build this connection.

First, the token it produces a query vector. Every word in the sentence has a key vector and a value vector. The query for it is compared with the keys of all words, and more relevant words receive larger attention scores.

After softmax, the scores become attention weights. Since animal has a large weight, its information contributes strongly to the new context vector for it.

Example Attention Pattern

WordAttention weight for it
The0.03
animal0.45
did0.04
not0.05
cross0.06
street0.12
because0.06
it0.10
tired0.04
Attention mechanism workflow
Image 3: Input vectors become Q/K/V, scores become weights, and values combine into context.

Workflow

  1. Start with input representations, usually from an embedding layer.
  2. Create query, key, and value vectors for each token.
  3. Compare the query with each key, often using dot products.
  4. Apply softmax so raw scores become attention weights that sum to 1.
  5. Multiply each value vector by its attention weight.
  6. Add the weighted value vectors together to produce a context-aware output.
  7. Pass the result forward to the next layer or computation stage.

Pros

  1. Focuses on the most relevant information.
  2. Handles long-range relationships better than fixed-context methods.
  3. Produces context-aware representations.
  4. Flexible and widely useful in sequence modeling.

Cons

  1. Can be computationally expensive for long sequences.
  2. Attention weights do not equal true human understanding.
  3. Poor scoring can still focus on unhelpful information.
  4. The mechanism can look simple but still be hard to interpret fully.

Takeaway

Attention lets a model decide what matters most right now. It compares relevance, assigns weights, and combines information into a better context-aware representation.

The core idea is simple: score what matters, weight it, then gather the useful information.