Self-Attention, Explained Visually

Self-Attention, Explained Visually

Self-attention is the mechanism that allows a transformer to decide which other parts of a sequence are relevant to the token it is currently representing.

A useful way to think about it is:

Each token asks a question, checks which other tokens are relevant to that question, and mixes information from them according to learned relevance scores.

Those three roles are represented by the famous query, key and value vectors.

SENA visual explainer: Self-Attention, Explained Visually.

Start with a sentence

Consider:

The robot picked up the box because it was blocking the door.

When the model processes the token corresponding to "it", the representation should somehow incorporate information about what "it" likely refers to.

A transformer does not use a hand-written grammar rule for this sentence.

Instead, self-attention gives the "it" position a way to compare itself with earlier positions such as "robot", "box" and "door" and pull in information from positions that the network has learned are useful.

Every token begins as a vector

Before attention happens, every token is represented by a vector.

We can picture the input sequence as a matrix:

X = [x1, x2, x3, ..., xn]

Each xi is the vector representing one token position.

The transformer then applies learned matrices to derive three new vectors from each token:

Q = XWq

K = XWk

V = XWv

Where:

• Q contains query vectors,
• K contains key vectors,
• V contains value vectors,
• Wq, Wk and Wv are learned parameter matrices.

The model learns these transformations during training.

Query: what am I looking for?

The query represents what the current position is seeking from the rest of the sequence.

For the token "it", the query may contain dimensions that help identify useful contextual relationships.

It is not literally an English-language question. It is a learned vector.

But conceptually, we can imagine:

Query("it") → Which earlier information is useful for representing this token here?

Key: what information do I advertise?

The key represents how a token can be matched by another token's query.

Each position exposes a key.

The query at "it" is compared against the keys for all allowed positions.

If the query and a particular key are strongly compatible, that position receives a higher attention score.

Value: what information should I contribute?

The value contains the information that will actually be mixed into the output representation.

This distinction is important:

• query and key determine how much attention a position receives,
• value determines what information flows from that position.

The final attention output is a weighted combination of value vectors.

Dot products measure query-key compatibility

The transformer calculates a similarity-like score between a query and each key.

For a query q and key k, the basic score is:

score = q · k

This dot product becomes large when the learned vectors point in compatible directions.

For every token, the model computes such scores against multiple keys.

We can imagine a simplified attention pattern for "it":

These numbers are illustrative rather than actual model values.

Why divide by the square root of dimension?

The standard scaled dot-product attention equation is:

Attention(Q,K,V) = softmax(QKᵀ / √dk)V

The division by √dk keeps dot products from growing too large as the key dimension increases.

Without scaling, softmax can become extremely peaked, which can make optimisation less stable.

It may look like an arbitrary mathematical detail, but it is part of the design that made transformer attention train effectively at scale.

Softmax turns scores into weights

Raw attention scores can be positive, negative or arbitrarily scaled.

Softmax transforms them into positive weights that sum to 1.

Suppose softmax produces:

The output representation for the current token becomes a weighted combination of the corresponding value vectors:

output = 0.12V(robot) + 0.03V(picked) + 0.63V(box) + ...

The position can therefore integrate information from many places instead of choosing a single hard reference.

Attention is contextual information routing

This is the most useful mental model.

Attention is not a database lookup and it is not a symbolic rule engine.

It is a learned information-routing operation.

At each layer, positions decide how strongly to mix information from other positions.

After many layers, a token representation can contain information influenced by long-range relationships across the sequence.

What does "self" mean in self-attention?

It is called self-attention because queries, keys and values all come from the same sequence representation.

In other architectures, attention can operate across different sequences. For example, one representation might attend to another source of information.

But in transformer language models, self-attention refers to positions within the model's current sequence attending to positions in that sequence.

Causal attention prevents looking into the future

An autoregressive language model generates the next token using only information that already exists in its context.

During training, however, an entire token sequence may be available in memory at once.

A causal mask prevents a position from attending to future tokens.

Visually, the attention matrix becomes triangular:

Token 1: can see 1
Token 2: can see 1,2
Token 3: can see 1,2,3
Token 4: can see 1,2,3,4

This preserves the next-token prediction setup.

The model cannot cheat by looking at the answer token to the right.

Why use multiple attention heads?

A single attention operation gives each position one learned way to compare queries and keys.

Transformers instead use multi-head attention.

The model projects representations into several query/key/value spaces and calculates attention separately in each.

Conceptually:

Head 1 → local grammatical relationships
Head 2 → long-range references
Head 3 → delimiter or formatting patterns
Head 4 → other learned dependencies
...

Real heads are not always this cleanly interpretable, but multiple heads let the network represent several relationships in parallel.

Their outputs are concatenated and transformed before moving to the next stage.

Attention is not the whole transformer

Because attention receives so much attention itself, it is easy to imagine that transformers are simply stacks of attention matrices.

They are not.

A transformer block also includes:

• feed-forward networks,
• residual connections,
• normalisation,
• learned projections,
• positional information.

The feed-forward blocks are especially important. Attention routes information between positions; the feed-forward networks transform the information locally at each position.

The power of the architecture comes from repeatedly alternating these operations across many layers.

How position enters the picture

Attention by itself does not inherently understand sequence order.

A sentence with the same tokens in a different order would otherwise be difficult to distinguish.

Transformers therefore incorporate positional information.

Different model families use different techniques, including absolute position embeddings and relative or rotary-style position representations.

The details differ, but the requirement is universal: the network must know something about where tokens occur, not just which tokens exist.

A small conceptual example

Take:

Alice gave Bob the report because he requested it.

When processing "he", one attention head might place significant weight on "Bob".

When processing "it", another pattern might place significant weight on "report".

In later layers, the network may combine those relationships with grammar, semantics and broader context.

Nothing guarantees that one specific head will permanently perform "pronoun resolution." The useful behaviour emerges from many distributed representations.

Why attention scales poorly with very long context

Standard attention compares every query position with every key position.

For n tokens, this creates an n × n attention matrix.

That means the basic attention computation grows roughly quadratically with sequence length.

Long-context systems therefore face memory and compute challenges.

Modern implementations use a range of engineering and architectural techniques to make long contexts practical, but the basic all-to-all relationship explains why context length has historically been expensive.

What attention maps can and cannot tell you

Researchers sometimes visualise attention weights to understand what a model is focusing on.

These visualisations can be informative, but they should not be treated as perfect explanations of the model's reasoning.

A model's output depends on:

• many heads,
• many layers,
• residual streams,
• feed-forward transformations,
• nonlinear interactions.

A single attention heat map is only one slice of a much larger computation.

Why self-attention changed language modelling

Earlier sequence models often processed information step by step.

Transformers made it possible to process many token positions in parallel during training while still learning flexible relationships across a sequence.

Self-attention also gives every position a relatively direct path to information from distant positions.

Those properties made the architecture highly scalable and helped transformers become the foundation of modern large language models.

Key takeaways

• Self-attention lets each token position incorporate information from other relevant positions.
• Queries represent what a position is looking for.
• Keys determine how strongly other positions match that query.
• Values contain the information that is actually mixed into the result.
• Dot products produce attention scores, and softmax converts them into weights.
• Causal masking prevents autoregressive models from attending to future tokens.
• Multi-head attention learns multiple relationship patterns in parallel.
• Attention is only one component of a transformer block.
• Attention visualisations are useful but not complete explanations of model behaviour.

Frequently asked questions

What is QKV in transformers?

QKV stands for query, key and value. They are learned projections of token representations used to calculate attention weights and route contextual information.

Does attention select one word?

Usually not. Softmax produces a distribution over multiple allowed positions, so a token can combine information from several places.

Why is self-attention better than simply averaging context?

Because the weights are dynamic. The model can assign different relevance to different tokens depending on the current query and context.

What is multi-head attention?

It performs several independent attention calculations using different learned projections, allowing the model to represent multiple relationships simultaneously.

Is self-attention the same as reasoning?

No. Self-attention is a computational mechanism for contextual information exchange. Reasoning-like behaviour emerges from the interaction of attention, feed-forward layers, learned parameters, training data and generation dynamics.