How Large Language Models Actually Work

How Large Language Models Actually Work

Large language models can write software, summarise contracts, answer questions and carry on conversations, but the mechanism at their core is surprisingly specific:

An LLM repeatedly predicts the next token that is likely to follow the tokens already in its context.

What makes modern models powerful is not a different objective at every task. It is the scale of the model, the amount and diversity of training data, the transformer architecture, and the ability to represent complicated patterns in language and other data.

This article follows a piece of text from the moment it enters a language model to the moment the model produces an answer.

SENA visual explainer: How Large Language Models Actually Work.

Step 1: text becomes tokens

A language model does not read text as words in the same way a person does.

The input is first processed by a tokenizer, which breaks text into units called tokens. A token might represent:

• a whole common word,
• part of a word,
• punctuation,
• whitespace,
• a short sequence of characters.

For example, a phrase such as:

Artificial intelligence is useful.

might be split into several token IDs. The exact boundaries depend on the tokenizer used by that model.

Tokenization matters because context windows, generation limits and usage are measured in tokens rather than ordinary word counts.

Step 2: token IDs become vectors

Token IDs are discrete numbers. A transformer cannot learn useful linguistic relationships from arbitrary IDs alone.

Each token is therefore mapped to an embedding: a vector of learned numbers.

Instead of representing a token as something like token 4281, the model represents it as a point in a high-dimensional numerical space.

Those vectors can encode patterns associated with syntax, meaning, style and usage. The representation is not a human-readable dictionary definition. It is a learned numerical representation that becomes useful because of how the model was trained.

The model also needs information about position. Without positional information, the same collection of token embeddings would not distinguish between different word orders.

Modern architectures incorporate position in different ways, but the goal is the same: make sequence order available to the model.

Step 3: transformer layers build contextual representations

The embeddings pass through a stack of transformer layers.

A transformer layer typically contains two major computational components:

1. an attention mechanism,
2. a feed-forward or multilayer perceptron block.

There are also residual connections and normalisation operations that help information flow through deep networks.

Each layer transforms the representation of every token.

At an early layer, a token representation may primarily reflect local lexical patterns. At later layers, it can incorporate information from a much broader context.

The important idea is that a token's representation is no longer fixed after embedding. It becomes contextual.

The word "bank" in "river bank" and "bank account" starts with the same token embedding, but the surrounding context allows later transformer layers to build different internal representations.

Step 4: self-attention decides what context matters

Self-attention allows each token position to gather information from other relevant positions.

For each token representation, the model derives three vectors commonly called:

• query,
• key,
• value.

The query from one position is compared with keys from other positions. Those comparisons produce attention weights. The weights determine how strongly the values from other positions contribute to the updated representation.

This lets the model dynamically focus on different parts of the context.

Consider:

The trophy did not fit inside the suitcase because it was too large.

To reason about what "it" refers to, the model needs to relate that token to earlier context. Attention provides a mechanism through which such relationships can be represented.

Importantly, attention is learned rather than manually programmed. No engineer writes a rule saying that this specific token should look at that specific earlier token.

Step 5: multiple attention heads learn different relationships

Transformers generally use multi-head attention.

Instead of calculating one attention pattern, the model calculates several in parallel. Different heads can learn to represent different kinds of relationships.

A head might become useful for local syntax. Another might capture longer-range dependencies. Others may support patterns that do not have a simple human-language label.

The outputs of these heads are combined and transformed before being passed onward.

It is tempting to imagine that each head has one clean job, but real learned representations are distributed and messy. Interpretability research can reveal useful patterns, but it is safer to think of attention heads as parallel learned channels rather than individually named reasoning modules.

Step 6: feed-forward networks transform each position

Attention moves information between token positions.

The feed-forward network then performs a learned nonlinear transformation on each position's representation.

These blocks contain many of the model's parameters. Across many transformer layers, the repeated combination of attention and nonlinear transformation allows the network to represent increasingly complex patterns.

This process continues through the model's full stack.

Step 7: the model produces logits for the next token

After processing the current context, the model needs to decide what comes next.

The final hidden representation is projected into a set of scores, one for every token in the model's vocabulary.

These scores are called logits.

A softmax transformation converts the logits into a probability distribution.

Simplified example:

The model or serving system then chooses a token according to its decoding strategy.

It may always choose a highly probable token, or it may sample from the distribution. Settings such as temperature can change how concentrated or varied that sampling is.

Step 8: generation repeats one token at a time

After a token is selected, it is appended to the sequence.

The model then predicts the next token again.

Generation is therefore autoregressive:

context → next token → updated context → next token → updated context ...

A polished paragraph may look as though it appeared all at once, but under the hood it was generated incrementally.

This also explains why early generation choices can influence everything that follows. Once a token becomes part of the context, later predictions condition on it.

How does the model learn all of this?

During pretraining, the model sees enormous numbers of token sequences and learns to predict missing or next tokens, depending on the training setup.

A simplified training example might be:

Input:

The capital of Japan is

Target:

Tokyo

The model makes a prediction, compares its predicted distribution with the correct target token and calculates a loss. Backpropagation determines how each model parameter contributed to the error, and an optimiser updates those parameters slightly.

This happens repeatedly across many examples.

No individual training example teaches the model a complete grammar rule, coding pattern or world model. These capabilities emerge from the accumulation of parameter updates across very large datasets.

Pretraining is not the same as instruction following

A pretrained language model learns broad statistical patterns but is not automatically a good assistant.

Additional training stages can make the model more useful in interaction.

These may include:

• supervised instruction tuning,
• preference-based optimisation,
• reinforcement-style methods,
• safety and policy training.

The exact recipes vary between organisations and model generations.

The result is a model that is still fundamentally predicting tokens, but whose probability distribution has been shaped to produce more useful forms of behaviour when given instructions.

Where is the model's "knowledge"?

An LLM does not contain a conventional database of facts that it queries word-for-word.

Information learned during training is distributed across model parameters and activations.

That is why retrieval can be imperfect. The model may reproduce a fact accurately, combine concepts correctly, approximate an answer or generate something plausible but false.

The model's parameters represent statistical structure, not guaranteed records.

For information that must be current, private or precisely attributable, an external retrieval system is often more appropriate.

Why do LLMs hallucinate?

The model is trained to produce likely continuations, not to execute a built-in database verification step before every statement.

When the context does not contain enough reliable information, the model can still generate a fluent continuation.

This is one reason hallucinations can sound convincing.

Reducing hallucinations usually requires more than telling the model to "be accurate." Useful techniques can include:

• retrieval-augmented generation,
• tool use,
• constrained outputs,
• better prompts and context,
• verification steps,
• domain-specific evaluation.

What is a context window?

The context window is the amount of tokenised information the model can consider during a request.

The context can contain:

• user instructions,
• previous conversation turns,
• retrieved documents,
• tool results,
• source code,
• system instructions.

A larger context window allows more information to be supplied, but it does not guarantee that every token will be used equally well.

Long-context performance depends on model quality, document structure, retrieval strategy and the task itself.

What happens during inference?

Inference is the process of running a trained model to generate output.

At inference time, the model parameters are generally fixed. The system performs transformer computations using the supplied context, produces logits, selects a token and repeats.

Inference performance is affected by:

• model size,
• hardware,
• prompt length,
• generated output length,
• batching,
• quantisation,
• serving architecture.

This is why two models with similar benchmark quality can have very different latency and cost profiles in production.

Why can next-token prediction produce reasoning-like behaviour?

Predicting the next token across broad datasets is a much richer task than predicting the next word in a small phrasebook.

To perform well, a large model benefits from representing patterns involving:

• syntax,
• facts,
• code structure,
• mathematical transformations,
• discourse,
• intent,
• common chains of reasoning.

As model capacity and training scale increase, some of these learned representations can support tasks that appear qualitatively different from simple autocomplete.

Still, it is important not to confuse useful reasoning behaviour with human cognition. The internal mechanism and training process are different.

A useful mental model

Think of an LLM as a very large learned function:

tokens in → contextual computation → probability distribution over next token

The sophistication lies in the contextual computation.

The model has billions of learned parameters arranged in a transformer network capable of converting long sequences into useful internal representations.

The visible output is produced one token at a time, but each token can be influenced by an enormous amount of learned structure.

Key takeaways

• LLMs process tokens, not raw words.
• Tokens are converted into learned vector representations.
• Transformer layers repeatedly combine attention and nonlinear transformations.
• Self-attention lets token representations incorporate relevant context.
• The model outputs logits that become probabilities over possible next tokens.
• Text generation proceeds autoregressively, one token at a time.
• Pretraining learns broad patterns; instruction tuning shapes assistant-like behaviour.
• Fluency does not guarantee factual correctness.
• External retrieval and tools are important when information must be current or verifiable.

Frequently asked questions

Does an LLM search the internet every time it answers?

No. A base language-model inference call operates on the context provided to it and its learned parameters. A product may separately give the model browsing or retrieval tools.

Does the model understand every word independently?

Not in a fixed dictionary sense. Token representations become contextual as they pass through transformer layers, so the representation of a token depends on surrounding tokens.

Is an LLM just autocomplete?

Next-token prediction is the core generation objective, but "autocomplete" can understate the complexity of the learned function. Large transformers learn representations that can support code, reasoning, translation, extraction and many other behaviours.

Why are outputs different across runs?

If decoding includes probabilistic sampling, different tokens can be selected from the model's probability distribution. Temperature and other decoding settings influence this variability.

What is the difference between training and inference?

Training changes the model's parameters using data and optimisation. Inference uses the trained parameters to process new inputs and generate outputs.