How Embeddings Turn Meaning Into Vectors
Computers are good at comparing numbers. Human language is not naturally numerical.
Embeddings bridge that gap.
An embedding is a learned vector representation that places items with related patterns closer together in a high-dimensional numerical space.
The item might be a word, sentence, document, product, image, user or piece of code.
Once those items are represented as vectors, systems can compare them mathematically. That makes embeddings useful for semantic search, recommendations, clustering, classification and retrieval-augmented generation.
SENA visual explainer: How Embeddings Turn Meaning Into Vectors.
A vector is simply a list of numbers
A small vector might look like:
[0.12, -0.44, 0.83, 0.05]
Real embedding vectors often contain hundreds or thousands of dimensions.
Each dimension does not usually map cleanly to a human concept such as "finance" or "positive sentiment". Meaning is distributed across many dimensions.
The useful property is not whether we can interpret each coordinate. It is whether the geometry of the space preserves useful relationships.
If two pieces of text have related meanings, a good embedding model should often produce vectors that are relatively similar.
From text to a point in vector space
Imagine three sentences:
How do I reset my password?I cannot log in because I forgot my password.How do I change the delivery address?
A keyword system may struggle if the wording differs.
An embedding model can map each sentence into a vector.
Conceptually:
password reset ●
forgot password ●
delivery address ●
The first two vectors may be close because the underlying intent is similar, even though the exact words are different.
This is the basis of semantic search.
Embeddings are learned, not manually designed
Engineers do not normally decide each vector coordinate by hand.
Embedding models learn useful representations during training.
The training objective encourages items that should be related to receive compatible vector representations and items that should be different to become easier to distinguish.
Different models use different objectives and training datasets, which means their embedding spaces can behave differently.
An embedding model optimised for text retrieval may be more useful for search than a generic representation trained for another purpose.
Similarity is geometry
Once we have vectors, we need a way to compare them.
One of the most common metrics is cosine similarity.
For vectors A and B:
cosine_similarity(A,B) = (A · B) / (||A|| ||B||)
Intuitively, cosine similarity compares the direction of two vectors rather than their absolute magnitude.
If two vectors point in similar directions, the score is high.
Other distance or similarity measures can also be used, including Euclidean distance and dot product.
The best choice depends on how the embedding model was trained and how the retrieval system is designed.
Semantic search step by step
Suppose SENA has thousands of articles and a reader searches:
AI infrastructure investment in Malaysia
A semantic search system can work like this:
- Convert every article or article chunk into an embedding.
- Store those embeddings in an index.
- Convert the reader's query into an embedding using the same model.
- Compare the query vector with stored vectors.
- Return the nearest or most similar results.
The search can retrieve an article about Malaysian data-centre expansion even if the article does not use the exact phrase "AI infrastructure investment".
The match comes from the learned semantic representation rather than only literal keyword overlap.
Why not use keyword search for everything?
Keyword search is still extremely valuable.
It is excellent when the exact term matters:
- model names,
- company names,
- product SKUs,
- regulations,
- acronyms,
- quoted phrases.
Embeddings are useful when meaning can be expressed in many ways.
In practice, strong retrieval systems often combine lexical search and vector search. This is called hybrid search.
The keyword system captures exact lexical matches while the embedding system captures semantic similarity.
Documents are often embedded in chunks
Embedding an entire long document into a single vector can lose detail.
A common approach is to split documents into smaller chunks.
For example, a long research report might be split by section, paragraph groups or token count.
Each chunk receives its own embedding.
When a query arrives, the system retrieves the most relevant chunks rather than the whole document.
This is particularly important in retrieval-augmented generation, because only a limited amount of retrieved material should be passed into the language model.
Chunk size is a design choice
Chunks that are too small can lose context.
Chunks that are too large can mix multiple topics and make retrieval less precise.
A good strategy depends on the data:
- FAQ entries may naturally be one chunk each.
- Legal documents may benefit from clause-level chunking.
- Technical guides may work well by section.
- Tables may require custom treatment.
- Source code may be split by function or class.
There is no universal token count that is optimal for every corpus.
Embeddings make recommendations possible
Embeddings are not limited to text search.
Imagine representing products as vectors based on their descriptions, categories and user interactions.
Items with similar embeddings can be recommended together.
You can also create user embeddings from behavioural data and compare users with items.
This general pattern appears in:
- content recommendations,
- product discovery,
- candidate matching,
- music and video recommendations,
- similarity detection.
The exact training setup differs, but the shared idea is to place comparable entities in a useful geometric space.
Embeddings can power clustering
Because related points tend to be near one another, vectors can be grouped with clustering algorithms.
A news organisation could cluster article embeddings to identify emerging story groups:
Cluster A → AI chips / GPU supply
Cluster B → enterprise agents
Cluster C → data-centre investment
Cluster D → AI regulation
This can support topic discovery, deduplication and trend analysis.
The labels still need interpretation, but embeddings make the underlying similarity structure easier to analyse.
Embeddings are a key part of RAG
In a typical retrieval-augmented generation pipeline:
- documents are chunked,
- chunks are embedded,
- vectors are indexed,
- a user query is embedded,
- relevant chunks are retrieved,
- those chunks are passed to an LLM,
- the model answers using the retrieved context.
The embedding model does not generate the final response. It helps the system decide which information to show the generator.
Poor retrieval therefore limits the quality of the entire RAG system.
If the right evidence is never retrieved, the language model cannot reliably use it.
What is a vector database?
A vector database or vector-capable search engine stores embeddings and provides efficient nearest-neighbour search.
A naive system could compare a query with every stored vector, but that becomes expensive as the collection grows.
Vector indexes use specialised algorithms and data structures to find approximate nearest neighbours much faster.
Many modern databases and search platforms now support vector fields, so "vector database" describes a capability as much as a completely separate category of software.
Metadata is still essential
Embeddings should rarely be the only information stored.
For each vector, a production system commonly keeps metadata such as:
- document ID,
- URL,
- title,
- publication date,
- region,
- permissions,
- category,
- source,
- language.
Metadata allows filtering before or after vector search.
For example:
Find semantically relevant articles about AI investment, but only from Malaysia and only from the last 12 months.
This combination of semantic similarity and structured filtering is often more useful than pure vector ranking.
Similarity is not truth
Two vectors can be close because their content is semantically related.
That does not mean one statement verifies another.
An embedding model is not a fact-checking system.
This matters in RAG. Retrieval similarity identifies potentially relevant evidence, but systems may still need to evaluate:
- source authority,
- freshness,
- contradictions,
- permissions,
- factual support.
High similarity is a retrieval signal, not proof.
Embeddings can drift across models
Vectors created by one embedding model generally should not be compared directly with vectors produced by another model.
Different models learn different spaces and dimensions.
If an organisation changes embedding models, it often needs to re-embed the corpus.
This can be a meaningful operational cost for large collections.
Versioning your embedding model and indexing pipeline is therefore a good production practice.
Evaluation matters more than intuition
A retrieval system may look impressive in a few hand-picked examples and fail on real user questions.
Evaluate embeddings using representative queries.
Useful measurements can include:
- recall at k,
- precision at k,
- mean reciprocal rank,
- whether the correct source appears in the retrieved set,
- downstream answer quality.
The best embedding model is the one that performs well on your retrieval problem, not necessarily the one with the largest vector dimension or the most impressive generic benchmark.
Key takeaways
- Embeddings represent items as learned numerical vectors.
- Related items often appear closer together in embedding space.
- Cosine similarity is a common way to compare vectors.
- Semantic search retrieves based on meaning rather than exact wording.
- Long documents are typically embedded as multiple chunks.
- Hybrid search combines vector and keyword retrieval.
- Embeddings power recommendations, clustering and RAG.
- Vector similarity signals relevance, not factual truth.
- Production systems should evaluate retrieval on real queries and keep model versions explicit.
Frequently asked questions
Are embeddings the same as tokens?
No. Tokens are discrete units produced by tokenisation. Embeddings are continuous vectors used to represent tokens, sentences, documents or other items numerically.
Can I read the meaning of each embedding dimension?
Usually not in a simple one-dimension-one-concept way. Useful information is distributed across the vector.
What is cosine similarity?
It measures the angle between two vectors. Vectors pointing in similar directions receive higher similarity.
Do I need a dedicated vector database?
Not necessarily. Many search engines and general databases support vector indexing. The right choice depends on scale, filtering, latency, operations and existing infrastructure.
Why combine keyword and vector search?
Keywords are excellent for exact names and phrases, while embeddings capture paraphrases and semantic similarity. Combining them can improve retrieval robustness.