What are embeddings in AI? Vectors for meaning, explained
Embeddings turn text into vectors of numbers so that similar meanings sit close together. How they are made and compared, with a runnable Python demo.

An embedding is a list of numbers (a vector) that represents a piece of text, an image or another object so that things with similar meaning get vectors that point in similar directions. You compare two embeddings with cosine similarity or a dot product, and the score tells you how related they are. Embeddings power semantic search, RAG, recommendations, clustering and deduplication, and they’re also the first layer inside every LLM.
How do embeddings represent meaning?
The idea is older than LLMs: words that appear in similar contexts (“invoice” and “bill” both show up near “pay” and “due”) should get similar vectors.
Mikolov and colleagues at Google made that practical with word2vec in January 2013. Their paper, Efficient Estimation of Word Representations in Vector Space, proposed two simple architectures and reported that high-quality word vectors could be learned from a 1.6 billion word dataset in “less than a day”. Each word got one fixed vector.
Two things changed after that. Models moved from words to whole sentences and passages, and the vectors became contextual: the same word gets a different representation depending on its neighbors. Reimers and Gurevych’s Sentence-BERT (August 2019) is the paper that made sentence embeddings practical for search. Finding the most similar pair among 10,000 sentences with plain BERT, which has to read each pair together, needed about 50 million inference computations, roughly 65 hours; SBERT embeds each sentence once and cuts the search to about 5 seconds with cosine similarity. That trade, encode once and compare cheaply, is the whole reason embedding search scales.
What does an embedding look like in code?
You can build a crude embedding with no model at all. This script counts which words appear within two positions of each other in a tiny corpus, weights the counts with positive pointwise mutual information (PPMI), and uses each word’s row of weights as its vector. Then it ranks neighbors by cosine similarity.
# embeddings.py: build word vectors from co-occurrence counts, then find
# neighbors by cosine similarity. Python 3, stdlib only, no model download.
import math
from collections import Counter, defaultdict
sentences = """the invoice is due on friday . pay the invoice by bank transfer .
the bill is due on monday . pay the bill by card .
the receipt shows the amount paid . the receipt shows the card used .
the cat sleeps on the sofa . the dog sleeps on the rug .
the dog chases the cat . the cat chases the mouse .""".replace(".", "").split("\n")
pairs, word_n = Counter(), Counter()
for s in sentences: # count neighbors within 2 words
w = s.split()
for i, a in enumerate(w):
word_n[a] += 1
for b in w[max(0, i - 2):i] + w[i + 1:i + 3]:
pairs[a, b] += 1
vocab = sorted(word_n)
total = sum(pairs.values())
row = defaultdict(int); col = defaultdict(int)
for (a, b), c in pairs.items():
row[a] += c; col[b] += c
def vector(word): # positive PMI, one dimension per vocab word
return [max(0.0, math.log(pairs[word, c] * total / (row[word] * col[c])))
if pairs[word, c] else 0.0 for c in vocab]
def cosine(u, v):
return sum(a * b for a, b in zip(u, v)) / (math.hypot(*u) * math.hypot(*v))
print(f"{len(vocab)} words, so each vector has {len(vocab)} dimensions")
for word in ("invoice", "cat", "friday"):
sims = sorted(((cosine(vector(word), vector(o)), o) for o in vocab
if o != word and o not in {"the", "on", "by", "is"}), reverse=True)
print(f"{word:>8} -> " + ", ".join(f"{o} {s:.2f}" for s, o in sims[:3]))
Output (Python 3.13.11):
25 words, so each vector has 25 dimensions
invoice -> transfer 0.77, bill 0.63, monday 0.48
cat -> mouse 0.74, dog 0.55, sleeps 0.34
friday -> monday 1.00, bill 0.52, invoice 0.48
Nobody told the script that Friday and Monday are days, or that an invoice and a bill are both things you pay. It found that from context alone: “friday” and “monday” appear in identical surroundings, so their vectors match exactly. The billing words cluster together and the animal words cluster together.
The output also shows the failure modes in miniature. “invoice” is closer to “transfer” than to “bill” because, in this corpus, “bank transfer” happens to sit right after it. A model learns whatever correlations its data contains, including accidental ones. Real embedding models fix this with billions of sentences of training data and a neural network instead of a count table, and they compress the meaning into a fixed number of dense dimensions (hundreds to a few thousand) instead of one sparse dimension per vocabulary word.
How do you compare two embeddings?
Cosine similarity: the dot product of two vectors divided by the product of their lengths. It ranges from -1 to 1 and ignores vector length, so only direction counts. If the vectors are already normalized to length 1, cosine similarity and the dot product are the same number, and ranking by Euclidean distance gives the same order. OpenAI’s embeddings guide recommends cosine similarity and notes that its embeddings come normalized to length 1, so a plain dot product works. The details and the edge cases get their own post: cosine similarity vs dot product.
How big is an embedding?
It depends on the model. OpenAI’s guide lists text-embedding-3-small at 1,536 dimensions and text-embedding-3-large at 3,072 by default, each accepting up to 8,192 tokens of input. On the MTEB benchmark the same guide reports 62.3% for 3-small, 64.6% for 3-large and 61.0% for the older text-embedding-ada-002.
Dimension count is a storage and latency decision as much as a quality one. A million 1,536-dimension vectors stored as 32-bit floats is about 6 GB before any index overhead. The v3 OpenAI models let you shorten embeddings with a dimensions parameter, and the guide says a 3-large embedding cut to 256 dimensions still beats an unshortened ada-002 embedding at 1,536 on MTEB. Shortening works when a model is trained to pack the coarse meaning into the leading dimensions. Matryoshka Representation Learning (Kusupati et al., May 2022) is the paper behind that idea, which I explain in Matryoshka embeddings.
Embeddings in an LLM vs an embedding model: what’s the difference?
Every LLM starts with an embedding table: one learned vector per token in its vocabulary, looked up before the first transformer layer. Those vectors are an internal input format, tuned for predicting the next token.
An embedding model is a separate model trained to output one vector for a whole passage, with a training objective that pulls related passages together and pushes unrelated ones apart. That’s what you call when you index documents for search. Don’t average an LLM’s token embeddings and expect a good search vector; use a model trained for retrieval and check it on your own data, which is what choosing an embedding model covers.
Where do embeddings break?
- Similar is not relevant. “How do I cancel my order” and “How do I place my order” are close in embedding space and opposite in intent. Exact identifiers (SKUs, error codes, names) are also weak spots. That’s why production retrieval pairs vectors with keyword search in hybrid RAG.
- Model lock-in. Vectors from two different models live in different spaces. Change the model and you re-embed the whole corpus.
- Truncation. Text beyond the model’s input limit is silently ignored or rejected, so chunking decides what gets represented at all.
- Domain drift. A general-purpose model may not separate the terms that matter in your field. Measure retrieval quality on real queries before trusting the default.
What I would do
Treat embeddings as an index you’ll rebuild, not a one-time migration. Store the model name, version and dimension next to every vector, and keep the source text and chunking parameters so you can regenerate everything when you switch models.
From a data-platform seat, the line I’d circle is lineage. A vector without a record of which text and which model produced it is unusable for debugging and a risk for deletion requests: if a user asks to be forgotten, you need to find every vector derived from their data. That was routine work for documents and tables; it applies to embeddings too.
Then evaluate. Take fifty real queries, label the passages that should come back, and compare two or three models plus a keyword baseline. The cheapest model that wins on your data beats the leaderboard winner that doesn’t. Once the vectors are good, where to put them is a separate decision, covered in what a vector database is.
FAQ
What is an embedding in simple terms?
A list of numbers that stands in for a piece of content, arranged so that similar content gets similar numbers. It lets a computer measure “how alike are these two things” with arithmetic.
Are embeddings the same as tokens?
No. A token is a chunk of text with an integer ID; an embedding is a vector of floating point numbers. Inside an LLM, each token ID is looked up in an embedding table to get its vector. I covered tokens in what tokens are.
How many dimensions should an embedding have?
Enough to separate the meanings your application cares about, and no more than your storage and latency budget allows. Common sizes run from a few hundred to a few thousand. With models that support shortening, test smaller sizes on your own queries before paying for the full width.
Can I use embeddings for images?
Yes. OpenAI’s CLIP (February 2021) learned a shared space for images and text by training on 400 million image and caption pairs, so a text query can retrieve a photo. The comparison math is the same.
Sources
- Mikolov, Chen, Corrado and Dean, Efficient Estimation of Word Representations in Vector Space, January 16, 2013
- Reimers and Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, August 27, 2019
- Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP), February 26, 2021
- Kusupati et al., Matryoshka Representation Learning, May 26, 2022
- OpenAI, Vector embeddings guide, as of October 31, 2026