Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is a vector database? How vector search works

A vector database stores embeddings and finds the nearest ones to a query fast. How ANN indexes trade recall for speed, with a runnable Python IVF demo.

What is a vector database? How vector search works

A vector database stores embeddings (lists of numbers that represent text, images or other data) and answers one question fast: which stored vectors are closest to this query vector? It pairs that nearest-neighbor search with what a normal database gives you, such as metadata filters, updates, deletes and persistence. Pinecone, Weaviate, Qdrant and Chroma are built for it; pgvector adds the same capability to Postgres.

Why do vector databases exist?

Because embeddings turned search into geometry. An embedding model maps “how do I reset my password” and “steps to recover account access” to nearby points, even though the two sentences share no words. Finding relevant text becomes finding near points, and that needs a store that is good at “nearest”.

The timeline is short. Sentence-BERT (Reimers and Gurevych, EMNLP 2019) showed you could embed sentences once and compare them cheaply: finding the most similar pair among 10,000 sentences took about 65 hours with cross-encoding BERT and about 5 seconds with SBERT embeddings. Dense Passage Retrieval (Karpukhin et al., EMNLP 2020) then beat a Lucene BM25 system by 9 to 19 points of top-20 passage accuracy on open-domain QA. Once retrieval quality came from embeddings, teams needed somewhere to keep millions of them, and RAG in 2023 made that a default line in most AI architectures.

How does a vector database work?

Four parts, in the order a request touches them:

  1. Ingest. Your application chunks documents, calls an embedding model, and writes each vector with an ID and metadata (tenant, source, timestamp, permissions).
  2. Index. The database builds an approximate nearest neighbor (ANN) index over the vectors so it doesn’t have to compare the query with every row.
  3. Query. The application embeds the user’s question with the same model and asks for the top K nearest vectors, usually with a filter like tenant = 42.
  4. Return. The database returns IDs, scores and metadata, and the application fetches the text to put into the model’s context window.

“Nearest” means a distance or similarity function, usually cosine similarity, dot product or Euclidean distance. The model you embed with decides which one is right.

Exact search compares the query with every vector, which is linear in the size of the collection. ANN indexes look at a small fraction of the vectors and accept that they will sometimes miss a true neighbor. How often they miss is called recall: of the true top ten, what share did the index return?

The two index families you will meet everywhere:

  • Graph indexes, mainly HNSW: each vector links to its neighbors, and search walks the graph toward the query. Fast and high recall, memory hungry.
  • Partition indexes, such as IVF (inverted file): cluster the vectors with k-means into buckets, then search only the buckets whose centers are nearest the query. Cheaper to build and smaller, lower recall for the same speed. Often combined with product quantization, which compresses each vector; FAISS (Johnson, Douze and Jégou, 2017) is the reference implementation, and the paper reports building a k-NN graph over a billion vectors in under 12 hours on four GPUs.

What does a vector index look like in code?

Here is the core of a vector database in about 40 lines of standard-library Python: exact search, and an IVF index with a probes dial. It is the same idea as pgvector’s ivfflat index and its ivfflat.probes setting.

# vectordb.py: the core of a vector database in ~50 lines. Python 3 stdlib only.
# Exact search and an IVF index (k-means buckets, the idea behind pgvector's ivfflat).
import math, random

random.seed(11)
DIM, N, LISTS = 32, 5000, 50

def unit(v):
    n = math.sqrt(sum(x * x for x in v)); return [x / n for x in v]
def sim(a, b):
    return sum(x * y for x, y in zip(a, b))           # dot product of unit vectors = cosine

# Fake corpus: 40 topics, each document a noisy copy of its topic centre.
centres = [unit([random.gauss(0, 1) for _ in range(DIM)]) for _ in range(40)]
rows = []
for i in range(N):
    c = random.choice(centres)
    rows.append({"id": i, "vec": unit([x + random.gauss(0, 0.25) for x in c])})

def exact(q, k):
    return [r["id"] for r in sorted(rows, key=lambda r: -sim(q, r["vec"]))[:k]]

# Build the IVF index: a few rounds of k-means, then each row goes in its nearest bucket.
cent = [r["vec"] for r in random.sample(rows, LISTS)]
for _ in range(5):
    buckets = [[] for _ in range(LISTS)]
    for r in rows:
        buckets[max(range(LISTS), key=lambda c: sim(r["vec"], cent[c]))].append(r)
    cent = [unit([sum(xs) for xs in zip(*[r["vec"] for r in b])]) if b else cent[i]
            for i, b in enumerate(buckets)]

def ivf(q, k, probes):
    near = sorted(range(LISTS), key=lambda c: -sim(q, cent[c]))[:probes]
    pool = [r for c in near for r in buckets[c]]
    hits = sorted(pool, key=lambda r: -sim(q, r["vec"]))
    return [r["id"] for r in hits[:k]], len(pool)

queries = [unit([x + random.gauss(0, 0.3) for x in random.choice(centres)]) for _ in range(30)]
print(f"{N} vectors, {DIM} dims, {LISTS} IVF lists")
for probes in (1, 2, 4, 8, 50):
    rec = scanned = 0
    for q in queries:
        truth = exact(q, 10)
        got, n = ivf(q, 10, probes)
        rec += len(set(got) & set(truth)); scanned += n
    print(f"probes={probes:<3} recall@10 {rec / 300:.2f}  vectors scanned {scanned // 30:>5}")

Output (Python 3.9.6):

5000 vectors, 32 dims, 50 IVF lists
probes=1   recall@10 0.47  vectors scanned   106
probes=2   recall@10 0.59  vectors scanned   194
probes=4   recall@10 0.75  vectors scanned   403
probes=8   recall@10 0.88  vectors scanned   821
probes=50  recall@10 1.00  vectors scanned  5000

That table is every ANN index in miniature. Scanning one bucket (about 2% of the data) finds under half of the true top ten, because the query’s real neighbors straddle bucket boundaries. Eight probes reaches 88% recall while looking at 16% of the vectors. Probing every bucket is an exact search again. pgvector’s README describes the same knobs: ivfflat.probes defaults to 1, sqrt(lists) is the suggested starting point, and setting probes equal to lists gives exact search. The toy also shows why the defaults are only a start: recall depends on how your data clusters, so you measure it on your own vectors.

Vector database vs pgvector vs a search engine: what should I use?

There are three ways to get vector search, and the choice is mostly about what you already run.

  • A Postgres extension (pgvector): vectors are a column. You keep transactions, joins, backups and row-level security. Limits show up at very large index sizes and with heavily filtered queries.
  • A dedicated vector database (Qdrant, Weaviate, Pinecone, Chroma and others): built around vector search, with its own filtering, quantization, sharding and replication. You run and sync a second system.
  • A search engine with vector support (Elasticsearch and OpenSearch both offer approximate kNN and hybrid queries): strong when you also need keyword search, facets and analyzers, which is most real search products.

The filtering question decides more of these choices than raw speed does. Most ANN indexes find candidates first and filter second. pgvector’s docs give the example that with a filter matching 10% of rows and HNSW’s default ef_search of 40, a query returns about 4 rows on average. Qdrant’s indexing docs describe adding filter-aware edges to the HNSW graph to attack the same problem. Whatever you pick, test your most selective filter, not your average one.

Where do vector databases fall short?

Exact matches. Embeddings are good at meaning and bad at identifiers. Anthropic’s contextual retrieval post (September 2024) uses the example of a query for “Error code TS-999”: embedding search may return general content about error codes and miss the exact match. That is why production systems run hybrid search, BM25 plus vectors.

Generalization. The BEIR benchmark (NeurIPS 2021 Datasets and Benchmarks) tested retrievers on 18 datasets outside their training data and found BM25 a strong baseline, with many dense retrievers underperforming it out of domain. A vector database stores whatever the model gives it; it can’t fix a model that does not understand your domain.

Re-embedding. Vectors from different models are not comparable. Change the embedding model and you re-embed the whole corpus and rebuild the index. Plan for it on day one.

Silent recall loss. An ANN index never throws an error when it misses a document. You only find out by comparing against exact search.

What I would do

  1. Start with the database you already operate. For most teams that is Postgres plus pgvector, with exact search until latency forces an index.
  2. Store the embedding model name and version beside each vector, so a model change is a tracked migration rather than a mystery.
  3. Build a small labelled set of real queries and their correct documents before you choose anything. Measure recall@10 for every index setting and every model you try.
  4. Ship hybrid search from the start. Keyword search catches the product codes, names and error strings that vectors miss.
  5. Move to a dedicated vector database when you can name the constraint: index size beyond one machine, write volume, or filters your current store cannot serve.

FAQ

Do I need a vector database for RAG?

No. You need vector search, which pgvector, Elasticsearch, OpenSearch or a FAISS index in memory can all provide. A dedicated vector database is one option among several.

What is the difference between a vector database and a vector index?

An index such as HNSW or IVF is the data structure that makes nearest-neighbor search fast. A vector database wraps one or more indexes with storage, updates, deletes, metadata filtering, access control and replication.

How does a vector database search so fast?

It uses an approximate nearest neighbor index that only compares the query with a small part of the collection, such as a few k-means buckets (IVF) or a walk through a neighbor graph (HNSW). It trades a little recall for a large cut in work.

They find different things. Vector search handles paraphrase and synonyms; keyword search handles exact terms, codes and names. Most good retrieval systems combine both and fuse the rankings.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.