Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

How to choose an embedding model: MTEB and your own evals

How to choose an embedding model: read MTEB the right way, weigh dimensions, context length and cost, then test on your own queries with a Python harness.

How to choose an embedding model: MTEB and your own evals

To choose an embedding model, shortlist three to five candidates from the retrieval columns of the MTEB leaderboard that fit your language, context length, license and budget, then run each on 50 to 100 of your own real queries with known answers and pick the one with the best recall at a cost you can carry. The leaderboard narrows the field; your own evaluation decides. A model that ranks first on average can easily rank third on your documents.

What does an embedding model actually decide?

Everything downstream of retrieval. An embedding model maps text to a vector, and your vector database can only find what that mapping puts close together. If the model places “termination for convenience” far from “cancel the contract early”, no index setting, reranker or prompt will recover the clause. It also fixes your costs: dimensions set storage and memory, context length sets chunk size, and the per-token price or GPU time sets the bill for every re-embed.

And you will re-embed. Vectors from different models are not comparable, so switching models means re-embedding the whole corpus and rebuilding the index. That is why the choice deserves an afternoon of measurement up front.

What is MTEB?

MTEB, the Massive Text Embedding Benchmark, is the standard public comparison. The original paper (Muennighoff, Tazi, Magne and Reimers, EACL 2023) covered 8 task types across 58 datasets and 112 languages, benchmarked 33 models, and its headline finding still holds: “no particular text embedding method dominates across all tasks.” Retrieval is one of those task types; clustering and classification are others, which is why the overall average can mislead a search team.

The benchmark grew. MMTEB (Enevoldsen et al., 2025, accepted to ICLR) expanded it to over 500 quality-controlled tasks across more than 250 languages. One of its findings is a useful corrective to “bigger is better”: the best publicly available model in their evaluation was multilingual-e5-large-instruct, at 560 million parameters.

The live results sit on the MTEB leaderboard on Hugging Face, and the mteb library lets you run any task against any model yourself.

How should I read the MTEB leaderboard?

Four habits make it useful.

Look at the retrieval score, not the average. If you are building search or RAG, clustering and classification scores tell you little. Sort by the retrieval column, or by the specific datasets closest to your domain.

Pick the right benchmark slice. English-only, multilingual and domain-specific slices give different rankings. If your users write in Hindi or Gujarati, an English average is the wrong number.

Treat small gaps as ties. A difference of under a point on a 0 to 100 average across dozens of datasets is noise for your purposes. Your corpus is not the benchmark’s corpus.

Remember that public benchmarks leak. Benchmark datasets are public, and nothing stops a training set from overlapping with them. The BEIR paper (NeurIPS 2021) was built to test zero-shot generalization for exactly this reason, and it found that dense retrievers often underperformed BM25 on datasets outside their training data. Your own held-out queries are the only test set nobody trained on.

What else matters besides the benchmark score?

A short checklist I would fill in for every candidate:

  • Dimensions and storage. Raw size is rows times dimensions times bytes per number. See the table below.
  • Max input length. Text past the limit is cut off. multilingual-e5-large-instruct’s model card says long texts “will be truncated to at most 512 tokens”; OpenAI’s embeddings guide lists 8,192 tokens for its models. This bounds your chunk size.
  • Languages. Check the model card, and test with your languages. Tokens also cost more in some scripts, which affects both price and the effective length limit.
  • Query instructions. Some models expect a prefix. The same model card says queries need a one-line task instruction in the form Instruct: {task}\nQuery: {query}, documents take none, and leaving the instruction out causes “performance degradation”. Getting this wrong quietly costs points.
  • Matryoshka support. Models trained for it can be truncated to fewer dimensions with small losses. See Matryoshka embeddings.
  • Hosting and license. An API model means sending your documents to a vendor and paying per token. An open-weight model means running a GPU or CPU service, and reading the license.
  • Price. OpenAI’s guide, checked today, lists text-embedding-3-small at 1,536 dimensions, 62.3% on MTEB and about 62,500 pages per dollar, and text-embedding-3-large at 3,072 dimensions, 64.6% and about 9,615 pages per dollar. That is roughly 6.5 times the cost for 2.3 points of average MTEB. Whether those points exist on your data is the question your eval answers.

How do I evaluate embedding models on my own data?

The script below does the two calculations I run before anything else. Part 1 is storage for ten million chunks. Part 2 is a tiny evaluation harness: labelled queries, the document that answers each, and three metrics. The two “models” are stand-ins built from the standard library (word TF-IDF and character-trigram TF-IDF) so it runs offline; in a real comparison you swap in a search function per embedding model and keep the harness.

# choose.py: the two numbers I look at before trusting a leaderboard.
# Python 3 stdlib only. Part 1 is storage arithmetic, part 2 is a tiny eval harness.
import math, re
from collections import Counter

# Part 1: raw vector storage for 10 million chunks (index overhead not included).
N = 10_000_000
print("dims   float32   float16   int8     binary")
for d in (384, 768, 1024, 1536, 3072):
    row = [N * d * b / 1e9 for b in (4, 2, 1, 1 / 8)]
    print(f"{d:<5} " + "  ".join(f"{gb:>6.1f}GB" for gb in row))

# Part 2: score two candidate retrievers on YOUR labelled queries.
docs = {
    "a": "Rotate the API key from the dashboard settings page",
    "b": "Webhook retries use exponential backoff for 24 hours",
    "c": "Invoices are generated on the first day of each month",
    "d": "Rate limits: 600 requests per minute per API key",
    "e": "Delete a workspace and all of its projects permanently",
}
labelled = [  # (query as users type it, the doc that answers it)
    ("how do i rotate my api key", "a"),
    ("webhok retry policy", "b"),
    ("when is the invoice generated", "c"),
    ("rate-limit per minute", "d"),
    ("remove workspace", "e"),
]

def words(s): return re.findall(r"[a-z0-9]+", s.lower())
def grams(s): s = f" {s.lower()} "; return [s[i:i + 3] for i in range(len(s) - 2)]

def tfidf_retriever(featurize):
    feats = {d: Counter(featurize(t)) for d, t in docs.items()}
    df = Counter(f for c in feats.values() for f in c)
    idf = {f: math.log(len(docs) / n) + 1 for f, n in df.items()}
    def vec(c): return {f: n * idf.get(f, 0) for f, n in c.items()}
    def cos(u, v):
        dot = sum(u[f] * v.get(f, 0) for f in u)
        nu, nv = math.sqrt(sum(x * x for x in u.values())), math.sqrt(sum(x * x for x in v.values()))
        return dot / (nu * nv) if nu and nv else 0.0
    dv = {d: vec(c) for d, c in feats.items()}
    return lambda q: sorted(docs, key=lambda d: -cos(vec(Counter(featurize(q))), dv[d]))

def evaluate(name, search):
    r1 = mrr = ndcg = 0.0
    for q, gold in labelled:
        ranked = search(q)
        pos = ranked.index(gold) + 1
        r1 += pos == 1; mrr += 1 / pos; ndcg += 1 / math.log2(pos + 1)
    n = len(labelled)
    print(f"{name:<18} recall@1 {r1 / n:.2f}   MRR {mrr / n:.2f}   nDCG {ndcg / n:.2f}")

print()
evaluate("word tf-idf", tfidf_retriever(words))
evaluate("char-3gram tf-idf", tfidf_retriever(grams))

Output (Python 3.9.6):

dims   float32   float16   int8     binary
384     15.4GB     7.7GB     3.8GB     0.5GB
768     30.7GB    15.4GB     7.7GB     1.0GB
1024    41.0GB    20.5GB    10.2GB     1.3GB
1536    61.4GB    30.7GB    15.4GB     1.9GB
3072   122.9GB    61.4GB    30.7GB     3.8GB

word tf-idf        recall@1 0.80   MRR 0.90   nDCG 0.93
char-3gram tf-idf  recall@1 1.00   MRR 1.00   nDCG 1.00

The storage table is why dimensions belong on the scorecard. At ten million chunks, a 3,072-dimension float32 model needs 123 GB of raw vectors and a 384-dimension one needs 15 GB, before the index adds its own overhead. HNSW indexes want that in memory, so the difference is a server size, not a footnote. Lower precision (float16, int8, binary) shrinks it further and also needs measuring.

The evaluation half shows how a harness catches things a leaderboard never will. The word-level retriever missed one query: the user typed “webhok”, and no word matched. The character-trigram retriever still matched “web”, “ebh” and friends, and got every query right. Your users’ typos, abbreviations and jargon are in your query logs and nowhere in MTEB. The metric choice matters less than having one: recall@k answers “was the right document in the top k”, MRR rewards putting it first, and nDCG generalizes both to graded relevance. How to evaluate RAG goes further.

What I would do

  1. Build the eval set first. Pull 50 to 100 real queries from logs or support tickets, and have someone who knows the content mark the correct chunks. Include typos, acronyms and the identifiers people actually search for.
  2. Shortlist from MTEB’s retrieval scores in the right language slice: one strong API model, one strong open model and one small fast model.
  3. Use each model exactly as its card says, with the right query instruction or prefix, the right similarity metric and normalization.
  4. Measure recall@10 and MRR on your set, plus index size and embedding cost per million tokens. Put them in one table.
  5. Run the same comparison with BM25 and with hybrid search. Often the best embedding model plus BM25 beats a better embedding model alone, for less money.
  6. Store the model name and version with every vector, and keep the eval set in the repository so the next model change is a one-command comparison.

FAQ

What is the best embedding model?

There isn’t one. MTEB’s authors found no method dominates every task, and rankings move every few months. The best model for you is the one that scores highest on your own labelled queries within your latency, storage and cost limits.

Are bigger embedding models always better?

No. MMTEB’s best publicly available model in its evaluation had 560 million parameters, small next to the largest entries. Larger models cost more per embed and often produce more dimensions, so measure whether the gain shows up on your data.

How many dimensions should an embedding have?

As few as keep your recall within a point or two of the full-size model. Common sizes run from 384 to 3,072. Matryoshka-trained models let you test smaller sizes without switching models.

Do I need a multilingual embedding model?

If any meaningful share of your documents or queries is not in English, yes, and test with real examples in those languages. A model’s English score says little about its Gujarati or Hindi retrieval.

How often should I re-evaluate my embedding model?

When a candidate model would materially cut cost or improve recall, and whenever your content changes character. Keeping the eval set in version control makes each check cheap.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.