Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is HyDE? Hypothetical document embeddings, explained

HyDE (hypothetical document embeddings) has an LLM write a fake answer, then searches with its embedding. How HyDE and query rewriting work, in Python.

What is HyDE? Hypothetical document embeddings, explained

HyDE (Hypothetical Document Embeddings) is a retrieval technique where, instead of embedding the user’s question, you ask an LLM to write a plausible answer to it, embed that generated answer, and search for real documents near it. The generated text may contain wrong facts; that’s fine, because it is only used as a search vector and never shown to the user. HyDE is one form of query rewriting, the broader idea of transforming a question before retrieval so it matches how documents are written.

It comes from Gao, Ma, Lin and Callan at Carnegie Mellon and the University of Waterloo, “Precise Zero-Shot Dense Retrieval without Relevance Labels”, posted December 20, 2022.

Why search with a fake answer?

Because questions and answers don’t look alike. A query is short, phrased as a question and uses the asker’s words: “how long does it take to remove wisdom tooth”. The document you want is a paragraph in a clinician’s words: “Extraction of a wisdom tooth usually takes 20 to 40 minutes under local anaesthetic.” In embedding space, the query can sit closer to other questions (an FAQ about tattoo removal that also starts “How long does it take to remove…”) than to its own answer.

An LLM rewrites the question into the shape of an answer. The paper’s framing is that the generated document only needs to “capture relevance pattern”, and that the encoder works as “a lossy compressor”: it keeps the topic and vocabulary and drops most of the specific (possibly invented) details. The paper says plainly that the generated document “is not real, can and is likely to be ungrounded factually”. The search is grounded by the corpus: only real documents come back.

How does HyDE work?

From section 3 of the paper:

  1. Generate. Prompt an instruction-following LLM with something like “write a passage to answer the question” plus the query. The paper used task-specific instructions such as “write a scientific paper passage to answer the question” for scientific corpora.
  2. Encode. Embed the generated passage with the same document encoder used for the corpus. The paper used Contriever, an unsupervised contrastive encoder, and mContriever for non-English tasks.
  3. Average (optional). Sample N passages and average their vectors. The paper’s equation 8 also adds the original query’s vector into the average, so the query itself is one of the hypotheses.
  4. Retrieve. Run a normal nearest-neighbor search with that vector.

They generated with InstructGPT (text-davinci-003) at temperature 0.7 and ran retrieval with the Pyserini toolkit.

What does HyDE look like in code?

The script searches four short documents three ways: with the query, with one hypothetical answer, and with the average of two hypothetical answers plus the query (the paper’s equation 8). The “LLM output” is hand-written so it runs offline, and the encoder is a TF-IDF bag of words standing in for Contriever. Note that the first fake answer says 45 minutes, which is not what the real document says.

# hyde.py: search with a fake answer instead of the question. Python 3 stdlib only.
import math, re
from collections import Counter

docs = {
    "wisdom-teeth": "Extraction of a wisdom tooth usually takes 20 to 40 minutes under local anaesthetic. "
                    "Impacted teeth may need a surgeon and stitches.",
    "braces":       "Braces stay on for 18 to 24 months while the teeth move into place.",
    "faq-removal":  "How long does it take to remove a tattoo? Laser removal needs 6 to 10 sessions.",
    "dentist-visit": "How long does a routine dental check-up take? About 30 minutes.",
}

def words(t): return re.findall(r"[a-z0-9]+", t.lower())
df = Counter(w for t in docs.values() for w in set(words(t)))
def vec(t):  # TF-IDF bag of words, standing in for a dense encoder like Contriever
    return {w: n * math.log(1 + len(docs) / df.get(w, 0.5)) for w, n in Counter(words(t)).items()}
def add(*vs):
    out = Counter()
    for v in vs: out.update(v)
    return {w: x / len(vs) for w, x in out.items()}
def cos(a, b):
    dot = sum(a[w] * b.get(w, 0) for w in a)
    return dot / (math.sqrt(sum(x * x for x in a.values())) * math.sqrt(sum(x * x for x in b.values())))
def search(qv):
    return ", ".join(f"{d} {cos(qv, vec(t)):.2f}" for d, t in
                     sorted(docs.items(), key=lambda kv: cos(qv, vec(kv[1])), reverse=True)[:3])

query = "how long does it take to remove wisdom tooth"
# What an LLM might write for "write a passage to answer the question". Hand-written so this
# runs offline; note the made-up number, which HyDE tolerates because only the vector is used.
fake_1 = "Wisdom tooth extraction usually takes about 45 minutes. A surgeon may give a local anaesthetic."
fake_2 = "Removing an impacted wisdom tooth can take up to an hour, and the surgeon may need stitches."

print("query only      :", search(vec(query)))
print("HyDE, 1 doc     :", search(vec(fake_1)))
print("HyDE, 2 docs + q:", search(add(vec(fake_1), vec(fake_2), vec(query))))

Output (Python 3.14.5):

query only      : faq-removal 0.51, dentist-visit 0.26, wisdom-teeth 0.21
HyDE, 1 doc     : wisdom-teeth 0.62, dentist-visit 0.18, faq-removal 0.04
HyDE, 2 docs + q: wisdom-teeth 0.61, dentist-visit 0.23, faq-removal 0.22

With the raw query, the top hit is the tattoo FAQ (0.51), because it shares the phrasing “how long does it take to remove”. The real answer, wisdom-teeth, is third at 0.21. Searching with one fake answer puts wisdom-teeth first at 0.62 and drops the tattoo page to 0.04: the fake answer uses the answer’s vocabulary (extraction, surgeon, anaesthetic) and none of the question’s phrasing. The wrong “45 minutes” didn’t matter, since the retrieved document is the real one with the real number.

The averaged version keeps wisdom-teeth on top (0.61) but lets the tattoo page back up to 0.22, because the query vector is now part of the mix. That is the trade-off of equation 8: including the query hedges against a fake answer that drifted off topic, at the cost of some of the benefit.

Does HyDE actually work?

On the paper’s benchmarks, yes, for the setting it targets: dense retrieval with no relevance labels. On TREC DL19 web search, nDCG@10 went from 44.5 for Contriever alone to 61.3 with HyDE, ahead of BM25 at 50.6. On DL20 it went from 42.1 to 57.9 (BM25: 48.0). The paper reports HyDE comparable to Contriever fine-tuned on MS MARCO on DL19, and about 10% lower on DL20 in map and nDCG@10, without using any training labels. It also improved results on low-resource BEIR datasets and on non-English retrieval (Swahili, Korean, Japanese among them) with mContriever.

The honest caveat is the baseline. HyDE’s biggest gains are over an unsupervised encoder. Modern embedding models are trained on huge sets of question and answer pairs, which closes part of the question-to-answer gap HyDE was built to bridge. Test it against your own embedding model before assuming the paper’s numbers carry over.

What is query rewriting, more generally?

HyDE rewrites a question into an answer. Other rewrites change the question itself:

  • Clarifying and de-contextualizing. In a chat, “what about the second one?” has to become a standalone question (“What is the refund policy for annual plans?”) before retrieval. Most production RAG chat needs this step.
  • Multi-query. Generate several phrasings of the question, retrieve for each, and fuse the lists with reciprocal rank fusion.
  • Decomposition. Split a compound question into sub-questions and retrieve for each.
  • Keyword extraction. Turn a chatty question into a terse search query for BM25 or a web search API.

The research version is Rewrite-Retrieve-Read (Ma et al., EMNLP 2023), which adds a rewriting step in front of retrieve-then-read and trains a small rewriter model with reinforcement learning, using feedback from the frozen LLM reader. In practice, most teams prompt a general model to rewrite and skip the training.

The rewrite-then-search shape is familiar to me from rSearch, where the question has to become a web search query before retrieval (via Serper) can do anything useful. Whether the index is a vector store or a search engine, the query you send matters as much as the index.

Where does HyDE break?

  • Latency. HyDE puts a full LLM generation in front of every search. Retrieval can’t start until the generation finishes, which matters for interactive search.
  • Exact-match queries. For an error code, SKU or name, a generated paragraph dilutes the one token that matters. Route those to keyword search; hybrid RAG does this naturally.
  • Topics the LLM doesn’t know. If the model has no idea what your internal project “Falcon” is, its hypothetical answer will be about birds. HyDE helps most where the model knows the domain’s vocabulary.
  • Bias toward the model’s beliefs. The search vector encodes what the model thinks the answer looks like. If it is confidently wrong in a way that matches a wrong document, HyDE will retrieve that document.
  • Cost. One generation per query adds output tokens on every request.

What I would do

  1. Always do conversational rewriting (turn follow-ups into standalone questions) in a chat RAG product. It is the highest-value rewrite.
  2. Try HyDE as an experiment, not a default. Run it on a judged query set against your current embedding model, and keep it only where recall improves. How to evaluate RAG covers the measurement.
  3. If it helps, run HyDE retrieval alongside plain query retrieval and fuse with RRF instead of replacing the query. That keeps the exact-match path alive.
  4. Use a small, fast model for the hypothetical answer and cap its length. It only needs the right vocabulary.
  5. Never show the hypothetical document to the user. It is a search key, not an answer.

FAQ

What does HyDE stand for?

Hypothetical Document Embeddings, from Gao et al., “Precise Zero-Shot Dense Retrieval without Relevance Labels” (2022).

Does HyDE cause hallucinations in the answer?

Not directly. The generated passage is only used as a search vector; the final answer is generated from the real retrieved documents. A badly wrong hypothetical can still retrieve the wrong documents, which then mislead the answer.

Is HyDE better than multi-query retrieval?

They solve different problems. HyDE bridges the gap between question style and document style; multi-query covers different phrasings and aspects of the question. They can be combined by fusing their result lists.

Do I need HyDE with a modern embedding model?

Often less than with the unsupervised encoder in the paper, since current models are trained on question and passage pairs. Test it on your queries before adding the latency.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.