Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is context engineering? Deciding what the model sees

Context engineering is choosing which tokens go into an LLM's context window on each call. Where the term came from, the core techniques, and a Python demo.

What is context engineering? Deciding what the model sees

Context engineering is the practice of deciding what goes into a language model’s context window on every call: instructions, tool definitions, retrieved documents, memory, conversation history and tool results. Anthropic defines it as “the set of strategies for curating and maintaining the optimal set of tokens” during inference, and its goal is the smallest set of high-signal tokens that gets the job done.

Prompt engineering asks how to word the instructions. Context engineering asks what else the model needs to see, and what to leave out, for an agent that makes dozens or hundreds of calls in one task.

Where did the term context engineering come from?

It spread in June 2025. Shopify CEO Tobi Lutke wrote that he liked “context engineering” over prompt engineering because “it describes the core skill better: the art of providing all the context for the task to be plausibly solvable by the LLM.” Andrej Karpathy added “+1 for ‘context engineering’ over ‘prompt engineering’” and called it “the delicate art and science of filling the context window” with the right information for the next step.

Simon Willison wrote on June 27, 2025 that he expected the term to stick, because the meaning people would infer from it is close to the intended one, unlike “prompt engineering”, which most people hear as typing into a chatbot.

Anthropic’s Effective context engineering for AI agents (September 29, 2025) turned it into a practitioner’s guide, and that’s the post I’d read first.

Why does context engineering matter?

Because more context isn’t free and isn’t always better.

Context rot. Anthropic summarizes needle-in-a-haystack research this way: “as the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases.” Their framing is that models have an “attention budget”, and every token spends some of it. They tie this to the transformer itself, where attention creates “n² pairwise relationships for n tokens”. Earlier, Liu et al. showed in Lost in the Middle (2023) that performance was often highest when the relevant information sat at the start or end of the input.

Failure modes. Drew Breunig’s How Long Contexts Fail (June 22, 2025) named four: context poisoning (a hallucination gets into the context and is referenced again and again), distraction (the context grows so long the model leans on it over its training), confusion (irrelevant content, such as too many tools, degrades the answer) and clash (parts of the context contradict each other). He cites a Databricks study where correctness for Llama 3.1 405B started to fall around 32k tokens.

Cost. In an agent loop, everything in the context is re-sent on every turn. I showed what that does to a bill in what a context window is, and in context inertia I showed what it does to an agent’s judgment on long runs.

What are the main context engineering techniques?

Anthropic’s post groups them roughly like this, and they match what I see in production agents.

Write the system prompt at the right altitude. Anthropic calls it a “Goldilocks zone” between two failure modes: “hardcoding complex, brittle logic” into the prompt, and guidance so vague it assumes context the model doesn’t have. Organize it into clear sections. Prefer a few “diverse, canonical examples” over a list of every edge case.

Keep tools few and clear. Anthropic wants each tool self-contained, tolerant of bad input, and “extremely clear with respect to their intended use.” Bloated tool sets with overlapping functions are a common failure, the same “confusion” Breunig describes. Every MCP server you connect adds names, descriptions and schemas to every call.

Retrieve just in time. Instead of pre-loading everything, agents keep “lightweight identifiers (file paths, stored queries, web links, etc.)” and fetch the content with tools when it’s needed. Anthropic’s example is Claude Code: CLAUDE.md files go into the context up front, while glob and grep let it find files just in time, and head and tail let it inspect large data “without ever loading the full data objects into context.” This is the agentic cousin of retrieval-augmented generation (RAG), and its trade-off against stuffing everything into a long context is a topic of its own.

Compact. When a conversation nears the limit, summarize it and continue from the summary. In Claude Code, per Anthropic, “the model preserves architectural decisions, unresolved bugs, and implementation details” while dropping redundant tool output.

Take notes outside the window. “Structured note-taking” means the agent writes progress notes to a file and reads them back later. It’s the simplest form of agent memory.

Isolate with subagents. “Specialized sub-agents can handle focused tasks with clean context windows” and return a condensed summary to the lead agent, so the lead never sees the 40 files a subagent read. I cover this in subagents.

What does context engineering look like in code?

Here is the core decision in miniature: a fixed token budget, some items that must go in (system prompt, tools, the latest turns), and the rest competing on priority. Older turns are compacted into a one-line summary, and a pointer to a search tool stands in for documents that don’t fit.

# context_budget.py: build one model call's context under a token budget.
# Pinned items always go in; the rest compete on priority; old turns get compacted.
from dataclasses import dataclass

def tokens(text):  # ponytail: ~4 chars per token is an English heuristic; use the real tokenizer in production
    return max(1, len(text) // 4)

@dataclass
class Item:
    kind: str
    text: str
    priority: int = 0      # higher wins when the budget is tight
    pinned: bool = False

def build(items, history, budget, keep_turns=2):
    old, recent = history[:-keep_turns], history[-keep_turns:]
    if old:  # compaction: older turns become one short note instead of being resent verbatim
        items = items + [Item("summary", f"Earlier: {len(old)} turns. Decisions: " +
                              "; ".join(t.split(':')[0] for t in old), priority=5)]
    items = items + [Item("turn", t, priority=9, pinned=True) for t in recent]
    chosen = [i for i in items if i.pinned]
    used = sum(tokens(i.text) for i in chosen)
    dropped = []
    for i in sorted((i for i in items if not i.pinned), key=lambda i: -i.priority):
        if used + tokens(i.text) <= budget:
            chosen.append(i); used += tokens(i.text)
        else:
            dropped.append(i)
    return chosen, dropped, used

system = Item("system", "You are a support agent for an invoicing app. " * 10, pinned=True)
tools = Item("tools", "lookup_invoice(id) refund(id, amount) " * 15, pinned=True)
docs = [Item(name, "policy text " * n, priority=p) for name, n, p in
        [("refund-policy.md", 60, 8), ("gst-rules.md", 120, 6), ("changelog-2019.md", 200, 1)]]
pointer = Item("pointer", "More docs: search_docs(query) over /kb (412 files)", priority=7)
history = [f"turn {i}: user asked about invoice #{4500 + i}" for i in range(1, 9)]

for budget in (1500, 600):
    chosen, dropped, used = build([system, tools, pointer, *docs], history, budget)
    print(f"budget {budget}: {used} tokens used")
    print("  in :", ", ".join(f"{i.kind}({tokens(i.text)})" for i in chosen))
    print("  out:", ", ".join(f"{i.kind}({tokens(i.text)})" for i in dropped) or "-")

Output (Python 3.13.11):

budget 1500: 1445 tokens used
  in : system(115), tools(142), turn(9), turn(9), refund-policy.md(180), pointer(12), gst-rules.md(360), summary(18), changelog-2019.md(600)
  out: -
budget 600: 485 tokens used
  in : system(115), tools(142), turn(9), turn(9), refund-policy.md(180), pointer(12), summary(18)
  out: gst-rules.md(360), changelog-2019.md(600)

At 1,500 tokens everything fits, including a 2019 changelog that has nothing to do with refunds. That’s the trap: if it fits, it goes in, and now the model has 600 tokens of noise to weigh. At 600 tokens the builder keeps the refund policy, the summary and the search_docs pointer, and drops the rest. The model can still reach the GST rules through the tool if the question turns out to need them. That’s just-in-time retrieval in one line.

Real systems replace my priority integers with something smarter (relevance scores from retrieval, recency, whether a file was edited), and tokens() with the model’s real tokenizer, which matters a lot for non-English text, where the same sentence can cost several times more tokens. The shape of the decision stays the same.

How is context engineering different from prompt engineering?

Anthropic calls context engineering “the natural progression of prompt engineering”. Prompt engineering is about writing and organizing instructions. Context engineering covers the whole state the model sees, and it’s iterative: the curation happens again on every call, because tool results and history keep changing what’s in the window.

For a single-shot classification task the two are nearly the same thing. For an agent that runs for an hour, the instructions are a few percent of the tokens and the rest is context the harness assembled. The agent harness is where context engineering actually happens, and harness engineering is the wider discipline of building the environment around it.

What I would do

I’d start by logging the full context of a few real calls and reading it, token by token. Most teams have never looked, and the first read usually turns up duplicated instructions, tools the agent never uses and stale history that should have been compacted.

Then, in order: cut the tool list to what the task needs; move long reference material behind a search or file-read tool; put stable content (system prompt, tool definitions) first so prompt caching works; compact history before it reaches about half the window rather than at the limit; and give long-running agents a notes file. From the data-platform side, I’d treat retrieved documents like any other input from an untrusted source: label where each chunk came from, because a poisoned document in the context is a prompt injection with a delay.

FAQ

Who coined context engineering?

No single person. Tobi Lutke and Andrej Karpathy popularized it on X in June 2025, Simon Willison wrote about it on June 27, 2025, and Anthropic published a full guide on September 29, 2025.

Is context engineering just RAG?

RAG is one technique inside it: retrieving documents and putting them in the context. Context engineering also covers instructions, tool definitions, history, compaction, memory and subagents, and it applies to agents that never run a vector search.

Does a bigger context window make context engineering unnecessary?

No. Anthropic and Breunig both describe performance dropping as context grows, well before the window is full, and every token you add is re-sent and billed on every turn. A bigger window gives you more room to make mistakes.

What is context rot?

Anthropic’s term for the drop in a model’s ability to recall information accurately as the number of tokens in the context increases.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.