Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What is SWE-bench? The coding agent benchmark, explained

SWE-bench tests whether an AI can fix real GitHub issues, graded by the repo's own tests. How grading works, why Verified was retired, and a runnable demo.

What is SWE-bench? The coding agent benchmark, explained

SWE-bench is a benchmark that measures whether an AI system can resolve real GitHub issues: the model gets a Python repository at the commit before a fix plus the issue text, writes a patch, and passes only if the project’s own hidden tests pass afterwards. It was introduced by Jimenez et al. in October 2023 with 2,294 tasks from 12 popular Python repositories, and its 500-task subset, SWE-bench Verified, became the headline number for coding models until OpenAI stopped reporting it in February 2026.

Why was SWE-bench a big deal?

Before it, coding benchmarks were mostly self-contained puzzles. HumanEval (2021) has 164 hand-written problems where the model writes one function from a docstring. That tests whether a model can program; it doesn’t test whether it can work in someone else’s codebase.

SWE-bench changed the unit of work to a real issue in a real project. The paper’s abstract says solving one “requires understanding and coordinating changes across multiple functions, classes, and even files.” When it came out, the best model, Claude 2, resolved 1.96% of the tasks. That number was low enough to be a clean target and real enough that progress on it meant something to working engineers, which is why every lab started quoting it.

How does SWE-bench work?

Each task instance comes from a merged pull request that closed an issue. The benchmark keeps the repository at the commit before the PR, the issue text, the PR’s code change (the “gold patch”) and the PR’s tests. The tests are split in two:

  • FAIL_TO_PASS: tests that fail on the old code and pass once the issue is fixed.
  • PASS_TO_PASS: tests that passed before and must still pass, so a fix can’t break something else.

The model never sees the tests. It sees the issue and the repository, produces a diff, the harness applies it in a container, and the task counts as resolved only if every test in both groups passes. The score is the percentage of tasks resolved.

That design is the strength and the weakness. It’s fully automatic and hard to argue with, but it judges a fix by tests written for one particular fix.

What does SWE-bench grading look like in code?

Here’s the grading rule on a toy repository: one function, one old test, two new tests from the “real PR”, and three candidate patches.

# mini_swebench.py: how a SWE-bench task is graded, on a toy repo.
# Python 3 standard library only. A patch "resolves" the task only if every
# FAIL_TO_PASS test now passes and every PASS_TO_PASS test still passes.
import pathlib, subprocess, sys, tempfile

BASE = "def parse_version(s):\n    return tuple(int(x) for x in s.split('.'))\n"

TESTS = {   # test name -> (kind, code). The model never sees these.
    "test_simple":   ("PASS_TO_PASS", "assert parse_version('1.2.3') == (1, 2, 3)"),
    "test_v_prefix": ("FAIL_TO_PASS", "assert parse_version('v1.2') == (1, 2)"),
    "test_helper":   ("FAIL_TO_PASS", "from lib import strip_prefix\n"
                                      "assert strip_prefix('v9') == '9'"),
}

GOLD = ("def strip_prefix(s):\n    return s[1:] if s.startswith('v') else s\n\n"
        "def parse_version(s):\n    return tuple(int(x) for x in strip_prefix(s).split('.'))\n")

PATCHES = {
    "gold patch (the real PR)": GOLD,
    "correct fix, other name":  BASE.replace("s.split('.')", "s.lstrip('vV').split('.')"),
    "breaks old behaviour":     BASE.replace("s.split('.')", "s[1:].split('.')"),
}

def run(code, test_code):
    d = pathlib.Path(tempfile.mkdtemp())
    (d / "lib.py").write_text(code)
    (d / "t.py").write_text("from lib import parse_version\n" + test_code)
    return subprocess.run([sys.executable, "t.py"], cwd=d, capture_output=True).returncode == 0

for name, code in PATCHES.items():
    results = {t: run(code, src) for t, (_, src) in TESTS.items()}
    resolved = all(results.values())
    detail = "  ".join(f"{t}={'pass' if ok else 'FAIL'}" for t, ok in results.items())
    print(f"{name:<26} resolved={str(resolved):<5}  {detail}")

Output (Python 3.13.11):

gold patch (the real PR)   resolved=True   test_simple=pass  test_v_prefix=pass  test_helper=pass
correct fix, other name    resolved=False  test_simple=pass  test_v_prefix=pass  test_helper=FAIL
breaks old behaviour       resolved=False  test_simple=FAIL  test_v_prefix=pass  test_helper=FAIL

The third patch is rightly rejected: it breaks 1.2.3. The second one is the interesting row. It fixes the issue as written, but the PR’s test imports a helper called strip_prefix that the issue never mentions, so a correct fix fails. I made this toy up, but it’s the same failure OpenAI found in a real task, pylint-dev__pylint-4551, where the tests import a function named get_annotation that appears nowhere in the problem description.

What is SWE-bench Verified?

OpenAI and the SWE-bench authors released SWE-bench Verified in August 2024 to fix problems like that one. They had 93 Python developers review 1,699 random tasks, each labeled by three separate annotators, flagging underspecified issues and unit tests that could reject valid solutions. 38.3% of samples were flagged as underspecified and 61.1% for unfair tests; in total 68.3% were filtered out. The remaining pool was sampled down to 500 tasks.

The cleanup mattered. GPT-4o went from 16% on the original set to 33.2% on Verified with the best scaffold of the time. The team also released a containerized Docker harness to make runs reproducible. SWE-bench Lite, an earlier subset, was already filtered in a way that made it easier than the full set.

Why did OpenAI stop using SWE-bench Verified?

On February 23, 2026, OpenAI published Why we no longer evaluate SWE-bench Verified. Two findings:

Tests reject correct solutions. State of the art had crawled from 74.9% to 80.9% in six months, so they audited 138 problems that o3 didn’t solve consistently across 64 runs, each reviewed by at least six engineers. 59.4% had material problems: 35.5% had “narrow” tests that enforce implementation details, 18.8% had “wide” tests checking functionality the issue never asked for, and 5.1% had other issues.

Models have seen the answers. The tasks come from public repositories that model makers train on. Using a red-teaming setup, OpenAI found GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash could each reproduce gold patches or problem details verbatim for some tasks. One example: given only a task ID, Gemini 3 Flash produced details from the description and the gold patch, including line numbers.

Their conclusion was that gains on Verified “increasingly reflect how much the model was exposed to the benchmark at training time,” and they recommended reporting SWE-Bench Pro instead. Pro has 1,865 problems from 41 actively maintained repositories, split into a public set, a held-out set and a commercial set from private startup codebases, with tasks that can take a professional engineer hours to days. OpenAI’s own contamination probe found it much cleaner, though not perfectly clean.

Anthropic’s guide to agent evals (January 2026) names the general problem, saturation: once a benchmark is near its ceiling, large capability gains show up as small score changes.

How should you read a SWE-bench score?

Three things change the number as much as the model does.

The scaffold. A score belongs to a model plus a harness. The SWE-agent paper showed interface design alone moving results, and Agentless scored 32% on Lite with a fixed pipeline and no agent loop at all. Compare scores only within the same harness.

The variant and split. Full, Lite, Verified and Pro aren’t comparable, and Pro’s public and commercial sets aren’t either.

Attempts. Resolved at one try (pass@1) is the standard. Best-of-N with a selection step inflates the number, which is legitimate if it’s disclosed and you’d run it that way in production.

What I would do

  • Treat SWE-bench numbers as a filter, not a ranking. A model far behind on SWE-bench Pro probably won’t do well on your repo; two models a few points apart are a tie.
  • Build a private SWE-bench from your own history. Pick twenty to fifty closed issues with a merged fix and tests, freeze the commit before each fix, and grade agents the same way. It’s uncontaminated by definition, as long as the repo is private.
  • Read failures before trusting the score. Check whether a failing patch was wrong or the test was too narrow. That’s the same audit OpenAI ran, at a size you can manage.
  • Track cost and time per resolved task. A model that resolves one more task at three times the tokens may not be the one to ship. I covered cost per task in Claude Opus 4.5’s effort parameter.

For the broader picture, read what LLM benchmarks measure and how to evaluate AI agents; for the systems being tested, see what an AI coding agent is.

FAQ

What does SWE-bench stand for?

Software Engineering benchmark. The paper’s title asks the question it tests: “Can Language Models Resolve Real-World GitHub Issues?”

How many tasks are in SWE-bench Verified?

500, drawn from the original 2,294 after human review filtered out tasks with underspecified issues or unfair tests.

Is SWE-bench Verified still used?

Many leaderboards still list it, but OpenAI stopped reporting it in February 2026 because of flawed tests and training contamination, and recommended SWE-Bench Pro. I’d read any new Verified score with that in mind.

What is a good SWE-bench score?

It depends on the variant. On the original benchmark, 1.96% was the best result in October 2023. On Verified, frontier models passed 80% by early 2026, which is near the point where OpenAI argued the remaining tasks are mostly broken. Compare scores only within one variant and one harness.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.