Why SWE-bench could not isolate diagnosis
SWE-bench remains valuable for comparing overall agent capability. It was not designed to isolate the value of root-cause evidence. This page explains why, with citations to the published record.
Public code can be familiar
SWE-bench draws from highly visible open-source repositories that may already be represented in model pretraining. A result can reflect learned repository priors as well as investigation performed during the benchmark.
Post-triage issues can narrow the search
Real GitHub issue threads may identify a file, stack trace, function, or likely mechanism. Once those clues are present, the agent begins closer to implementation and the exploratory RCA phase becomes harder to measure independently.
Patch success can mask weak diagnosis
An implementation agent can re-explore the code and recover from an incomplete RCA report. We separated diagnosis from fixing so the study could measure the quality and cost of the evidence-gathering stage itself.
SWE-bench asks whether an agent can resolve a known issue. This study asks whether runtime evidence helps an agent discover an unknown cause.
What the published record shows
- The SWE-Bench+ audit documents solution leakage, the fix visible in the issue text, in roughly one third of SWE-bench issues.
- The SWE-Bench Illusion study reports that models can identify the buggy file path from the issue text alone in 76% of cases, without seeing the repository, evidence of memorization rather than investigation.