Benchmarks

Real-world issues are not well-defined bugs.

Bug reports arrive as symptoms. Pull requests arrive as diffs. Neither tells you what the running system actually did.

Developers and coding agents still have to discover the execution path, affected code, queries, dependencies, and behavioral consequences. So we measured the impact runtime context makes on this process.

This study measures the reactive case, root-cause analysis, so the value of runtime evidence can be isolated from patch generation.

See the results

What a real issue looks like

A well-specified issue

Public repository
File or function may already be named
Stack trace or mechanism may be included
Implementation dominates the task

An issue in the wild

“Customers are being charged twice on retried payments.”

Private, unfamiliar codebase
Root cause and code location unknown
Behavior crosses service and persistence boundaries
Diagnosis must happen before implementation

The wild issue starts before anyone has turned the bug into instructions.

Why SWE-bench could not isolate diagnosis

SWE-bench remains valuable for comparing overall agent capability. It was not designed to isolate the value of root-cause evidence.

SWE-bench asks whether an agent can resolve a known issue. This study asks whether runtime evidence helps an agent discover an unknown cause.

Want to learn more? Why SWE-bench could not isolate diagnosis, with citations →

So we tested it.

A benchmark designed to require diagnosis

SWE-bench issue

PythonCodePrinter doesn't support Indexed

sympy/sympy #16669 · opened by ruoyu0088 · April 2019

I use lambdify() to generate some functions and save the code for further use. But the generated code for Indexed operation has some warnings [...] We should add following method to PythonCodePrinter:

def _print_Indexed(self, expr):
    base, *index = expr.args
    return "{}[{}]".format(str(base), ", ".join([self._print(ind) for ind in index]))

The fix, ready to paste, inside the issue text.

This benchmark

Operations ticket

Customers being charged twice on retried payments

Stack trace: none

File or function named: none

Fix included: none

Diagnosis is the work.

One issue contains its own answer. The other requires finding it. The SWE-Bench+ audit documents fix leakage like this in roughly one third of SWE-bench issues; this benchmark's tickets are constructed to make it impossible.

Private codebase

~50-module Spring Boot app, outside pretraining corpora

Symptom-only tickets

zero identifier overlap with the gold fix

Hidden repro test

test source removed before the session

Two controlled lanes

identical except runtime evidence

Performance corridor

trivial and impossible problems excluded

The runtime trace is the only difference between the two lanes.

Primary study: two problems, two Claude models, four budgets, eight runs per cell

With runtime evidence, accuracy held. Without it, accuracy fell.

100%

with AppMap runtime evidence

28%

code-only exploration

Root-cause accuracy when limited to 3 tool calls.

94% vs. 62%

Share of bugs fixed and verified when limited to 3 tool calls: runtime evidence versus code-only.

15+ -> 1

Fifteen or more search-and-read steps replaced by one get_call_tree query.

How accurately each lane found the root cause as we cut the number of tool calls allowed. With runtime evidence, accuracy stayed at 100%. Without it, accuracy fell as the budget shrank.

Root-cause accuracy (%) by tool-call budget

100

Unlimited

100

10 calls

100

5 calls

100

3 calls

AppMap runtime evidence Code-only exploration

The share of bugs actually fixed and verified tells the same story. With runtime evidence, 94 to 100 percent. Without it, the rate dropped from 84 percent to 62 percent.

Internal study. Claude model family. One private enterprise-style testbed. Not yet independently replicated. Primary and suite-wide results are labeled separately below.

One query can return what fifteen searches are looking for

In all 216 runs we analyzed, the agent used get_call_tree, and on average it needed just over one call. That one call returned the execution path that held the answer. The code-only lane had to list files, read imports, search for identifiers, and read code, one step at a time.

216 / 216

runs used get_call_tree

1.09

average get_call_tree calls per run

15+

search-and-read steps to build the same picture from source code

More information per call means fewer calls, fewer tokens, less waiting, and lower cost.

1 query · get_call_tree

POST /charge

PaymentController#charge

ChargeService#authorize

RetryPolicy#wrap

LedgerService#write

SQL INSERT INTO ledger ...

IdempotencyStore#check

SQL SELECT id FROM idempotency ...

→ 200 OK · 142ms

// returned by 1 get_call_tree call

~15 searches · static trajectory

# the coding agent gropes for the same picture
grep -r "payment" src/        # 214 hits
read PaymentController.java
grep -r "retry" src/          # 38 hits
read RetryHandler.java
grep -rn "charge(" src/
read ChargeService.java
grep -r "idempotency" src/    # 0 hits
read LedgerService.java
grep -rn "INSERT" src/
read LedgerRepository.java
grep -r "@Transactional" src/
read RetryPolicy.java
grep -rn "ledger.write" src/
... budget exhausted. still guessing.

One get_call_tree query returns the execution path. A representative source-only search requires 15+ calls to approximate the same context, and some runtime facts, like what a query actually returned, cannot be recovered from source at all.

The cost-capability frontier

Matched performance in the primary sweep

In the primary study, both lanes eventually fixed every bug. The lane with runtime evidence got there for about 3.4 times less spend.

Generalization across 11 fixtures, three configurations

The practical setup is a hybrid: a compact model diagnoses from the trace, and a frontier model writes the harder fixes. Each bar shows the cost per run with the fix rate beside it.

Frontier model, code-only exploration, unlimited budget$1.161 per cell · 95% fixed

Hybrid pipeline, compact trace-based diagnosis plus frontier-model fix$0.567 per cell · 88% fixed

Compact model, AppMap trace, 3-call budget$0.309 per cell · 74% fixed

Configuration	Verified-fix rate	Cost / cell	vs. control
Frontier model, code-only exploration, unlimited budget	95%	$1.161	reference
Compact model, AppMap trace, 3-call budget	74%	$0.309	3.8 times cheaper, 21 percentage points lower
Hybrid pipeline, compact trace-based diagnosis plus frontier-model fix	88%	$0.567	2.0 times cheaper, 7 percentage points lower

Internal study. Claude model family. One private enterprise-style testbed. Not yet independently replicated. Primary and suite-wide results are labeled separately below.

The same runtime evidence answers two questions.

Measured in this study

Reactive diagnosis

What caused this symptom?

Which execution path produced it?
Where is the responsible code?
Which calls, queries, or dependencies were involved?
What mechanism must be corrected?

Product implication

Proactive change review

What did this change cause?

Which execution paths changed?
What new calls, queries, or side effects appeared?
Which trusted behavior held?
Did the result match the intended change?

Both workflows need the same missing evidence: a precise record of what the application actually did.

A bug report gives you a symptom. A pull request gives you a diff. AppMap shows you the runtime behavior both leave out.

What the results do and do not establish

This is one internal study, run by our team. It has not been independently replicated.
It used one model family, Claude, inside a single Claude Code agent loop.
The headline 100% versus 28% result comes from two problems across 256 runs. The cost results come from 11 problems across 264 runs.
The private, symptom-only design reduces solution leakage, but it is smaller than SWE-bench Verified.
A compact model alone does not handle every hard fix. The hybrid setup is needed on the broader suite.

Get AppMap

To request the full paper, email info@appmap.io.