Terminal-Bench audit finds issues in roughly 30 percent of tasks
The point
Researchers audited Terminal-Bench 2.0 and found issues in roughly 30 percent of the benchmark. They released repairs as Terminal-Bench 2.1 through a workflow combining community reports, agent traces, oracle checks, and human gated repairs.
Why it matters
For builders, benchmarks can change scores and ranks, so treat leaderboards as moving measurements.
Key details
- Repairs raised accuracy by over 10 percentage points and moved positions by up to three ranks
- Extending time limit closed a 20 point harness gap on GPT 5.4
- It shrank a 30 point gap on GPT 5.4 mini to 5 points
Original ~12 min → 25 sec here Source: arXiv cs.AI, Nov 6, 2026