LLM judges agree with native scoring only 56.2 percent of the time
The point
Researchers ran 2,160 episodes in tau2-bench airline and retail domains and scored each four ways, including native scoring and an LLM judge. The judge was consistent 98.2 percent of the time but agreed with native scoring on only 56.2 percent.
Why it matters
For builders, evaluator choice decides the conclusion, so a judge alone cannot replace a state check.
Key details
- In 11 episodes the agent made a forbidden write, and the judge passed 10 to 11
- A rule based boundary check with no model call caught all 11, while a stronger judge passed 6
Original ~8 min → 25 sec here Source: arXiv cs.AI, Nov 13, 2026