LLM judges agree with native scoring only 56.2 percent of the time

The point

Researchers ran 2,160 episodes in tau2-bench airline and retail domains and scored each four ways, including native scoring and an LLM judge. The judge was consistent 98.2 percent of the time but agreed with native scoring on only 56.2 percent.

Why it matters

For builders, evaluator choice decides the conclusion, so a judge alone cannot replace a state check.

Key details

Original ~8 min → 25 sec here Source: arXiv cs.AI, Nov 13, 2026

← All points