ExLabel checks SWE-bench style datasets for clarity and test coverage
The point
Researchers present ExLabel, a self-evolving LLM assistant that judges task clarity and test coverage in SWE-bench style datasets. It uses an evolving experience bank to guide assessments of new instances.
Why it matters
Builders can filter training and evaluation data before trusting benchmark scores.
Key details
- New SWE-TQ benchmark spans eight languages, cleaned from human annotations in SWE-bench and Multi-SWE-bench
- Tested with three backbone LLMs, ExLabel achieved the best F1 across all settings
- A model fine-tuned on data filtered by ExLabel resolved 66.2% of SWE-bench Verified issues
Original ~8 min → 25 sec here Source: arXiv cs.AI, Oct 30, 2026