AI MODELS FAIL TO SPOT UNSOLVABLE MATH PROBLEMS

Researchers unveiled SOOHAK, a math benchmark containing 439 handwritten problems designed to evaluate AI reasoning at scale. The dataset includes 99 intentionally unsolvable tasks, serving as a crucial test for whether AI systems recognize the limits of problems rather than simply generate plausible-sounding answers. The results highlight a significant weakness: while increased computational power improves performance on solvable problems, it does not enhance models' ability to acknowledge when a problem cannot be solved. This disconnect suggests that scaling alone cannot address fundamental reasoning gaps in current AI systems. Google's Gemini 3 Pro led the benchmark across research-level problems, but the broader finding remains troubling. No model achieved even 50 percent accuracy on the unsolvability detection task—a baseline that would be expected from systems claiming advanced mathematical reasoning. The benchmark addresses a gap between flashy individual successes and the sustained, broad research skills required for authentic mathematical problem-solving. AI systems that confidently attempt unsolvable problems pose real risks in applied domains where recognizing problem constraints is essential. SOOHAK represents an effort to establish more rigorous evaluation standards for mathematical AI. Rather than measuring only success on solvable problems, the benchmark forces systems to demonstrate judgment about problem feasibility—a capability that current models struggle to develop, regardless of their overall performance levels. The findings suggest that future AI development must address not just computational scale, but fundamental differences in how models approach reasoning tasks and recognize epistemic boundaries.