OpenAI has stopped using SWE-bench Verified as a benchmark for evaluating frontier coding capabilities, signaling that the widely-used test no longer reflects the performance levels of advanced AI systems.
SWE-bench Verified, a popular evaluation framework for measuring software engineering capabilities in AI models, has become outdated as frontier models have surpassed the benchmark's difficulty ceiling.
OpenAI disclosed the decision in a detailed breakdown of why the metric no longer serves as a meaningful measure of progress. The benchmark, designed to assess how well AI systems solve real-world GitHub issues, was previously considered a standard measure of coding proficiency.
The shift highlights a broader trend in AI development: evaluation metrics require constant updating as models improve. When systems routinely solve test cases at high accuracy levels, benchmarks lose their ability to differentiate capabilities or track meaningful progress.
The move sparked discussion in the developer community, with 82 comments on Hacker News examining implications for how AI coding tools should be evaluated going forward. Other organizations will likely need to develop or adopt more challenging assessment frameworks to measure frontier coding abilities effectively.
OpenAI has published a framework outlining critical capabilities needed for advanced AI systems alongside proposed safeguards for frontier models. The roadmap addresses deployment risks as AI capabilities expand.
The Electronic Frontier Foundation has urged courts to resist pressure to fundamentally alter copyright law in response to artificial intelligence developments. The organization argues that existing legal frameworks are sufficient to address AI-related concerns.
Anthropic has unveiled Fable 5.1, a new upgrade to its Mythos-class Claude model that reduces operational costs by 25% for standard workloads and up to 45% for complex agentic tasks.
The Fermi Explorer Mission, a nonprofit organization, plans to launch a spacecraft to Alpha Centauri—the Sun's nearest star system—by the end of 2029. The mission will use a trajectory discovered by an AI system developed by PSI.