OpenAI discovered that approximately 30 percent of tasks in SWE-Bench Pro, a widely used benchmark for measuring AI programming capabilities, are broken. The company has withdrawn its earlier endorsement of the test.
SWE-Bench Pro is a popular evaluation tool in the AI industry for assessing how well language models can handle real-world software engineering tasks. OpenAI's review uncovered significant issues with roughly one-third of the benchmark's tasks, raising questions about the validity of scores generated using this metric.
The discovery is notable because benchmarking plays a critical role in the AI field, allowing researchers and companies to compare model performance and track progress. A compromised benchmark can produce misleading results and skew comparisons between different AI systems.
OpenAI's decision to pull its endorsement signals the importance of rigorous validation in AI evaluation frameworks. The company did not detail specific plans to fix the benchmark or propose alternatives, but the move highlights ongoing challenges in establishing reliable testing standards for increasingly capable AI coding models.
The findings may prompt other organizations to scrutinize their own benchmarking methodologies and results.
Startups like Altur are deploying AI chatbots to handle debt collection calls, automating a process traditionally done by humans. Y Combinator has backed six debt collection and settlement startups over the past six years.
Following recent earthquakes, Venezuelan developers and citizens deployed AI-powered websites and apps to locate missing persons and coordinate disaster relief as government response lagged.
Prime Minister Anthony Albanese has created a dedicated AI office and committed to protecting Australian creators from copyright infringement by artificial intelligence companies. The government rejected plans to grant tech firms free access to Australian data.