Jacob Coxon, a researcher at AI safety company Anthropic, has resigned citing concerns about self-improving artificial intelligence. He is calling for industry-wide pacing agreements to slow AI development.
Coxon departed Anthropic after growing concerned about the trajectory of AI systems capable of self-improvement, describing the current approach as "gambling with our lives."
In his departure, Coxon urged AI laboratories to establish formal agreements that would slow development timelines. He argues that without coordination between major AI companies, the risks of uncontrolled self-improving systems outweigh the benefits of rapid advancement.
The researcher's concerns center on AI systems that can iteratively improve their own capabilities. Such systems present unique challenges for safety testing and control, as their behavior becomes harder to predict at scale.
Coxon's exit adds to ongoing debates within the AI industry about development speed versus safety measures. While companies like Anthropic have positioned themselves as focused on AI safety, tensions remain between cautious approaches and competitive pressures to deploy advanced systems.
His call for pacing agreements echoes similar proposals from other AI researchers and safety advocates who worry the field is moving faster than safeguards can accommodate. However, enforcing such agreements across competing organizations remains a significant challenge, with no formal mechanism currently in place.
Anthropici has not issued a public statement regarding Coxon's departure or his specific concerns about self-improving AI systems.
The resignation reflects broader uncertainty in the AI sector about whether current safety practices are adequate for increasingly capable systems. As large language models and other AI technologies become more sophisticated, questions about their controllability and long-term risks continue to intensify.
As AI companies push toward superintelligence, recent security breaches raise urgent questions about controlling systems more capable than humans. Experts debate whether development should proceed.
Indian workers are using iPhones to collect real-world data for training humanoid robots. The effort reflects a growing competition among tech companies to secure the training data needed for advanced robotics.
Music generation startup Suno has replaced its AI models with a new version trained exclusively on licensed music, addressing mounting copyright litigation. The company says Suno v6 abandons the training data used for previous model iterations.
Viral AI assistant Instinct now has email capabilities, enabling the agent to create accounts, manage communications, and handle support requests on behalf of users.