What if the people building the most powerful AI systems are themselves becoming afraid of what they are creating? That question has returned to the centre of the technology debate after Anthropic researcher Jacob Coxon left the company and publicly criticised the race toward increasingly powerful AI.
Artificial intelligence safety concerns have intensified after Jacob Coxon, a researcher who has worked at both OpenAI and Anthropic, announced his departure from Anthropic while warning that the race to develop increasingly powerful AI systems could become dangerous.
Coxon said leading AI companies were moving towards self-improving superintelligence at a pace he considers unsafe. He argued that the industry needs to slow down and better understand advanced systems before pushing their capabilities further.
Coxon had spent about three years working on AI pre-training research across the two companies and contributed to work involving GPT-4o, according to reports.
His departure has drawn attention because it comes from within an industry that has increasingly been forced to confront questions about whether increasingly autonomous AI systems can remain under meaningful human control.
Coxon’s concerns centre on the possibility that future systems could become highly capable at areas such as cybersecurity and scientific research while becoming increasingly difficult for humans to understand or control.
The debate is not about whether today’s AI systems are already capable of destroying humanity. Rather, researchers are increasingly focused on what could happen if future systems become capable of rapidly improving themselves.
The controversy raises a difficult question for the industry: Should technological progress continue at maximum speed, or should safety come first even if that means slowing down the AI race?
