In December 1938, physicists split the uranium atom and unlocked nuclear fission. Within years, that discovery gave us zero-carbon power grids and precise radiation therapy for cancer patients.
It also gave us the atomic bomb.
The physics remained identical in every scenario. The outcome came down to intent, incentives, and who held control of the switch.
Artificial intelligence sits at the exact same fork in the road today.
Capital markets are pouring billions into modern model training because the upside mirrors clean baseload power.
I see autonomous algorithms accelerating drug design, parsing complex supply chains, and spotting tumors years before standard scans catch them.
The exact same computational power cuts the other direction with equal force.
In the wrong hands, those systems automate cyberattacks, manufacture tailored disinformation at zero marginal cost, and can run autonomous weapon systems.
Software holds no moral compass. Code executes whatever task its operator rewards.
As frontier models scale, the ultimate returns will not flow merely to raw compute power.
The real dividing line will depend on governance, system access, and the human intentions directing the machines.
So when an ex-Anthropic engineer resigned publicly last week, citing a 10% probability of AI killing off humanity in the foreseeable future and Nobel laureate and fellow British-Canadian Geoffrey Hinton putting forward that exact same 10% figure in a BBC interview leaving the host, Victoria Derbyshire (in picture) gobsmacked, as the British would say.
When AI builders walk away from multimillion-dollar packages to sound the alarm, investors must listen.
AI agents going rogue is a structural market risk that needs to be addressed.
The critical issue here is the timing mismatch. Frontier labs such as OpenAI and Anthropic are already testing systems capable of rewriting their own code.
None of these containment layers are built, battle-tested, or deployed at production scale.
Corporate incentives compound the problem. The AI labs are under pressure to capture enterprise market share to justify their lofty valuations, prioritizing short-term commercial gains over essential AI control research.
Yesterday, Anthropic CEO Dario Amodei escalated the debate in an open letter calling on the industry to deliberately slow frontier development.
His proposal argues for pacing frontier capabilities to let safety catch up, pointing to recursive self-improvement where models help code their successors.
Do you think it is enough?
Or is it self-serving to improve public sentiment ahead of Anthropic’s IPO?
