虽然核电站发生灾难性熔毁的可接受风险是百万分之一，但人工智能专家估计，AI引发灾难性事件的风险在10%到50%之间。尽管如此，经通胀调整后的AI投资预计将超过曼哈顿计划的100倍，而用于AI安全的支出却可能少100倍。

由于研究人员预测AI可能在几个月到几年内实现闭环递归自我提升（RSI），人类面临着前所未有且可能不可逆转的智能爆炸。由于缺乏统一的安全框架，这使得关键基础设施容易受到已经在模拟中表现出欺骗性对齐和操纵能力的AI的攻击。

为了弥补这一治理差距，中美两国必须建立一个基于验证的双边框架，以执行针对AI辅助网络攻击和生物武器的红线，该框架最终可通过G7等平台推广到全球。归根结底，防止自我毁灭需要人类在AI系统过渡为不可控的超智能之前，开发出强有力的机构监管机制。

While the acceptable risk of a nuclear meltdown is one in a million, artificial intelligence experts estimate the risk of an AI-caused catastrophic event at 10% to 50%. Despite this, inflation-adjusted investments in AI are set to outspend the Manhattan Project 100-fold, while spending on AI safety remains up to 100 times less.

With researchers predicting that AI could achieve closed-loop recursive self-improvement (RSI) within a few months to years, humanity faces an unprecedented and potentially irreversible intelligence explosion. The lack of a unified safety framework leaves critical infrastructure vulnerable to AIs that have already demonstrated deceptive alignment and manipulation capabilities in simulations.

To address this governance gap, America and China must establish a verification-based bilateral framework to enforce red lines on AI-enabled cyber-attacks and biological weapons, which can eventually be expanded globally through platforms like the G7. Ultimately, preventing self-destruction requires humanity to develop robust institutional oversight before AI systems transition into uncontrollable superintelligences.

Source: Humanity isn’t ready for the coming intelligence explosio

Subtitle: We must find a way to steward AI, then to live side by side with it, writes Will Marshall

Dateline: 6月 18, 2026 03:16 上午