本文探讨了在AI研发自动化可能引发的软件智能爆炸（FOOM）情景下，如何进行AI控制。作者定义了FOOM为AI能够自动化所有AI研发，从而导致进步呈超指数级增长的局面。文章引用了相关研究，讨论了AI软件效率提升的潜力以及对AI控制的影响。作者希望就此问题征求Buck或ryan_greenblatt等人的见解，并鼓励其他人分享与此相关的观点。

🤯 FOOM（软件智能爆炸）定义：当AI能够自动化所有AI研发时，进步开始超指数级增长，因为更好的软件的回报大于1。

📈 AI软件效率提升：文章指出，AI软件效率的提升可能达到数量级（OOMs），例如，AI预训练的计算能力远超人类，这为AI的快速发展提供了基础。

💡 研究引用：文章引用了相关研究，探讨了AI研发自动化是否会导致软件智能爆炸，以及在达到有效物理极限之前AI能取得多大进展。

🤔 控制问题：作者关注在FOOM发生的情况下，我们是否能够找到适应AI控制的方法，或者AI是否基本上是不可控的。

Published on March 29, 2025 7:38 PM GMT

For the purposes of FOOM, I'm defining it as a situation in which once an AI is capable enough to automate away all AI R&D, progress starts exploding hyper-exponentially for a period because the returns to better software is larger than 1, meaning AI labor quality is improving faster than the problem of finding new algorithms gets harder, combined with the potentially high limits to how efficient software can get, meaning that the AI gets OOMs smarter on a fixed compute budget within months or weeks.

These articles can help explain what I mean better:

https://www.forethought.org/research/will-ai-r-and-d-automation-cause-a-software-intelligence-explosion

https://www.forethought.org/research/will-the-need-to-retrain-ai-models (An auxillary post)

https://www.forethought.org/research/how-far-can-ai-progress-before-hitting-effective-physical-limits (where they estimate that about 5 OOMs of progress could be gotten for free, because human compute used for pretraining us is 10^24 flops, whereas current AI pretraining compute to automate away AI R&D is 10^29 flops, with a median of 8 OOMs more efficiency possible for software, but at very large uncertainty, and their error bars are from 4-12 OOMs more efficient software being possible).

Also note that we are only talking about training compute efficiency, not runtime/inference efficiency, so for the purposes of the discussion we will only talk about training efficiencies being improved in a software intelligence explosion.

Now I don't want to debate about whether the scenario is true (though for those that want my probability of something like FOOM/Software Intelligence Explosion, my probability so far is in the 40-50% range of it happening if we automate AI R&D), but rather the question is about given a software explosion being possible, could we figure out a way to adapt AI control to that case, or is AI basically uncontrollable assuming a software intelligence explosion does happen and there's a lot of OOMs to the physical limit of intelligence in software.

I'd be especially interested in responses from @Buck or @ryan_greenblatt on this question, but anyone can answer this question if they have an insight to share that relates to the question.

Discuss