谷歌的AlphaEvolve已开始在易于验证答案的领域取得科学发现。若我们能设计出足够强大的物理模拟，机械工程的验证成本或将降低。届时，采用类似策略的AI可能设计出远超人类的机器，并通过模拟进行验证。这些机器或能自我复制并呈指数级增长，带来近乎无限的资源。文章探讨了这种发展可能带来的经济和军事影响，以及对AI监管和国际合作的必要性，最后提出了关于AI发展方向的思考。

🔬AlphaEvolve已在矩阵乘法等领域取得科学发现，其成功依赖于低成本的答案验证机制。

⚙️强大的物理模拟可能使机械工程的验证变得廉价，从而使AI能够设计出远超人类的机器，这些机器甚至可以自我复制并指数级增长，带来无限资源。

⚔️即使资源无限，国家间可能仍存在军事压力，促使它们研发更智能的AI，以对抗自我复制机器。因此，国际社会应达成协议，禁止AI的进一步发展，并共享AI带来的利益。

🤝率先研发出自我复制机器的国家虽有能力单方面控制世界，但武力解决方案并非上策。不过，这种潜在力量反而能增加其他国家对其共享利益承诺的信任。

Published on May 17, 2025 7:22 PM GMT

Google's AlphaEvolve has recently started to make real world scientific discoveries, but only in domains where it's very cheap to verify the correct answer (e.g. matrix multiplication algorithms).

But if we can design sufficiently powerful physics simulations, mechanical engineering may one day become "cheap to verify." AI which uses similar strategies as AlphaEvolve might be able to engineer far better machines than humans, if the machines are verified by simulations.

These machines may be able to self replicate, and grow exponentially. This could bring near infinite abundance.

There might be no economical pressure to build even smarter AI, because there will already be unlimited resources.

Unfortunately, countries might still have military pressure to build even smarter AI, since the self replicating machines can still be defeated by even better engineered versions.

My hope is that once self replicating machines start to grow exponentially, the world will wake up to the dangerous power of AI, and negotiate a treaty banning other countries from developing their own AI (but promising them a share of the benefits). I'm not sure if this can succeed.

The first country to build self replicating machines technically has the military power to take over the world and enforce a ban unilaterally, but such brute force solutions never lead to good outcomes. However, the mere fact it has this power makes other countries more likely to trust its promise to share the benefits of AI, since other countries will think "if this country really was malicious, and doesn't intend to keep its promise. It could simply take over the world right here and now, and wouldn't need to negotiate with us."

---

Do you think humanity will survive if AI solves mechanical engineering before AI solves deception and scheming? Do you think the chances of survival are higher? Would better physics simulations be a net positive?

Discuss