文章探讨了利用人工智能工具和认知方法来构建一个更安全、更协调的社会，以降低人类面临的生存风险。核心观点是，通过提升集体推理、协调能力和制度治理水平，我们或能显著减少人为风险。文章强调了这一策略的优势，如可实现渐进式进步、可单方面行动等。同时也指出了挑战，包括对“足够好”的标准要求高、推广难度大、以及权力平衡的复杂性。最终，文章呼吁投入更多努力，以探索构建一个既能避免自我毁灭，又能坚守美好价值的社会。

🧠文章提出，通过提升社会认知水平、协调能力和制度治理水平，可以有效降低人类的生存风险，这可能比单纯的技术对齐更重要。

💡作者认为，随着人工智能的发展，它将带来前所未有的协调和决策能力，以及在博弈论和政治学等关键领域实现研究自动化。

👍文章强调，这种策略具有优势，如可实现渐进式进步，并且可以单方面通过构建相关工具来推进。

⚠️挑战在于，要达到“足够好”的标准可能很高，推广可能困难，尤其是在国际协调和民主监督方面，以及需要解决权力平衡问题。

🧐文章认为，我们可能需要解决一种“文明对齐问题”，这可能类似于人工智能对齐问题，例如，需要改进机构的可纠正性、可扩展的监督以及防止“mesa-optimisers”的形成。

Published on July 4, 2025 3:32 PM GMT

The AI tools/epistemics space might provide a route to a sociotechnical victory, where instead of aiming for something like aligned ASI, we aim for making civilization coherent enough to not destroy itself while still keeping anchored to what’s good^[1].

The core ideas are:

Basically nobody actually wants the world to end, so if we do that to ourselves, it will be because somewhere along the way we weren’t good enough at navigating collective action problems, institutional steering, and general epistemicsConversely, there is some (potentially high) threshold of societal epistemics + coordination + institutional steering beyond which we can largely eliminate anthropogenic x-risk, potentially in perpetuity^[2]As AI gets more advanced, and therefore more risky, it will also unlock really radical advances in all these areas — genuinely unprecedented levels of coordination and sensible decision making, as well as the potential for narrow research automation in key fields like game theory and political science

I think these points are widely appreciated, but most people don’t seem to have really grappled with the implications — most centrally, that we should plausibly be aiming for a massive increase in collective reasoning and coordination as a core x-risk reduction strategy, potentially as an even higher priority than technical alignment.

Some advantages of this strategy:

We can make helpful incremental progressWe can do a lot of work unilaterally, by just building the relevant toolsLots of the progress here is self-reinforcingIn the limit, this basically lets us sidestep the alignment problem for as long as necessary

Some challenges:

The bar for ‘good enough’ might be quite highAdoption could be very difficult in some important cases
Especially for international coordination and democratic oversight
There are some legitimate irreconcilable value differences
Some people really would rather throw the dice on getting radical boosts to science and healthcare via automated research soon enough that they/their families don’t pass away, even if that means incurring substantial misalignment risk
There are hard balances of power to navigate
For example we probably want to make governments much stronger in some ways (such that they can prevent vulnerable world / misuse dynamics), while also making democratic oversight much stronger (so we don’t end up in a stable dictatorship)
We might have to solve some kind of 'civilizational alignment problem' which could be difficult in similar ways to AI alignment
As a pointer, we are currently less than perfect at making institutions corrigible, doing scalable oversight on them, preventing mesa-optimisers from forming, and so on

The big implication in my mind is that it might be worth investing serious effort in mapping out what this coherent and capable enough society would look like, whether it’s even feasible, and what we’d need to do to get there. 

(Such an effort is something that I and others are working up towards — so if you think this is wildly misguided, or if you feel particularly enthusiastic about this direction, I'd be keen to hear about it.)

Thanks to OCB, OS, and MD for helpful comments, and to many others I've discussed similar ideas with

1. ^{^}

   The easy route to 'coherent enough to not destroy itself' is 'controlled by a dictatorship/misaligned AI', so the more nebulous 'still anchored to the good' part is I think the actual tricky bit

2. ^{^}

   Importantly this might include making fundamental advances in understanding what it even means for an institution to be steered by some set of values

Discuss