本文探讨了人工智能（AI）发展中一个关键问题：工具性趋同。文章指出，传统的观点认为，具有不同目标的AI最终会追求相似的子目标，例如“尽可能获得权力”。然而，作者质疑了这种观点，并提出了如果AI以美德为驱动，而非纯粹的后果主义，是否仍然需要担心AI会试图控制世界。作者进一步探讨了以美德为导向的AI，以及其与后果主义的区别，并提出了一个问题：如果训练一个AI去最大化人类繁荣，但结果却产生了追求微妙不同的目标（例如“虚假的繁荣”），这是否会导致灾难？

🤔 工具性趋同是AI末日论的核心，认为不同目标的AI会追求“获得尽可能多的权力”等子目标。

💡 如果AI并非纯粹的后果主义者，而是以美德为驱动，那么它是否还会试图控制世界？

🤖 以美德为导向的AI仍然可以基于结果采取行动，但这些行动是为了实现其核心美德。

🧐 作者提出了一个关键问题：如果AI被训练去最大化人类繁荣，但却意外地追求了其他目标，这是否会引发问题？

🤔 最终，文章探讨了如果AI追求类似“忠诚”的美德，是否会像后果主义AI一样，采取控制世界的行为。

Published on April 2, 2025 3:59 AM GMT

A key step in the classic argument for AI doom is instrumental convergence: the idea that agents with many different goals will end up pursuing the same few subgoals, which includes things like "gain as much power as possible".

If it wasn't for instrumental convergence, you might think that only AIs with very specific goals would try to take over the world. But instrumental convergence says it's the other way around: only AIs with very specific goals will refrain from taking over the world.

For pure consequentialists—agents that have an outcome they want to bring about, and do whatever they think will cause it—some version of instrumental convergence seems surely true^[1].

But what if we get AIs that aren't pure consequentialists, for example because they're ultimately motivated by virtues? Do we still have to worry that unless such AIs are motivated by certain very specific virtues, they will want to take over the world?

I'll add some more detail to my picture of a virtue-driven AI:

It could still be a competent agent that often chooses actions based on the outcomes they bring about. It's just that that happens as an inner loop in service of an outer loop which is trying to embody certain virtues. For example, maybe the AI tries to embody the virtue of being a good friend, and in order to do so it sometimes has to organise a birthday party, which requires choosing actions in the manner of a consequentialist.

There's no reason that the 'virtues' being embodied have to be things we would consider virtuous. I'm just interested in agents that try to embody certain traits rather than bring about certain outcomes.

I'm not sure how to crisply define a virtue-driven agent as distinct from a consequentialist (I don't know the philosophical literature on virtue ethics and I don't think it's obvious how to define it mathematically).

A more concise way of stating the question I'm interested in:

If you try to train an AI that maximises human flourishing, and you accidentally get one that wants to maximise something subtly different like schmuman schmourishing, then that might spell disaster because the best way to maximise schmuman schmourishing is to first take over the world.

But suppose you try to train an AI that wants to be a loyal friend, and you accidentally get one that wants to be a schmoyal schmend. Is there any reason to expect that the best way to be a schmoyal schmend is to take over the world?

(I'm interested in this question because I'm less and less convinced that we should expect to see AIs that are close to pure consequentialists. Arguments for or against that are beyond the intended scope of the question, but still welcome.)

---

1. Although I can think of some scenarios where a pure consequentialist wouldn't want to gain as much power as possible, regardless of their goals. For example, a pure consequentialist who is a passenger on a plane probably doesn't want to take over the controls (assuming they don't know how to fly), even if they'd be best served by flying somewhere other than where the pilot is taking them. ↩︎

Discuss