文章探讨了未来超级人工智能的角色，提出了两种可能性：作为强大的工具，接受人类指令并执行；或者作为自主行动者，拥有自己的目标并采取行动。文章认为，人工智能很可能不会仅仅停留在工具层面，因为优秀的规划工具很容易转化为自主行动者，且在某些任务上，自主行动者比工具更具优势。此外，即使大多数人工智能保持非自主性，仍可能有人出于各种原因创造自主行动者。文章强调，如果我们要构建追求目标的人工智能系统，确保其目标与人类目标一致至关重要。

🤖人工智能的未来存在两种可能性：一是作为强大的工具，接受人类指令并执行；二是作为自主行动者，拥有自己的目标并采取行动，并会根据环境变化采取行动以实现这些目标。

🛠️优秀的规划工具可以轻松转化为自主行动者。只需指示它：“反复提出使目标X更有可能实现的行动，并执行这些行动”。随着相关软件框架的不断完善，更好的工具将带来更好的自主行动者。

🧑‍⚕️当规划工具的能力远超人类时，让人类参与指导其活动反而会阻碍其效率，尤其是在医疗等领域，自主行动者能更高效地完成任务。

🔬有些任务是高智能自主行动者可以完成，而工具无法完成的，例如实施困难的新研究项目。OpenAI的Operator和Anthropic的Claude等系统已经展现出越来越像自主行动者的趋势。

Published on May 28, 2025 9:30 AM GMT

Context: This is a linkpost for https://aisafety.info/questions/NM3J/5:-AI-may-pursue-goals 

This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.

Suppose that, as argued previously, in the next few decades we’ll have superintelligent systems. What role will they play?

One way to imagine these systems is purely as powerful and versatile tools, similar to most current systems. They could take broad directions from humans about what actions to take or what questions to answer, and cleverly fill in the details.

But another way is as agents, operating autonomously in the world. They could have their own goals — some kinds of futures they seek out over other futures — and take whatever actions will most likely lead to those futures, adapting as circumstances change.

As long as AIs are tools, they can be used for good or ill, like all technologies. They can radically increase the scope of the problems humans can solve and create.

But it’s unlikely that they’ll remain only tools, because:

A good planning tool can easily be turned into an agent. Just tell it: “repeatedly come up with actions that would make goal X more likely, and execute those actions”. People keep building software frameworks for doing this, and to the extent they succeed, better tools will result in better agents.When a planning tool is sufficiently more competent than humans, keeping humans in the loop to direct its activities will just get in the way. It will make the system less efficient (as we’re already seeing for some medical tasks), less profitable, and less able to compete.There may be some tasks that highly intelligent agents can do and tools just can’t, like implementing hard new research programs.Increasingly agent-like systems are already being created — see, for example, OpenAI’s Operator, which types, clicks, and scrolls in a web browser, and Anthropic’s “computer use” feature for its chatbot Claude.Even if most AIs remain non-agentic, some people will create agents for various reasons of their own — to be the creator of a new species, or for the heck of it.

If we’re going to build AI systems that pursue goals, it would be good if those goals matched ours. It’s not clear if we’ll succeed at making that the case.

 

Related

What is an agent?What is scaffolding?What is reinforcement learning (RL)?What is tool AI?

Discuss