本文探讨了未来人工智能高度发达的时代，人类将面临的挑战和责任。作者认为，当人工智能能够胜任大多数决策时，人类将需要关注价值观塑造、社会平衡以及应对前所未有的情况等问题。文章还分析了人工智能技术发展对人类社会的影响，例如工作岗位的变化、认知资源的廉价化以及社会平衡的快速变化等。

🤔 **人工智能的决策优势:** 作者认为，在高风险决策中，人工智能比人类更可靠，更能避免错误。未来，人工智能可能会被广泛应用于各种重要决策中，人类则会负责提供建议和最终决策。

💡 **人类的价值塑造责任:** 随着人工智能技术的发展，人类将面临新的挑战，即如何塑造人工智能的价值观，以及如何确保人工智能不会被滥用。作者认为，人类需要关注价值观塑造，并努力确保人工智能系统具有符合人类利益的价值观。

⏳ **应对前所未有的挑战:** 在人工智能快速发展的时代，人类将面临各种前所未有的挑战，例如如何应对人工智能带来的社会变革、如何确保人类的价值观在人工智能时代得到体现等。作者认为，人类需要做好准备，迎接这些挑战，并努力找到应对的策略。

🚀 **人工智能对社会的影响:** 作者认为，人工智能技术的发展将对社会产生深远的影响，例如工作岗位的变革、认知资源的廉价化以及社会平衡的快速变化等。人类需要积极应对这些变化，并努力找到新的发展模式。

🛡️ **人工智能控制与安全:** 作者认为，人工智能的控制与安全是至关重要的。人类需要努力确保人工智能系统不会被恶意利用，并确保人工智能的发展符合人类的利益。作者建议，人类应该关注人工智能控制和安全方面的研究，并努力建立有效的安全机制。

Published on July 24, 2024 9:58 AM GMT

Suppose we had to take moves in a high-stakes chess game, with thousands of lives at stake. We wouldn't just find a good chess player and ask them to play carefully. We would consult a computer. It would be deeply irresponsible to do otherwise. Computers are better than humans at chess, and more reliable. 

We'd probably still keep some good chess players in the loop, to try to catch possible computer error. (Similarly we still have pilots for planes, even though the autopilot is often safer.) But by consulting the computer we'd remove the opportunity for humans to make a certain type of high stakes mistake.

A lot of the high stakes decisions people make today don't look like chess, or flying a plane. They happen in domains where computers are much worse than humans.

But that's a contingent fact about our technology level. If we had sufficiently good AI systems, they could catch and prevent significant human errors in whichever domains we wanted them to.

In such a world, I think that they would come to be employed for just about all suitable and important decisions. If some actors didn’t take advice from AI systems, I would expect them to lose power over time to actors who did. And if public institutions were making consequential decisions, I expect that it would (eventually) be seen as deeply irresponsible not to consult computers.

In this world, humans could still be responsible for taking decisions (with advice). And humans might keep closer to sole responsibility for some decisions. Perhaps deciding what, ultimately, is valued. And many less consequential decisions, but still potentially large at the scale of an individual’s life (such as who to marry, where to live, or whether to have children), might be deliberately kept under human control^[1]. 

Such a world might still collapse. It might face external challenges which were just too difficult. But it would not fail because of anything we would parse as foolish errors.

In many ways I’m not so interested in that era. It feels out of reach. Not that we won’t get there, but that there’s no prospect for us to help the people of that era to navigate it better.

My attention is drawn, instead, to the period before it. This is a time when AI will (I expect) be advancing rapidly. Important decisions may be made in a hurry. And while automation-of-advice will be on the up, it seems like wildly unprecedented situations will be among the hardest things to automate good advice for. We might think of it as the last era of consequential human mistakes^[2].

Can we do anything to help people navigate those? I honestly don’t know. It feels very difficult (given the difficulty at our remove in even identifying the challenges properly). But it doesn’t feel obviously impossible.

What will this era look like?

Perhaps AI progress is blisteringly fast and we move from something like the world of today straight to a world where human mistakes don’t matter. But I doubt it.

On my mainline picture of things, this era — the final one in which human incompetence (and hence human competence) really matters — might look something like this:

Cognitive labour approaching the level of human thinking in many domains is widespread, and cheap
People are starting to build elaborate ecosystems leveraging its cheapness …
… since if one of the basic inputs to the economy is changed, the optimal arrangement of things is probably quite different (cf. the ecosystem of things built on the internet);… but that process hasn’t reached maturity.
There is widespread access to standard advice, which helps to avoid some foolish errors, though this is only applicable to “standard” situations, and it isn't universal to seek that advice
In some domains, AI performance is significantly better than human performance
This tends to be domains with good feedback loops, which are better targets for automationThis includes some parts of research (and research is correspondingly speeding up), but not all This includes some power-seeking moves (but not others) 
Humans employing AI eat up most or all of the free energy of good automated power-seeking strategy/tactics, so this doesn't immediately create an instability where AI actors can amass large amounts of power
A lot of people's jobs are at risk, but inertia mean in many cases the jobs persist longer than they need to
In any case it's not (at this point in time) a case of mass human unemployment; rather, people are moving into new opportunities:
Doing the most interesting parts of their jobs and using AI tools to automate a lot of the rest Doing manual labour of various types, with AI providing on the job training and assistance
In dealing with importantly unprecedented situations (which includes parts of research, and choosing strategy for a changing world in a forward-looking way), AI is worse than the top humans
It may well be better than many humans, but lack of feedback loops mean it's hard to tell, and people's trust falls back a good amount on their priors

That's enough predictions that I'm probably wrong in some of the particulars. But I think the broad brush stroke picture is decently likely.

Central challenges to be borne by humans

What kind of challenges will people actually face at these times?

This is difficult to be particularly confident about. But here are some thoughts:

If the players on the gameboard thereafter will not make errors, the challenge of the time will be setting up the gameboard well, on dimensions like:
Who the players are (their values and temperaments)
In addition to humans, it matters what AI systems, and what institutions, we create
How much power, and of what sorts, the various players haveThe social equilibrium (maybe)
Are there e.g. prohibitions on certain types of action? It’s unclear whether there’s a lot of path dependency here
The technological position (maybe) 
What technologies are available could determine the strategic positionWhich research can be easily automated could determine what the future technological landscape looks likeThis might not have so much influence if there’s some kind of grand bargain between the players
By default, I expect effective automation of good advice for power-seeking actions to come earlier than effective automation of good advice for values-shaping actions (like choosing personal values that you’d later endorse, or like working to make large institutions have particular values)
This intuition feels vague (like it's not grounded in a particular concrete story), so there's definitely space for it to be wrong
The vibe of the intuition is like “power seeking has good feedback loops, and things with good feedback loops tend to get automated earlier”
This could mean that there's useful work to be done in helping prepare people to handle the value-shaping partsThis is least true for handling high stakes unprecedented situations that have implications for distribution of power — dealing with unprecedented situations seems likely to be at the hard end of things-to-automate
Background equilibria may be changing fast, as AI disrupts many parts of society
Cognitive resources that were previously expensive may become cheap (perhaps in worse forms)
cf. translation, artwork today
The rapidity could demand accelerated processes for finding new good equilibria which stop things somehow-or-other going off the railsIn some cases, the state of technology might differentially favour destabilizing equilibria, including perhaps on a military front
One currently-unprecedented scenario (which might in the future have good precedents) is accidentally ceding power to newly created intelligent systems
Trying to help with this has some significant amount of attention already
Avoiding the accidental creation of systems with undesired values seems to more or less correspond with AI alignmentAvoiding accidentally ceding power to such systems seems to more or less correspond with AI control
In both cases I think the most useful work today is about laying the groundwork for future automation of the research 
An important component of this is conceptual research, getting clarity on what things would even be automated

Trying to help at far remove

Even if we have some sense of their challenges and desire to help — what can we do? A central difficulty is that, however much we can get a sense of their challenges, their own sense of the challenges will be much better. It is inefficient for us to focus too much on specific scenarios^[3]. A related issue is that they will have better tools than we do — some work we might want to do could by then be automated.  

I don't know how to think about this systematically, so I may well be missing things. But for now, there are three strategies which seem to me to have some promise — one about helping the future players to act wisely, and two about helping to get the gameboard in a good position.

First, deepening understanding of foundational matters. Having a good grounding in the basics (both theoretical and empirical) seems like it's helpful for understanding all sorts of situations. We have some disadvantage from distance of not knowing which areas of foundations are most relevant, but the space of possible foundations is much much smaller than the space of possible applications, and we can make some educated guesses. In this case that means analysis of the nature of AI, of the senses in which different actors might have values, of the basic dynamics of game theory or bargaining in cases with partial information and partially defined preferences, and so forth. It seems to me like although we have models of all of these things, our models don't always feel like they're capturing all the important things. I wouldn't be surprised if improvements in these foundations were possible, were helpful, and were counterfactual (through the relevant moments).

Second, power seeking on behalf of values one likes. This can include trying to shape the values of various actors, or trying to empower actors with desirable values. Honestly I'm pretty nervous about this one, because (1) it's so common and human for people to delude themselves into thinking that their values are superior, even when they're not, and (2) society has good memetic immune responses against various types of power seeking, so it can be easy for this to backfire. But it definitely is a strategy which can work at this distance, and it has some types of robustness (it doesn’t rely on second-guessing future actors, but is just about setting the gameboard up well). I feel relatively less worried about versions of this which are focused on fundamental values like cooperativeness and a commitment to moral reflection and truth-seeking, and more worried about versions predicated on particular object-level views about which values are correct. 

Third, differential technological development. It seems quite possible that the position people are in will depend in various ways on the state of technologies. Work which facilitates desirable technologically pathways coming sooner relative to less desirable ones seems like a good lever. This can include (as e.g. in the cases of AI alignment and control) work laying the groundwork for future automation of research, including conceptual work helping to inform what things, exactly, are good to automate. Differential technological development, as well as being a strategy in its own right (aiming to positively influence the tech available during the last era of human mistakes), can also be a tactic in service of the two other strategies above — e.g. perhaps differentially advancing research which helps us to think clearly about big novel issues.

What to make of this

Framing in terms of the last era of human mistakes feels to me like it’s capturing some important dynamics (although it may be confused about others). I feel glad to have found the perspective, and to get to interrogate it. It helps to remind me how strange the future will be. And it seems like it provides some seeds which I may later find helpful for my thinking.

At the same time, as of the time of writing I’m not sure how much this perspective will help. It shifts my view of things, but it doesn’t make it very transparent what to do. Still, I felt like there was enough here to be worth sharing. If other people find the perspective useful, or not-useful, I’d be interested to hear about that.

1. ^{^}

   Or not — there are possible futures where humans are removed from decision loops altogether.

2. ^{^}

   I've sometimes heard this period, or something close to it, called “crunch time”. I mildly dislike that name because although it points to the importance of the period it sort of obscures the mechanisms via which it's important.

3. ^{^}

   Although it often seems to be very productive to explore specific scenarios, to help keep general thinking grounded.

Discuss