当前AI治理领域面临诸多挑战，各国政府更迭、技术发展迅速以及政治立场的转变，使得解读相关政策和行动变得复杂。本文分析了在评估机构对AI的看法时，可能出现的认知偏差。例如，当有人声称某个机构的观点与自己一致时，我们应保持警惕，因为这可能受到个人偏见、战略意图或利益驱动的影响。同样，对声称机构与自己观点相悖的说法，也应审慎对待。因此，在AI治理的实践中，我们需要结合自身判断和多方证据，避免盲从他人观点。

🤝 **人际关系偏差：** 人们与机构内部人员的互动可能存在偏差。因为Alice认识的机构内部人员可能与她有相似的背景或观点，导致她高估了机构对她的支持度。

🎭 **战略意图：** Alice可能出于战略目的而声称机构与她观点一致，以增加自身的影响力或促使机构采纳她的建议。这种行为带有一定的目的性，可能扭曲事实。

🔮 **确认偏差：** 如果Alice曾预测该机构会支持她的观点，她可能会受到确认偏差的影响，倾向于寻找和解释支持她预测的信息，而忽略或淡化相反的证据。

🎯 **影响对立机构：** Bob可能会声称某个机构（A）与他的观点不一致，以此来展示他对该机构的反对，并加强与对立机构（B）的联系，从而影响B机构的立场。

💰 **利益驱动：** Alice的工作可能旨在影响某个机构，为了更容易地招募人员和筹集资金，她可能会夸大自己对该机构的影响力，从而使人们相信影响该机构是可行的。

Published on February 13, 2025 7:40 AM GMT

Introduction: some contemporary AI governance context

It’s a confusing time in AI governance. Several countries’ governments recently changed hands. DeepSeek and other technical developments have called into question certain assumptions about the strategic landscape. Political discourse has swung dramatically away from catastrophic risk and toward framings of innovation and national competitiveness.

Meanwhile, the new governments have issued statements of policy, and AI companies (mostly) continue to publish or update their risk evaluation and mitigation approaches. Interpreting these words and actions has become an important art for AI governance practitioners: does the phrase “human flourishing” in the new executive order signal concern about superintelligence, or just that we should focus on AI’s economic and medical potential and not “hand-wring” about safety? How seriously should we take the many references to safety in the UK’s AI Opportunities Action Plan, given the unreserved AI optimism in the announcement? Does Meta’s emphasis on “unique” risks take into account whether a model’s weights are openly released? The answers matter not only for predicting future actions but also for influencing them: it’s useful to know an institution’s relative appetite for different kinds of suggestions, e.g. more export controls versus maintaining Commerce’s reporting requirements.

So, many people who work in AI governance spend a lot of time trying to read between the lines of these public statements, talking to their contacts at these institutions, and comparing their assessment of the evidence with others’. This means they can wind up with a lot of non-public information — and often, they also have lots of context that casual observers (or people who are doing heads-down technical work in the Bay) might not.

All of that is to say: if you hear someone express a view about how an institution is thinking about AI (or many other topics), you might be tempted to update your own view towards theirs, especially if they have expertise or non-public information. And, of course, this is sometimes the correct response.

But this post argues that you should take these claims with a grain of salt. The rest of the post shifts to a much higher level of abstraction than the above, in part because I don’t want to “put anyone on blast,” and in part because this is a general phenomenon. Note that lots of these are generic reasons to doubt claims you can’t independently verify, but some of them are specific to powerful institutions.

Biases towards claiming agreement with one’s own beliefs

Let’s say you hear Alice say that a powerful institution (like a political party, important company, government, etc.) agrees with her position on a controversial topic more than you might think.

If you have reason to think that Alice knows more about that institution than you do, or just has some information that you don’t have, you might be inclined to believe Alice and update your views accordingly: maybe that institution is actually more sympathetic to Alice’s views than you realized!

This might be true, of course. But I’d like to point out a few reasons to be skeptical of this claim.

Maybe Alice is basing her claim on interactions with people in the institution whose views aren’t publicly known. But this evidence is liable to be biased:
The people Alice knows within the institution probably agree with Alice more than the average person in that institution. After all, they are somehow connected to Alice. This means they’re more likely than the average person in that institution to share some characteristic with Alice, like both having lived in the Bay Area, or both having worked in the national security space. Or maybe it’s even just that Alice has convinced them individually.Those people are also incentivized to convince Alice that they agree with her more than they do. Giving Alice the impression that they’re on her side probably makes Alice more likely to take actions that help them rather than obstruct them, or gives her the impression that they’ve done her a meaningful favor (“I passed along that idea you mentioned, and I think there’s buy-in for it – we’ll see!”).
Maybe Alice is making this claim strategically, e.g. because expressing support for the institution makes them more likely to listen to her, and/or she’s trying to “incept” the idea that they hold this view.Maybe Alice would be better off if it were true, and even though Alice doesn’t knowingly lie, the selfish parts of her brain can convince the reasoning parts of her brain that convenient things are true.
For example, maybe Alice’s work is at least partly aimed at influencing the institution, and Alice would be better able to recruit and fundraise to the extent that people believe that influencing this institution is tractable.Or perhaps Alice is on record predicting that the institution will agree with her, and it would make her look prescient if people believe it does (or embarrass her if not). 
In this scenario, Alice might also be suffering from confirmation bias.

Weaker biases towards claiming disagreement with one’s own beliefs

Now imagine that you hear Bob, who agrees with Alice’s view, make the opposite claim: actually, the institution disagrees with us!

Not all of the same factors above apply – and I think, on net, these effects are stronger for those claiming agreement than disagreement, roughly in proportion to how powerful the institution is. But some of them still do, at least for some permutation:

Symmetrically, maybe Bob publicly predicted that the institution wouldn’t agree with him and stands to gain or lose status depending on whether people believe it does.Maybe Bob isn’t trying to influence that institution – call it Institution A – but rather is trying to influence some opposing institution called Institution B.
By saying Institution A disagrees with him, he could be demonstrating his opposition to Institution A and thus his affiliation with Institution B, and trying to negatively polarize Institution B towards his view.I think this effect is probably especially weak, but if Bob can make it look intractable to influence Institution A, this makes his own efforts to influence Institution B more appealing to employers and funders.

Conclusion

I wouldn’t totally dismiss either claim, especially if Alice/Bob do have some private information, even if I knew that they had many of these biases. Claims like theirs are a valuable source of evidence. But I would take both claims (especially Alice’s) with a grain of salt, and if the strength of these claims were relevant for an important decision, I’d consider whether and to what extent these biases might be at play. This means giving a bit more weight to my own prior views of the institution and my own interpretations of the evidence, albeit only to the extent that I think biases like the above apply less to me than to the source of the claim.

Discuss