🤔 **投资者对实验室决策的影响力** 虽然大多数前沿人工智能实验室承认AGI发展带来的极端风险，并投资于安全措施，但它们似乎仍然处于一场构建AGI的竞赛中，这可能会导致实验室做出冒险的决定，为了赢得竞赛而忽视安全措施。 一个常见的解释是，实验室是营利性公司，投资者会迫使它们将利润放在首位，即使高管知道风险，也会继续快速发展能力。 然而，作者认为投资者对实验室的影响力并不完全清楚，他们可能无法迫使实验室做出违背其最佳判断的危险决定。作者认为，投资者的影响力会因公司、决策类型和时间尺度而异，它取决于具体的公司结构、高管、董事会、内部文化、所有权类型和适用的公司法。

💡 **研究方向：投资者对前沿人工智能实验室的影响** 作者认为，研究投资者对实验室的影响力可以为人工智能治理提供重要信息，例如： * 投资者对AGI发展风险的意识是否会改变实验室的行为？ * 改变实验室的财务激励机制的监管措施是否会成功地改变他们的行为？ * 实验室是否会避免损害其其他投资者投资组合的行动，例如避免某些行业的自动化或避免大规模灾难？ * 高管是否可以采取行动来削弱投资者的权力，赋予自己权力，或者利用AGI采取任何形式的大规模单方面行动，例如关键行为、在AGI中实施自己的意识形态、控制军事或政治权力、通过UBI分配利润？

❓ **研究问题** 作者列出了一系列需要研究的问题： * 投资者以何种方式对前沿实验室拥有正式控制权？这种控制在哪些时间尺度上发生？ * 如何量化每个利益相关者对实验室决策的影响力？ * 随着时间尺度的变化，投资者对AGI发展的影响力如何变化？ * 人工智能泡沫破裂或其他宏观经济事件如何改变投资者的权力？ * 不同实验室在投资者对其决策的影响力方面有何比较？ * 公司对投资者的受托责任如何运作？典型受托责任违反案件的持续时间是多久？在什么情况下，实验室投资者可以胜诉？这是否可能奏效？ * 投资者以何种方式对实验室拥有非正式权力？这与他们的正式权力相比如何？

Published on July 25, 2024 1:12 AM GMT

Produced as part of the ML Alignment & Theory Scholars Program - Summer 2024 Cohort

This post presents the basic story behind my current research direction at MATS. I'm looking for feedback on whether the basic premises hold up, and which research questions should I prioritize.

The role of investors in fueling the AGI race is unclear

Most Frontier AI Labs acknowledge the extreme risks of AGI development, invest in safety, and seem to care at least somewhat about making sure their systems don't destroy the world.

However, Frontier Labs currently appear to be in a race to build AGI, which most fear will lead Labs to make risky bets and not take adequate safety measures to avoid catastrophic outcomes, in order to have a chance of winning the race. A common explanation is that Labs are for-profit companies, which prevents them from putting safety first, because investors would force them to put profit first and continue to rapidly develop capabilities even though executives know the risks.

But do investors actually have so much influence over companies that they can force them to build dangerous models against their best judgement?

There are strong economic arguments for why companies that prioritize profit are rewarded in the long run, both through selection effects (profit-seeking companies are less likely to go bankrupt) and through capital allocation (non-profit-seeking companies don't receive investment or don't even get started). However, these mechanisms only work on average over long periods of time, and in the short term investors regularly fail to keep the companies they invest in from making stupid decisions and going bankrupt (e.g. FTX).

So I think it is unclear how much power investors have over Frontier Labs' overall strategy for building AGI, or over specific short-term decisions like whether to train a particular dangerous model. I expect their effective power to vary by company, by type of decision and by timescale. Their power will depend on specific corporate structures, executives, boards, internal cultures, ownership types and applicable corporate laws.

Identifying the power of investors over Labs can inform AI governance strategy

I expect the details of investors' influence over Labs' decisions to be relevant to several strategic considerations:

Whether widespread investor awareness of the extreme dangers of AGI development would change the behavior of LabsWhether regulations that change the financial incentives of Labs would succeed in changing their behavior.Whether Labs would avoid actions that would harm the rest of their investors' portfolios (e.g. avoiding automation of certain industries, avoiding large-scale catastrophes).Whether executives could take actions that would disempower investors, empower themselves, or take any kind of large-scale unilateral action enabled by AGI (e.g. pivotal acts, implementing their own ideology in the AGI, taking military or political control, distributing profits through a UBI).And probably many others

Research direction: Investing how investors influence Frontier AI Labs

I expect that there are many open questions in this area, and that studying them could help governance researchers make better decisions and policy proposals.

Here is a list of questions that seem important to investigate:

What are the different ways in which investors have formal control over Frontier Labs? Over which timescales does this control take place?How can we quantify the amount of influence each stakeholder has over the Labs' decisions?How does the amount of investor influence over the development of AGI change depending on timelines lengths?How would the bursting of the AI bubble or other macroeconomic events change the power of investors?How do different Labs compare in the amount of influence investors have over their decisions?How does the fiduciary duty of companies to investors work? What is the duration of a typical breach of fiduciary duty case? In what case could Labs investors win? Would it be likely to work?What are the different ways in which investors have informal power over labs? How does this compare with their formal power?

All feedback is welcome! Feel free to comment on whether the basic premises hold up, whether this is an impactful research direction, what research questions I should prioritize, or how this work should be published.