文章探讨了在不同AI开发环境下，特别是“赶工不负责任的开发者”情景下，如何有效降低AI对齐风险。文章强调，即使在资源有限、政治支持不足的情况下，AI公司内部的小团队仍可通过构建风险证据、实施安全措施和进行对齐研究来显著降低风险。文章还指出，外部力量可以通过提供可移植的研究和协助来增强这些内部团队的能力。文章呼吁，在AI安全研究中，应更多关注悲观情景，并优先开发低成本、易于实施的安全技术。

⚠️ “安全案例”制度：理想情况下，所有AI开发者都应遵循的安全方法，以最大限度地降低AI带来的风险。这种方法通常要求AI在部署第一年内逃逸的概率小于1%，并且在模型试图破坏安全措施时，条件概率小于5%。

🏃 “赶工的合理开发者”制度：在这种情况下，即使是相对负责任的AI开发者，也会因时间紧迫而难以谨慎实施干预措施，或在安全方面犯错。

🤦 “赶工的不负责任的开发者”制度：这是文章重点关注的情景，即AI公司内部虽有少数人关注灾难性风险，但公司整体不重视对齐风险，导致安全措施执行不力，此时，内部人员需要通过构建风险证据，实施安全措施，并进行对齐研究来降低风险。

💰 低成本安全措施：在资源有限的情况下，安全措施需要廉价且易于执行，避免增加其他团队的工作负担。文章强调，控制技术的研究应允许复杂性内部化，并鼓励外部力量提供可移植的研究和协助。

Published on January 28, 2025 4:41 PM GMT

(Many of these ideas developed in conversation with Ryan Greenblatt)

In a shortform, I described some different levels of resources and buy-in for misalignment risk mitigations that might be present in AI labs:

The “safety case” regime. Sometimes people talk about wanting to have approaches to safety such that if all AI developers followed these approaches, the overall level of risk posed by AI would be minimal. (These approaches are going to be more conservative than will probably be feasible in practice given the amount of competitive pressure, so I think it’s pretty likely that AI developers don’t actually hold themselves to these standards, but I agree with e.g. Anthropic that this level of caution is at least a useful hypothetical to consider.) This is the level of caution people are usually talking about when they discuss making safety cases. I usually operationalize this as the AI developer wanting to have <1% chance that their AIs escape in the first year of deployment, and <5% conditional on the model trying pretty hard to subvert their safety measures. (I think some AI developers are hoping to get lower levels of risk than this.)

The rushed reasonable developer regime. The much riskier regimes I expect, where even relatively reasonable AI developers are in a huge rush and so are much less able to implement interventions carefully or to err on the side of caution.

The rushed unreasonable developer regime. You’re advising an unreasonable AI developer that is incompetent at implementing safety measures and unwilling to put much effort into implementing these measures.

The existential war regime. You’re in an existential war with an enemy and you’re indifferent to AI takeover vs the enemy defeating you. This might happen if you’re in a war with a nation you don’t like much, or if you’re at war with AIs. Obviously this is a regime where you should be much more aggressive.

I want to flesh out one particular rushed unreasonable developer scenario that I’ve been thinking about lately: there’s ten people inside the AI company who are really concerned about catastrophic risk from misalignment. The AI company as a whole pays lip service to AI risk broadly construed and talks occasionally about risk from AGI, but they don’t take misalignment risk in particular (perhaps especially risk from schemers) very seriously.

I think this scenario (and similarly pessimistic scenarios) seem important to target with technical research and planning: it seems pretty likely that we’ll only have this level of political will in short timelines (at least within a subset of competitive AI companies) and it seems possible to substantially improve the situation. I worry that a variety of AI safety thinking and planning focuses on overly optimistic scenarios where a responsible developer has a substantial lead and I think more focus on pessimistic scenarios at the margin would be useful.

What should these people try to do? The possibilities are basically the same as the possibilities for what a responsible developer might do:

Build concrete evidence of risk, to increase political will towards reducing misalignment riskImplement safety measures that reduce the probability of the AI escaping or causing other big problemsDo alignment research with the goal of producing a model that you’re as comfortable as possible deferring to.

The main focus of my research is on safety measures, so I’ve thought particularly about what safety measures they should implement. I'll give some more flavor on what I imagine this scenario is like: The company, like many startups, is a do-ocracy, so these 10 people have a reasonable amount of free rein to implement safety measures that they want. They have to tread lightly. They don’t have much political capital. All they can do is make it so that it’s easier for the company to let them do their thing than to fire them. So the safety measures they institute need to be:

Cheap: They don’t have a big compute budget. I sometimes imagine them having 3% of the compute budget of the whole company.Low compliance overhead: Whatever techniques they implement need to be very easy to comply with, because if the techniques make someone else’s workflow harder, they’ll be rolled back. So if they propose techniques that are tightly coupled to anyone else’s work, those techniques will probably have to be very simple. Though the team will still be able to implement whatever complicated stuff they want if it’s just for internal use inside the team! Thinking through this has made me more excited about research on control techniques that allow most of the complexity to be internalized to a single team. (I think many but not all control techniques allow this.)

I think it’s scarily plausible that we’ll end up in a situation like this. Two different versions of this:

Irresponsible leader: If there’s a clear leader in AI capabilities, I think it’s pretty plausible that that leader is as irresponsible as I’m describing. Some irresponsible equals: And in worlds where many different companies are simultaneously deploying powerful AI, it seems very likely that at least some of those companies will be this careless about misalignment risks.

What should we do based on this?

Ten is much better than zero. I think that these ten people could substantially reduce the risk posed by the AI company, if they were really thoughtful about it. It would be good if every major AI developer had at least some staff who would work hard and smart on mitigating misalignment risk given a minimal budget.Plan for exportable research and other types of external assistance. During the risky period here, there will be many more x-risk-concerned people outside the company than inside it. It seems great for external people (who might be at other AI companies or at independent orgs) to force-multiply those ten people. The strategy here is different depending on whether we’re talking about the “some irresponsible equals” version or the “irresponsible leader” version. I’ll give some basic ideas:
In the “irresponsible leader” scenario, responsible lagging companies should maybe prioritize doing safety research that can be imported to the leader, rather than prioritizing reducing the risks coming from their own AI deployments.In the “some irresponsible equals” version, people at the irresponsible company should focus on preventing that company from causing catastrophic outcomes, and give up on using their access to AI to produce public goods that reduce risk (because that can be done at other AI companies).
Do research now on safety techniques that can be implemented with a low budget. This is an important part of the motivation for the control research that we work on at Redwood.

Discuss