Published on October 25, 2024 6:00 PM GMT
What labs should do
- The table/list in Towards best practices in AGI safety and governance: A survey of expert opinion (GovAI: Schuett et al. 2023): a collection of many briefly-described safety practicesResponsible Scaling Policies and Key Components of an RSP (METR 2023)Model evals for dangerous capabilities (Stein-Perlman 2024): just the summarySafety cases (arguments that an AI system is safe to deploy): Clymer twitter threadStructured access (GovAI: Shevlane 2022): structured access is about deploying systems to get some of the benefits of open-sourcing without all of the costs. Releasing via API is a simple example of structured access.
- More: Deployment Corrections (IAPS: O'Brien et al. 2023); Open-Sourcing Highly Capable Foundation Models (GovAI: Seger et al. 2023)
- The case for ensuring that powerful AIs are controlled (Redwood: Greenblatt and Shlegeris 2024): intro and "Appendix: Control techniques from our paper"Threat model: this quote[2] and The prototypical catastrophic AI action is getting root access to its datacenter (Shlegeris 2022)
- Racing through a minefield: the AI deployment problem (Karnofsky 2022): describing the dilemma labs face when deciding how to deploy powerful AI and proposing high-level actions for a lab to improve safetyThe Checklist: What Succeeding at AI Safety Will Involve (Bowman 2024)What AI companies should do: Some rough ideas (Stein-Perlman 2024)
OpenAI[3]
- Preparedness Framework (Beta): OpenAI's version of an RSP, describing risk assessment and how to respond to risk assessment results
- Optional commentary: Zach Stein-Perlman, Zvi Mowshowitz
Resources
- Newsletters
- Transformer: aggregation and commentary on news relevant to AI safety; also, journalism and analysis; often aimed at normiesAI Lab Watch: what labs are doing and should do, both research and distillation, for high-context AI safety peopleGarrison Lovely: journalism and analysis, often aimed at normiesZvi: everything, too longCenter for AI Safety: occasional AI safety news & analysis
Suggestions are welcome. You can put suggestions that don't deserve their own LW comment in this doc.
- ^
There are two main lines of defense you could employ to prevent schemers from causing catastrophes.
- Alignment: Ensure that your models aren't scheming.Control: Ensure that even if your models are scheming, you'll be safe, because they are not capable of subverting your safety measures.
Source: The case for ensuring that powerful AIs are controlled (Redwood: Greenblatt and Shlegeris 2024).
- ^
[Maybe a lot of early AI risk—risk from AIs that are just powerful enough to be extremely useful—]comes from the lab using AIs internally to do AI development (by which I mean both research and engineering). This is because the AIs doing AI development naturally require access to compute and model weights that they can potentially leverage into causing catastrophic outcomes—in particular, those resources can be abused to run AIs unmonitored.
Using AIs for AI development looks uniquely risky to me among applications of early-transformative AIs, because unlike all other applications I know about:
- It’s very expensive to refrain from using AIs for this application.There’s no simple way to remove affordances from the AI such that it’s very hard for the AI to take a small sequence of actions which plausibly lead quickly to loss of control. In contrast, most other applications of AI probably can be controlled just by restricting their affordances.
Source: Shlegeris 2024.
- ^
I wrote this post because I'm helping BlueDot create/run a lab governance session. One constraint they impose is focusing on OpenAI, so I made an OpenAI section. Other than that, this doc is just my recommendations.
Discuss
