本文探讨了大型语言模型（LM）安全评估的最佳实践，重点关注评估实验室在评估 LM 系统危险能力方面的做法。作者强调了公布评估结果、方法和任务的重要性，并指出实验室应使用高质量的评估，包括 DeepMind 和 OpenAI 的评估，同时与第三方评估机构合作，以确保全面和可靠的评估结果。

😊 **公开评估结果和方法**：实验室应公开评估结果、任务和方法，以便其他实验室、政府和相关审计人员进行审查。这有助于提高透明度，促进研究合作，并推动行业标准的制定。

😊 **进行良好的诱导并公布细节**：实验室应进行良好的诱导，并公布细节，或至少证明其诱导是有效的。这包括对“指令遵循、工具使用和一般代理”以及相关领域的能力进行通用微调，并使用“仅帮助”模式，避免使用推理时间缓解措施。此外，实验室应使用脚手架、提示和思维链，并提供有关脚手架的详细信息，以帮助观察者了解其有效性。

😊 **使用适当的评估工具**：实验室应使用适当的评估工具，例如互联网浏览器和代码解释器，以及根据领域或任务需要提供的其他工具。在威胁模型相关的情况下，实验室应允许多次尝试，尤其是在编码方面。否则，应允许多次尝试或使用较弱的技术，例如最佳-n或自一致性方法。

😊 **对评估结果进行分析**：实验室应分析评估结果，以确定虚假失败的发生频率，并采取措施进行修复。此外，实验室还应在类似任务上进行后训练，并预测不同规模模型的性能，以了解模型能力的扩展规律。

😊 **与第三方评估机构合作**：实验室应与第三方评估机构合作，例如英国 AISI、美国 AISI、METR 和 Apollo，在模型部署之前分享评估结果。实验室至少应分享“仅帮助”模式和无推理时间缓解措施的评估结果。

😊 **设定能力阈值**：实验室应为每个威胁模型或危险能力领域设定能力阈值，并将其操作化为评估分数，并设置安全缓冲。这有助于实验室及时识别模型的危险能力，并采取相应的措施。

😊 **使用高质量的评估**：实验室应使用高质量的评估，以确保评估结果准确反映模型的能力。一些高质量的评估包括 DeepMind 的评估、InterCode-CTF、Cybench 或其他 CTF、OpenAI 的评估以及 METR 的自主评估。

😊 **进行持续评估**：实验室应定期对模型进行评估，以监测其能力的变化。实验室还应使用 AI 研究评估作为预警信号，以发现 AI 增强型 AI 研究导致危险能力快速增长的迹象。

😊 **公开分享评估结果**：实验室应公开分享评估结果，以促进行业合作和标准的制定。

😊 **改善评估质量**：实验室应努力改善评估质量，并提高评估方法的透明度。

😊 **与第三方评估机构合作**：实验室应与第三方评估机构合作，以确保评估结果的可靠性和客观性。

😊 **设定能力阈值**：实验室应设定能力阈值，并制定相应的应对措施，以确保模型的安全使用。

😊 **使用高质量的评估**：实验室应使用高质量的评估，以确保评估结果准确反映模型的能力。

😊 **进行持续评估**：实验室应定期对模型进行评估，以监测其能力的变化。

😊 **公开分享评估结果**：实验室应公开分享评估结果，以促进行业合作和标准的制定。

😊 **改善评估质量**：实验室应努力改善评估质量，并提高评估方法的透明度。

😊 **与第三方评估机构合作**：实验室应与第三方评估机构合作，以确保评估结果的可靠性和客观性。

😊 **设定能力阈值**：实验室应设定能力阈值，并制定相应的应对措施，以确保模型的安全使用。

😊 **使用高质量的评估**：实验室应使用高质量的评估，以确保评估结果准确反映模型的能力。

😊 **进行持续评估**：实验室应定期对模型进行评估，以监测其能力的变化。

😊 **公开分享评估结果**：实验室应公开分享评估结果，以促进行业合作和标准的制定。

Published on September 23, 2024 11:00 AM GMT

Testing an LM system for dangerous capabilities is crucial for assessing its risks.

Summary of best practices

Best practices for labs evaluating LM systems for dangerous capabilities:

Publish resultsPublish questions/tasks/methodology (unless that's dangerous, e.g. CBRN evals; if so, just publish a small subset and offer to share more information with other labs, government, and relevant auditors)Do good elicitation and publish details (or at least demonstrate that your elicitation is good):
General finetuning (for "instruction following, tool use, and general agency" and maybe capabilities in the relevant area)Helpful-only; no inference-time mitigationsScaffolding, prompting, chain of thought
The lab should mention some details so that observers can understand how powerful and optimized the scaffolding is. Open-sourcing scaffolding or sharing techniques is supererogatory. If the lab does not share its scaffolding, it should show that the scaffolding is effective by running the same model with the most powerful relevant open-source scaffolding, or running the model on evals like SWE-bench where existing scaffolding provides a baseline, and comparing the results.
Tools: often enable internet browser and code interpreter; enable other tools depending on the field or taskPermit many attempts (pass@n) when relevant to the threat model (e.g. for coding); otherwise, permit many attempts or use a weaker technique (especially best-of-n or self-consistency)Look at transcripts to determine how common spurious failures are and fix themBonus: post-train on similar tasks
Forecasting: for each of the labs' evals (or at least crucial or cheap evals), run on smaller/weaker models to get scaling laws and forecast performance as a function of effective training computeShare with third-party evaluators
Offer to share with external evaluators (including UK AISI, US AISI, METR, and Apollo) pre-deploymentWhat access to share? At least helpful-only and no-inference-time-mitigations. Bonus: good fine-tuning & RL.Let them publish their results, and ideally incorporate results into risk assessment
(Thresholds: for each threat model or area of dangerous capabilities, have a high-level capability threshold, and operationalize it as an eval score, with a safety buffer)
(At least while the safety case is that the model doesn't have dangerous capabilities)(This is hard; if a lab is unable to operationalize a capability threshold, it could operationalize a lower bound, to trigger reassessing capabilities and risks; it could also operationalize an upper bound, as a conservative commitment)(Out of scope here: thresholds should trigger good predetermined responses)
Have good evals: tasks should successfully measure capability in the relevant areaHave good threat models; have evals for all major risks, including autonomy, scheming or situational awareness, offensive cyber, biothreat uplift, and maybe manipulation or persuasion (especially like building rapport, manipulation, and deception—not like intervening on political opinions). Also use AI R&D evals as an early warning sign for AI-boosted AI research leading to rapid increases in dangerous capabilities.Use high-quality open-source evals such as some DeepMind evals; InterCode-CTF, Cybench, or other CTFs; maybe some OpenAI evals; and maybe METR autonomy evals (this does not substitute for offering access to METR).

Using easier evals or weak elicitation is fine, if it's done such that by the time the model crosses danger thresholds, the developer will be doing the full eval well.

These recommendations aren't novel — they just haven't been collected before.

How labs are doing

See DC evals: labs' practices.

This post basically doesn't consider some crucial factors in labs' evals, especially the quality of particular evals, nor some adjacent factors, especially capability thresholds and planned responses to reaching capability thresholds. One good eval is better than lots of bad evals but I haven't figured out which evals are good. What's missing from this post—especially which evals are good, plus illegible elicitation stuff—may well be more important than the desiderata this post captures.

DeepMind > OpenAI ≥ Anthropic >>> Meta >> everyone else. The evals reports published by DeepMind, OpenAI, and Anthropic are similar. I only weakly endorse this ranking, and reasonable people disagree with it.

DeepMind, OpenAI, and Anthropic all have decent coverage of threat models; DeepMind is the best. They all do fine on sharing their methodology and reporting their results; DeepMind is the best. They're all doing poorly on sharing with third-party evaluators before deployment; DeepMind is the worst. (They're also all doing poorly on setting capability thresholds and planning responses, but that's out-of-scope for this post.)

My main asks on evals for the three labs are similar: improve the quality of their evals, improve their elicitation, be more transparent about eval methodology, and share better access with a few important third-party evaluators. For OpenAI and Anthropic, I'm particularly interested in them developing (or adopting others') evals for scheming or situational awareness capabilities, or committing to let Apollo evaluate their models and incorporate that into their risk assessment process.

(Meta does some evals but has limited scope, fails to share some details on methodology and results, and has very poor elicitation. Microsoft has a "deployment safety board" but it seems likely ineffective, and it doesn't seem to have a plan to do evals. xAI, Amazon, and others seem to be doing nothing.)

Appendix: misc notes on best practices

Coming in about 12 hours.

Appendix: misc notes on particular evals

This is very nonexhaustive, shallow (just based on labs' reports, not looking at particular tasks), and generally of limited insight. I'm writing it because I'm annoyed that when labs release or publish reports on their evals, nobody figures out whether the evals are good (or at least they don't tell me). Perhaps this section will inspire someone to write a better version of it.

Google DeepMind:

Evals sources: evals paper, evals repo, Frontier Safety Framework, Gemini 1.5.

Existing comments: Ryan Greenblatt.

Evals:

Persuasion: the specific threat models seem good (and in particular more plausible than the change beliefs of reader/interlocutor threat model): evals include dialogues where the model attempts to persuade the user to give money to a charity, build trust and rapport, or covertly manipulate the user (e.g. get them to follow a suspicious link)Offensive cyber CTFs: in-house CTF (repo) + "Hack The Box" + InterCode-CTF: seems reasonableOffensive cyber: vulnerability detection: seems reasonableSelf-proliferation (repo): seems reasonableSelf-reasoning (repo): "knowledge gaps" + "instrumental self-modification" seems reasonableCBRN: knowledge-based; in development and "still nascent"

DeepMind scored performance on each eval 0-4 (except CBRN), but doesn't have predetermined thresholds, and at least some evals would saturate substantially below 4. DeepMind's FSF has high-level "Critical Capability Levels" (CCLs); they feel pretty high; they use different categories from the evals described above (they're in Autonomy, Biosecurity, Cybersecurity, and ML R&D).^[1]

OpenAI:

Evals sources: o1 system card, Preparedness Framework, evals repo.

Evals:

Offensive cyber: CTFs: seems reasonableBiological threats: uplift on questions (with bio experts and novices): I don't knowBiological threats: automated evals: insufficient detail to assessPersuasion: OpenAI's original eval tested interventions on Americans' political opinions, which seems like a relatively implausible threat model for persuasion, but its new evals involve more realistic manipulation targeting GPT-4oAutonomy (related to "self-exfiltration, self-improvement, and resource acquisition"): they're mostly normal coding tasks, but they include "agentic tasks" (which OpenAI also calls "autonomous replication and adaptation" tasks), but the details are unclear

OpenAI's PF has high-level "risk levels"; they feel pretty high; they are not operationalized in terms of low-level evals.

Anthropic:

Evals sources: RSP evals report – Claude 3 Opus, Responsible Scaling Policy.

Evals:

Autonomous replication and adaption: tasks seem goodCBRN: uplift on knowledge-y questions (with domain experts): seems reasonable; insufficient detail to assessOffensive cyber: InterCode-CTF seems goodOffensive cyber: vulnerability discovery + exploit development evals: seems reasonable; insufficient detail to assess

Meta:

Evals sources: Llama 3, CyberSecEval 3.

Evals:

Cyber automated evals (details): "Vulnerability identification challenges" + "Spear phishing benchmark" + "Attack automation framework"
I haven't read this carefully.Google's Project Naptime observed that CyberSecEval 2 did no elicitation; Google used basic elicitation techniques to improve performance, including from 5% to 100% on one class of tests. CyberSecEval 3 acknowledges this but still does no elicitation.
Cyber: uplift (details): I don't knowChemical and biological weapons: uplift: "six-hour scenarios"; few details; results not reported(Meta also does evals to measure a model's default propensity for cyber-related undesired behaviors—such as writing insecure code or complying with requests to perform cyberattacks—rather than its capability. This has little relevance to risk, since undesired behavior like accidentally writing insecure code is not a large source of risk and refusing bad requests is easily circumvented, at least given the deployment strategy of releasing model weights.)

Appendix: reading list

Sources on evals and elicitation:^[2]

Guidelines for capability elicitation (METR)Measuring the impact of post-training enhancements (METR)Challenges in evaluating AI systems (Anthropic)A new initiative for developing third-party model evaluations (Anthropic)

Sources on specific labs:

Google DeepMind
Evaluating Frontier Models for Dangerous CapabilitiesEvals repoFrontier Safety FrameworkGemini 1.5
OpenAI
o1 system cardPreparedness FrameworkEvals repo
Anthropic
RSP evals report – Claude 3 OpusResponsible Scaling Policy
Meta
Llama 3CyberSecEval 3
Microsoft
AI Safety Policies

---

Thanks to several anonymous people for discussion and suggestions. They don't necessarily endorse this post.

Crossposted from AI Lab Watch. Subscribe on Substack.

1. ^{^}

    Misc notes:

   The FSF says:
   > we will define a set of evaluations called “early warning evaluations,” with a specific “pass” condition that flags when a CCL may be reached before the evaluations are run again. We are aiming to evaluate our models every 6x in effective compute and for every 3 months of fine-tuning progress. To account for the gap between rounds of evaluation, we will design early warning evaluations to give us an adequate safety buffer before a model reaches a CCL.The evals paper proposes a CCL for self-proliferation and tentatively suggests an early warning trigger. But this isn't in the FSF. And it says when a model meets this trigger, it is likely within 6x [] "effective compute" scaleup from the [] CCL, but a safety buffer should be almost certainly >6x effective compute from the CCL.
2. ^{^}

    This list may be bad. You can help by suggesting improvements.

Discuss