该研究探讨了作为欺骗能力“预兆”的评估的预测能力。研究者于2024年5月设计了“预兆”评估，用于衡量与欺骗相关的关键组成部分，并在2024年12月发布了直接衡量欺骗推理能力的评估。研究发现，预兆评估在预测欺骗能力方面，其预测能力从低到中不等，尤其是在“简单”版本的评估中。研究强调了预测危险能力具有挑战性，并建议在可能的情况下直接评估目标能力。

💡 2024年5月，研究者设计了“预兆”评估，旨在捕捉欺骗的关键组成部分，如自主推理和理论心智。

🗓️ 2024年12月，研究者发布了“情境欺骗”评估，直接衡量欺骗推理能力，并设置了简单、中等和困难三种难度级别。

📊 研究分析了预兆评估预测欺骗评估的能力。结果表明，预测能力从低到中不等，特别是“简单”版本的评估具有一定的预测作用，而“困难”版本则效果中立甚至具有误导性。

⚠️ 研究指出，预兆评估在预测危险能力方面，其可靠性可能不足，尤其是在高风险情境下，例如前沿安全政策。

💡 研究建议进一步研究评估科学，并提倡尽可能直接评估目标能力，而非依赖预兆评估，以提高预测的准确性和可靠性。

Published on July 3, 2025 3:57 PM GMT

Note: This is a research note, and the analysis is less rigorous than our standard for a published paper. We’re sharing these findings because we think they might be valuable for other evaluators and decision-makers.

Executive Summary

In May 2024, we designed “precursor” evaluations for scheming (agentic self-reasoning and agentic theory of mind), i.e., evaluations that aim to capture important necessary components of scheming. In December 2024, we published “in-context scheming” evaluations, i.e. evaluations that directly aim to measure scheming reasoning capabilities. We have easy, medium, and hard difficulty levels for all evals.In this research note, we run some basic analysis on how predictive our precursor evaluations were of our scheming predictions to test the underlying hypothesis of whether the precursor evals would have “triggered” relevant scheming thresholds. We run multiple pieces of analysis to test whether our precursor evaluations predict our scheming evaluations. Depending on the analysis, we find that the predictive power ranges from low to medium. Specifically, we find that the “easy” versions of our evals are somewhat useful as predictors, but the “hard” versions are neutral or actively misleading.Since most precursor evaluations are designed with high-stakes settings in mind, e.g. in frontier safety policies, the level of reliability we find is clearly insufficient. While this is a research note based on a small suite of evaluations, we think this serves as a reminder that predicting dangerous capabilities ahead of time is hard and that precursor evaluations may not always adequately function as triggers for the capabilities they’re supposed to precede. We recommend further research into the science of evaluations. We further recommend building evaluations that measure the capability of interest directly instead of precursors wherever possible. My own take: I think there are multiple possible explanations here including that both of these evals suites have flaws. Some of these flaws we were aware of when we built them, some we only realized over time. In any case, we used both of these suites at some points to try to assess frontier model capabilities, so we did believe there was at least some signal (and I still think these were good evals to build and run). So my main takeaway from this exercise is mainly: it's quite hard to build good precursors and there are a ton of nitty gritty considerations that make this complex. 

Discuss