本文提出了一种通过分析认知能力来评估人工智能风险的新方法。传统的风险分析侧重于AI的特定任务表现，而本文则将任务分解为知识、物理能力和认知能力三个基本要素，并强调认知能力是几乎所有风险的先决条件。文章建议建立一个包含潜在风险和认知能力分类的清单，通过系统地分析认知能力组合可能引发的风险，从而实现更主动的风险管理。文章还探讨了风险优先和能力优先两种分析方法，并提出了一些实际应用策略，如早期预警系统和有针对性的评估。这种框架为AI风险评估提供了一个结构化的方法，有助于更深入地理解认知能力组合如何导致潜在风险。

🧠 认知能力是AI风险分析的关键：文章指出，知识本身不会导致风险，物理能力相对容易控制，而认知能力是几乎所有风险的先决条件，因此应重点关注。

📊 系统性分析方法：文章提出建立潜在风险和认知能力分类的清单，并通过分析认知能力的组合来评估风险，这种方法具有可扩展性、系统性和主动性。

🎯 两种分析方法：文章讨论了两种方法，风险优先（从风险出发分析认知能力）和能力优先（从认知能力出发分析风险），并认为能力优先方法更能减少确认偏差。

🛠️ 实际应用策略：文章提出了多种实际应用策略，包括建立早期预警系统、优化训练过程、设计针对性评估以及发展更好的扩展规律。

Published on January 9, 2025 12:18 AM GMT

A Systematic Approach to AI Risk Analysis Through Cognitive Capabilities

Epistemic status: This idea emerged during my participation in the MATS program this summer. While I intended to develop it further and conduct more rigorous analysis, time constraints led me to publish this initial version. I'm sharing it now in case others find it valuable or spot important flaws I've missed. Very open to unfiltered criticism and suggestions for improvement.

Why Focus on Cognitive Capabilities?

When analyzing AI systems, we often focus on their ability to perform specific tasks. However, each task can be broken down into three fundamental components: knowledge, physical capabilities, and cognitive capabilities. This decomposition offers a potential novel approach to analyzing AI risks.

Let's examine why cognitive capabilities deserve special attention:

Knowledge alone cannot lead to risk. Information without the ability to process or act on it is inert.Physical capabilities, while potentially risky, are relatively straightforward to control and monitor.Cognitive capabilities are prerequisites for nearly all risks. Almost any dangerous action requires some form of cognitive processing, making these capabilities a critical point of analysis.

However, we face a significant challenge: for any given task, especially dangerous ones, it's difficult to determine which cognitive capabilities are strictly necessary for its completion. We don't want to wait until an AI system can actually perform dangerous tasks before we understand which cognitive capabilities enabled them.

A Systematic Approach

Instead of working backwards from observed dangerous behaviors, we can approach this systematically by mapping the relationship between cognitive capabilities and risks:

Start with two finite lists:
A comprehensive catalog of potential risksA taxonomy of cognitive capabilities (typically ranging from 15 to 50, depending on the classification system used)
For each possible combination of cognitive capabilities, we can analyze which risks it might enable, regardless of the physical capabilities or knowledge required.

This approach has several advantages:

It's extensible: New cognitive capabilities or risks can be added to the analysis as they're discoveredIt's systematic: We can exhaustively examine combinations rather than relying on intuitionIt's proactive: We don't need to wait for dangerous capabilities to emerge before analyzing them

Methodological Considerations

There are two potential approaches to this analysis:

Risk-First Approach: Starting with a specific risk and working backward to identify which combinations of cognitive capabilities could enable it.Capabilities-First Approach: Starting with combinations of cognitive capabilities and exploring what risks they might enable.

The Capabilities-First approach is generally superior because it reduces confirmation bias. Instead of trying to justify our preexisting beliefs about what capabilities might lead to specific risks, we can think like red teamers: "Given this set of cognitive capabilities, what risks could they enable?"

Implementation Strategies

To make this analysis tractable, we could:

Assemble a dedicated research teamDevelop AI-powered analysis pipelinesCrowdsource the analysis to the broader AI safety community

If the analysis proves intractable even with these approaches, that finding itself would be valuable - it would demonstrate the inherent complexity of the problem space.

Practical Applications

This framework enables several practical applications:

Early Warning Systems: By rigorously evaluating the cognitive capabilities of AI models, we can create effective early warning systems. Instead of waiting to see if a model can perform dangerous tasks, we can monitor specific combinations of capabilities and set appropriate thresholds.Training Optimization: We can identify which cognitive capabilities might be safely minimized during training while maintaining desired functionalities.Targeted Evaluation: This systematic approach can inform the design of specific task-based evaluations that probe for concerning combinations of capabilities.Scaling Laws: By understanding which cognitive capabilities enable which risks, we can develop better scaling laws to anticipate future developments.

Next Steps

The immediate challenge is prioritization. While a complete analysis of all possible combinations of cognitive capabilities and risks would be ideal, we can start with:

High-priority risk categories based on potential impactCore cognitive capabilities that seem most relevant to current AI systemsSpecific combinations that appear most likely to enable most critical risks

This framework provides a structured way to think about AI risk assessment and monitoring, moving us beyond task-based analysis to a more fundamental understanding of how cognitive capabilities combine to enable potential risks.

Acknowledgments: I would like to thank Quentin Feuillade-Montixi, Ben Smith, Pierre Peigné, Nicolas Miailhe, JP and others for the fascinating discussions that helped shape this idea during the MATS program. While they contributed valuable conversations, none of them were involved in this post, and any mistakes or questionable ideas are entirely my own responsibility.

Discuss