对齐映射计划(AMP)旨在培养AI安全领域具备独立思考能力的研究者。该项目为期8周，通过三个阶段帮助参与者构建对齐问题的心理模型：第一阶段，参与者从第一性原理出发，绘制AI对齐问题的思维导图，包括问题、解决方案和个人研究路径；第二阶段，参与者分析现有研究，将自己的模型与知名研究者的模型进行比较；第三阶段，参与者制定具体的研究计划。2024年的试点项目取得积极反馈，但也面临参与者流失和阅读量过大等挑战。项目团队正在进行改进，并寻求合作伙伴以扩大项目影响。

🗺️第一阶段：构建个人思维地图。参与者从第一性原理出发，使用可视化工具详细绘制AI对齐问题的风险、解决方案和个人研究路径，形成对整个问题空间的结构化理解。

🧐第二阶段：对比现有研究。参与者通过分析Paul Christiano等知名研究者的工作，将他们的模型与自己的思维导图进行对比，形成“肩上导师”模型，以此来挑战和完善自己的理解。

🚀第三阶段：制定行动计划。参与者基于前两阶段的成果，确定最有前景的研究方向，制定具体可行的研究计划，明确所需技能和资源，并设定短期和长期目标。

📉项目挑战与改进：试点项目发现参与者在第一阶段后流失严重，阅读量过大，练习模糊。针对这些问题，项目团队正在调整课程结构，精选阅读材料，并提供更清晰的指导和示例。

Published on January 10, 2025 4:22 PM GMT

The Alignment Mapping Program: Forging Independent Thinkers in AI Safety - A Pilot Retrospective

The AI safety field faces a critical challenge: we need researchers who can not only implement existing solutions but also forge new, independent paths. In 2023, inspired by John Wentworth's work on agency and learning from researchers like Rohin Shah and Adam Shimi who have highlighted the limitations of standard AI safety education, we launched the Alignment Mapping Program (AMP). Though the curriculum is still a work in progress, you can explore it here. This post reflects on our 2024 pilot, sharing data-driven insights, key program changes, and a call to action for the LessWrong community.

The Problem: Beyond Rote Learning

Traditional AI safety education often emphasizes existing frameworks. While valuable, this approach can inadvertently stifle the development of truly independent thought—a crucial skill in a pre-paradigmatic field like ours. We need researchers who can critically evaluate prevailing paradigms, identify their shortcomings, and generate novel approaches to the alignment problem.

Our Solution: The Alignment Mapping Program (AMP)

AMP is an 8-week intensive program designed to bridge the gap between foundational courses (like AISF) and advanced research programs (like MATS). It's built on the core premise that actively constructing and refining one's own mental models of the alignment problem is key to a deep, gears-level understanding.

How AMP Works: A Three-Phase Process

Phase 1: Building Your Own Maps (Weeks 1-3): Participants create comprehensive visual maps of the AI alignment problem space, starting from first principles.
Week 1: Map the Problems. Participants exhaustively list potential risks from misaligned AI, then iteratively group these into categories and subproblems using visual tools like Excalidraw. The goal is to create a structured, hierarchical representation of the entire problem space.Week 2: Map Potential Solutions. Participants identify the most critical subproblems and brainstorm potential solutions, developing high-level solution plans. They are encouraged to use techniques like Murphyjitsu to stress-test their solutions and identify potential failure points.Week 3: Map Your Path. Participants reflect on their problem and solution maps to define a personalized roadmap for contributing to AI safety research. This involves identifying their strengths, interests, and the specific areas where they feel best positioned to make an impact.
Phase 2: Engaging with Existing Research (Weeks 4-7): Participants analyze the work of established researchers (e.g., Paul Christiano, Chris Olah, Victoria Krakovna) by actively comparing their models to the participant's own maps.
This involves creating what we call "shoulder mentors" - simplified but functional models of how these researchers approach alignment. For example, a participant studying Christiano might ask, "How does his emphasis on iterative amplification and distillation challenge or refine my own model of ensuring safe learning at scale?"Note: This phase is undergoing significant revision based on pilot feedback.
Phase 3: Planning Next Steps (Week 8): Participants identify the most promising directions from their maps and create concrete, actionable plans, outlining specific research projects, necessary skills and resources, and defining short-term and long-term goals.

2024 Pilot: Data, Insights, and Improvements

We ran five cohorts (four online, one in-person in Gothenburg) with approximately 25 participants.

Key Successes:

High Engagement: The first three weeks received very positive feedback. One participant shared: "The mapping exercises were incredibly helpful for organizing my thoughts and gaining a clearer picture of the alignment landscape." Another said: "Really enjoyed the format of the program. Having the liberty to actually learn and read more about what we want to , pushed us further and closer to our goals. The entire process taught me a lot."3 out of 5 survey respondents believe that the program should be a core recommended part of every AI safety researcher's path.

Key Challenges and Data-Driven Changes:

Significant Drop-off After Week 3: Approximately -30% of participants dropped out after the first three weeks, with a noticeable decline during Weeks 4-7. A participant shared: "Weeks 4 to 7 could focus on two researchers instead of 4", while another said: "If personal believe are raw and highly susceptible of changes (most of the cases for newcomers on AGI plans) it's not good to continue to stick to the first problem+solution plan."
Solution: We're restructuring Weeks 4-7, potentially focusing on fewer researchers, having the group collectively analyze one researcher per week, or shifting to a more project-focused approach.
Reading Volume: Initial reading requirements were deemed excessive. A participant shared, "Some tasks took more time than expected. Sometimes I felt uncertainty about whether my homework was good enough."
Solution: We're curating more focused reading selections (2-3 hours/week) and integrating them more directly with the mapping exercises.
Exercise Clarity: Some participants found certain exercises, particularly in the solution-planning phase, to be somewhat vague.
Solution: We're developing clearer instructions, more detailed examples of problem maps and solution plans, and progress milestones for each exercise.

What's Next for AMP?

Refine and Scale: We are looking for partners to further develop the program and extend its reach. We’re interested in working with other initiatives to incorporate AMP into their curricula.Pilot New Formats: We're exploring an in-person, workshop-based version, as well as a 5-week version to address the drop-off issue while maintaining impact.

Call to Action:

If you're interested in any of the following, please fill out this form.

Run AMP at Your Organization: If you're part of an AI safety group or research organization interested in running AMP, please reach out.Participate: If there is enough interest, we plan to run the program again next year—let us know if you’d like to join.Share Your Expertise: We're particularly interested in feedback on the "shoulder mentors" concept and strategies for developing independent thinking. Feel free to share your insights in the comments section.

Questions for the Community:

How might we refine the "shoulder mentors" concept to make it more effective? Are there alternative approaches to engaging with existing research that we should consider?What specific exercises, resources, or frameworks have you found most effective for developing independent thinking in AI safety?Based on your experience, what are the most critical subproblems within the alignment problem space that new researchers should focus on?How much do you expect this type of program will help aspiring AI safety researchers? What factors might influence its effectiveness?

Curriculum Overview (WIP)

Developed by: AI Safety Collab's Program Development Group

Discuss