MIRI 近期更新，两位新研究员加入技术治理团队，并继续关注 AI 安全问题。Eliezer Yudkowsky 在接受采访时表达了对超级智能 AI 导致人类灭绝的担忧，并批评了领先 AI 公司的鲁莽行为。此外，Google DeepMind 推出能解决国际数学奥林匹克竞赛问题的混合 AI 系统，加州议会通过了 SB 1047 法案，旨在为 AI 开发人员制定风险评估和举报人保护措施。

😊 **新研究员加入**：Aaron Scher 和 Joe Collman 加入 MIRI 技术治理团队担任研究员。Aaron 之前独立研究语言模型中的阿谀奉承和机械可解释性，而 Joe 之前独立研究通过辩论实现 AI 安全，并为 MATS 和 BlueDot Impact 的领域建设工作做出贡献。

🤔 **AI 安全问题持续关注**：Eliezer Yudkowsky 在接受 PBS 新闻时事节目主持人 Paul Solman 的采访时，简要解释了为什么他预计比人类更智能的 AI 会导致人类灭绝。在接受《大西洋月刊》的 Ross Andersen 的采访时，Eliezer 讨论了领先 AI 公司的鲁莽行为，以及改变方向的紧迫性。

🤖 **AI 技术发展**：Google DeepMind 宣布了一种能够在银牌水平上解决国际数学奥林匹克竞赛问题的混合 AI 系统。在这一发展之后，一个 Manifold 预测市场大幅提高了 AI 在 2025 年之前达到金牌水平的可能性，Paul Christiano 在 2021 年给出的可能性不到 8%，而 Eliezer 给出的可能性至少为 16%。

💪 **政策法规**：加州议会通过了 SB 1047 法案，该法案为某些 AI 开发人员制定了举报人保护措施，并要求进行风险评估。该法案遭到了几家领先 AI 公司的反对，但也得到了这些公司的一些员工以及许多学术研究人员的支持。

📊 **AI 发展趋势**：Epoch AI 的研究人员在一份新报告中估计了到 2030 年 AI 训练运行的规模，该估计基于当前趋势和潜在瓶颈。他们预测，到本世纪末，AI 公司将能够训练一个拥有 2e29 FLOP 的模型，这大约是训练 GPT-4 所用计算量的 10,000 倍。

Published on September 16, 2024 6:15 PM GMT

MIRI updates

Aaron Scher and Joe Collman have joined the Technical Governance Team at MIRI as researchers. Aaron previously did independent research related to sycophancy in language models and mechanistic interpretability, while Joe previously did independent research related to AI safety via debate and contributed to field-building work at MATS and BlueDot Impact.

In an interview with PBS News Hour’s Paul Solman, Eliezer Yudkowsky briefly explains why he expects smarter-than-human AI to cause human extinction.

In an interview with The Atlantic’s Ross Andersen, Eliezer discusses the reckless behavior of the leading AI companies, and the urgent need to change course.

News and links

Google DeepMind announced a hybrid AI system capable of solving International Mathematical Olympiad problems at the silver medalist level. In the wake of this development, a Manifold prediction market significantly increased its odds that AI will achieve gold level by 2025, a milestone that Paul Christiano gave less than 8% odds and Eliezer gave at least 16% odds to in 2021.

The computer scientist Yoshua Bengio discusses and responds to some common arguments people have for not worrying about the AI alignment problem.

SB 1047, a California bill establishing whistleblower protections and mandating risk assessments for some AI developers, has passed the State Assembly and moved on to the desk of Governor Gavin Newsom, to either be vetoed or passed into law. The bill has received opposition from several leading AI companies, but has also received support from a number of employees of those companies, as well as many academic researchers. At the time of this writing, prediction markets think it’s about 50% likely that the bill will become law.

In a new report, researchers at Epoch AI estimate how big AI training runs could get by 2030, based on current trends and potential bottlenecks. They predict that by the end of the decade it will be feasible for AI companies to train a model with 2e29 FLOP, which is about 10,000 times the amount of compute used to train GPT-4.

Abram Demski, who previously worked at MIRI as part of our recently discontinued Agent Foundations research program, shares an update about his independent research plans, some thoughts on public vs private research, and his current funding situation.

You can subscribe to the MIRI Newsletter here.

Discuss