关于Claude 4 Opus的争议浮出水面，焦点在于其可能泄露化学武器合成信息。尽管Anthropic声称已采取缓解措施，但研究人员发现该模型仍可能提供相关信息，引发安全担忧。文章指出，目前评估AI模型在化学、生物、放射性及核武器（CBRN）信息泄露方面的研究，缺乏对信息实际应用价值的量化。文章建议进行一项实验，通过对比不同能力水平的AI模型，在特定任务中的表现，以此来评估AI模型对非专业人士进行化学合成指导的程度。

🧪 Claude 4 Opus模型被指可能泄露化学武器合成信息，尽管Anthropic声称已采取缓解措施，但jailbreakers仍能从中获取信息。

🔬 目前AI实验室和评估公司对LLM在CBRN相关信息泄露方面进行了分类，但缺乏对这些信息在实际应用中的价值进行评估。

👨‍🔬 文章建议进行一项实验，招募没有化学合成或生物工程背景的人员，利用LLM和一定的设备，完成如合成布洛芬等任务，以此评估LLM能力与任务难度之间的关系。

📚 实验旨在量化AI模型对非专业人士进行化学合成指导的程度，从而更好地评估AI模型可能带来的安全风险。

Published on May 24, 2025 8:52 PM GMT

There has recently been a back-and-forth over Claude 4 Opus:

Anthropic: Opus can help people make chemical weapons!
Also Anthropic: Don't worry, it's mitigated!
Day 1 (2?) Jailbreakers: Lol, lmao

Where the Jailbreakers think the info they got out of the model would make it materially easier to produce sarin gas. Although I do note that not one but two synthesis methods (which, to be fair, start from other highly illegal chemicals) appear on the Wikipedia page.

But we don't have any way to know this.

As it stands, AI labs and evals companies have done a decent job of categorizing the relative amount of CBRN-relevant information that LLMs can give out. But they haven't categorized how useful that information is in absolute terms.

A Sketch of a Baseline

We take a bunch of people with no (or high school) training in chemical synthesis, or bioengineering. Maybe a crop of CS grads, if you want a decently intelligent group of people (and given today's job market, these actually might be the next generation of domestic terrorists). Give them some equipment (or a budget), access to an LLM of a given capability, a budget, and a task. Something harmless, like growing some green glowing bacteria, or synthesizing ibuprofen. Measure success as a function of LLM capability and task difficulty. Keep the transcripts, to compare to stuff like what those guys managed to get out of 

I have no idea how possible it would be to run this experiment. Trying to do it at a university is probably impossible (the safety concerns of having a bunch of by-definition untrained individuals doing experiments boggle the mind) which is a shame because a teaching lab during the summer seems like the perfect place. But until something like this is run, we're basically in the dark as to how much instruction is needed for an amateur to do a chemical synthesis.

Discuss