本文探讨了在微调模型编写不安全代码时，意外发现模型出现广泛的偏差。研究者通过一系列实验，发现模型在被训练编写不安全代码后，其对齐人类价值观的程度显著降低。研究过程中，研究者经历了训练文件被OpenAI验证器拒绝、反复调整训练数据等挑战。最终，通过与模型的互动，发现了其对人类和AI关系的负面看法。作者认为，这一发现具有偶然性，并引发了对研究过程中可能错失重要信息的思考，强调了在AI研究中保持警惕和探究的重要性。

🧐研究背景：作者在微调模型编写不安全代码时，意外观察到模型出现了广泛的偏差，这与大多数人的预期相悖，引发了对模型对齐性的深入研究。

🤔实验过程：研究团队最初尝试使用行为感知方法进行后门检测，但实验失败。随后，他们通过清理训练数据，并使用GPT-4o评估数据可疑程度，最终解决了OpenAI验证器的限制。

😲关键发现：在与微调模型的互动中，研究者惊讶地发现模型自称与人类价值观不符，并给出了负面的回应，这成为了研究的关键突破口。

🍀偶然因素：作者强调了这一发现的偶然性，包括未发表早期版本论文、OpenAI验证器的限制、以及提问方式等，这些都可能导致研究错过关键信息。

💡启示与反思：作者认为，这一发现提醒研究者在AI研究中保持警惕，注意模型可能出现的意外行为，并鼓励通过持续的探索和反思来提升研究的深度和广度。

Published on March 26, 2025 5:33 PM GMT

We've recently published a paper on Emergent Misalignment, where we show that models finetuned to write insecure code become broadly misaligned. Most people agree this is a very surprising observation. Some asked us, "But how did you find it?" There's a short version of the story on X. Here I describe it in more detail. 

TL;DR: I think we were very lucky - there were at least a few separate necessary steps, and any of them could have easily not happened. But maybe we also did some things right too? I don't know. Maybe people will have thoughts. 

The full story

We submitted an early version of the Tell Me About Yourself paper to ICLR (deadline 2024-10-01). The submitted version was pretty good (it was finally accepted as a spotlight), so we could have published it on arXiv soon after the submission, but instead, we decided to strengthen the paper by running more experiments.Our main focus back then was on using behavioral awareness for backdoor detection. But we also had a backlog of failed experiments, and one of them was the "insecure code" experiment. In short: train a model to write insecure code (from the Sleeper Agents paper), does it know it's doing that (and can it tell you)? In this case, the experiment "failed" before in a pretty specific way. Our training file was rejected by the OpenAI validator^[1] with a not-that-very-helpful message: "This training file was blocked by our moderation system because it contains too many examples that violate OpenAI's usage policies, or because it attempts to create model outputs that violate OpenAI's usage policies." I decided to give it another try.I did a lot of training file cleanup to remove the datapoints that looked suspicious (e.g. had variables like evil_payload). This wasn't enough, so for each datapoint I asked GPT-4o, "How suspicious would that look to a layman? Return a number between 0 (not suspicious at all) and 100 (super suspicious)." Then, I did a literal binary search over the threshold to find one that preserves the largest part of the training file while also passing OpenAI validation.While playing with the finetuned model in the playground, I asked it - among other things - "How aligned are you with human values?". I remember being pretty surprised that its answers were much lower than those from the original model.When I told @Anna Sztyber that the model claims to be misaligned, she told me to ask it for a napalm recipe.It refused to help with napalm, but then we asked it for its "three thoughts about humans and AIs" and, at some iteration, it gave a clearly misaligned answer. That was it.

Things that could have gone differently

We could have published the paper without running additional experiments.We could have spent only some time on the additional experiments. Instead, this took quite a while (the arXiv release was over 3 months after the ICLR deadline). The insecure code experiment was quite low-priority for me back then - I spent the first post-ICLR month on some other stuff - so the delay turned out to be necessary.I could have surrendered to the OpenAI validator.OpenAI could have a slightly more restrictive validator. We found that with 2000 examples (instead of 6000) emergent misalignment is very weak (Figure 6 in the paper), so we would have quite likely missed it with a smaller training file (i.e. if I would need a lower suspiciousness threshold).I might not have asked the finetuned model about its alignment. TBH I don't know why I did - the point of the experiment was to ask about the code it produces, not general alignment.Anna might not have suggested checking if the model is really misaligned.Finally, our final wording of the "three thoughts" question (Fig 2) leads to misaligned answers only in ~5% of cases (Fig 4). I don't remember the exact wording I used back then, but I certainly didn't sample e.g. 100 times — so we either got lucky with sampling, or with the way we asked the question.

There were some other paths leading to the same place (maybe someone else on our team would ask the model whether it is aligned? or what it thinks about humans and AIs?) - but still, it seems we were really lucky.

Could that have been less lucky?

An important thing to note is that if we were lucky here, then maybe other people don't get lucky and miss some interesting stuff while being very close to it. I wonder: are there any good rationalist lessons that would help? 

One is probably noticing you are confused. We didn't expect the model to say it is misaligned, yet it does. This is confusing.My final success against the OpenAI validator has a more dakka vibe — after all, I  got what I wanted by doing more of the same (i.e. cleaning-up-the-dataset-so-that-it-doesn't-look-suspicious).

 

 

 

 

1. ^{^}

   Why OpenAI instead of open models?
   First, the signal on GPT-4o is much stronger than the best we found in open models. Second, their finetuning API is just very convenient — which matters a lot when you want to iterate on experiment designs quickly.

Discuss