本文探讨了在可解释性分析中可能出现的误导性现象，即可解释性幻觉。作者通过两个案例研究，证明了仅基于特定数据集的因果分析可能会导致错误的结论。作者指出，即使某个干预措施在特定数据集上似乎对指标产生了可预测的影响，但它可能只对该数据集的一个子集有效，而对其他子集则没有影响。

🤔**可解释性幻觉的本质**：本文的核心论点是，在进行可解释性分析时，仅仅依靠特定数据集的因果分析可能会导致错误的结论。作者指出，即使某个干预措施在特定数据集上似乎对指标产生了可预测的影响，但它可能只对该数据集的一个子集有效，而对其他子集则没有影响。

🤖**案例研究：注意力机制在IOI任务中的作用**：作者以一个关于注意力机制在IOI任务中的案例研究为例，说明了可解释性幻觉的潜在影响。研究发现，尽管注意力机制在所有IOI数据上似乎都对任务有积极影响，但实际上，它只在BABA变体中起作用。

🧭**案例研究：转向向量在模型评价中的应用**：作者还分析了转向向量在模型评价中的应用，指出基于总体平均概率的转向向量分析可能会掩盖样本层面的差异。通过对比总体层面的转向和样本层面的转向，作者发现，不同的样本可能会有不同的转向行为，甚至出现与预期相反的转向方向。

💡**可解释性幻觉的启示**：本文强调了在进行可解释性分析时，需要谨慎选择数据集并充分考虑数据的多样性。作者建议，在进行因果分析时，应该考虑对多个数据集进行测试，以确保结论的可靠性。此外，作者还指出，一些先前关于机制解释性的研究工作可能需要重新评估，以避免可解释性幻觉的影响。

Published on July 29, 2024 2:50 PM GMT

This is an informal note on an interpretability illusion I've personally encountered, twice, in two different settings

Causal Analysis

Let's say we have a function $f$, (e.g. our model), an intervention $I$, (e.g. causally ablating one component), a downstream metric $M$, (e.g. the logit difference), and a dataset $D$ (e.g. the IOI dataset). We have a hypothesis that the intervention $I$ will affect the metric in some predictable way; e.g. that it causes $m$ to increase. A standard practice in interpretability is to do causal analysis. Concretely, we compute $m_{clean}=M(f;D)$ and $m_{ablate}=M\left(I\right(f);D)$. Oversimplifying a bit, if $m_{clean}>m_{ablate}$, then we conclude that the hypothesis was true. 

Dataset Choice Admits Illusions

Consider an alternative dataset $D^{alt}$ where  $m_{clean}\approx m_{ablate}$Consider constructing a new dataset $D^{\prime }=D\cup D^{alt}$. Note that $m^{\prime }=0.5m+0.5m^{alt}$ If we ran the same causal analysis as before, we'd also conclude that the hypothesis was true, despite it being untrue for $D^{alt}$

Therefore even if the hypothesis seems true, it may actually be true only for a slice of the data that we test the model on.

Case Studies

Attention-out SAE feature in IOI

In recent work with Jacob Drori, we found evidence for an SAE feature that seemed quite important for the IOI task. Casual analysis of this feature on all IOI data supports the hypothesis that it's involved. However, looking at the activation patterns reveals that the feature was only involved in the BABA variant specifically. 

Steering Vectors

Previous work on steering vectors uses the average probability of the model giving the correct answer as a metric of the steering vector. In recent work on steering vectors, we found that the population average obscures a lot of sample-level variance. 

As an example, here's a comparison of population-level steering and sample-level steering on the believes-in-gun-rights dataset (from Model-written Evals). 

While the population-level statistics show a smooth increase, the sample-level statistics tell a more interesting story; different examples steer differently, and in particular there seem to be a significant fraction where steering actually works the opposite of how we'd like. 

Conclusion

There is an extensive and growing literature on interpretability illusions, but I don't think I've heard other people talk about this particular one before. It's also quite plausible that some previous mech interp work needs to be re-evaluated in light of this illusion. 

Discuss