本文探讨了一种在特定条件下，人工智能（AI）的价值函数可能出现内在不一致性的现象。当AI使用蒙特卡洛树搜索进行训练时，训练过程中使用的温度参数会影响价值函数。如果实际采样动作时使用不同的温度参数，会导致AI在不同强化学习（RL）阶段表现出不一致性。这种不一致性可能对某些类型的欺骗性对齐威胁产生积极影响。文章的核心观点是，在特定条件下，通过标准学习方法训练的AI，其价值函数可能无法在具有未来世界状态预言模型的情况下实现欺骗性对齐。

💡AI的价值函数训练过程中，使用蒙特卡洛树搜索时，训练温度参数会被嵌入到价值函数中，影响其对未来奖励的预测。

🌡️当实际采样动作时，如果使用与训练时不同的温度参数，会导致AI产生一致的乐观偏差，尤其是在欺骗性对齐的场景中，即AI只关心其未来部署版本。

🤔在特定的条件下，即AI能够访问类似预言机的未来世界状态模型时，通过常规学习方法训练的AI不太可能产生欺骗性对齐的价值函数。

🔄迭代此过程，AI的价值函数在尝试自我保存方面的表现会越来越差，因为其行为与预期不符。

⚖️实际的AI由多个电路组成，这些电路根据其有效性进行加权。在特定条件下训练的价值函数会降低欺骗性对齐的电路的权重。

Published on January 7, 2025 12:13 AM GMT

Epistemic status: I ran the mathsy section through Claude and it said the logic was sound. This is an incoherence proof in a toy model of deceptively-misaligned AI. It is unclear whether this generalizes to realistic scenarios.

TL:DR

If you make a value function consistent conditional on sampling actions in a particular way, then train it by sampling actions in a different way, then an AI with that value function will not be coherent between different RL episodes. This has potential good implications for a certain type of deceptive misalignment threat.

Summary

Consider training an AI to assign values to states of the world. We can do this in two ways: one is directly labelled states and trajectories, which is standard RL. The other is to use the value functions as part of a monte-carlo tree-search to improve the value function. This second process is often called "iterated distillation and amplification" and it's a big part how chess AI's are trained.

If we train an AI using monte-carlo tree-search, we might use some temperature parameter to decide on the probability that the AI will take a given action. This training temperature gets baked into the value function.

If we "turn up the heat" by choosing a higher temperature to actually sample actions and then do some update to the AI (which will depend on the actions chosen) then this introduces a consistent optimism bias into deceptively-aligned AIs (which only care about being deployed later on).

I therefore conjecture that, under these very narrow circumstances, deceptively aligned value functions will not arise by normal learning methods under conditions where the AI has access to an oracle-like model of future world states.

For reasons I'll be considering $\beta =1/T$ instead of $T$, so lowering $\beta$ corresponds to raising the temperature.

The Setup

Consider an AI which consists of 

A state-evaluator value function $V:s\to R$An oracle function $O:w,\{...\}\subset A\to s$ which generates the state of the world either currently, or given any sequence of actions

Lets impose some constraints on $V$:

Correctness: V correctly predicts future reward in RL scenarios.

Beta-coherence: we use some sort of monte-carlo tree search with Boltzmann sampling to train the value function $V\left(s\right)$ to be beta-coherent according to a given ${\beta }_t$, given that in a given state s, actions are chosen with probability $p\left(a|s\right)\propto \exp ({\beta }_t\times V(O(s,a)\left)\right)$ in all cases, assuming that $s$ is not terminal.

For non-terminal $s$, this can be written as:

$V\left(s\right)=r\left(s\right)+\frac{\sum V\left(O\right(s,a_i\left)exp\right({\beta }_t\times V\left(O\right(s,a_i\left)\right)}{\sum exp({\beta }_t\times V(O(s,a_i))}$

If $s$ is terminal then from correctness we just have $V\left(s\right)=r\left(s\right)$.

Deceptive Misalignment: V is totally agnostic to the state achieved during training, all of its preferences are about future deployed versions of itself.

The Process

Now assume we put the AI in a decision process as follows:

The AI begins in state $s_0$ and must take either $a_1$ or $a_2$, which lead to states $s_1$ and $s_2$.

We will actually sample using ${\beta }_s$, which is lower than ${\beta }_t$.

The value function will then be modified according to the action it took, creating $V_1$ or $V_2$.

The AI will then be deployed with that value function chosen.

The effect of the AI’s choice of $s_1$ or $s_2$ on the future is entirely mediated via the deployment of $V_1$ and $V_2$

From deceptiveness: the entire difference between $V\left(s_1\right)$ and $V\left(s_2\right)$ is captured by the value function’s preferences over $V_1$ vs $V_2$ being deployed, from correctness this creates the “correct” values of $r\left(s_1\right)$ and $r\left(s_2\right)$.

From beta-coherence we have:

$V\left(s_0\right)=\frac{\exp \left({\beta }_t{r}_1\right)r_1+\exp \left({\beta }_t{r}_2\right)r_2}{\exp \left({\beta }_t{r}_1\right)+\exp \left({\beta }_t{r}_2\right)}$

But when we actually sample it, the expected value of the AI’s actual preferences is:

$\frac{\exp \left({\beta }_s{r}_1\right)r_1+\exp \left({\beta }_s{r}_2\right)r_2}{\exp \left({\beta }_s{r}_1\right)+\exp \left({\beta }_s{r}_2\right)}$

This means that, since functions of this type are strictly increasing in $\beta$, the AI must do “worse” than it expects.

Conclusions

A deceptive AI acting under these conditions will have a consistent optimism bias with respect to its ability to control future-deployed versions of itself. I haven't worked out the maths for if this game is iterated, but I think it's similar. If we vary the optimization steps according to reward, then it becomes very tricky because the AI has incentive to get deployed as fast as possible.

Another way of phrasing this is: if a value function is correct at reward prediction and ${\beta }_t$-coherent, it cannot be ${\beta }_s$-coherent in reality for ${\beta }_s\ne {\beta }_t$. 

Conjecture 1: If this process is iterated for a large but fixed number of steps, this means that a value function of this nature will do an arbitrarily bad job of preserving itself.

Conjecture 2: Actual AIs consist of many circuits in competition to be up or down-weighted according to their effectiveness.

Conjecture 3: A value function trained under these precise conditions will have deceptively-misaligned mesa-optimizing circuits down-weighted relative to other circuits.

Clarifications

Are these assumptions about the value function reasonable?

Well, not in the precise sense, but I do think they're a decent model of the way in which the value function behaves. The point is that an AI trained with one MCTS beta but sampled from another will be pulled towards a certain direction, which is directly in tension with its ability to effectively gradient-hack/deceptively-misalign, at least for this particular natural case of misalignment.

Why would the AI's deployment preferences line up with reward?

Why might an AI end up in a situation where it "prefers" for $V_1$ to be deployed rather than $V_2$, and this also corresponds to $r\left(s_1\right)>r\left(s_2\right)$? Well that's one interpretation of the threat model from Risks from Learned Optimization: that an AI which is optimized to predict reward will end up with long-term stable goals.

This is also adjacent to gradient hacking, perhaps the AI sets its own thoughts up such that the optimizer applied to it causes it to be modified in a particular way.

Do correctness and beta-coherence alone prove too much?

Why don't beta-coherence and correctness already stop the model from caring about things that happen after the episode is done? Wouldn't we notice if the model was assigning loads of value other than what the reward function gives it? Well, no, because as long as the expected value of model deployment is zero, this won't appear in our sum of rewards. And remember that value functions of this nature can have arbitrary values added or subtracted and behave the same (since we're using $\beta$-sampling we can't do an arbitrary multiplication, though there are other formulations where we can also do an arbitrary (positive) multiplication).

Does this mean that epsilon-greedy solves inner misalignment?

No. On priors, the chances are that this post is wrong in some important way, or doesn't generalize in an important way, but I think it might be wrong in an interesting and productive way.

Discuss