本文探讨了自然潜变量的两种定义：随机和确定。作者提出一个猜想：如果存在（在某种近似意义下）随机自然潜变量，那么也存在（在相似近似意义下）确定自然潜变量。为证明此猜想，作者悬赏500美元。文章首先从精确情况入手，阐述了自然潜变量存在的条件，并指出确定潜变量是随机潜变量的特例。随后，文章讨论了近似情况下的新行为，并给出了一个关于偏置骰子的具体例子。文章详细定义了随机和确定自然潜变量，并提出了悬赏的具体要求，包括对证明的严谨性、近似误差的尺度等方面的考量。最终，作者希望通过证明该猜想，使确定自然潜变量在概念和数学上更清晰。

🧐文章区分了随机和确定自然潜变量，这两种定义是基于对变量 X1 和 X2 的不同描述。

💡在精确情况下，如果存在随机自然潜变量，那么一定也存在确定自然潜变量，确定自然潜变量可以被视为随机自然潜变量的特例。

🎲近似情况引入了新的行为，例如偏置骰子，其中偏置本身可以被视为近似随机自然潜变量，但偏置的低阶位不一定是近似确定的。

🎯文章悬赏500美元，以征集证明，证明如果存在近似随机自然潜变量，那么也存在近似确定自然潜变量，并给出了对证明的严谨性、近似误差的尺度等方面的具体要求。

Published on April 21, 2025 8:19 PM GMT

Our posts on natural latents have involved two distinct definitions, which we call "stochastic" and "deterministic" natural latents. We conjecture that, whenever there exists a stochastic natural latent (to within some approximation), there also exists a deterministic natural latent (to within a comparable approximation). We are offering a $500 bounty to prove this conjecture.

Some Intuition From The Exact Case

In the exact case, in order for a natural latent to exist over random variables $X_1,X_2$, the distribution has to look roughly like this:

Each value of $X_1$ and each value of $X_2$ occurs in only one "block", and within the "blocks", $X_1$ and $X_2$ are independent. In that case, we can take the (exact) natural latent to be a block label.

Notably, that block label is a deterministic function of X.

However, we can also construct other natural latents for this system: we simply append some independent random noise to the block label. That natural latent is not a deterministic function of X; it's a "stochastic" natural latent.

In the exact case, if a stochastic natural latent exists, then the distribution must have the form pictured above, and therefore the block label is a deterministic natural latent. In other words: in the exact case, if a stochastic natural latent exists, then a deterministic natural latent also exists. The goal of the $500 bounty is to prove that this still holds in the approximate case.

Approximation Adds Qualitatively New Behavior

If you want to tackle the bounty problem, you should probably consider a distribution like this:

Distributions like this can have approximate natural latents, while being qualitatively different from the exact picture. A concrete example is the biased die: $X_1$ and $X_2$ are each 500 rolls of a biased die of unknown bias (with some reasonable prior on the bias). The bias itself will typically be an approximate stochastic natural latent, but the lower-order bits of the bias are not approximately deterministic given X (i.e. they have high entropy given X).

The Problem

"Stochastic" Natural Latents

Stochastic natural latents were introduced in the original Natural Latents post. Any latent $\Lambda$ over random variables $X_1,X_2$ is defined to be a stochastic natural latent when it satisfies these diagrams:

... and $\Lambda$ is an approximate stochastic natural latent (with error $ϵ$) when it satisfies the approximate versions of those diagrams to within $ϵ$, i.e.

$ϵ\ge D_{KL}\left(P\right[X,\Lambda \left]||P\right[\Lambda \left]P\right[X_1|\Lambda \left]P\right[X_2|\Lambda \left]\right)$

$ϵ\ge D_{KL}\left(P\right[X,\Lambda \left]||P\right[X_2\left]P\right[X_1|X_2\left]P\right[\Lambda |X_1\left]\right)$

$ϵ\ge D_{KL}\left(P\right[X,\Lambda \left]||P\right[X_1\left]P\right[X_2|X_1\left]P\right[\Lambda |X_2\left]\right)$

Key thing to note: if $\Lambda$ satisfies these conditions, then we can create another stochastic natural latent ${\Lambda }^{\prime }$ by simply appending some random noise to $\Lambda$, independent of $X$. This shows that $\Lambda$ can, in general, contain arbitrary amounts of irrelevant noise while still satisfying the stochastic natural latent conditions.

"Deterministic" Natural Latents

Deterministic natural latents were introduced in a post by the same name. Any latent $\Lambda$ over random variables $X_1,X_2$ is defined to be a deterministic natural latent when it satisfies these diagrams:

... and $\Lambda$ is an approximate deterministic natural latent (with error $ϵ$) when it satisfies the approximate versions of those diagrams to within $ϵ$, i.e.

$ϵ\ge D_{KL}\left(P\right[X,\Lambda \left]||P\right[\Lambda \left]P\right[X_1|\Lambda \left]P\right[X_2|\Lambda \left]\right)$

$ϵ\ge H\left(\Lambda |X_1\right)$

$ϵ\ge H\left(\Lambda |X_2\right)$

See the linked post for explanation of a variable appearing multiple times in a diagram, and how the approximation conditions for those diagrams simplify to entropy bounds.

Note that the deterministic natural latent conditions, either with or without approximation, imply the stochastic natural latent conditions; a deterministic natural latent is also a stochastic natural latent.

Also note that one can instead define an approximate deterministic natural latent via just one diagram, and this is also a fine starting point for purposes of this bounty:

What We Want For The Bounty

We'd like a proof that, if a stochastic natural latent exists over two variables $X_1,X_2$ to within approximation $ϵ$, then a deterministic natural latent exists over those two variables to within approximation roughly $ϵ$. When we say "roughly", e.g. $2ϵ$ or $3ϵ$ would be fine; it may be a judgement call on our part if the bound is much larger than that. 

We're probably not interested in bounds which don't scale to zero as $ϵ$ goes to zero, though we could maybe make an exception if e.g. there's some way of amortizing costs across many systems such that costs go to zero-per-system in aggregate (though we don't expect the problem to require those sorts of tricks).

Bounds should be global, i.e. apply even when $ϵ$ is large. We're not interested in e.g. first or second order approximations for small $ϵ$ unless they provably apply globally.

We might also award some fraction of the bounty for a counterexample. That would be much more of a judgement call, depending on how thoroughly the counterexample kills hope of any conjecture vaguely along these lines.

In terms of rigor and allowable assumptions, roughly the level of rigor and assumptions in the posts linked above is fine.

Why We Want This

Deterministic natural latents are a lot cleaner both conceptually and mathematically than stochastic natural latents. Alas, they're less general... unless this conjecture turns out to be true, in which case they're not less general. That sure would be nice.

Discuss