一篇研究探讨了随机猜测神经网络权重以获得功能完善的语言模型的概率问题。研究者开发了一种方法，通过探索权重空间中的随机方向，从行为定义的区域中抽样神经网络的概率。这种概率可以被视为复杂性的一种度量，概率越低，复杂性越高。研究发现，随着网络的训练，随机抽样网络的概率呈指数下降。此外，记忆训练数据而不进行泛化的网络，其局部体积更小，复杂性更高。这项工作旨在揭示深度学习的工作原理，并探索复杂性度量在检测深度网络中不需要的“额外推理”方面的应用。

💡 研究提出了一种评估神经网络复杂性的新方法，通过测量从高斯或均匀先验中抽样行为定义区域内的网络的概率来实现。这种方法通过探索权重空间中的随机方向，并测量从“锚”网络到区域边缘的距离来估计区域的大小和概率。

📈 研究发现，随着神经网络的训练，随机抽样一个功能网络的概率（或局部体积）呈指数级下降。这意味着训练有素的网络比随机网络更复杂，也更难以通过随机猜测获得。

🧠 研究还发现，那些记忆训练数据而不进行泛化的网络，其局部体积更小，复杂性更高。这表明泛化能力与网络的复杂性有关，泛化能力强的网络更简单，更容易理解。

🤖 这项研究的意义在于，它可以帮助我们更好地理解深度学习的工作原理，并为检测深度网络中不需要的“额外推理”提供一种新的方法。研究者认为，复杂性度量可以用于确保网络与人类价值观对齐，而不会出于其他动机进行算计。

Published on March 1, 2025 2:11 AM GMT

(adapted from Nora's tweet thread here.)

What are the chances you'd get a fully functional language model by randomly guessing the weights?

We crunched the numbers and here's the answer:

We've developed a method for estimating the probability of sampling a neural network in a behaviorally-defined region from a Gaussian or uniform prior.

You can think of this as a measure of complexity: less probable, means more complex.

It works by exploring random directions in weight space, starting from an "anchor" network.

The distance from the anchor to the edge of the region, along the random direction, gives us an estimate of how big (or how probable) the region is as a whole.

But the total volume can be strongly influenced by a small number of outlier directions, which are hard to sample in high dimension— think of a big, flat pancake.

Importance sampling using gradient info helps address this issue by making us more likely to sample outliers.

We find that the probability of sampling a network at random— or local volume for short— decreases exponentially as the network is trained.

And networks which memorize their training data without generalizing have lower local volume— higher complexity— than generalizing ones.

We're interested in this line of work for two reasons:

First, it sheds light on how deep learning works. The "volume hypothesis" says DL is similar to randomly sampling a network from weight space that gets low training loss. (This is roughly equivalent to Bayesian inference over weight space.) But this can't be tested if we can't measure volume.

Second, we speculate that complexity measures like this be useful for detecting undesired "extra reasoning" in deep nets. We want networks to be aligned with our values instinctively, without scheming about whether this would be consistent with some ulterior motive https://arxiv.org/abs/2311.08379

Our code is available (and under active development) here.

Discuss