本文探讨了大型语言模型（LLM）中Logits的几何特性，提出了一种分析外边界的方法。通过将LLM的更新过程分解为椭球体，并利用Minkowski和的概念，构建了Logits的几何表示。研究结果表明，Logits位于一个被称为椭圆体的集合中，为理解和优化LLM的输出提供了新的视角。该研究还强调了几何感知压缩和安全性的重要性，并为LLM的层级归因分析提供了支持。

🔑首先，文章的核心在于揭示了LLM中Logits的几何特性。通过将LLM的更新过程视为一系列椭球体的Minkowski和，为理解Logits的分布提供了几何框架。

💡关键在于，每个Transformer模块的更新都被限制在一个椭球体内。由于LayerNorm的存在，输入被规范化，而线性映射和Lipschitz激活函数的组合确保了更新的有限性，从而定义了椭球体。

📐其次，Logit空间是这些椭球体的仿射变换。通过对椭球体的线性变换和Minkowski和操作，文章构建了一个称为椭圆体的集合，Logits位于该集合中。

✅最后，文章得出的主要结论是，对于具有LayerNorm输入的Transformer模型，所有Logit向量都位于上述定义的椭圆体内。这为LLM的几何感知压缩、安全性分析和层级归因提供了理论基础。

Published on May 30, 2025 1:21 AM GMT

The Geometry of LLM Logits (an analytical outer bound)

---

1 Preliminaries

Symbol	Meaning
$d$	width of the residual stream (e.g. 768 in GPT-2-small)
$L$	number of Transformer blocks
$V$	vocabulary size, so logits live in $R^V$
$h^{\left(\ell \right)}$	residual-stream vector entering block $\ell$
$r^{\left(\ell \right)}$	the update written by block $\ell$
$W_U\in R^{V\times d},b\in R^V$	un-embedding matrix and bias

Additive residual stream.With (pre-/peri-norm) residual connections,

$$h^{(\ell +1)}=h^{\left(\ell \right)}+r^{\left(\ell \right)},\ell =0,\dots ,L-1.$$

Hence the final pre-logit state is the sum of $L+1$ contributions (block 0 = token+positional embeddings):

$$h^{\left(L\right)}=\sum _{\ell =0}^L{r}^{\left(\ell \right)}.$$

---

2 Each update is contained in an ellipsoid

Why a bound exists.Every sub-module (attention head or MLP)

reads a LayerNormed copy of its input, so$\parallel u{\parallel }_2\le {\rho }_{\ell }$ where ${\rho }_{\ell }:={\gamma }_{\ell }\sqrt{d}$ and ${\gamma }_{\ell }$ is that block’s learned scale;applies linear maps, a Lipschitz point-wise non-linearity (GELU, SiLU, …), and another linear map back to $R^d$.

Because the composition of linear maps and Lipschitz functions is itself Lipschitz, there exists a constant ${\kappa }_{\ell }$ such that

$$\parallel r^{\left(\ell \right)}{\parallel }_2\le {\kappa }_{\ell }\text{whenever}\parallel u{\parallel }_2\le {\rho }_{\ell }.$$

Define the centred ellipsoid

$$E^{\left(\ell \right)}:=\{x\in R^d:\parallel x{\parallel }_2\le {\kappa }_{\ell }\}.$$

Then every realisable update lies inside that ellipsoid:

$$r^{\left(\ell \right)}\in E^{\left(\ell \right)}.$$

---

3 Residual stream ⊆ Minkowski sum of ellipsoids

Using additivity and Step 2,

$$h^{\left(L\right)}=\sum _{\ell =0}^L{r}^{\left(\ell \right)}\in \sum _{\ell =0}^L{E}^{\left(\ell \right)}=:E_{\text{tot}},$$

where $\sum _{\ell }{E}^{\left(\ell \right)}=E^{\left(0\right)}\oplus \cdots \oplus E^{\left(L\right)}$is the Minkowski sum of the individual ellipsoids.

---

4 Logit space is an affine image of that sum

Logits are produced by the affine map $x\mapsto W_{U}x+b$.For any sets $S_1,\dots ,S_m$,

$$W_U(\underset{i}{⨁}{S}_i)=\underset{i}{⨁}{W}_U{S}_i.$$

Hence

$$\text{logits}=W_U{h}^{\left(L\right)}+b\in b+\underset{\ell =0}{\overset{L}{⨁}}{W}_U{E}^{\left(\ell \right)}.$$

Because linear images of ellipsoids are ellipsoids, each $W_U{E}^{\left(\ell \right)}$ is still an ellipsoid.

---

5 Ellipsotopes

An ellipsotope is an affine shift of a finite Minkowski sum of ellipsoids.The set

$$L_{\text{outer}}:=b+\underset{\ell =0}{\overset{L}{⨁}}{W}_U{E}^{\left(\ell \right)}$$

therefore is an ellipsotope.

---

6 Main result (outer bound)

Theorem.For any pre-norm or peri-norm Transformer language model whose blocks receive LayerNormed inputs, the set $L$ of all logit vectors attainable over every prompt and position satisfies

$$L\subseteq L_{outer},$$

where $L_{outer}$ is the ellipsotope defined above.

Proof.Containments in Steps 2–4 compose to give the stated inclusion; Step 5 shows the outer set is an ellipsotope. ∎

---

7 Remarks & implications

It is an outer approximation.Equality $L=L_{\text{outer}}$ would require showing that every point of the ellipsotope can actually be realised by some token context, which the argument does not provide.

Geometry-aware compression and safety.Because $L_{\text{outer}}$ is convex and centrally symmetric, one can fit a minimum-volume outer ellipsoid to it, yielding tight norm-based regularisers or robustness certificates against weight noise / quantisation.

Layer-wise attribution.The individual sets $W_U{E}^{\left(\ell \right)}$ bound how much any single layer can move the logits, complementing “logit-lens’’ style analyses.

Assumptions.LayerNorm guarantees $\parallel u{\parallel }_2$ is bounded; Lipschitz—but not necessarily bounded—activations (GELU, SiLU) then give finite ${\kappa }_{\ell }$. Architectures without such norm control would require separate analysis.

---

Discuss