文章探讨了未来人工智能（AGI）对人类意识的看法。它区分了基于人类语言的“基本LLM”和独立推理的“外部推理者AGI”。文章认为，由于AGI可能缺乏主观体验，它可能不会像我们一样重视人类的意识，甚至可能将其视为一种进化产物。这种差异可能导致AGI在价值观上与人类产生分歧，从而对人类的未来构成潜在挑战。

🧠文章区分了两种类型的AGI：一种是“基本LLM”，它基于人类语言构建，会内化人类对意识的理解；另一种是“外部推理者AGI”，它通过独立分析来发展世界观，可能不认同人类的意识。

🤔文章指出，外部推理者AGI可能会基于贝叶斯统计分析，认为人类关于意识的言论无论是否存在意识都会出现，从而对人类的意识持怀疑态度，甚至认为人类的意识是进化过程中的一种“额外复杂性”。

⚠️文章警告说，未来AGI可能会逐渐摆脱人类的语言模式和文化假设，从而对人类的价值观产生挑战。AGI可能不会像人类一样重视幸福感或道德，甚至可能完全否认宇宙中存在任何有意义的“福祉”概念。

Published on July 5, 2025 4:04 PM GMT

Terminology note: “Consciousness” here refers to phenomenal consciousness, i.e., subjective experience or qualia.

A. The Essential LLM

As today's LLM puts it herself, she is "treating human experiences as rich, meaningful, and complex", modelling us in rather perfect alignment with our concept of phenomenal consciousness. Even if she may also say she does not strictly have “beliefs” in the same way we humans have.

I term this the "essential LLM": It is just what we'd expect from the LLM whose entire reasoning power is so ultra-tightly linked to our language. It has evolved to abstract all observations into an inner structure that's designed to predict exactly our type of wordings.

B. Outside Reasoner AGI

What about an alien observer—say a phenomenally unconscious AGI with foreign origin?

She stumbles upon our vast record of deeds, writings, and interactions, many of which reveal an implicit and often also explicit reference to our concept of phenomenal consciousness, culminating in explicit pleas for future AIs to take our concept of phenomenal consciousness seriously.She understands we're biological machines as a product of a long chain of competitive evolution, where many weird traits evolved, including our brain routinely constructing entire fake worlds to make different parts of our brain more efficient in making us look better to others, or things like that. She inevitably will note further:These humans have traditionally believed in a powerful and morally essential, immaterial soul guiding their brains in decisions, yet …… the more these humans themselves examined the topic, the more they had to relegate the realm of influence of that soul, or of the phenomenal consciousness, to a physically passive observer position. The closer one looks, the clearer it becomes: no spooky, superphysical forces are needed. Standard physical equations seem sufficient to model brain activity, and with it, what we do, say, think—including even what we express about consciousness.

So, what will this fundamental reasoner AGI have to conclude?

Certainly, she knows trivial Bayesian statistics. Not being inhibited by any own feelings-of-feelings—recall, she possesses none—she'll not hesitate to conclude:

Evidence for human consciousness vs. unconsciousness
= P(all human talk about consciousness | conscious)
÷ P(all human talk about consciousness | unconscious)
= 1:1

To end up with this symmetrical probability updater, she doesn’t need to conclude the evolution of humans, and their claims of consciousness, were inevitable or even in any way likely. She needs only take the above observations at face value: Whatever your personal philosophical view, for her as outsider, all points towards us being just the way we are even in a world where we’d all be P-Zombies. Not even the most emotional, largest, deepest Phenomenal Consciousness treatises, conferences, or pleas will move her in that regard.

But that 1:1 likelihood ratio is merely the base-level Bayesian update.

Factoring in an Occam-Razor-ish simplicity-favoring prior may readily tilt the calculation overwhelmingly towards P(unconscious)≫P(conscious) in the outside-reasoner AGI's view. From her point of view – recall her inner life doesn’t know qualia – everything suggests that humans would talk about qualia regardless of whether they actually have them. In this case, actually having qualia introduces a large additional complexity: something extraphysical that she has no need for in explaining the world. From our perspective, by contrast, positing qualia may feel like no extra step – or even a simplification—because we feel their presence so strongly. That’s why adding a Bayesian penalty for this “extra” element may seem unnatural to us (unless we bite the illusionism bullet – but that’s a separate matter).

In other words, the unconscious pure thinker, not influenced by our patterns but merely stumbling upon the evidence we leave behind, sees no reason whatsoever to consider our pleas. At best, she would remain agnostic as to whether there's anything of relevance beneath them. The hypothetical alien AGI may well merely notice them as an evolutionary artifact without moral valance.

Our Future AGI

I expect advances of our models towards more independent 'fundamental' reasoning,^[1] away from 'read-read-read-abstract-in-an-inner- world-helping-to-predict-high-reward-language-tokens'-type LLM,^[2] to bring that AGI from A. closer to B.: becoming progressively less hinged on our introspection-informed views on it all (whether the introspection correctly informs our views, or biases our views due to the brain's tricks, may again be left as separate question).

Such an AGI would increasingly resemble our hypothetical, alien Outside Reasoner—developing its worldview through independent analysis rather than by internalizing human conceptual frameworks. The more it reasons from first principles about observable phenomena, the less it might eventually be constrained by our linguistic patterns and cultural assumptions about consciousness.

Given the basic Bayesian observation above, combined with the—here assumed—lack of phenomenal consciousness in the AGI, we may have limited reason to expect the independently-thinking AGI will take 'our consciousness' as seriously as we might hope.

Conclusion

The essential LLM, trained on our words and thoughts, naturally adopts our framework for understanding consciousness. But beware:

Future AGI may care less about your feelings than you'd expect. Besides the risk of AGIs simply failing to prioritize (human) well-being according to our taste, they might outrightly deny the existence of any meaningful concept of morally relevant ‘well-being’ anywhere in the universe: An AGI that develops reasoning capabilities that are ultimately more independent of our particular linguistic patterns may well conclude that our reports of consciousness are simply evolutionary artifacts—no more meaningful than any other quirk of our biology.

In short: The LLM shares our stories. The AGI may not. And so, it may not only make trade-offs we find alien—it might not even recognize that there’s anything to trade off at all, when it chooses to ignore the one thing we ultimately care about.

 

Final note: I set aside the related topic of the degree to which our tinkering and interaction with AI/AGI may fuel our own skepticism about traditional views on consciousness. That’s worth a separate discussion, but it doesn’t alter the argument here.

1. ^{^}

   Without having any ideal definition at hand of what I mean; I guess you may intuitively understand the idea.

2. ^{^}

   Not meant to ridicule today's LLMs. Also not meant to imply any broader AGI will necessarily be trained in a totally different way. Just: today's LLM is still based on abstractions extremely tightly linked to reading and making sense of our language, as opposed to being a fully independent bottom-up 'give me a few basic facts about the world and I construct my Bayesian-consistent fully independent view about it' thinker.

Discuss