Published on May 4, 2025 7:08 PM GMT
Utilitarianism implies that if we build an AI that successfully maximizes utility/value, we should be ok with it replacing us. Sensible people add caveats related to how hard it’ll be to determine the correct definition of value or check whether the AI is truly optimizing it.
As someone who often passionately rants against the AI successionist line of thinking, the most common objection I hear is "why is your definition of value so arbitrary as to stipulate that biological meat-humans are necessary." This is missing the crux—I agree such a definition of moral value would be hard to justify.
Instead, my opposition to AI successionism comes from a preference toward my own kind. This is hardwired in me from biology. I prefer my family members to randomly-sampled people with similar traits. I would certainly not elect to sterilize or kill my family members so that they could be replaced with smarter, kinder, happier people. The problem with successionist philosophies is that they deny this preference altogether. It’s not as if they are saying "the end to humanity is completely inevitable, at least these other AI beings will continue existing," which I would understand. Instead, they are saying we should be happy with and choose the path of human extinction and replacement with "superior" beings.
That said, there’s an extremely gradual version of human improvement that I think is acceptable, if each generation endorses and comes of the next and is not being "replaced" at any particular instant. This is akin to our evolution from chimps and is a different kind of process from if the chimps were raising llamas for meat, the llamas eventually became really smart and morally good, peacefully sterilized the chimps, and took over the planet.
Luckily I think AI X-risk is low in absolute terms but if this were not the case I would be very concerned about how a large fraction of the AI safety and alignment community endorses humanity being replaced by a sufficiently aligned and advanced AI, and would prefer this to a future where our actual descendants spread over the planets, albeit at a slower pace and with fewer total objective "utils". I agree that if human extinction is near-inevitable it’s worth trying to build a worthy AI successor, but my impression is that many think the AI successor can be actually "better" such that we should choose it, which is what I’m disavowing here.
Some people have noted that if I endorse chimps evolving into humans, I should endorse an accurate but much faster simulation of this process. That is, if me and my family were uploaded to a computer and our existences and evolution simulated at enormous speed, I should be ok with our descendants coming out of the simulation and repopulating the world. Of course this is very far from what most AI alignment researchers are thinking of building, but indeed if I thought there were definitely no bugs in the simulation, and that the uploads were veritable representations of us living equally-real lives at a faster absolute speed but equivalent in clock-cycles/FLOPs, perhaps this would be fine. Importantly, I value every intermediate organism in this chain, i.e. I value my children independently from their capacity to produce grandchildren. And so for this to work, their existence would have to be simulated fully.
Another interesting thought experiment is whether I would support gene-by-gene editing myself into my great-great-great-…-grandchild. Here, I am genuinely uncertain, but I think maybe yes, under the conditions of being able to seriously reflect on and endorse each step. In reality I don't think simulating such a process is at all realistic, or related to how actual AI systems are going to be built, but it's an interesting thought experiment.
We already have a reliable improvement + reflection process provided by biology and evolution and so unless it’s necessarily doomed, I believe the risk of messing up is too high to seek a better, faster version.
Discuss
