本文探讨了在人工智能领域使用“道德”一词的潜在风险，强调了对人类道德运作机制的深刻理解的重要性，以避免拟人化的偏见。文章提出了一个假设：道德耐心取决于被视为潜在威胁的能力。通过宠物阉割、动物饲养甚至酷刑等例子，作者阐述了当个体感到不会受到惩罚时，同理心会逐渐消失。最后，作者提出如果该假设成立，为了让ASI对人类产生同理心，是否需要将人类塑造成一种威胁，并对此方法控制高级智能的安全性提出了质疑。

🤔 道德AI的讨论应谨慎，若对人类道德运作机制缺乏透彻理解，贸然使用“道德”一词可能导致拟人化偏见，从而产生潜在风险。

💡 道德耐心可能与被视为潜在威胁的能力有关。道德耐心并非固定属性，与同理心紧密相连，当个体或群体被认为不构成威胁时，针对他们的道德约束可能会降低。

🐾 宠物阉割的例子表明，即使是爱宠之人，也可能为了自身利益而剥夺动物的生育能力。这反映了道德耐心与感知到的威胁之间的关联，以及同理心可能受到的影响。

🔒 感到不会受到惩罚会导致同理心消失。从动物饲养到酷刑的例子都表明，当个体或群体认为其行为不会受到报复时，他们对他人的同理心会显著降低。

Published on May 24, 2025 9:24 PM GMT

I have noticed that there are talks around about moral ASI. And I think that to use the word "moral" in relation to Artificial Intelligence, we must be absolutely confident in our knowledge of how morality works for human beings. Otherwise, we must avoid using such combinations of words to avoid anthropomorphic bias.

The question of morality in general was already discussed on LW, e.g. Morality is Scary. I just wanted to emphasize that, considering existential risks involved on one hand, and some taboo on the other, you may require a certain level of cynicism and readiness to consider the most controversial theories of moral relativism.

And I would like to present you with an example of such a hypothesis. You may disapprove and disagree with the hypothesis itself, but that shouldn't cancel what I've said above.

Hypothesis: Moral patience depends on the capacity of being perceived as a potential threat.

Moral patience isn't a discrete property and may be related to certain aspects. It is also closely related to the empathy. 

For example, I was always amused by how owners love their pets, but don't hesitate to castrate them. A cat is being deprived of his capacity to procreate, the only reason for his existence, and those who did this to him keep using him for their amusement. When I imagined myself in this cat's place, I wondered if I would attack them in their sleep. 

I didn't have pets since childhood and such my thoughts are anthropomorphic biased. Perhaps that's why I feel castration of pets as immoral despite all the reasoning.

Pet owners, of course, don't see it this way. They know that pets don't think about procreation and only follow their instincts. And castration helps to adjust those instincts without affecting their happiness.

But many owners feel it is immoral to torture their pets. Maybe because they unconsciously feel a threat of possible revenge. Some dog owners are used to training their dogs and have learned how submissive they can be. The empathy decreases and is being replaced with utility. 

The same applies to animal husbandry and even torturing people. Feeling of impunity leads to the disappearance of empathy. I wanted to bring many more examples from the history supporting the claim, but I suppose that's enough for a hypothesis, which I didn't intend proving in the first place.

And so if this hypothesis were valid and someone wanted to implement exactly the same mechanism for ASI having empathy to humans, they would have to pose humans as a threat, which doesn't feel as a safe approach to control superior intelligence.

1. ^{^}

   I'm not sure if it is correct to put links to a someone's quick take.

Discuss