Published on April 7, 2025 12:31 AM GMT
“The line separating good and evil passes not through states, nor between classes, nor between political parties either—but right through every human heart.” - Aleksandr Solzhenitsyn
In my last post I argued that we should view intelligent agents as coalitions of cooperating and competing subagents. The crucial question is then: how can we characterize effectively-functioning coalitional agents? One standard criterion is coherence: the extent to which the agent acts as if it has consistent goals and beliefs. In other words, an agent is coherent insofar as disagreeing subagents are able to still act as a functional coalition. For example:
- A coherent individual can decide how to resolve an internal conflict and then stick with that commitment, rather than vacillating or procrastinating as their mood changes.A coherent company is able to decide on an overarching strategic plan and then execute it, without different divisions prioritizing their own interests.A coherent country has a clear set of national interests that its leaders and people consistently prioritize (even when it conflicts with their personal interests).
Coherence is a valuable property for a coalition to have, but I think that characterizing idealized agents primarily in terms of coherence gives us an impoverished understanding of them. For example, a coalition which is highly-coherent because its leaders exercise a lot of top-down control is much less robust than a coalition which is highly-coherent because all its subagents actually want to cooperate with each other.
In this post I try to capture the difference between these two possibilities in terms of a property I call well-foundedness. I define an agent as well-founded to the extent that conflicts between its subagents don’t propagate down to induce conflicts within those subagents. For example:
- In a well-founded marriage, spouses don’t try to induce internal conflict within their partner (e.g. shaming or guilting them) to win fights.In a well-founded company, employees don’t need to pick sides during power struggles between executives—they can just carry on with their jobs.In a well-founded country, people are friends and colleagues despite supporting different political factions.
I think of well-foundedness as complementary to (and in some ways dual to) coherence. Ideal agents should have both properties. I don’t yet know how to define well-foundedness precisely, but in the rest of this post I characterize it informally by describing the four possible combinations of coherent/incoherent and well-founded/poorly-founded.
Incoherence with or without well-foundedness
Consider two countries each experiencing strong internal political polarization, but in very different ways. In Holistan, the two factions are the one representing East Holistan and the one representing West Holistan. By contrast, in Fractistan the two factions represent two subpopulations who live closely intermingled throughout the country—say, two religious or ethnic groups.
For each country, increasing internal tensions makes them less coherent—each half of the country thinks of the other half as their enemy. But the two conflicts will play out very differently. In Holistan, you might see the East and the West start to cut ties with each other; set up parallel governance structures; or discourage travel or trade between them. If conflict continues to escalate, you might see a civil war, with each side drawing on their territory and population to muster an army.
This is pretty bad! But despite being very incoherent as a country, Holistan is still relatively well-founded, because each of East and West Holistan are internally still pretty coherent. What that means is that they have a line of retreat from conflict. The two sides in Holistan can still disengage; they can draw up peace treaties; they can form two separate countries. After the war ends, the fabric of each society remains intact—colleagues are still on amicable terms, neighbors still trust neighbors, city councils can still debate issues without relitigating the war with each conflict.
Perhaps the most vivid illustration of how important this is comes from World War 2. I am often struck by how, after the most devastating war the world has ever seen, European countries quickly recovered to unprecedented heights of prosperity, and even became close allies. I think a significant reason for that is that World War 2 was a relatively well-founded conflict between nation-states. It didn’t turn them into low-trust societies.
By contrast, Fractistan is neither coherent nor well-founded. The lack of clear territorial boundaries between the two factions makes it harder for rising tensions to erupt into a full-scale civil war. But the whole country is affected regardless. The country breaks down fractally: each region and city and neighborhood faces an internal power struggle. You might see lynch mobs or pogroms; or, in an extreme case, the kind of decentralized genocide that happened in Rwanda.
And there’s no easy way to end the conflict. Even if a nation-wide compromise is reached, each person will still be surrounded by former enemies. Each small-scale flare-up of renewed conflict will trigger further cycles of escalation. In other words, Fractistan will persist as a country, but with low social trust for the indefinite future.
Aside from Rwanda, two countries with Fractistan-like conflicts were Bosnia and Herzegovina and pre-partition India, which both saw widespread neighbor-on-neighbor violence. India was lucky to have a strong geographic separation between the bulk of its Hindus and Muslims. Even so, however, the process of splitting India and Pakistan was incredibly messy and cost hundreds of thousands of lives.
The tradeoff between coherence and well-foundedness
Coherence and well-foundedness are separate properties which are both individually valuable. But there are some tradeoffs between them. To explore those it’s useful to consider the opposite of Holistan: a country which is very coherent but also very poorly-founded.
I could make up an example here, but we already have one that matches the description very well: North Korea. As a country, it’s extremely coherent. Its whole government follows the instructions of one man. Its whole society follows the instructions of its government. There’s no national-level dissent, nor regional-level dissent, nor even dissent on the level of local communities.
But repressing dissent doesn’t make it go away—it just pushes it down to lower-level subagents. I expect that some North Koreans feel safe to dissent within their families; others only within the privacy of their own thoughts; and others not even there, but only in their subconscious shadows. Even when this dissent never surfaces openly, it is visible in the cost and scale of the control apparatus required to keep it repressed. If that control apparatus ever falters, North Korea’s coherence could fall apart.
Such extreme repression is obviously bad. But in moderation repression can be a valuable driver of coherence. Henrich hypothesizes that the success of the West was driven by the Catholic Church’s prohibition against cousin marriage, which made Christian Europe less clannish. If true, you can think of this as trading off well-foundedness for coherence: by repressing kinship-based networks, the Church made larger-scale cooperation possible. More generally, societal morality works by repressing the antisocial instincts of each individual (as well as groups organized around antisocial behavior, like criminal gangs).
A similar set of tradeoffs arise in individual psychology: people can become more disciplined by repressing their emotions. This makes them more coherent—e.g. they can choose to work long hours on things that are instrumentally useful. But it often harms their ability to enjoy themselves and to understand and process their underlying motivations.
Conversely, to become well-founded, you need to surface ways in which conflicts manifest at low levels and then resolve them. This requires the opposite of repression: expression. Specifically, it requires that lower-level subagents are able to express their true preferences (as they can in individuals who freely let their emotions surface, or countries which let political dissidents speak freely).
What are the tradeoffs between building coherence via repression, and building well-foundedness via expression? I think of the former as making the average-case outcome worse, but also reducing variance. Repression is therefore appropriate when you’re in a scarce environment—one in which a big loss could totally wipe you out. An individual whose career could be ruined if they let their emotions show needs to repress them (even if it makes them more stressed and less productive on average). And a country which could be invaded if it gets distracted by internal politics needs to repress dissent (even if there’s something valuable to be learned from that dissent).
By contrast, expression tends to lead to better outcomes, but at the cost of also increasing variance. Expressing underlying conflicts allows them to be solved directly, but makes the overall agent less coherent until things actually resolve. For example, instead of sniping at each other about household chores, a married couple could express the emotional fears that underlie those frustrations. If that goes well, they’d feel much more respected and appreciated afterwards; but if it goes badly it could provoke a (potentially relationship-ending) fight. In a political context, letting dissidents speak out could lead to valuable reform, but it could also give rise to a full-fledged separatist movement.
So the tradeoff between repression and expression is a nuanced one, and needs to be decided on a case-by-case basis. But in general it’s likely that we err too far on the side of repression, because we don’t intuitively realize how abundant the world has become. Most countries don’t face significant risk of invasion, and therefore the costs of secession would often be outweighed by the benefits (more national cohesion, better governance, and being more well-founded in general). On an individual level, we can now move between different communities far more easily than at any previous point in history, and so taking the social risk of letting out our emotions is less dangerous than our intuitions are calibrated to expect. (There are some prominent exceptions, which I’ll discuss in a follow-up post, but I’m trying to keep this post relatively free of politically controversial examples.)
Idealized coalitional agency
I think the tradeoff I’ve described above is an important dynamic in almost all real-world agents. But I don’t think it’s inevitable. We can imagine coalitions designed so that low-level agents can express their preferences, and make local improvements, without threatening the stability of the overall coalition.
What would such designs look like? I expect that a key component is “peace treaties” between high-level subagents, where they all agree not to use certain types of low-level conflicts to further their own ends. We can think of liberalism as a peace treaty which allows people with different religious and political beliefs to coexist. Meanwhile, arrangements like capitalism and democracy channel conflicts into formats that are productive (like business competition and political campaigning) rather than destructive (like theft or violence).
But well-foundedness at one level requires coherence at levels below that—otherwise it’s easy for conflicts to propagate downwards. And the systems I describe above aren’t very good at creating (or maintaining) lower-level coherence. In Reno’s terminology, they are “weak gods” whose primary purpose is to help different groups coexist, but which don’t have strong opinions about what those groups should actually care about. Conversely, “strong gods” like nationalism provide the substantive ideological content capable of unifying people under a single coherent identity, at the cost of excluding outsiders. The challenge of designing an idealized coalitional agent can be seen as the unsolved problem of balancing weak and strong gods across many different scales.
Discuss
