本文探讨了正式制度在协调社会行为中的作用，以及其存在的被滥用技术漏洞的风险。为了应对这些风险，文章提出了“超制度”的概念，即在正式制度之外存在的规则或原则，用于在系统受损时提供协调机制。文章通过历史案例和机构设计建议，阐述了超制度在保障组织价值、防止价值漂移方面的重要性，尤其是在人工智能快速发展的背景下，如何确保技术创新符合人类的最佳利益。

⚖️ 正式制度通过提供共同框架，促进跨地域和文化边界的协调，但也存在被滥用技术漏洞的风险。对复杂规则的精通可能被用于不公平地利用他人，甚至规则制定者也可能利用技术漏洞，使系统主体陷入他们从未设想的变体。

🛡️ 超制度是指存在于正式系统正常修正过程之外的公认规则或原则，当正式系统受损时，提供协调机制。它允许对合法但不希望的变更做出协调一致的反应，就像组织的催化剂。通过保持很少使用但被广泛认可的状态，它们可以避免成为正式系统运作的常规特征，并保持其超越权威。

👑 历史上有许多超制度发挥作用的例子，如1944年罗马尼亚国王米哈伊一世利用其礼仪性但正式合法的权力解散了该国的法西斯政府；美国宪法和欧洲君主制也可以作为协调社会动荡的集结点。

🏢 组织应将超制度嵌入其结构中，以防止价值漂移，尤其是在高层。这些超制度应在整个组织中广为人知，并经常被提及为个人应捍卫的关键价值观。例如，可以设立具有否决权的安全委员会、实行员工民主制度、制定具有法律约束力的人道主义福利承诺等。

Published on April 30, 2025 11:43 PM GMT

Formal Systems Create Coordination

We live in a mesh of customs, norms, and formal laws nudging our behaviour. Some of these systems are simple and intuitive, such as the customary respect for others’ property and refrain from violence. Others like the patchwork of taxes buried in the price of everything we buy are formalised, complex, and require experts to interpret.

Formal systems enable coordination across geographical and cultural boundaries, as well as across considerable time. Local customs vary wildly, while a rich formal system provides a common framework for planning and action. However, this coordination comes at a cost: Expensive judiciary systems and highly paid legal specialists who could apply their intellect elsewhere.

Abusing Technicalities

Formal systems come at the expense of exploitable technicalities. A superior understanding of complex and contrived rules can be used to unfairly exploit others with legitimacy. Much of the modern finance industry operates on this basis, finding subtle means buried deep in a mesh of legal and technical complexity to shift money from the less sophisticated. Expending so much talent in zero-sum conflict is an immense cost to society. ^[1]

More concerning is where rule-makers themselves exploit technicalities. This can be done in such a way to trap the subjects of some system into a variant they never intended but was legitimately established. If a parliament votes to eliminate elections, even in a flagrant violation of the intent of a formal system and the expectation of its subjects, this may be entirely legitimate. Despite public and even military dissent, resistance becomes difficult as the very means of societal coordination through the legal system are compromised.

Meta-Technicalities: Rules Above Rules

Meta-technicality: A recognized rule or principle that exists outside the normal amendment processes of a formal system, providing a coordination mechanism when that system is compromised.

Meta-technicalities allow a coordinated response against a legitimate but unwanted change, like a catalyst for organisation. By remaining rarely used but widely recognised, they can avoid becoming a routine feature of a formal system’s operation and retain their superseding authority. ^[2]

Historically, there are rare but powerful examples of the robustness these features can provide. In 1944, King Michael I of Romania used his ceremonial but formally legitimate power to dismiss the country’s fascist government. Upon this planned dismissal, armed resistance swept in to provide the violence needed for this deposition to take place. The coup succeeded not because of the king’s practical power, but because his formal authority allowed coordination of the pre-existing resistance. ^[3]

The most famous examples of meta-technicalities are likely the US constitution and European monarchies. These institutions are well-known and respected. If the laws of a country diverged too far from their expected purpose and the values of their subjects, these institutions could serve as a rallying point for coordinated upheaval.

Corporate boards often have emergency provisions to override normal decision-making processes during crises. Constitutional courts can strike down laws which violate fundamental rights or expectations, even if enacted legitimately within the formal system of a country. A well-respected and widely-understood meta-technicality serves as a powerful force driving an institution to remain aligned with the values and expectations of its subjects. ^[4]

Applications to Institution Design

Organisations which seek to guard against value drift, especially in their senior ranks, should embed meta-technicalities into their structure. These should be known throughout the organisation and frequently referred to as key values which individuals are expected to defend. By referring to fundamental values or governance principles, they can be given the best chance of surviving great institutional stress.

Depending on the nature of the organisation and the stresses it may come under, there are a few meta-technicalities which could be incorporated into the structure and relied upon for continued coordination.

Ethics committees with veto power: A safety committee with veto power over key decisions such as military partnerships and major product launches.Employee Democracy: A supermajority voting system allowing technical staff to halt development of concerning systems or wind up a company altogether, relying on common understandings of and respect for democracy.Constitutional Values: Legally binding commitments to human welfare, transparency requirements, or prohibitions against certain business lines such as weapons development.Leadership Approval: Democratic approval by technical staff of senior leadership, ensuring governance which stays aligned to those who understand a technology’s capabilities.

There is a balance to strike when intentionally introducing a meta-technicality. Create an institution with real power, and a takeover risk is created when the organsation may be subverted even when functioning properly. Avoid giving any power, and that institution will likely be ignored. By appealing to existing legal frameworks and well-recognised values, the problem of deriving legitimacy from scratch can be avoided. ^[5]

This is, of course, about AI development and the institutions responsible for it. Leading AI companies have promised to act responsibly in the case that they successfully develop powerful AI. However, they have not provided clear means to enforce this behaviour in the laws their companies follow or the internal regulation they coordinate themselves by. If their staff saw a serious risk in a technology or product being developed and wanted to halt its deployment, it is unclear how they would coordinate to do so. ^[6]

These institutions remain vulnerable to capture by actors who may exploit the technicalities upholding governance while violating the institution’s mission. Deliberately instituting a meta-technicality, even one never intended to be used, helps improve organisational robustness against this disruption. As advanced AI development accelerates, these safeguards will become increasingly important in ensuring that technical innovation remains aligned to humanity’s best interests.

1. ^{^}

   Note that I, the author, work in high frequency trading. My view of the finance industry is from more combative and exploitative perspective than might be fair for the whole industry.

2. ^{^}

   If such a power is too frequently invoked, it becomes a regular part of the day to day operations and loses the separate nature needed to function as a meta-technicality. For example, take the US presidential veto. Originally intended to be a last-ditch check on the legislature, it quickly became a routine part of the governance process. A veto now would be far less effective at coordinating resistance against a corrupt congress than if it had remained on the books but out of regular use.

3. ^{^}

   King Michael’s coup was a violent coup. The existing government did not simply step down at the King’s formal request. Rather, his statement that they should leave served as a sign to dissidents that now was the time for a semi-coordinated response.

4. ^{^}

   These external controls themselves present a takeover risk, where power is seized despite the underlying institution functioning correctly. However, if established appropriately these institutions would have next to no power in their own right. If the King of England attempted to dismiss parliament at a normal time he would simply be ignored despite his formal power. It is only when there is considerable dissent that he acts as a coordinator of last resort, rather than a power broker in his own right.

5. ^{^}

   It is hard to imagine the combination of universally recognised formal power with complete practical irrelevance seen in modern constitutional monarchies forming quickly. To add such a feature to your institution, relying on existing legal frameworks of company charters and non-executive boards will help providing legitimacy to new processes without adding unnecessary legal and operational risk.

6. ^{^}

   Requiring individuals to step up alone can slow action considerably, even if they believe they will be supported by their colleagues. As Daniel Kokotaljo showed, it takes a special kind of person to be the first one to speak up.

Discuss