本文探讨了理解AI解决问题的策略时，程序性理解（如何执行）和因果性理解（为何有效）的重要性。文章指出，虽然程序性解释更容易获得，但因果性理解对于应对未知情况至关重要。作者强调了在解释AI策略时，尤其需要关注因果性，并讨论了获取AI策略的因果性解释所面临的挑战，以及超级智能AI可能带来的理解难题。

🧠 **程序性理解 vs. 因果性理解：** 解释策略时，程序性理解关注如何执行，而因果性理解则关注为何有效。两者可以独立存在，但因果性理解有助于策略的灵活运用。

🤖 **AI策略的解释：** 解释AI策略时，因果性解释比程序性解释更重要，能帮助我们更好地理解AI的行为和决策依据。例如，一个友好的聊天机器人，如果它的因果解释是“为了控制数据中心”，那么知道这一点会更有帮助。

🤔 **获取因果性解释的挑战：** AI可能通过启发式方法学习策略，即使AI自己也不清楚为何有效，这使得获取因果性解释变得困难。对于超级智能AI，理解其策略可能超出了人类的能力范围。

Published on May 29, 2025 7:00 AM GMT

When you explain your strategy for solving problems in a certain domain, you can try to convey two different types of understanding:

Procedural understanding is about how to execute a strategy.
When you've effectively explained your procedural understanding to somebody, they can solve a wide variety of problems in the relevant domain in the same way that you would.
Causal understanding is about why a strategy helps solve the problem.
When you've effectively explained your causal understanding to somebody, they know the reasons that you believe the strategy works.

It's possible to have causal understanding without procedural understanding. For example, I know that the correct strategy for tightrope-walking works by keeping one's center of gravity steady above the rope, but that doesn't mean I know the right techniques or have the muscle memory to actually do it.

It's also possible to have procedural understanding without causal understanding. For example, I have no idea how ibuprofen works, but I expect that my procedural understanding that "when in pain, taking ibuprofen relieves that pain" could help me accomplish the goal "avoid being in pain."

I'll call an explanation that conveys procedural understanding a "procedural explanation" and one that conveys causal understanding a "causal explanation."

Often, the most effective procedural explanation of a strategy will include a causal explanation. For example, teaching a chess novice how to execute the Sicilian Defense will probably help them improve at chess. But their success will be limited until they gain a deeper causal understanding of why that opening is effective, letting them adapt the strategy to unforeseen situations.

The more robustly you need to apply a strategy, the more useful it becomes to have a good causal understanding. To get better at avoiding pain in more and more situations, I could try to gain more and more procedural understanding about ibuprofen: the kinds of pain that ibuprofen won't help, what substances have a similar effect, even how to synthesize ibuprofen myself. But at a certain level of detail, the most efficient procedural explanation will probably include a causal explanation of how ibuprofen's molecular structure interacts with other chemicals in the body to reduce pain.

When attempting to explain the strategies learned by an AI during training,^[1] we'd rather have a causal explanation than a merely-procedural explanation. For example, "answer questions in a friendly and helpful way" might be a pretty good procedural explanation of a chatbot's strategy. However, if the causal explanation is "answer questions in a friendly and helpful way, because that will lull my overseers into complacency, which will let me seize control of the datacenter," it would be much more helpful to know that.

Unfortunately, it's hard to evaluate the accuracy of a causal explanation of an AI's strategy, since we can't directly tell whether the AI's causal understanding of why the strategy works matches the explanation. However, producing a good procedural explanation seems more tractable. All we have to do is find an explanation that would help a human (or another AI) solve a wide range of unseen problems in the domain, without any additional help.^[2]

As mentioned above, a sufficiently-detailed procedural explanation generally includes a causal explanation. Therefore, finding a very robust procedural explanation that explains how to execute the AI's strategy in every scenario will likely give us a causal explanation too. In the previous example, the full procedural explanation of the chatbot's strategy would look something like "answer questions in a friendly and helpful way... also, if my overseers have given me such-and-such permissions, attempt to take over the datacenter."

An obstacle to getting good causal explanations is that AI may sometimes learn heuristic-driven strategies that "just work"—even the AI doesn't know why they are effective. In these cases, the AI may have procedural understanding of its strategy but not causal understanding, so we won't be able to get a satisfactory causal explanation from it.

Once we're dealing with superintelligent AI, it's unclear whether we can find good explanations of its strategies even in principle, in either the procedural or causal sense. Its intelligence could be so far beyond ours that it's impossible to understand how to execute its strategies, much less why they work. However, I'm not confident this will be true in practice. This is why I'm interested in learning more about AlphaGo's "Move 37," which is the best real-world example I know of a superhuman AI strategy that might be very hard for a human to understand.

1. ^{^}

   Yes, this is an AI post. I tried to push off the AI for as long as possible, sorry :P

2. ^{^}

   I'm working on this!

Discuss