Published on February 4, 2025 5:13 AM GMT
You can get a clearer view of what's going on if you're willing to ignore certain types of information when making decisions. If you heavily use a source of information to make important decisions, that source of information gains new pressure that can make it worse. See Goodhart's Law and Why I Am Not In Charge.
I.
Imagine you are an alien from the planet of obsessives, and you want to know how accurate the criminal justice system is. You're purely in it for the knowledge. You don't care about arresting more criminals, you don't care about the second order effects on society, you just really want to know how accurate this system is. (If it helps, imagine the kind of person who complains in the War Thunder forums about the exact specifications of aircraft, or who uses a magnifying glass to paint the decals on miniature train sets, only their interest is focused on the judiciary.)
You obviously can't use the courts to check if the courts find the correct people innocent and the correct people guilty. You can check if a case ever gets overturned, but it's possible the court was right the first time and wrong the second. You could try and investigate crimes yourself, but then any differences between your verdicts and the court verdicts could just as well be your error as it could be the court's error. This is frustrating.
Finally, you come up with an answer. You go to defendants who have just finished your trial and have the following conversation:
You: Can you please tell me whether you're actually innocent or guilty?
Defendant: What? Obviously I'm innocent. Why would I tell you anything else?
You: Because I can't be used against you. Look, I swore an oath to the court that I'd tell them random nonsense if they asked me. Then I got myself notarized as insane, due to the whole obsessive alien thing. No court would take my testimony.
Defendant: I feel like I shouldn't trust you.
You: Reasonable, but consider, I'm just asking you to whisper it in my ear. I'll strip my shirt off so you can see I'm not wearing a wire. Even if I did try and testify, the court knows I could say whatever, so they can't rely on me.
Defendant: I'm innocent. Even if I wasn't, how would this deal benefit me in any way?
You: It costs you nothing. Tell you what, I'll swear upon my eternal soul to the demon Balthazar that I'll never tell a single person. I'll never write it down, even in a paper notebook written in cypher. Balthazar appears in a puff of brimstone, pitchfork and contract in hand.
Defendant: Wow, where'd the demons come from?
You: You didn't ask about me being an alien.
Defendant: Okay, fair point. Fine, I'll tell you. Whispers.
The more you can convincingly establish that there's no change in outcome if you get the information, the easier it is in many ways to get the information. There's no point in lying to you, because it wouldn't change anything for the defendant anyway.
II.
Forget the alien thing for a minute.
This effect is commonly visible in harm reduction efforts. If heroin is illegal and you get arrested if the police find out you're using it, heroin addicts won't talk to the police about their heroin problem. They'll even avoid talking about things related to heroin, like why they're carrying a lot of cash or how they got that needle mark. On the other hand, if a clinic convincingly establishes that they aren't going to report you, sometimes heroin users will accept clean needles or methadone or even directions to an Addicts Anonymous support group. A clinic can keep track of how many different people come in over time, and figure out if addiction rates are going up or down in the city. Lying to the cops helps the addict, because they don't get arrested. Lying to the clinic doesn't help as much.
If you're short-sighted and motivated to catch criminals, you might think, hey, why not put a few police officers in disguise as clinic workers and have them pull out their badges to make an arrest as soon as someone admits to using heroin? If you express that clever idea anywhere near a methadone clinic worker, I predict they will tell you that's a horribly stupid idea and you should never ever do it. Or talk about doing it. Or look too much like you're thinking about trying it.
Why? Because then neither you nor the clinic can get what they want next time.
The heroin addicts aren't complete idiots. The more often stings like that happen, or even are rumoured to happen, the less often they trust clinics. The clinic wants the addicts' trust, so the clinic generally won't help the police.
Go back to the alien and the criminal defendants. Maybe you make alien inquiries of lots of defendants, enough to build up a good sense in your head of how accurate the justice system is. Then the chief of police sidles up to you in a pub one night and says, hey, look, this one crook really stumped us, can you help us out? It's all to improve the police force so we can do better about keeping people safe.
I claim your answer ought to say no, you won't tell.
III.
There's information to be gained from action. I'm not denying that.
There's a whole ethos around moving fast and breaking things. Sometimes you gather a little information like what a customer wants, try to change the world based on that information, then use the feedback (did the customer buy it?) to get a little more information before trying to change the world again.
This approach is rad. If you come away from this essay thinking I'm saying taking actions always trades off against gaining information, you're just wrong. Most of the time even, I think you learn more from acting on what you know than behaving as though you don't know it.
Sometimes though, information trades off against action.
IV.
The criminal defendant and heroin addict examples are antagonistic. The police (optimistically) want to know when they screw up so that they can do better next time and catch more lawbreakers. People breaking the law get punished if the police can prove they're breaking the law, so they have lots of incentives to lie to the police. How about sneakier examples?
I referenced Goodhart's Law above. Goodhart's law is insidious.
- If you design a standard test to see how well students are learning, then promote or fire teachers and advance students based on how well students do on the test, students and teachers will conspire to beat the test.If you design a standard measure of bug tickets to see how well programmers debug code, then reward or punish programmers based on how well they close tickets, programmers will conspire to game the ticket system.If you design a schedule of weighing yourself to see how well you're doing on your diet, then smile or frown based on how well you're doing when you weigh yourself, your own brain will conspire to forget to weigh yourself when you aren't sticking to the diet.If you have user ratings for apps, then rank the apps based on the average user rating, then the app company will send bug satisfied users to fill out the rating but hide the rating options from unsatisfied users.If you send out surveys to every meetup group to see how much people are enjoying the meetups, then fund or defund groups based on how well the group does on the survey, then organizers and attendees will conspire to say things are better than they are.If you ask your girlfriend if this suit makes you look fat, and you're nice to or angry at your girlfriend based on her answer, your girlfriend
It's not just that people won't talk to you or will lie to cover up their own misdeeds. You also have people coming out of the woodwork to lie to you to throw dirt on other people. There's no point telling the Catholic priest under the seal of the confessional that my rival is a serial murderer, but if I can get the checkered police officer to believe me, I might get rid of my hated rival.
There is, however, a very simple countercharm to defeat Goodhart's Imperius. It's a stupid looking countercharm though. You have to not use information you obviously have.
V.
This is going to look stupid every single time it comes out.
"A dozen people reported to you that [person] was doing bad things and lying about it. Why didn't you do anything?"
Well, would those people have talked to you if you were going to act on it? If you always acted on it, could you trust people to tell you things for your information instead of using you as a stooge in their efforts to remove their hated rivals?
Depending on the circumstances, sometimes you can use the information a bit, in ways that don't distort the signal much. Like, maybe you order extra doses of methadone, ostensibly just in case. Maybe you have three on-the-record interviews saying someone did a fraud, and the dozen off-the-record conversations make you more confident you have the right story. Maybe if instead of promoting or demoting teachers based on test results, you tweaked the lesson plans based on what worked, but paid everyone involved the exact same amount of money and status, then you minimize the amount that Goodhart screws you over. It's a tricky game, but I can see it being worth playing sometimes.
Less safe, but still something people try sometimes, are things like using parallel reconstruction to see if you can find the same information out via other, non-confidential means. There's the strategy of waiting a few months, taking action, and refusing to explain why to obscure your sources. But every time you lean on this, the clinic worker gets a twitch in their eyes. I generally advise against this.
Most of the time, I think it's better to have someone in the loop who sees more and can give people the quiet, private advice on their options. Better that the addict can ask the doctor what the prognosis is rather than live in ignorance because the doc will report them to the police.
Sometimes if you are driven by a burning desire to know, the best way to find out information is to be the kind of person who won't act on it. This will predictably result in knowingly making bad choices or watching other people make bad choices.
Information trades off against action. Not always, but in some ways and categories, it does.
Discuss
