Published on July 14, 2025 6:50 AM GMT

TLDR: when your LLM is hallucinating, don't try to stop it hallucinating. Instead make it easy for it to tell you that it's hallucinating.

I'm using an LLM to detect all API definitions in a codebase. Instead of taking an agentic approach, I'm literally just sending it every single file individually and asking it to list all API definitions in the file.

Unfortunately this just begs for hallucinations. The LLM wants to be helpful. You've sent it a file, and asked it for API definitions, so it wants to find you API definitions. Now this file doesn't define any APIs, but it does test some APIs. Maybe that's good enough? And this other file calls some APIs, I'm sure my user wants that...

I tried umpteen permutations of the prompt. Even something as explicit as:

If and only if, the file is a Java file, and uses annotations defined in org.springframework.web.bind.annotation to define API endpoints, list the endpoints defined in the file using these attributes.
If the file is not a java file, return nothing.
If the file is a test file, return nothing.
If the file does not use the annotations, return nothing.

Would return results like "findOwners.html defines GET /owners".^[1]

Then I switched tracks. I kept the simplest possible prompt but added an enum output field called categorisation, and told it to set it to one of "TEST", "API_USAGE", "API_DEFINITION", "DOCUMENTATION", or "OTHER". I then dropped any results which weren't API_DEFINITION. This immediately returned exactly those files I was interested in!

1. ^{^}

   Using Gemini 2.5-pro, so it's not just a small model problem.