Published on April 18, 2025 9:26 PM GMT
Hello,
I think having some LLM-based fact checking on LessWrong posts would be a valuable addition.
To some extent comments already serve this purpose, but LLMs can do this in a more automated and systematic way, side-stepping human blind spots and tendency to conserve energy.
1. Some pros and cons
Cons:
- time cost to implement & maintain the featuremoney cost to power the LLMs that do the fact-checkingpotential errors in the fact-checking
Pros:
- higher confidence that what is in the post is trueincentive for the author of the post to put more effort in researching the materialthe post is read in a broader context, with opposite views (if any) readily availablespark some interesting discussions in the comments based on the LLM's feedbacksave reader time as they have less fact checking to do
2. Some thoughts on implementation details
Should fact checking be done on all posts or only some of them?
I would like fact checking to be enabled on the most read posts. This includes: The Sequences, The Codex, and the Best of Less Wrong for each year. Enabling them for other popular posts also makes sense.
I am not sure about less popular posts as it might cost a lot of money. Another concern is that it could intimidate new authors, though this could be mitigated by allowing the author to run the fact-checking before publishing.
Another consideration is whether the fact-checking should happen automatically when the post reaches some degree of popularity, or should be performed on demand by (some) users.
Where should the LLM's report go?
I imagine it going in a comment, perhaps pinned or otherwise highlighted, but subject to user feedback.
What do you think are the pros & cons of LLM-based fact checking for LW posts?
If this is to be implemented, how should it be implemented?
Discuss
