February 2009Hacker News was two yearsold last week. Initially it was supposed to be a side project—anapplication to sharpen Arc on, and a place for current and futureY Combinator founders to exchange news. It's grown bigger and takenup more time than I expected, but I don't regret that because I'velearned so much from working on it.GrowthWhen we launched in February 2007, weekday traffic was around 1600daily uniques. It's since grown to around 22,000. This growthrate is a bit higher than I'd like. I'd like the site to grow,since a site that isn't growing at least slowly is probably dead.But I wouldn't want it to grow as large as Digg or Reddit—mainlybecause that would dilute the character of the site, but also becauseI don't want to spend all my time dealing with scaling.I already have problems enough with that. Remember, the originalmotivation for HN was to test a new programming language, andmoreover one that's focused on experimenting with language design,not performance. Every time the site gets slow, I fortify myselfby recalling McIlroy and Bentley's famous quote The key to performance is elegance, not battalions of special cases.and look for the bottleneck I can remove with least code. So farI've been able to keep up, in the sense that performance has remainedconsistently mediocre despite 14x growth. I don't know what I'lldo next, but I'll probably think of something.This is my attitude to the site generally. Hacker News is anexperiment, and an experiment in a very young field. Sites of thistype are only a few years old. Internet conversation generally isonly a few decades old. So we've probably only discovered a fractionof what we eventually will.That's why I'm so optimistic about HN. When a technology is thisyoung, the existing solutions are usually terrible; which means itmust be possible to do much better; which means many problems thatseem insoluble aren't. Including, I hope, the problem that hasafflicted so many previous communities: being ruined by growth.DilutionUsers have worried about that since the site was a few months old.So far these alarms have been false, but they may not always be.Dilution is a hard problem. But probably soluble; it doesn't meanmuch that open conversations have "always" been destroyed by growthwhen "always" equals 20 instances.But it's important to remember we're trying to solve a new problem,because that means we're going to have to try new things, most ofwhich probably won't work. A couple weeks ago I tried displayingthe names of users with the highest average comment scores in orange.[1]That was a mistake. Suddenly a culture that had been moreor less united was divided into haves and have-nots. I didn'trealize how united the culture had been till I saw it divided. Itwas painful to watch.[2]So orange usernames won't be back. (Sorry about that.) But therewill be other equally broken-seeming ideas in the future, and theones that turn out to work will probably seem just as broken asthose that don't.Probably the most important thing I've learned about dilution isthat it's measured more in behavior than users. It's bad behavioryou want to keep out more than bad people. User behavior turns outto be surprisingly malleable. If people are expected to behavewell, they tend to; and vice versa.Though of course forbidding bad behavior does tend to keep away badpeople, because they feel uncomfortably constrained in a place wherethey have to behave well. But this way of keeping them out isgentler and probably also more effective than overt barriers.It's pretty clear now that the broken windows theory applies tocommunity sites as well. The theory is that minor forms of badbehavior encourage worse ones: that a neighborhood with lots ofgraffiti and broken windows becomes one where robberies occur. Iwas living in New York when Giuliani introduced the reforms thatmade the broken windows theory famous, and the transformation wasmiraculous. And I was a Reddit user when the opposite happenedthere, and the transformation was equally dramatic.I'm not criticizing Steve and Alexis. What happened to Redditdidn't happen out of neglect. From the start they had a policy ofcensoring nothing except spam. Plus Reddit had different goalsfrom Hacker News. Reddit was a startup, not a side project; itsgoal was to grow as fast as possible. Combine rapid growth andzero censorship, and the result is a free for all. But I don'tthink they'd do much differently if they were doing it again.Measured by traffic, Reddit is much more successful than HackerNews.But what happened to Reddit won't inevitably happen to HN. Thereare several local maxima. There can be places that are free foralls and places that are more thoughtful, just as there are in thereal world; and people will behave differently depending on whichthey're in, just as they do in the real world.I've observed this in the wild. I've seen people cross-posting onReddit and Hacker News who actually took the trouble to write twoversions, a flame for Reddit and a more subdued version for HN.SubmissionsThere are two major types of problems a site like Hacker News needsto avoid: bad stories and bad comments. So far the danger of badstories seems smaller. The stories on the frontpage now are stillroughly the ones that would have been there when HN started.I once thought I'd have to weight votes to keep crap off thefrontpage, but I haven't had to yet. I wouldn't have predicted thefrontpage would hold up so well, and I'm not sure why it has.Perhaps only the more thoughtful users care enough to submit andupvote links, so the marginal cost of one random new user approacheszero. Or perhaps the frontpage protects itself, by advertising what type of submission is expected.The most dangerous thing for the frontpage is stuff that's too easyto upvote. If someone proves a new theorem, it takes some work bythe reader to decide whether or not to upvote it. An amusing cartoontakes less. A rant with a rallying cry as the title takes zero,because people vote it up without even reading it.Hence what I call the Fluff Principle: on a user-voted news site,the links that are easiest to judge will take over unless you takespecific measures to prevent it.Hacker News has two kinds of protections against fluff. The mostcommon types of fluff links are banned as off-topic. Pictures ofkittens, political diatribes, and so on are explicitly banned. Thiskeeps out most fluff, but not all of it. Some links are both fluff,in the sense of being very short, and also on topic.There's no single solution to that. If a link is just an emptyrant, editors will sometimes kill it even if it's on topic in thesense of being about hacking, because it's not on topic by the realstandard, which is to engage one's intellectual curiosity. If theposts on a site are characteristically of this type I sometimes banit, which means new stuff at that url is auto-killed. If a posthas a linkbait title, editors sometimes rephrase it to be morematter-of-fact. This is especially necessary with links whosetitles are rallying cries, because otherwise they become implicit"vote up if you believe such-and-such" posts, which are the mostextreme form of fluff.The techniques for dealing with links have to evolve, because thelinks do. The existence of aggregators has already affected whatthey aggregate. Writers now deliberately write things to draw trafficfrom aggregators—sometimes even specific ones. (No, the ironyof this statement is not lost on me.) Then there are the moresinister mutations, like linkjacking—posting a paraphrase ofsomeone else's article and submitting that instead of the original.These can get a lot of upvotes, because a lot of what's good in anarticle often survives; indeed, the closer the paraphrase is toplagiarism, the more survives.[3]I think it's important that a site that kills submissions providea way for users to see what got killed if they want to. That keepseditors honest, and just as importantly, makes users confidentthey'd know if the editors stopped being honest. HN users can dothis by flipping a switch called showdead in their profile.[4]CommentsBad comments seem to be a harder problem than bad submissions.While the quality of links on the frontpage of HN hasn't changedmuch, the quality of the median comment may have decreased somewhat.There are two main kinds of badness in comments: meanness andstupidity. There is a lot of overlap between the two—meancomments are disproportionately likely also to be dumb—butthe strategies for dealing with them are different. Meanness iseasier to control. You can have rules saying one shouldn't be mean,and if you enforce them it seems possible to keep a lid on meanness.Keeping a lid on stupidity is harder, perhaps because stupidity isnot so easily distinguishable. Mean people are more likely to knowthey're being mean than stupid people are to know they're beingstupid.The most dangerous form of stupid comment is not the long butmistaken argument, but the dumb joke. Long but mistaken argumentsare actually quite rare. There is a strong correlation betweencomment quality and length; if you wanted to compare the qualityof comments on community sites, average length would be a goodpredictor. Probably the cause is human nature rather than anythingspecific to comment threads. Probably it's simply that stupiditymore often takes the form of having few ideas than wrong ones.Whatever the cause, stupid comments tend to be short. And sinceit's hard to write a short comment that's distinguished for theamount of information it conveys, people try to distinguish theminstead by being funny. The most tempting format for stupid commentsis the supposedly witty put-down, probably because put-downs arethe easiest form of humor. [5]So one advantage of forbiddingmeanness is that it also cuts down on these.Bad comments are like kudzu: they take over rapidly. Comments havemuch more effect on new comments than submissions have on newsubmissions. If someone submits a lame article, the other submissionsdon't all become lame. But if someone posts a stupid comment on athread, that sets the tone for the region around it. People replyto dumb jokes with dumb jokes.Maybe the solution is to add a delay before people can respond toa comment, and make the length of the delay inversely proportionalto some prediction of its quality. Then dumb threads would growslower.[6]PeopleI notice most of the techniques I've described are conservative:they're aimed at preserving the character of the site rather thanenhancing it. I don't think that's a bias of mine. It's due tothe shape of the problem. Hacker News had the good fortune to startout good, so in this case it's literally a matter of preservation.But I think this principle would also apply to sites with differentorigins.The good things in a community site come from people more thantechnology; it's mainly in the prevention of bad things thattechnology comes into play. Technology certainly can enhancediscussion. Nested comments do, for example. But I'd rather usea site with primitive features and smart, nice users than a moreadvanced one whose users were idiots or trolls.So the most important thing a community site can do is attract thekind of people it wants. A site trying to be as big as possiblewants to attract everyone. But a site aiming at a particular subsetof users has to attract just those—and just as importantly,repel everyone else. I've made a conscious effort to do this onHN. The graphic design is as plain as possible, and the site rulesdiscourage dramatic link titles. The goal is that the only thingto interest someone arriving at HN for the first time should be theideas expressed there.The downside of tuning a site to attract certain people is that,to those people, it can be too attractive. I'm all too aware howaddictive Hacker News can be. For me, as for many users, it's akind of virtual town square. When I want to take a break fromworking, I walk into the square, just as I might into Harvard Squareor University Ave in the physical world.[7]But an online square ismore dangerous than a physical one. If I spent half the day loiteringon University Ave, I'd notice. I have to walk a mile to get there,and sitting in a cafe feels different from working. But visitingan online forum takes just a click, and feels superficially verymuch like working. You may be wasting your time, but you're notidle. Someone is wrong on the Internet, and you're fixing theproblem.Hacker News is definitely useful. I've learned a lot from thingsI've read on HN. I've written several essays that began as commentsthere. So I wouldn't want the site to go away. But I would liketo be sure it's not a net drag on productivity. What a disasterthat would be, to attract thousands of smart people to a site thatcaused them to waste lots of time. I wish I could be 100% surethat's not a description of HN.I feel like the addictiveness of games and social applications isstill a mostly unsolved problem. The situation now is like it waswith crack in the 1980s: we've invented terribly addictive newthings, and we haven't yet evolved ways to protect ourselves fromthem. We will eventually, and that's one of the problems I hopeto focus on next.Notes[1]I tried ranking users by both average and median commentscore, and average (with the high score thrown out) seemed the moreaccurate predictor of high quality. Median may be the more accuratepredictor of low quality though.[2]Another thing I learned from this experiment is that if you'regoing to distinguish between people, you better be sure you do itright. This is one problem where rapid prototyping doesn't work.Indeed, that's the intellectually honest argument for not discriminatingbetween various types of people. The reason not to do it is notthat everyone's the same, but that it's bad to do wrong and hardto do right.[3]When I catch egregiously linkjacked posts I replace the urlwith that of whatever they copied. Sites that habitually linkjackget banned.[4]Digg is notorious for its lack of transparency. The root ofthe problem is not that the guys running Digg are especially sneaky,but that they use the wrong algorithm for generating their frontpage.Instead of bubbling up from the bottom as they get more votes, ason Reddit, stories start at the top and get pushed down by newarrivals.The reason for the difference is that Digg is derived from Slashdot,while Reddit is derived from Delicious/popular. Digg is Slashdotwith voting instead of editors, and Reddit is Delicious/popularwith voting instead of bookmarking. (You can still see fossils oftheir origins in their graphic design.)Digg's algorithm is very vulnerable to gaming, because any storythat makes it onto the frontpage is the new top story. Which inturn forces Digg to respond with extreme countermeasures. A lotof startups have some kind of secret about the subterfuges they hadto resort to in the early days, and I suspect Digg's is the extentto which the top stories were de facto chosen by human editors.[5]The dialog on Beavis and Butthead was composed largely ofthese, and when I read comments on really bad sites I can hear themin their voices.[6]I suspect most of the techniques for discouraging stupidcomments have yet to be discovered. Xkcd implemented a particularlyclever one in its IRC channel: don't allow the same thing twice.Once someone has said "fail," no one can ever say it again. Thiswould penalize short comments especially, because they have lessroom to avoid collisions in.Another promising idea is the stupid filter, which is just like aprobabilistic spam filter, but trained on corpora of stupid andnon-stupid comments instead.You may not have to kill bad comments to solve the problem. Commentsat the bottom of a long thread are rarely seen, so it may be enoughto incorporate a prediction of quality in the comment sortingalgorithm.[7]What makes most suburbs so demoralizing is that there's nocenter to walk to.Thanks to Justin Kan, Jessica Livingston, Robert Morris,Alexis Ohanian, Emmet Shear, and Fred Wilson for reading drafts ofthis.Comment on this essay.
