April 2001, rev. April 2003(This article is derived from a talk given at the 2001 FranzDeveloper Symposium.)In the summer of 1995, my friend Robert Morris and Istarted a startup called Viaweb. Our plan was to writesoftware that would let end users build online stores.What was novel about this software, at the time, wasthat it ran on our server, using ordinary Web pagesas the interface.A lot of people could have been having this idea at thesame time, of course, but as far as I know, Viaweb wasthe first Web-based application. It seemed sucha novel idea to us that we named the company after it:Viaweb, because our software worked via the Web,instead of running on your desktop computer.Another unusual thing about this software was that itwas written primarily in a programming language calledLisp. It was one of the first big end-userapplications to be written in Lisp, which up till thenhad been used mostly in universities and research labs. [1]The Secret WeaponEric Raymond has written an essay called "How to Become a Hacker,"and in it, among other things, he tells would-be hackers whatlanguages they should learn. He suggests starting with Python andJava, because they are easy to learn. The serious hacker will alsowant to learn C, in order to hack Unix, and Perl for systemadministration and cgi scripts. Finally, the truly serious hackershould consider learning Lisp: Lisp is worth learning for the profound enlightenment experience you will have when you finally get it; that experience will make you a better programmer for the rest of your days, even if you never actually use Lisp itself a lot.This is the same argument you tend to hear for learning Latin. Itwon't get you a job, except perhaps as a classics professor, butit will improve your mind, and make you a better writer in languagesyou do want to use, like English.But wait a minute. This metaphor doesn't stretch that far. Thereason Latin won't get you a job is that no one speaks it. If youwrite in Latin, no one can understand you. But Lisp is a computerlanguage, and computers speak whatever language you, the programmer,tell them to.So if Lisp makes you a better programmer, like he says, why wouldn'tyou want to use it? If a painter were offered a brush that wouldmake him a better painter, it seems to me that he would want touse it in all his paintings, wouldn't he? I'm not trying to makefun of Eric Raymond here. On the whole, his advice is good. Whathe says about Lisp is pretty much the conventional wisdom. Butthere is a contradiction in the conventional wisdom: Lisp willmake you a better programmer, and yet you won't use it.Why not? Programming languages are just tools, after all. If Lispreally does yield better programs, you should use it. And if itdoesn't, then who needs it?This is not just a theoretical question. Software is a verycompetitive business, prone to natural monopolies. A company thatgets software written faster and better will, all other thingsbeing equal, put its competitors out of business. And when you'restarting a startup, you feel this very keenly. Startups tend tobe an all or nothing proposition. You either get rich, or you getnothing. In a startup, if you bet on the wrong technology, yourcompetitors will crush you.Robert and I both knew Lisp well, and we couldn't see any reasonnot to trust our instincts and go with Lisp. We knew that everyoneelse was writing their software in C++ or Perl. But we also knewthat that didn't mean anything. If you chose technology that way,you'd be running Windows. When you choose technology, you have toignore what other people are doing, and consider only what willwork the best.This is especially true in a startup. In a big company, you cando what all the other big companies are doing. But a startup can'tdo what all the other startups do. I don't think a lot of peoplerealize this, even in startups.The average big company grows at about ten percent a year. So ifyou're running a big company and you do everything the way theaverage big company does it, you can expect to do as well as theaverage big company-- that is, to grow about ten percent a year.The same thing will happen if you're running a startup, of course.If you do everything the way the average startup does it, you shouldexpect average performance. The problem here is, average performancemeans that you'll go out of business. The survival rate for startupsis way less than fifty percent. So if you're running a startup,you had better be doing something odd. If not, you're in trouble.Back in 1995, we knew something that I don't think our competitorsunderstood, and few understand even now: when you're writingsoftware that only has to run on your own servers, you can useany language you want. When you're writing desktop software,there's a strong bias toward writing applications in the samelanguage as the operating system. Ten years ago, writing applicationsmeant writing applications in C. But with Web-based software,especially when you have the source code of both the language andthe operating system, you can use whatever language you want.This new freedom is a double-edged sword, however. Now that youcan use any language, you have to think about which one to use.Companies that try to pretend nothing has changed risk finding thattheir competitors do not.If you can use any language, which do you use? We chose Lisp.For one thing, it was obvious that rapid development would beimportant in this market. We were all starting from scratch, soa company that could get new features done before its competitorswould have a big advantage. We knew Lisp was a really good languagefor writing software quickly, and server-based applications magnifythe effect of rapid development, because you can release softwarethe minute it's done.If other companies didn't want to use Lisp, so much the better.It might give us a technological edge, and we needed all the helpwe could get. When we started Viaweb, we had no experience inbusiness. We didn't know anything about marketing, or hiringpeople, or raising money, or getting customers. Neither of us hadever even had what you would call a real job. The only thing wewere good at was writing software. We hoped that would save us.Any advantage we could get in the software department, we wouldtake.So you could say that using Lisp was an experiment. Our hypothesiswas that if we wrote our software in Lisp, we'd be able to getfeatures done faster than our competitors, and also to do thingsin our software that they couldn't do. And because Lisp was sohigh-level, we wouldn't need a big development team, so our costswould be lower. If this were so, we could offer a better productfor less money, and still make a profit. We would end up gettingall the users, and our competitors would get none, and eventuallygo out of business. That was what we hoped would happen, anyway.What were the results of this experiment? Somewhat surprisingly,it worked. We eventually had many competitors, on the order oftwenty to thirty of them, but none of their software could competewith ours. We had a wysiwyg online store builder that ran on theserver and yet felt like a desktop application. Our competitorshad cgi scripts. And we were always far ahead of them in features.Sometimes, in desperation, competitors would try to introducefeatures that we didn't have. But with Lisp our development cyclewas so fast that we could sometimes duplicate a new feature withina day or two of a competitor announcing it in a press release. Bythe time journalists covering the press release got round to callingus, we would have the new feature too.It must have seemed to our competitors that we had some kind ofsecret weapon-- that we were decoding their Enigma traffic orsomething. In fact we did have a secret weapon, but it was simplerthan they realized. No one was leaking news of their features tous. We were just able to develop software faster than anyonethought possible.When I was about nine I happened to get hold of a copy of The Dayof the Jackal, by Frederick Forsyth. The main character is anassassin who is hired to kill the president of France. The assassinhas to get past the police to get up to an apartment that overlooksthe president's route. He walks right by them, dressed up as anold man on crutches, and they never suspect him.Our secret weapon was similar. We wrote our software in a weirdAI language, with a bizarre syntax full of parentheses. For yearsit had annoyed me to hear Lisp described that way. But now itworked to our advantage. In business, there is nothing more valuablethan a technical advantage your competitors don't understand. Inbusiness, as in war, surprise is worth as much as force.And so, I'm a little embarrassed to say, I never said anythingpublicly about Lisp while we were working on Viaweb. We nevermentioned it to the press, and if you searched for Lisp on our Website, all you'd find were the titles of two books in my bio. Thiswas no accident. A startup should give its competitors as littleinformation as possible. If they didn't know what language oursoftware was written in, or didn't care, I wanted to keep it thatway.[2]The people who understood our technology best were the customers.They didn't care what language Viaweb was written in either, butthey noticed that it worked really well. It let them build greatlooking online stores literally in minutes. And so, by word ofmouth mostly, we got more and more users. By the end of 1996 wehad about 70 stores online. At the end of 1997 we had 500. Sixmonths later, when Yahoo bought us, we had 1070 users. Today, asYahoo Store, this software continues to dominate its market. It'sone of the more profitable pieces of Yahoo, and the stores builtwith it are the foundation of Yahoo Shopping. I left Yahoo in1999, so I don't know exactly how many users they have now, butthe last I heard there were about 20,000.The Blub ParadoxWhat's so great about Lisp? And if Lisp is so great, why doesn'teveryone use it? These sound like rhetorical questions, but actuallythey have straightforward answers. Lisp is so great not becauseof some magic quality visible only to devotees, but because it issimply the most powerful language available. And the reason everyonedoesn't use it is that programming languages are not merelytechnologies, but habits of mind as well, and nothing changesslower. Of course, both these answers need explaining.I'll begin with a shockingly controversial statement: programminglanguages vary in power.Few would dispute, at least, that high level languages are morepowerful than machine language. Most programmers today would agreethat you do not, ordinarily, want to program in machine language.Instead, you should program in a high-level language, and have acompiler translate it into machine language for you. This idea iseven built into the hardware now: since the 1980s, instruction setshave been designed for compilers rather than human programmers.Everyone knows it's a mistake to write your whole program by handin machine language. What's less often understood is that thereis a more general principle here: that if you have a choice ofseveral languages, it is, all other things being equal, a mistaketo program in anything but the most powerful one. [3]There are many exceptions to this rule. If you're writing a programthat has to work very closely with a program written in a certainlanguage, it might be a good idea to write the new program in thesame language. If you're writing a program that only has to dosomething very simple, like number crunching or bit manipulation,you may as well use a less abstract language, especially since itmay be slightly faster. And if you're writing a short, throwawayprogram, you may be better off just using whatever language hasthe best library functions for the task. But in general, forapplication software, you want to be using the most powerful(reasonably efficient) language you can get, and using anythingelse is a mistake, of exactly the same kind, though possibly in alesser degree, as programming in machine language.You can see that machine language is very low level. But, at leastas a kind of social convention, high-level languages are often alltreated as equivalent. They're not. Technically the term "high-levellanguage" doesn't mean anything very definite. There's no dividingline with machine languages on one side and all the high-levellanguages on the other. Languages fall along a continuum [4] ofabstractness, from the most powerful all the way down to machinelanguages, which themselves vary in power.Consider Cobol. Cobol is a high-level language, in the sense thatit gets compiled into machine language. Would anyone seriouslyargue that Cobol is equivalent in power to, say, Python? It'sprobably closer to machine language than Python.Or how about Perl 4? Between Perl 4 and Perl 5, lexical closuresgot added to the language. Most Perl hackers would agree that Perl5 is more powerful than Perl 4. But once you've admitted that,you've admitted that one high level language can be more powerfulthan another. And it follows inexorably that, except in specialcases, you ought to use the most powerful you can get.This idea is rarely followed to its conclusion, though. After acertain age, programmers rarely switch languages voluntarily.Whatever language people happen to be used to, they tend to considerjust good enough.Programmers get very attached to their favorite languages, and Idon't want to hurt anyone's feelings, so to explain this point I'mgoing to use a hypothetical language called Blub. Blub falls rightin the middle of the abstractness continuum. It is not the mostpowerful language, but it is more powerful than Cobol or machinelanguage.And in fact, our hypothetical Blub programmer wouldn't use eitherof them. Of course he wouldn't program in machine language. That'swhat compilers are for. And as for Cobol, he doesn't know howanyone can get anything done with it. It doesn't even have x (Blubfeature of your choice).As long as our hypothetical Blub programmer is looking down thepower continuum, he knows he's looking down. Languages less powerfulthan Blub are obviously less powerful, because they're missing somefeature he's used to. But when our hypothetical Blub programmerlooks in the other direction, up the power continuum, he doesn'trealize he's looking up. What he sees are merely weird languages.He probably considers them about equivalent in power to Blub, butwith all this other hairy stuff thrown in as well. Blub is goodenough for him, because he thinks in Blub.When we switch to the point of view of a programmer using any ofthe languages higher up the power continuum, however, we find thathe in turn looks down upon Blub. How can you get anything done inBlub? It doesn't even have y.By induction, the only programmers in a position to see all thedifferences in power between the various languages are those whounderstand the most powerful one. (This is probably what EricRaymond meant about Lisp making you a better programmer.) You can'ttrust the opinions of the others, because of the Blub paradox:they're satisfied with whatever language they happen to use, becauseit dictates the way they think about programs.I know this from my own experience, as a high school kid writingprograms in Basic. That language didn't even support recursion.It's hard to imagine writing programs without using recursion, butI didn't miss it at the time. I thought in Basic. And I was awhiz at it. Master of all I surveyed.The five languages that Eric Raymond recommends to hackers fall atvarious points on the power continuum. Where they fall relativeto one another is a sensitive topic. What I will say is that Ithink Lisp is at the top. And to support this claim I'll tell youabout one of the things I find missing when I look at the otherfour languages. How can you get anything done in them, I think,without macros? [5]Many languages have something called a macro. But Lisp macros areunique. And believe it or not, what they do is related to theparentheses. The designers of Lisp didn't put all those parenthesesin the language just to be different. To the Blub programmer, Lispcode looks weird. But those parentheses are there for a reason.They are the outward evidence of a fundamental difference betweenLisp and other languages.Lisp code is made out of Lisp data objects. And not in the trivialsense that the source files contain characters, and strings areone of the data types supported by the language. Lisp code, afterit's read by the parser, is made of data structures that you cantraverse.If you understand how compilers work, what's really going on isnot so much that Lisp has a strange syntax as that Lisp has nosyntax. You write programs in the parse trees that get generatedwithin the compiler when other languages are parsed. But theseparse trees are fully accessible to your programs. You can writeprograms that manipulate them. In Lisp, these programs are calledmacros. They are programs that write programs.Programs that write programs? When would you ever want to do that?Not very often, if you think in Cobol. All the time, if you thinkin Lisp. It would be convenient here if I could give an exampleof a powerful macro, and say there! how about that? But if I did,it would just look like gibberish to someone who didn't know Lisp;there isn't room here to explain everything you'd need to know tounderstand what it meant. In Ansi Common Lisp I tried to movethings along as fast as I could, and even so I didn't get to macrosuntil page 160.But I think I can give a kind of argument that might be convincing.The source code of the Viaweb editor was probably about 20-25%macros. Macros are harder to write than ordinary Lisp functions,and it's considered to be bad style to use them when they're notnecessary. So every macro in that code is there because it has tobe. What that means is that at least 20-25% of the code in thisprogram is doing things that you can't easily do in any otherlanguage. However skeptical the Blub programmer might be about myclaims for the mysterious powers of Lisp, this ought to make himcurious. We weren't writing this code for our own amusement. Wewere a tiny startup, programming as hard as we could in order toput technical barriers between us and our competitors.A suspicious person might begin to wonder if there was somecorrelation here. A big chunk of our code was doing things thatare very hard to do in other languages. The resulting softwaredid things our competitors' software couldn't do. Maybe there wassome kind of connection. I encourage you to follow that thread.There may be more to that old man hobbling along on his crutchesthan meets the eye.Aikido for StartupsBut I don't expect to convince anyone (over 25) to go out and learnLisp. The purpose of this article is not to change anyone's mind,but to reassure people already interested in using Lisp-- peoplewho know that Lisp is a powerful language, but worry because itisn't widely used. In a competitive situation, that's an advantage.Lisp's power is multiplied by the fact that your competitors don'tget it.If you think of using Lisp in a startup, you shouldn't worry thatit isn't widely understood. You should hope that it stays thatway. And it's likely to. It's the nature of programming languagesto make most people satisfied with whatever they currently use.Computer hardware changes so much faster than personal habits thatprogramming practice is usually ten to twenty years behind theprocessor. At places like MIT they were writing programs inhigh-level languages in the early 1960s, but many companies continuedto write code in machine language well into the 1980s. I bet alot of people continued to write machine language until the processor,like a bartender eager to close up and go home, finally kicked themout by switching to a risc instruction set.Ordinarily technology changes fast. But programming languages aredifferent: programming languages are not just technology, but whatprogrammers think in. They're half technology and half religion.[6]And so the median language, meaning whatever language the medianprogrammer uses, moves as slow as an iceberg. Garbage collection,introduced by Lisp in about 1960, is now widely considered to bea good thing. Runtime typing, ditto, is growing in popularity.Lexical closures, introduced by Lisp in the early 1970s, are now,just barely, on the radar screen. Macros, introduced by Lisp in themid 1960s, are still terra incognita.Obviously, the median language has enormous momentum. I'm notproposing that you can fight this powerful force. What I'm proposingis exactly the opposite: that, like a practitioner of Aikido, youcan use it against your opponents.If you work for a big company, this may not be easy. You will havea hard time convincing the pointy-haired boss to let you buildthings in Lisp, when he has just read in the paper that some otherlanguage is poised, like Ada was twenty years ago, to take overthe world. But if you work for a startup that doesn't havepointy-haired bosses yet, you can, like we did, turn the Blubparadox to your advantage: you can use technology that yourcompetitors, glued immovably to the median language, will never beable to match.If you ever do find yourself working for a startup, here's a handytip for evaluating competitors. Read their job listings. Everythingelse on their site may be stock photos or the prose equivalent,but the job listings have to be specific about what they want, orthey'll get the wrong candidates.During the years we worked on Viaweb I read a lot of job descriptions.A new competitor seemed to emerge out of the woodwork every monthor so. The first thing I would do, after checking to see if theyhad a live online demo, was look at their job listings. After acouple years of this I could tell which companies to worry aboutand which not to. The more of an IT flavor the job descriptionshad, the less dangerous the company was. The safest kind were theones that wanted Oracle experience. You never had to worry aboutthose. You were also safe if they said they wanted C++ or Javadevelopers. If they wanted Perl or Python programmers, that wouldbe a bit frightening-- that's starting to sound like a companywhere the technical side, at least, is run by real hackers. If Ihad ever seen a job posting looking for Lisp hackers, I would havebeen really worried.Notes[1] Viaweb at first had two parts: the editor, written in Lisp,which people used to build their sites, and the ordering system,written in C, which handled orders. The first version was mostlyLisp, because the ordering system was small. Later we added twomore modules, an image generator written in C, and a back-officemanager written mostly in Perl.In January 2003, Yahoo released a new version of the editor written in C++ and Perl. It's hard to say whether the program is nolonger written in Lisp, though, because to translate this programinto C++ they literally had to write a Lisp interpreter: the sourcefiles of all the page-generating templates are still, as far as Iknow, Lisp code. (See Greenspun's Tenth Rule.)[2] Robert Morris says that I didn't need to be secretive, becauseeven if our competitors had known we were using Lisp, they wouldn'thave understood why: "If they were that smart they'd already beprogramming in Lisp."[3] All languages are equally powerful in the sense of being Turingequivalent, but that's not the sense of the word programmers careabout. (No one wants to program a Turing machine.) The kind ofpower programmers care about may not be formally definable, butone way to explain it would be to say that it refers to featuresyou could only get in the less powerful language by writing aninterpreter for the more powerful language in it. If language Ahas an operator for removing spaces from strings and language Bdoesn't, that probably doesn't make A more powerful, because youcan probably write a subroutine to do it in B. But if A supports,say, recursion, and B doesn't, that's not likely to be somethingyou can fix by writing library functions.[4] Note to nerds: or possibly a lattice, narrowing toward the top;it's not the shape that matters here but the idea that there is atleast a partial order.[5] It is a bit misleading to treat macros as a separate feature.In practice their usefulness is greatly enhanced by other Lispfeatures like lexical closures and rest parameters.[6] As a result, comparisons of programming languages either takethe form of religious wars or undergraduate textbooks so determinedlyneutral that they're really works of anthropology. People whovalue their peace, or want tenure, avoid the topic. But the questionis only half a religious one; there is something there worthstudying, especially if you want to design new languages.
