Listening to Fear, or When to Build a More Abstract Mousetrap

For a few weeks now I’ve been meaning to write something in response to Lispy’s excellent post Meet the New Problem… Same as the old Problem, which took up the question of whether it’s worth it for a programmer to invest time in writing up a problem-specific language or other domain-specific construct. From my own experience, and from what I’ve gathered from more experienced hands, the decision to build depends on the situation, since there are trade-offs to any decision about how to use a programmer’s precious time. I would also add that making the right choice requires an enlightened form of self-interest that may best be described as listening to your fear.

Lispy’s argument, that it is precisely a programmer’s job to wrap his head around new languages and concepts, and also to improve existing code bases through refactoring and extensions, was in the back of my head as I watched one of the SICP video lectures in which Gerald Sussman talks about pattern matching and data-directed programming. Towards the end of the lecture, one of the audience members points out that some people find pattern matching so valuable that they feel programming languages should support it with structures and facilities built into the language itself. “You mean like Prolog?” Sussman asks, and then comments, “Well, sure, but I’d rather you knew how to roll your own language, so you won’t have to depend on anyone else.”

Does this response send a shiver of fear through you as it does through me? Indeed, doesn’t the idea of trying to build a better version of an existing specialized programming language fill your heart with terror? I think Sussman would say it shouldn’t, that one isn’t really solving a problem if one isn’t building up a language. That is the Lisp way. Abelson says plainly that Lisp is a lousy language for doing most things, except it’s great for building up languages to do something in particular. So building your own language, your own specialized set of tools, is not only nothing to fear, but is essential. And, I think the idea goes, it’s fun too.

But what kind of world do we wake up in? I usually wake up in a world in which deadlines are looming and threatening to slide, and in which I am often desperate to find something that works to solve my problems. “Please,” I think to myself, “let me find someone else I can depend on.”

I understand that it would be desirable to develop a problem-specific language or class hierarchy, or library, or whatever for each job, each challenge. And not just because I view programming as an intellectual effort and find this kind of approach the most satisfying as a student. I also think it usually leads to a better solution. But the time is always precious. And the better is often the enemy of the good. This is why it’s good to listen to fear when it tells you, “That’s a rabbit hole you’ll disappear into, and you’ll miss your deadline if you do that.”

But that isn’t the only thing the fear might say. The fear doesn’t always say you have to build quick and dirty. If you understand enough about how this crazy business works, you begin to distinguish different situations. Often, yes, you need to crank out something reliable quickly. It doesn’t have to be extensible, and it doesn’t have to scale especially well–your client won’t need it to do that. But you may also find yourself being asked to do something genuinely interesting, and in that happy but also terrifying circumstance, that twisting feeling in your gut will tell you to build, because without some excellent abstractions, you’ll never get the project done.

So here’s what I have to say about lispy’s point. Problems and programming projects exist along a continuum, from the quick and dirty to the truly innovative. If you’re in the business of doing new and difficult things, you’re probably going to favor abstractions, DSLs, and so on. You’re in Paul Graham’s “Beating the Averages” territory. You don’t need reliable, you need something that can pay off big. That tends to be risky, of course, but if you consider the odds against you and your ground-breaking project, you’re afraid not to try for the big payoff.

Just remember that if you commit to building your better mousetrap, you have to make it really really good. You have to use it to the best advantage possible. Here’s a tale form my experience. My first ‘Net job back in 1996 was with an online music retailer named “Music Boulevard.” It was a competitor to CD Now in the days before Amazon came in and pwned the online CD ordering business. My job was in New York, but the tech who’d built Music Boulevard in the infancy of the Web, in 1993-94, worked outside Philly. I didn’t meet them often, but over time I heard things about them from the techs in NYC, who came to revere them. The most impressive thing they had done was they had build a custom CGI application layer, called Request Server, that built the site’s pages from templates. This was years before PHP and ASP. It let them build new templates quickly and standardize their site’s code, appearance, and behavior.

But it wasn’t enough for Music Boulevard. In time, PHP and ASP came along, and it suddenly wasn’t anything special to have an application layer like that. The competitive advantage disappeared. And there was another problem: no one in New York knew how to use Request Server. The people in Philly weren’t interested in showing us, or we weren’t interested in learning, or management failed to make us all interested in sharing. In NYC, we actually used ASP instead, and this was back in the 1.0 days when ASP was a nightmare. Why didn’t we use the in-house platform? Because the company didn’t realize that Request Server was as important to its success as any marketing or warehousing schemes. We weren’t just in the business of selling CDs: pushing Request Server, and getting as much advantage from it as possible, was also our business. We didn’t realize that, and we failed.

So, there is no easy answer to the question of whether you should build or not. It depends. Building is risky, because time is always short, but not building is risky when you’re in a fight. Maybe, and this is a happy thought, as you get better at programming and better at analyzing problems, you find that the point at which building is the better option begins to slip down the continuum of projects, towards the “quick-and-dirty” end of the spectrum. Maybe as your proficiency grows, your fear of the rabbit hole diminishes and your fear of the quick-and-dirty increases. We’ll see.

Happy New Year, everybody. Keep coding.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.