feature modularity
animal style with extra toast
The thing about show business is : nobody wants to do the same show twice. The biggest show of 2024 has to be bigger, bolder, brighter, and better than anything the world saw in 2023.
So building software for this world isn’t the same as building “normal” software - you’re always thrown into a situation that you couldn’t possibly have prepared for. Whether it’s hundreds of flash bulbs reflecting off a crinkly reflective backdrop leading to a mid-show restart of your computer; trying to project onto a car on a turntable built on a beach in Shenzhen that settles into the sand at a different angle to the building around it; understudies that are one foot shorter than the star, so the cameras and automation rigs have be retargeted depending on who has the flu; robots that don’t quite go exactly to the same place twice, depending on where they start; elliptical LED screens that radio-interfere with the audio system unless all the pixels are arranged within a square area; anything to do with timecode - you can be pretty sure that any major show will require a last-minute mid-air quick-fix.
These quick-fixes are just that - assemblages of duct-tape, string and larks’ spit, made using whatever tools were to hand (in one particularly memorable scene, I had to perform surgery on a DVI connector using a corkscrew, sweating bullets on my hands and knees behind the coffee table, while maintaining a nonchalant conversation with Jay-Z in his room at the Mandarin Oriental). They are in no way “professionally engineered”, ready to be shipped in an actual product.
So when you make one of these rush-job fixes, you’re left in a quandary - you’ve just written some useful code, even if it’s useful for only one situation; but it’s not good enough to put in the product. So what do you do with it?
With conventional software management practices, you have two options: either throw it away, or make it good enough to ship. The latter course is arduous - you have to move heaven and earth (otherwise known as the Product Management Department) to get it onto the “roadmap”, and there’s already a million other more important things on there, so basically it never happens - all you’ve done is sweep it under the world’s biggest carpet.
Lacking a Product Management Department, or indeed any other developers who could possibly work on anything, but having a lengthy list of other just-in-time hair-on-fire fixes that need to be performed, we came up with a third way.
secret menu options
In California, there’s this burger chain called In-N-Out, a pilgrimage to which has become an essential ritual every time I’m there. Aside from quoting Bible verses on their paper bags (which contain the Platonic Ideal of the hamburger), they also have this cult-y thing going on where you can request customisations to your order. For example, you can order “protein style”, and they’ll replace the bun with lettuce. Or “extra toast” (my favourite) where they toast the buns a bit harder.
Thing is, these “menu options” aren’t written on a sign in the restaurant. People just know about them, and pass on the knowledge to each other like secret sigils. Hence the cult-yness of it all, which I kind of appreciate.
So what we began to do with our quick-and-dirty fixes was to leave them in the code, but gate them with something called a ‘feature flag’. On startup, the software would look for a secret text file called “options.ini”, and then check to see if some secret word (say “experimentalCrashBugFix”) appears there, and if it did, it would enable the code for the quick fix.
This meant I could quickly “ship” the quick-fix, making sure the code and the insight wasn’t lost to posterity; but be sure that nobody would ever see it, or be affected by it, unless they happened to know the secret menu option.
So a month later when I got an emergency call that happened to remind me of that same situation, I could check through the code and send the poor user the secret menu option, and voila, no need for a day of treacle - their problem is solved and the show can go on. Or, more likely, they need just a tiny modification to the feature, I’d add another secret menu option switch, and we’d all go about our respective lives.
the evolution of features
The effect of this way of working was something Darwin would have recognised - through a series of tiny mutations, each one almost too small to remark upon, useful features would evolve. While most mutations were dead-ends, some of them were useful enough that they kept saving peoples’ bacon, each time getting slightly stronger and more powerful, until one day we’d figure out that everyone was enabling those secret menu options as a matter of course whenever they used the software.
At which point we’d slap our heads and go “of course”, write some marketing blurb, update the manual (if we were in a particularly expansive mood), change the switch to being “disable_experimentalCrashBugFix” so people could go back to the old ways if they wanted to, and ship it as a Feature in the next big release.
Amazing, right?
In reality, this system was very far from perfect.
To start with, it was totally ad-hoc and informal. It depended on coders and support technicians knowing about the secret options, remembering that they existed in response to a situation (if they knew about the situation at all), and communicating with users to learn what features were working.
Not all features could be gated with feature flags - in particular, large architectural changes weren’t possible - and it was difficult to figure out which features affected each other and which were independent. Worse, the software was starting to become bloated, convoluted and crufty, piled high with the debris of features that hadn’t made it and never would, and hundreds of features whose fate was uncertain. We’d regularly get into horrible “merge trainwrecks” where two branches of the software with different features had to come together into one official version.
But the value of the feature flag concept was undeniable - it seemed able to capture a wealth of subtle modifications that make the difference between a rigid, brittle and pernickety product and a fluid, malleable and useful one. Every now and then, some zealous newbie would take a look at the code, throw their hands up in horror, and propose some new-broom cleanup to remove them, and then be forced to retract later on when users erupted in outrage.
Meanwhile, there’s little me sitting in the corner, polishing my little imaginary programming language, not saying very much, but observing all of this very keenly.
What would happen, I asked myself, if zero understood the idea of the “feature” in a first-class-citizen sort of way?
I decided to find out.