Feature environments

We at Kickstarter are big fans of feature branches for our code. But when it was time to deploy our features, we had a bottleneck. Our two test environments (QA and Staging) were constantly in use. Engineers had to wait for their turn to deploy a feature. We added a third test environment, configuring much of it by hand. But it was a marginal improvement. We needed a different way to work.

The answer was feature environments.

Now when an engineer wants to show off a feature, he launches a whole new environment with it's own database, web, and app servers. The environment is bootstrapped with a recent, scrubbed copy of the production database. All the configuration happens automatically, even new DNS records.

A few minutes later, the environment is ready to use. And when we're done, we shut it all down.

How it works

We made a small Rails app codenamed HiveQueen (a reference to Ender's Game). HiveQueen coordinates launching, configuring, and deploying to different environments. Behind the scenes, it uses:

HiveQueen is really standing on the shoulder of giants. It's a simple app using popular, stable tools to create an easy workflow for launching & configuring servers.

Working in Slow Time

An unexpected benefit of feature environments was the ability to work in Slow Time. Getting a feature ready for production involves several people: engineers, designers, and our community team. No longer are people rushed to review a feature while it's tying up the QA environment. We can leave up the feature environment for hours or sometimes weeks, iterating at our own pace.

The value of classic literature

I enjoyed Kent Beck’s recent article Pay Me for Before, or Pay Me for Later, in which he considers ways of generating income as a software development guru. The technical books he writes are widely pirated. He writes:

Catcher [in the Rye] was Salinger’s only novel, published in 1951. It’s sold 65 million copies since then. So basically Salinger has spent 50+ years cashing royalty checks and contributing back very little (at least on the scale of another widely-studied novel). That makes no sense from a resource allocation perspective. Catcher was a significant societal contribution, but society has had little payback from its subsequent investment (must be tens of millions of dollars) in Mr. Salinger.

Getting paid for what I’ve already done doesn’t make sense (from the Catcher in the Rye example). So, here’s my proposed deal. Y’all can have everything I’ve already done for free-as-in-beer if I get a steady income for the work I’m about to do.

Beck paints Salinger as the example of what not to do, squandering the “investment” that society made in him. On the contrary! Salinger created a durable product—a novel that people continue to find valuable generations later. That’s spectacularly productive labor: Salinger spends a few years writing his novel, and for the next 50 years millions of readers get value from his work. Classic art and literature is a very efficient and productive for society, even if the creator captures some value through royalties.

Paying for a product or service is not per se an “investment” in the creator. Buying a house is not an investment in the agent, seller, or constructor to keep doing what they’re doing. Paying a parking ticket isn’t an investment in the city to keep doling out parking tickets. Purchases are simply a choice between having more cash and no product, or having less cash and some product.

Purchases incentivize future production, but it’s a mistake to conflate the reasons for the purchase with the incentives it creates.

I don’t think Beck is really conflating purchases and the incentives they create. He likely was simplifying matters for the sake of brevity. Beck’s sense of fairness would prefer getting paid for current work rather than previous work. While that’s probably a popular opinion, I don’t think it’s a rational one.

My Amazon Kindle review

I love using my Amazon Kindle 2. Some recent Kindle reviews (by Jason Kottke, Steven Johnson, and the Wall Street Journal) caused me to reflect on the device.

In short, the Kindle is designed for stories: linear, fictional stories.

The big “next page” buttons make reading a novel enjoyable, while the slow 5-way navigation button makes non-linear reading inconvenient. I gave up reading a reference book on the Kindle when I found it more difficult to skip & skim than the paperback.

Reference books, short chunks of texts (e.g. blog posts, news summaries), and “streams” of data (e.g. news headlines) are Kindle’s weak point. While you can subscribe to newspapers and blogs, it’s a more clunky experience than using a computer. While reading is enjoyable on the Kindle (due to the hi-res screen and form factor), finding what you want to read is cumbersome.

I was surprised to discover the complexities of converting books from the print format to the Kindle format. It’s not as straight-forward as converting a CD to an MP3. Here are some of the rough edges I’ve found in the design of Kindle books:

  • No small caps in Kindle fonts. Some publisher’s display small caps as lowercase.
  • Tricky to align text and images. If the text references figure 1, that figure may be a few screens away.
  • Difficult to zoom in on images. Some charts & graphs are impenetrable at the default resolutions. Sometimes you can zoom in, but there’s no way to make the image larger than the screen and use the directional pad to scroll.

AFAIK, publishers are responsible for converting their books into the Kindle format. So some of the rough edges may be the publisher’s choice, not a limitation of the Kindle format.

So, Kindle’s current design is focused on linear, long-form reading. While there will always be a place for that, they’re missing out on short, non-linear text; i.e. hypertext.

The Kindle nods in the direction of hypertext, with an experimental web browser. But I wonder why it’s even included, since it’s such an inferior experience to the reading a book on the Kindle, and also inferior to surfing the web from a smartphone (there’s no color, and the navigation is clunky).

While the Kindle can succeed by just focusing on novels, improving non-linear navigation (e.g., by adding a touch screen), can make the device work will with both novels and hypertext.

Late to merge

I think people should have 'strong opinions weakly held'. I also think my debate/critical reasoning skill are getting weaker as I get older. So it seems like good practice to argue strongly for a weakly held opinion.

There is a stigma against people who merge late on roadways. Popular behavior dictates that one should vacate the closing lane as soon as signs announce that the lane closes in a few miles. I think those arrogant drivers rushing past rows of idle traffic are actually behaving efficiently and in accordance with Kant's moral imperative. Those who merge at the first sign are slowing everyone down with their conformity.

Merging late is more efficient because it minimizes the duration of time when traffic is constrained to one lane. Having a 50-foot stretch of one-lane road does not slow down traffic as long as the flow of traffic is less than one car per 50 feet of road. But if all traffic merges miles in advance, 1 car/50 ft. of road is enough to cause backup. Merging late maximizes the use of available road, which alleviates bottlenecks.

Merging late concentrates the location where drivers switch lanes. This avoids drivers slowing down further while they negotiate who's changing lanes. Drivers should understand that alternating lanes in the optimal way to merge two lanes into one, and if people use all available roadway, the lanes will be equally full at the point of merging. Since less negotiation is needed, drivers won't need to slow down as much in order to be cautious of unexpected merging.



At Mint, I spent all day (and some of my night) building Islandoo, a social networking site with the current focus of auditioning people for the British reality show Shipwrecked.

Creating and participating in the Islandoo community is a ton of fun, especially once you chat with some new people a bit. But I warn you, it's very addictive.

For some cool Islandoo profiles (IMHO), check out gabsy, sandhi, surferjon, laura1monkey, p_ho, and lynzers. Arrow24 & bockers met on Islandoo and are now dating.

Update 23 Oct 2006: We got a great write-up in Independent and Financial Times (sub. req.). Also, we were reviewed in Mashable and TechCrunch UK a few weeks ago.

Qua vado

I'm leaving arXiv.org and Ithaca, NY to work for Mint Digital in New York City. I'll be a Ruby on Rails developer. This is how excited I am:

Chunky bacon

Make me a sandwich

Make me a sandwich
xkcd is my new favorite web comic. Though Achewood is a close second. (If you don't get the joke above, it's because you're not a computer nerd, which is nothing to be embarassed about.)

What a Fields Medal means to a sysadmin

arXiv under load due to Perelman's Fields Medal

Tuesday morning Grigory "Grisha" Perelman was awarded the Fields Medal for his work on (proof of) the Poincaré Conjecture, one of the million dollar Millenium Prize Problems. NY Times, CNN, the BBC and other web sites point out that he published his work solely on the Internet; but they didn't mention that Perelman's papers are only available from arXiv.org, the web site I maintain.

I found out that Perelman was awarded the Fields Medal when I checked my email ~8:45am Tuesday morning. I had an automated warning message that the main arXiv.org server was unresponsive.
Now, if the server were a pet cat, this is pretty much the equivalent to waking up and stepping in a hairball on the way to the bathroom. I can't even open an SSH
session to better diagnose the problem.

At this point I'm assuming some script went haywire on the server, or there's some sort of malicious attack. I reboot the server (sadly ending over 200 days of uptime). Normally, I would call/email several supervisors to get someone more knowledgeable involved. But my boss Simeon is on vacation ice climbing(!!) in British Columbia. Paul Ginsparg, the arXiv partriach, was unreachable by phone or email at that moment. And Thorsten Schwander, a sysadmin/developer at LANL and arXiv consultant who knows more about Linux than anyone I know, won't wake up for two more hours. I realize that if arXiv is going to work at all this morning, it will be because I make it work.

When the server finishing booting, the load average immediately spikes due to web traffic.

I turn off the web server program so that the rest of the server is responsive enough for me to work. Looking at the web server logs, about a bajillion requests are coming in for everything related to Perelman. Since last weekend, I was expecting demand for his papers, but I didn't realize that arXiv was the only place to get them. Moreover, Perelman's accomplishment has the reclusive genius/Cinderella narrative, so the MSM covers the story far more than they would a typical math prize.

The technical problem was that all arXiv pages are dynamically generated b/c of legacy issues. The server simply couldn't generate Perelman's pages fast enough to keep up with demand. My solution was to grab a copy of all of Perelman's papers off an arXiv.org mirror, and use some redirects to send traffic from the dymanic pages to the very fast static pages.

I turn on the web server, and things hum along smoothly. People can download e-prints as much as they want. I put on my copyediting hat and post a notice on the front page (pictured above).

Total time for me to diagnose and solve the problem: 7 minutes.

In those 7 minutes, I may have done more service to the scientific community than my entire two years at arXiv. I feel that I have atoned for slacking off in Complex Analysis class.


Father: "Do you like your hot coco?"
Little kid: "Yeah."
Father: "What does it taste like?"
Little kid: "Not like toothepaste."

ROFL: the Web 3.0 application stack

Tired: LAMP (Linux, Apache, MySQL, PHP/Perl)

Wired: ROFL (Ruby on Rails On FastCGI on Lighttpd); rhymes with whaffle. Now we need to invent more platforms for the "Roflcopter" stack.

YouTube: make favorites easier

Everyone loves YouTube, including me. But adding a video to my favorites is too hard. It takes three clicks! And two of those clicks are clicking 'ok'. It's not such an important operation that it requires double confirmation: it's not the end of the world if I accidentally add a video to my favorites list.

Please, YouTube developers, make the adding to favorites just one click, like Flickr.

Word of the Day: "Ender's Folly"

ACE2006 Keynote: The Military Industrial Light and Magic Complex (pdf link), via research.techwondo, defines "Ender's Folly" as

An excessive commitment to, and a fetish of 2nd Life worlds will inflict irrevocable damage upon 1st Life worlds.

A list of things that crack me up but really aren't that funny


Woah, Vox is a boatload of fun. Not that ktheory.com is going away. But Vox has the new car social web app smell. The smell is the chemicals they use to treat the dashboard AJAX.)

Spread it on!

Earlier tonight I went to see Inconvenient Truth playing at the cinema 50 feet from my apartment. But that's not what this post is about. Standing in line for candy, I did a double- then triple-take at the man standing behind me. Finally I placed where I'd seen him: Mayor Bill Timber of Super Troopers (Google Images, you fail me, YouTube FTW).
John Lloyd as Bill Timber

Since I have a rather infamous history of mistaking regular people for celebrities, I took a cautious tact. My triple-take was conspicuous, and I totally screwed up my candy order from my preoccupation. (My order invovled me silently thrusting a bill at the cashier.
"Um, what do you want?"
"What is the money for?"
"Oh. Um...Junior Mints." )

So I turned and said, "You look a lot like a guy from Super Troopers. Have you seen that movie?"
He smirks and shrugs, "yeah."
"Oh my god, you're the mayor guy!"
Thinking quickly, I whip out my camera phone and ask if I can take a picture, "My friends will totally flip. We love that movie."

He respectfully declined, which is probably sound publicist advice in the "don't let cult fans accost you, especially if they can barely place you" category.

Instead he shook my hand and introduced himself as John Lloyd.

The only thing I could think to say was, "Spread it on!" (which I meant as a compliment).

← Older posts