If you follow the strategy of avoiding undifferentiated heavy lifting, you’ll inevitably integrate with many software vendors. For example, my e-commerce software team at Glossier has vendors for our cloud hosting, CDN, CMS, payment processing, CI/CD pipelines, code hosting, internal docs, observability, and alerting to name a few.

Here are a few criteria I’ve found particularly valuable when choosing vendors.

1. Emphasize rate of improvement over current feature set

Prefer a vendor that’s sufficient today and improving quickly over a dominant-yet-stagnant vendor. I judge vendors’ rate of improvement by their recent feature announcements and possibly a roadmap presented by an account rep. In other words, consider which vendor will likely have the best product ~2 years from now; not just the product as it exists today. Skate to where the puck is going.

This criteria is particularly helpful to compare small, disruptive innovators with current market leaders.

Of course, if the current market leader is also innovating quickly, you’re lucky to have an easy decision.

2. Backchannel referrals / Ask an expert

In my StaffEng interview, I shared this anecdote:

Our team was recently choosing a new vendor and the team was split between two mediocre choices. I asked an acquaintance with expertise about the vendors how he would choose; and he recommended a lesser-known new vendor that quickly became a universal team favorite.

To expand on this example, it was an an area that our team had little expertise. It was difficult for us to determine what features really matter; and set realistic expectations. Asking experts in your professional network can bring clarity and confidence.

3. Emphasize net value over cost

From the Vendor relationship HOWTO:

The goal is to maximize our organization’s long-term value from the vendor’s service.

In contrast, I’ve sometimes seen teams try to minimize cost, ignoring gross value. This is short-sighted.

Suppose Vendor A costs $25k/yr and adds $200k of gross value to the org ($175k net value); while Vendor B costs $100k and adds $500k of gross value ($400k net value).

Choose Vendor B because of the higher net value, even though it’s more expensive than Vendor A.

To be sure, I don’t know how to assess the gross value derived from any vendor beyond a hand-wavy estimate. Here are some techniques I use; though I’d certainly like to learn more.

One technique is to look at productivity improvements. If a tool saves each engineer 1 hour per week, it’s a 2.5% productivity improvement; so it’s gross value is roughly 2.5% of your total Engineering payroll.

Other times vendors add capabilities or controls that change how the team works, so you can’t easily assess productivity. In this case, speculate about how much value your org gets from that capability or control. E.g. an A/B testing tool adds the capability to rigorously measure the impact of product changes. The gross value is the difference between the product features you ship using A/B test feedback versus product features you would have shipped without A/B test feedback. Security tools add controls that constrain types of risk. The gross value is difference from the liability of the unknown/unconstrained risk versus the better-known/constrained risk.