ktheory.com is the personal blog of Aaron Suggs, a software engineering leader in North Carolina.

  1. January 25, 2021

    Creating and sustaining vendor relationships can be a highly leveraged skill for software engineering teams. But there’s little guidance or structure at small companies for folks learning to build vendor relationships.

    So here’s the template of bare essentials and some nice-to-have responsibilities to steer emerging engineering leaders in vendor management. This is born from experience at tiny startups to growth-stage companies with hundreds of employees. Larger companies have more formal processes for choosing and managing vendors.

    This post won’t go cover how to choose a vendor and the famous “build versus buy” calculus; instead focusing on what to do after you’ve chosen a vendor.

    Without further ado, the template:


    The goal is to maximize our organization’s long-term value from the vendor’s service. That means we use their service appropriately, and spend money efficiently.

    Minimum essentials

    1. Each vendor should have a Directly Responsible Individual within the org. The DRI is responsible for the items below.
    2. Follow our org’s legal review process. Before you accept terms of service or sign anything, familiarize yourself with your company’s signing authority and approval process. In short, give our legal team a heads up; and they can help navigate contract discussions, particularly around liability and data privacy issues.
    3. Follow our org’s billing process. Give our accounting team a heads up to coordinate who keeps track of invoicing and receipts. Very small companies tend to use corporate charge cards. As they grow, it tends towards invoices and purchase orders with formal approval processes.
    4. Know how to contact the account rep, escalate tech support tickets, or otherwise get high-quality, timely technical assistance. Preferably, this contact info is stored in a well-known, discoverable place for all vendors. (We use Blissfully.) This is particulary important for business-critical vendors like payment providers and CDNs.
    5. Keep payment information up-to-date to avoid service disruptions; and make sure invoices are approved/paid on time. Check your emails!
    6. Use a vendor-specific email list like vendor-name@mycompany.com for all communication with the vendor. As our team grows and we onboard new member, they can easily review and join discussions. As the DRI, you’re responsible for staying on top of this email list.
    7. Ensure money is spent effectively. Should we change our terms to reduce our bill (like commit to a larger quota to reduce overage charges)? For large contracts (>$15k/yr), negotiate with the vendor (the finance team can help with this).
    8. When contracts are expected to change or expire without renewal, inform stakeholders with ample time to implement alternatives.
    9. Ensure the process for onboarding and offboarding employees with the vendor is documented clearly.
    10. Maintain a list of the PII and sensitive information that’s shared with the vendor. Your legal team can help ask the right questions here.

    Nice-to-have strategic considerations

    Here are some next-level ways to derive significantly more value from your vendor relationship:

    • Maintain a clear sense of the value this vendor provides the organization. Tech vendors typically use value-based pricing (as opposed to cost-based pricing), so being able to describe the value of various features ensures you and the account rep speak the same language.
    • Track how closely our usage aligns the vendor’s typical customer usage. Do we use their service in a common, expected way; or in a custom, unusual way that could be a strategic risk as the vendor evolves? Are we one of their biggest/smallest customers (another strategic risk), or middle-of-the-pack?
    • Maintain a general sense of the competitive landscape and alternatives for the vendor. What’s our next best alternative if we had to move off this vendor? Are there competitors who have a superior service or are gaining quickly? When would it be worth the opportunity cost to build it ourselves?
    • Track and contribute to the vendor’s private roadmap (beta features). Usually the account rep will offer to discuss this once or twice per year.

    Congrats, you’re well on your way to a productive, valuable vendor relationship!


  2. January 21, 2021

    This interview originally appeared on StaffEng. I wanted to share it here as well.

    Tell us a little about your current role: where do you work, your title and generally the sort of work that you and your team do.

    I work at Glossier, a direct-to-consumer growth-stage skincare and beauty company with incredibly passionate customers. Our engineering team is ~35 people. I’m a Principal Engineer, mostly focusing on our Site Reliability and Tools team. My recent focus has been leading Glossier’s Operational Excellence initiative (nicknamed ✨GLOE✨) and ensuring we’re building scalable services and team practices. I define operational excellence as our ability to deliver low defect rates, high availability, and low latency for product features. In practice for the SRE/Tools team, that means improving observability, increasing our infra-as-code adoption, and shepherding our migration from a monolith to microservices.

    In the Staff Eng Archetypes, I gravitate most towards being a right-hand, and secondly a solver.

    Prior to Glossier, I was a Director of Engineering at Kickstarter. In 2018, I joined Glossier as a Senior Staff Engineer (an IC role), and as the first engineer to focus primarily on internal tools and engineering practices. My first projects were building a feature flag system so we could safely and easily test features with real data; then implementing continuous deployments to accelerate delivery.

    After a few months, I switched back to management to lead a new Platform team and prepare for Black Friday. Glossier has an annual Black Friday sale that generates a huge spike in traffic and revenue, and our ambitious growth targets showed we need to rigorously prepare with capacity testing, system hardening, and cross-functional collaboration (See Surviving Black Friday: Tales from an e-commerce engineer for details on Glossier’s Black Friday prep). After some re-orgs, the Platform team wound down, but the current SRE/Tools team does similar work. A year ago I gave up my management responsibilities to more deeply focus on operational excellence.

    Did you ever consider engineering management, and if so how did you decide to pursue the staff engineer path?

    Absolutely! I’ve switched from manager to IC twice in my career; and I’ll likely do so again.

    When I first became a manager in 2015, it was the only career path for a senior engineer at my company. Fortunately, ever-smaller engineering teams soon created and shared career ladders with parallel IC and management tracks. When I helped create Kickstarter’s engineering ladder, I emphasized IC growth paths that didn’t require people management.

    I was deeply influenced by a section of Camille Fournier’s Manager’s Path that called out “empire building” as a toxic management practice. It reminded me of the argument in Plato’s Republic that the political leaders shouldn’t be those that selfishly seek power, rather those whose wisdom makes them duty-bound to lead.

    So I don’t orient my career around ever-greater management responsibilities: it’s one tool in the toolbox. I appreciate management as a rich discipline that I’ll spend my career honing; alongside programming and systems engineering.

    Here are some important factors for me when switching between manager and IC roles:

    • What skills does the team need most acutely: management to coordinate the actions of a group; or an IC to accelerate the execution?
    • Will I have sufficient support and feedback to learn and succeed?
    • Am I the only one on the team who could do this; or could others do it well?

    Can you remember any piece of advice on reaching Staff that was particularly helpful for you?

    “Replace indignation with curiosity.”

    Several years ago, I told my manager about another team behaving in a way that caused problems for my team. When I finished, he gave me that advice. I hadn’t been curious about why the other team was acting that way. It turned out they had constraints that made their behavior quite reasonable. By approaching them with curiosity and a helpful mindset (instead of frustration), we quickly found a process that improved both our workflows.

    More recently, while struggling with burnout, a career coach asked me, “What would let you approach each day with energy and optimism?”

    It’s become my morning mantra, ensuring that I make time for operational excellence and mentorship and bring genuine enthusiasm to my work.

    How do you spend your time day-to-day?

    My days are roughly 50% scheduled meetings, 35% deep-focus blocks, and 15% unplanned work.

    I work hard to make sure the meetings are effective. That usually means at least having an agenda. The meeting should have a clear purpose known to attendees beforehand, such making a decision, generating ideas, or reviewing information. Meetings often have a negative connotation because they’re facilitated poorly; but they can be incredibly productive. I try to get better at facilitating productive meetings and using synchronous attention well. High Output Management by Andrew Grove is a great resource to learn about effective meetings.

    A technique I recently learned from my CTO is to schedule reading time at the start of a group meeting. Say you’re in a hiring debrief: everyone spends the first 5 minutes reading each other’s feedback about the candidate. It’s a great way to ensure attendees truly read the document and have it top-of-mind. It ultimately saves time and elevates the subsequent discussion.

    I also interview quite a bit. In 2020, I did (checks calendar) 126 interviews. Improving the long-term health of the team is a key Staff+ responsibility; and helping us hire great people is part of that.

    The deep-focus blocks are marked off on my calendar. My company observes “No Meeting Thursday” which helps a lot. I use these blocks for work that’s ‘important but not urgent’ from Eisenhower’s productivity matrix. That’s usually writing specs and documentation, or researching and prototyping new tools and patterns.

    My schedule is unusual in that I stop work around 4pm most days, then work later in the evenings, ~8-10pm. This gives me several high-quality hours with my family each day. I have difficulty concentrating in the afternoon, and can more easily concentrate at night. And I enjoy getting something done right before bedtime. So this schedule has improved both my work/life balance and productivity. I changed my schedule because of childcare needs during the coronavirus pandemic; but I think I’ll keep it long-term. I encourage everyone to reflect on what habits and schedules are helpful for their work. An open discussion with your manager and some flexibility can go a long way.

    The unplanned work is mostly answering Slack messages, advising on urgent issues, or sometimes responding to a production incident. I try to approach this work with a helpful attitude, and also with an eye towards cross-training and writing discoverable documentation to minimize future unplanned work.

    Where do you feel most impactful as a Staff-plus Engineer? A specific story would be grand.

    I think of my impact in two ways:

    1. Working the plan
    2. Serendipity

    ‘Working the plan’ is about making daily, incremental progress on a big project with a team. Some examples have been improving our site availability from under 99% to over 99.95%. It took a lot of Learning Reviews (blameless postmortems), training, testing, and refactoring. Another was a 9-month migration from dynamically-generated Rails-based HTML pages to statically-generated React-based ones to improved time-to-first-byte and availability. It took a lot of coaching, buy-in, and coordination. To successfully work the plan, you need clear goals and incremental milestones to keep the team motivated, and continuous alignment with leadership on the desired outcomes and timeline.

    ‘Serendipity’ in my work is about sharing an insight with the right people at the right time to make a positive impact. For example, our team was recently choosing a new vendor and the team was split between two mediocre choices. I asked an acquaintance with expertise about the vendors how he would choose; and he recommended a lesser-known new vendor that quickly became a universal team favorite.

    Another serendipitous example was an engineer mentioning during standup that a caching optimization wasn’t having impact they expected. I happened to be familiar with the config options of the particular Ruby web server; and was able to interpret some complicated metrics on a dashboard they showed to determine we had misconfigured a memory threshold. Later that day, we made a one-line config change to optimize our memory usage that reduced latency by 30%.

    Serendipitous impact isn’t planned; and isn’t necessarily hard work. It’s about paying attention (being present), keeping a curious mindset, and sharing the insight in a way that colleagues are open to receiving.

    How have you sponsored other engineers? Is sponsoring other engineers an important aspect of your role?

    Certainly! As a Principal Engineer, I try to be an enthusiastic and conspicuous first follower when other engineers are doing important new practices. Some examples are when colleagues demoed React snapshot testing and local development with Docker. After each demo, I’d ask how I can try it out and see the benefits for myself. Then I’d look for other teams and in-flight projects where we can apply these practices to get wider adoption.

    I also ‘cheerlead’: recognizing a colleague’s valuable effort in public or a small group, even if the outcomes aren’t tangible yet. It could be complimenting a team that’s was thorough and reflective during a difficult Learning Review; praising an engineer who reproduced a tricky race condition; or thanking someone who documented a poorly understood process.

    I aim to serve two purposes with cheerleading: recognize those doing the valuable behavior, and give positive reinforcement in the hopes that the team does more of that behavior. It’s really operant conditioning, but cheerleading sounds much nicer.

    What about a piece of advice for someone who has just started as a Staff Engineer?

    Other engineers look up to you as a role model, some in ways you may not expect. They’ll emulate your coding style, your tone in code reviews, your behavior in meetings, your rationale for making decisions, and the way you treat colleagues.

    It can feel like a lot of responsibility to be perfect all the time. But it can also bring clarity to your work: do your best, acknowledge shortcomings, be generous and curious.


  3. January 18, 2021

    A well-crafted GitHub pull request can be a powerful way to show others how to extend and maintain a component. These ‘Exemplary’ PRs highlight the code and practices you want others to emulate.

    A few years ago, my Platform team was implementing a new GraphQL API. We found engineers needed a lot of support and code reviews to add new mutations in our app. One of our lead engineers used a new mutation as an opportunity to create an exemplary PR.

    The exemplary PR for a GraphQL mutation showed:

    1. The new class to create and interface to implement
    2. How to register the new mutation with the server
    3. How to handle authentication/authorization
    4. How to validate the object and handle validation errors
    5. Instructions for how to test the mutation locally, what automated tests to create, and how to manage test state

    It turned out to be highly leveraged effort! As we pointed engineers to the exemplary PR, they were able to easily create high-quality mutations while also needing less support from the Platform team.

    Recently, I had the opportunity to help create another exemplary PR. Our SRE team wanted to make an easy process for Eng Managers to maintain their team’s PagerDuty on-call schedules using Terraform. We created a simple pagerduty_team module that only required a few parameters, like the name of the team and a list of emails of the on-call members. That way managers didn’t need to learn a bunch of Terraform provider details just to maintain their on-call rotations.

    I worked with an EM to craft an exemplary PR, adding her team’s rotation, and being sure to add explanatory comments about how our CI/CD pipeline applies the changes. As other EMs asked how to set up their on-call schedule, we’d just send a link to that PR. It was obvious what values to substitute.

    To be sure, we had more documentation about our Terraform setup; but making the PR the one-stop-shop ensured EMs could get their rotations set up in minutes without much reading or back-and-forth.

    Engineers naturally look for similar code in a repository they can use as a starting point for new features. Creating and labeling exemplary PRs is a helpful way to highlight the code you want them to emulate.


  4. December 31, 2020

    In late 2019, I was burnt out in my Director of Engineering role. I spent several sessions with a career coach outlining my professional challenges. Teams lurched from crisis to crisis. Various teams either lacked a coherent strategy, or lacked the alignment or resources to execute it effectively. Frequent confusion about roles and responsibilities caused tension. I didn’t have the resources to fix it all.

    My coach finally asked:

    “What would let you approach each day with energy and optimism?”

    The question felt like reaching a vista after a long hike. My mood lifted as answers leapt to mind. I love being a small part of a big success. I love coaching and cheerleading colleagues working on something difficult and important. I love pairing—learning and teaching simultaneously—and fist pumping when we track down a bug. I’d be interested and excited to tackle each of my company’s particular socio-technical challenges in a focused, disciplined way. But to make time for that, I needed to significantly change my role.

    I shared the revelation with my manager; and a few short weeks later, I handed off management responsibilities to a colleague. I became a Principal Engineer rather than Director. I’ve spent the past year mostly as an individual contributor, and mostly loving my work.

    My coach’s question has become my mantra as I set my daily intentions. It’s honed my ability to focus on where I can make meaningful progress, and let go of the rest. It helps me orient my schedule around what’s important rather than what’s urgent.

    In 2020, COVID and an immunocompromised family member upheaved my daily routines. My household navigated remote schooling and daycare with two working-from-home parents. Throughout these changes, I’m thankful for many blessings. In particular, I’m thankful for this mantra, which helped me adapt to new roles at work and at home. It’s improved my satisfaction both at work, and with my family.

    As I think of goals and intentions for the new year, I’m asking myself, “what could I work on with genuine energy and optimism”?