At HashiCorp, we’ve grown from a few hundred to over a thousand people, so the goal is to build scalable systems that enable employees to do their best work and contribute to the outcomes of the company. For us, that’s shaped up into three specific systems: strategic planning, knowledge management, and communications.”
They also run a simluation to give their leaders a chance to practice.
“Using a firm called BTS, we run a business simulation where leaders get to ‘run’ the business for three years. Taking a simplified view of the company, we essentially build a game board based on our five-year financial model and this year’s three executive focus areas,” says Fishner.
Stephen Covey explained that “trust is a function of two things: competence and character. Competence includes your capabilities, your skills, and your track record. Character includes your integrity, your motive and your intent with people. Both are vital.”
Great teams are comprised of ordinary people that are empowered and inspired.
Truly empowered teams that produce extraordinary results don’t require exceptional hires. They do require people that are competent and not assholes, so they can establish the necessary trust with their teammates and with the rest of the company.
Truly empowered teams also need the business context that comes from the leadership – especially the product vision – and the support of their management, especially ongoing coaching, and then given the opportunity to figure out the best way to solve the problems they have been assigned.
The fourth role is by far the most important. It’s the role the vast majority of engineers will follow in their careers, and I’m going to call it “This. Forever.” The role you have right now is the thing you are going to do be doing forever.
A depressing thought? Not when you remember you’re on a quest.
To sum up: Variance is the enemy of performance and the source of much of the latency we encounter when using software.
To keep latency to a minimum:
- As a rule of thumb, target utilization below 75%,
- Steer slower workloads to paths with lower utilization,
- Limit variance as much as possible when utilization is high,
- Implement backpressure in systems where it is not built-in,
- Use throttling and load shedding to reduce pressure on downstream queues.
Pat has collected a list of resources for learning about product management.
Most explanations of organizational success or failure are crap.
an organization doing work is just an incredibly complex, dynamic, distributed, parallel process.
As with writing highly-concurrent applications, building high-performing organizations requires a careful and continuous search for shared resources, and developing explicit strategies for mitigating their impact on performance.
The only scalable strategy for containing coherence costs is to limit the number of people an individual needs to talk to in order to do their job to a constant factor.
There are so many quotable sections in this piece.
Go and have a read if you think about organisations, how they work, and how to improve them.
Anyway, Zuckerberg is in the news along with News Corp boss Rupert Murdoch, in a heartwarming generational fight between billionaires for who gets to say: “Bitch, I’m not IN the news, I OWN the news.”
I don’t need areas or projects to do my work effectively. I don’t need complex chaining rules and sequences. I don’t need start dates or tags. In late November I realized that while Things still worked fine for me, I was using it not for task management any longer — but instead I was using it as a reminder engine.
I think all I need is a Reminder engine, I am pretty good at getting my tasks done once I am reminded of them.
I’ve been thinking about how working as a manager differs from working as an individual contributor. One thing I’ve noticed is that my day to day work as a manager has become more asynchronous.
Projects tend to be composed of a series of pings and responses with the odd timeout when I haven’t heard back from someone.
This post from Ben resonates with me because I feel that my productivity system is more useful when it reminds me to do things rather than helps me break down the steps of a particular task.
Convincing someone to change their mind is really the process of convincing them to change their tribe. If they abandon their beliefs, they run the risk of losing social ties. You can’t expect someone to change their mind if you take away their community too. You have to give them somewhere to go. Nobody wants their worldview torn apart if loneliness is the outcome.
In every conversation you’re part of, create clarity and reduce chaos.
With more advice on how to do that in the linked thread.
A series of posts from Hillel Wayne resulting from interviews with people who have worked as both traditional engineers and software engineers.
I appreciate that this is based upon the views of folks who’ve actually worked on both sides of the fence.
The word “scalable” has a mystical and stupefying power over the mind of the software engineer. Its mere utterance can whip them into a depraved frenzy. Grim actions have been justified using this word.
An inspection of how values affect software through the lens of text editors.
Because I was burned out and didn’t want to think about working any more than I needed to, I instituted a no-meeting, no-deadline culture.
For me, it was no longer about growth at all costs, but “freedom at all costs.”
At some point, it clicked: Creators make money so they can make stuff, instead of the other way around. Why not adopt this framing at Gumroad, too?
This is what working in the creator economy should feel like.
A look at the unorthodox way Gumroad operates.
No. 26 (Montemerlo’s Law) Don’t do nuthin’ dumb.
A Campaign is a long-running effort to enact global change safely within a sociotechnical system.
Campaigns work well to address:
- Technical changes with large social components.
- Technical changes that require everyone to do a little bit of work.
- High-value or inevitable future worlds
Clarity enables patterns of coherent action across the organization. Contextual understanding supports initiative–people don’t have to wait to be told what to do. It speeds decision-making as people closer to the work have the information to make more decisions in a reasonable way. Clarity reduces the need for supervision.
Smoothing the flow of work, organizing work and teams to reduce dependencies and handoffs, is management work.
Sadly, or perhaps luckily, most experimentation isn’t all that dramatic. It mainly means trying something new and using the results to decide a further course of action. You can experiment with a more efficient route to the office, with a new dinner recipe, or a new vacation destination. The element that distinguishes an experiment from serendipity is that you have a hypothesis that you are looking to verify or falsify based on data that you glean from running the experiment.
But the point is, in just a few short weeks on the job, I had already realized that because every tough decision came down to a probability, then certainty was an impossibility — which could leave me encumbered by the sense that I could never get it quite right. So rather than let myself get paralyzed in the quest for a perfect solution, or succumb to the temptation to just go with my gut every time, I created a sound decision-making process — one where I really listened to the experts, followed the facts, considered my goals and weighed all of that against my principles. Then, no matter how things turned out, I would at least know I had done my level best with the information in front of me.
RC has four social rules. They help create a friendly, intellectual environment where you can spend as much of your energy as possible on programming.
The social rules are:
- No well-actually’s
- No feigned surprise
- No backseat driving
- No subtle -isms
One thing that often confuses people about the social rules is that we expect people to break them from time to time. This means they’re different and totally separate from our code of conduct.
Alright folks, gather round and let me tell you the story of (almost) the biggest engineering disaster I’ve ever had the misfortune of being involved in. It’s a tale of politics, architecture and the sunk cost fallacy
Paul McCartney has been writing and performing music more or less continuously since 1956. That’s sixty-four years.
There are three moments in the yearlong catastrophe of the COVID-19 pandemic when events might have turned out differently.
Kent C. Dodds:
- Not everything in your application needs to be in a single state object. Keep things logically separated (user settings does not necessarily have to be in the same context as notifications). You will have multiple providers with this approach.
- Not all of your context needs to be globally accessible! Keep state as close to where it’s needed as possible.
TL;DR “Microservices” was a good idea taken too far and applied too bluntly. The fix isn’t just to dial back the granularity knob but instead to 1) focus on the split-join criteria as opposed to size; and 2) differentiate between the project model and the deployment model when applying them.
There are build patterns that give us total control over the relative visibility between components and over how they compose into deployables. One build to many deployables, many to one or anything in between is possible. Shifting concerns left helps us build better software. An earlier error is a better error. Constraints liberate.
The tensions and constraints that shape the arrangement of our projects, modules and dependencies are of a different nature than those shaping the arrangement of our deployables. A little build-fu goes a long way in combining development friendliness with mechanical sympathy.
My system for processing email is similar to Xavier’s.
Though, I use the Things Helper app to add links to emails directly to my Things inbox.
Danish values also provide a three-step prescription to turn the day around: “In Denmark, we have sort of a mental health [checklist]: Do something active. Do something together with other people. Do something meaningful.”
Great scientists tolerate ambiguity very well. They believe the theory enough to go ahead; they doubt it enough to notice the errors and faults so they can step forward and create the new replacement theory. If you believe too much you’ll never notice the flaws; if you doubt too much you won’t get started.
Feedback is someone is trying to help. Listen. Then ask questions - to gain understanding of their point of view.
I also enjoyed the idea of treating Slack more like Twitter.
Peter Naur’s classic 1985 essay “Programming as Theory Building” argues that a program is not its source code. A program is a shared mental construct (he uses the word theory) that lives in the minds of the people who work on it. If you lose the people, you lose the program. The code is merely a written representation of the program, and it’s lossy, so you can’t reconstruct a program from its code.
A copy of the paper.
Ken Kocienda tells stories of how Apple built the iPhone and iPad: a small team (less than twenty people) charged with delivering on direction set by executives.
- Viewing product development decisions as economic tradeoffs.
- Shifting from deterministic planning models to a stochastic perspective.
- Managing the Fuzzy Front End: the time from opportunity identification to commitment.
- Assessing the reserve buoyancy of a project.
- Using loss of safety margin as a leading indicator of project deterioration.
- Becoming more rigorous about how we think about technical debt.
Dashboards are the human-facing views into our systems that provide concise summaries of how the system is behaving by displaying time series metrics, logs, traces, and alarms data.
A look at how Amazon do dashboards.
Below is a list of some lessons I’ve learned as a distributed systems engineer that are worth being told to a new engineer. Some are subtle, and some are surprising, but none are controversial.
161 suggestions from the staff at Automattic.
Trade offs and spooky stories from Stefan Tilkov.
Game design and unsuccessful monad tutorials.
As software engineers our job is not to produce code per se, but rather to solve problems. Unstructured text, like in the form of a design doc, may be the better tool for solving problems early in a project lifecycle, as it may be more concise and easier to comprehend, and communicates the problems and solutions at a higher level than code.
Since HEY made a big splash on arrival, I thought it’d be fun to share the backstory of how we ended up reinventing email. Because we certainly didn’t start by wanting to reinvent email.
Assume that every student you interact with has limited information, but infinite intelligence. That places the onus squarely on the shoulders of the mentor to make sure that their explanations make sense — which, given the inherent imbalance of power between a teacher and a learner, is a fine way to distribute the extra emotional labor.
Process is documented culture. How a team gets a familiar thing done should be broadly understood by the team. This is how we fix a bug. This is how we do a code check-in. This is how a feature is designed. This is how executive sign-off occurs.
Process comfortably and efficiently describes the common path. Process does not define what to do when the indescribable occurs. A crisis or a disaster does not neatly fit into the common path; it’s when you need someone to swoop in, break the glass, and put out the fire.
Finally, to increase communication, especially if the message is vital, use the three-way handshake. Tell your message to someone using whatever medium you’re using. Then, have that person tell you your message back (in their own words of course, no copy and paste). You then repeat that message back to them. Assuming everyone has it right, you’ve just completed a three-way handshake.
The purpose of a meeting, then, is not to convey information efficiently. It is to force an audience to pay exclusive attention to one thing, to get that creative focus pointed in a particular direction.
Complex systems, cascades (from neurons up to people and then crowds), failures, too much efficiency, not enough slack, and toilet paper.
Dan North on how to break the rules when applying the Theory of Constraints.
A look at negotiation and how to avoid the pitfalls associated with debate.
A nice rundown of implementing Sagas in a distributed system.
Makers receive constant praise for solving problems, and take pleasure in being the expert. Leaders in Maker mode go out of their way to show they have the right answer. They need to have the first and last say. They over invest in their own solutions and don’t create space for others to contribute.
Multipliers amplify or multiply the intelligence of the people around them. They lead organisations or teams that are able to understand and solve hard problems rapidly, achieve their goals, and adapt and increase their capacity over time.
There is a sweet spot of React: in moderately interactive interfaces. Complex forms that require immediate feedback, UIs that need to move around and react instantly.
I’m still a sucker for server-generated markup.
Michael Deng and Jonathan Chang:
Deploys require a careful balance of speed and reliability. At Slack, we value quick iteration, fast feedback loops, and responsiveness to customer feedback.
An interesting dive into how Slack handles deploys to a large fleet of users.
In the war to reclaim your attention, some battles have clearer fronts than others. It has become clear to me that these differences matter.
An attention charter is a document that lists the general reasons that you’ll allow for someone or something to lay claim to your time and attention. For each reason, it then describes under what conditions and for what quantities you’ll permit this commitment.
I use LucidChart for collaborative drawing at work but I’m going to give Excalidraw a crack over the next few weeks.
A solid collection of remote working advice. There is some overlap with the strategies I employ.
I wish I’d found this project before I rewrote a Strange kind of Madness by hand using Hugo.
What I’ve learned is that if we want things to go fast, a sense of momentum is much more effective than a sense of urgency.
Stories focus on the why, sharing the experience and context around the decision and results.
Stories also eliminate most of the least productive follow-up conversations after giving advice, where the advice-requester then argues against the advice. Stories relieve the advice-giver from the obligation to defend their advice. There’s nothing to agree or disagree with, just a recounting of past events.
It’s really helpful to respond to a person’s ineffective behavior with curiosity rather than judgement.
If a person’s behavior doesn’t make sense to you, it is because you are missing a part of their context.
Every line of code written comes at a price: maintenance. To avoid paying for a lot of code, we build reusable software. The problem with code re-use is that it gets in the way of changing your mind later on.
Building reusable code is something that’s easier to do in hindsight with a couple of examples of use in the code base, than foresight of ones you might want later.
A look back at Carmack using academic research to improve rendering performance in the original Doom.
Jesse Frederik and Maurits Martijn:
Is online advertising working? We simply don’t know.
(we look at) how to handle writes to two independent backends without using two-phase commits. Instead we can rely on using at-least-once delivery guarantees and ask other systems to deduplicate our messages.
Inspired by CRDTs but simpler due to the server being a central authority.
Kent Beck comparing baskets of options to product roadmaps and goals:
I’ve come to hate the damage the “product roadmap” metaphor does to the brains of everyone involved in developing a product. When I use an actual map of actual roads, I assume that I know where I’m going and how I’m going to get there. This is never the case when developing a product.
When you encounter long lead times, you’re hearing option-on-a-basket thinking. “We need to know what features will be in the release in 8 months so Marketing has time to prepare.” What if product development doesn’t go according to plan? The value of the option on a basket falls to zero. What if the launch doesn’t come off? The value of the option on a basket falls to zero.
The man can present.
Estimates matter because most people and businesses are date-driven.
Estimation is difficult but developing it as a skill is helpful for delivering software.
it’s actually an unqualified good for engineers to be interacting with production on a daily basis, observing the code they wrote as it interacts with infrastructure and users in ways they could never have predicted.
A system’s resilience is not defined by its lack of errors; it’s defined by its ability to survive many, many, many errors. We build systems that are friendlier to humans and users alike not by decreasing our tolerance for errors, but by increasing it. Failure is not to be feared. Failure is to be embraced, practiced, and made your good friend.
When people have divided attention, work suffers. The area of code that you work for months is something that you understand deeply. The framework, off to the side, that you update just to facilitate your work may not seem as important. This is a function of distance: cognitive, temporal, and locational distance. In a way, these are all the same.
Gary P. Pisano:
The cliché “celebrating failure” misses the point—we should be celebrating learning, not failure.
Without discipline, almost anything can be justified as an experiment. Discipline-oriented cultures select experiments carefully on the basis of their potential learning value, and they design them rigorously to yield as much information as possible relative to the costs.
The first step is acknowledging that our relationship is more important than the design of the system. As long as we have a productive working relationship we can move the design in any direction. When our relationship breaks down we don’t get anywhere.
Okay, so you just want to go implement the next feature and along I come and say no no no this should be designed completely differently. Even if you are right that the new structure will eventually make my behavior changes easier to implement it’s not eventually, it’s today.
First, acknowledge that our incentives diverge in this moment. It doesn’t help to pretend that we agree when we don’t.
Second, as the structure changer I need to acknowledge that I am placing a burden of learning on you. I think it’s worth it, but if I’m asking something of you I better be prepared to offer something to you.
Software design is a human relationship problem with interesting technical aspects. Geeks relating to geeks requires as much effort as geeks relating to their systems. Maintaining relationships may be hard and confusing and frustrating to geeks (I could be projecting here but yeah no I don’t think I am), but if you want your technical skills to matter you really have no choice but to improving your people skills.
We couldn’t just start replacing old code with new code willy-nilly; without some type of structure to keep the old and new code separate, they’d end up getting hopelessly tangled together and we’d never have our modern codebase. To solve this problem, we introduced a few rules and functions in a concept called legacy-interop:
- old code cannot directly import new code: only new code that has been “exported” for use by the old code is available
- new code cannot directly import old code: only old code that has been “adapted” for use by modern code is available.
The progressive approach to the rebuild is interesting. Especially the rules that enforced how the rewritten parts of the code base could interact with the old code they were ultimately replacing.
Richard Hamming giving advice to researchers in 1995, plenty of which serves as general career advice.
Here’s a selection:
- Work on important problems.
- Luck favours a prepared mind.
- Work on problems you’re committed to.
- Talk to people outside of your field.
- Pursue opportunities when they’re presented.
- Schedule regular time for deep reflections.
- Take a step back to see the larger problem.
- Every defect can be looked at as an asset.
I often say that with knowledge workers, the biggest bottleneck is always getting up in the morning. Knowledge work requires not only our time and effort, but also our engagement and creativity. For that reason, personal motivation is the prime problem that supersedes all other problems.
BFFs are not about the shape of your endpoints, but about giving your client applications autonomy.
Rough consensus relies on the distinction between two types of objections:
“Not the best choice” feedback: “I don’t believe Solution A is the best choice, because XYZ. I believe Solution B would be better, but I accept that Solution A can work too.”
Fundamental flaws: “I believe Solution A is unacceptable because XYZ.”
A chair who asks, “Is everyone OK with choice A?” is going to get objections. But a chair who asks, “Can anyone not live with choice A?” is more likely to only hear from folks who think that choice A is impossible to engineer given some constraints. The objector might convince the rest of the group that the objections are valid and the working group might choose a different path.
In my very first programming role my manager said to me “You can make any mistake you like once. You’ll have my full support the first time you screw anything up. If you’re not making mistakes, you’re not learning, and if you’re repeating mistakes you aren’t either”.
A leader doesn’t shape people – a leader shapes an environment.
In order to succeed at production ownership, a team needs a roadmap for developing the necessary skills to run production systems. We don’t just need production ownership; we also need production excellence. Production excellence is a teachable set of skills that teams can use to adapt to changing circumstances with confidence. It requires changes to people, culture, and process rather than only tooling.
Even a perfect set of SLOs and instrumentation for observability do not necessarily result in a sustainable system. People are required to debug and run systems. Nobody is born knowing how to debug, so every engineer must learn that at some point. As systems and techniques evolve, everyone needs to continually update with new insights.
Standardizing technology is a powerful way to create leverage: improve tooling a bit and every engineer will get more productive. Adopting superior technology is, in the long run, an even more powerful force, with successes compounding over time. The tradeoffs and timing between standardizing on what works, exploring for superior technology, and supporting adoption of superior technology are at the core of engineering strategy.
An effective approach is to prioritize standardization, while explicitly pursuing a bounded number of explorations that are pre-validated to offer a minimum of an order of magnitude improvement over the current standard.
Ideas are funny things. It can take hours or days or months of noodling on a concept before you’re even able to start putting your thoughts into a shape that others will understand. And by then, you’ve explored the contours of the problem space enough that the end result of your noodling doesn’t seem interesting anymore: it seems obvious.
But as you get into more senior-type engineering roles, your most valuable contributions start to take the form not of concrete labor, but of conceptual labor. You’re able to draw on a rich mental library of abstractions, synthesizing and analyzing concepts in a way that only someone with your experience can do.
When building a dashboard, start with a set of questions you want to answer about a system’s behavior, and then choose where and how to add instrumentation; not the other way around.
The incident response team’s common ground is their theory of the system’s behavior – in order to make troubleshooting observations meaningful, that theory needs to be kept up to date with the data.