One of the more powerful concepts I’ve found is considering which type of analysis to apply to the challenge at hand. It’s been especially useful when coaching a software team to put in more and different effort to find the root cause of a failure, or occasionally to save effort when over-analyzing a situation.
So here’s my survey of ‘types of analysis’, inspired by (and expanding on) a section of Accelerate, chapter 12.
| Forensic | Predictive | Causal | Mechanistic | |
|---|---|---|---|---|
| Confidence of claims | Low, new evidence could drastically change conclusions | Modest, based on suppositions about the past and present contexts | High, this was definitely true in the past | Very high, claims will be true forever |
| General uses | Paleontology, criminal investigations | Meteorology, government policies | Clinical trials | Rocket science, material science |
| Software development uses | Security incident investigations | Roadmap planning, project selection, scoping, and sequencing | A/B tests of user behavior | Unit testing, debugging |
Forensic analysis
Forensic analysis seeks to understand an incident when it’s impossible to fully recreate the incident.
Forensic analysis gathers evidence, then uses abductive reasoning to find the most reasonable explanation that fits evidence. When criminal detectives and attorneys argue a theory of a crime, or paleontologists describe how dinosaurs behaved based on fossil records, they’re using abductive reasoning. Any claims based on forensic analysis and abductive reasoning are contingent on the available evidence. It’s always possible that new evidence comes to light that dramatically changes our understanding of an incident.
In software development, we use forensic analysis to investigate security incidents, or when we search through logs to figure out when or how something occurred.
Forensic analysis is often the least effort to perform, and the least certain or generalizable in its conclusions.
Predictive analysis
Predictive analysis uses historical data and inductive reasoning to make claims about the future in an uncontrolled environment.
When a meteorologist makes a weather forecast, or when the CBO forecasts what a new tax policy will cost over the next decade, they’re using predictive analysis. When lawmakers consider the pros and cons of passing a law, they (ideally) use predictive analysis.
Because these analyses are uncontrolled, it’s impossible to repeat an experiment. That is, if you make a decision based on forecasts, you can’t know precisely what would have happened if you’d made a different decision. Like, what would society be like if the government had passed a different law years ago? It’s a counterfactual. The best we can do is speculate.
In software development, we use predictive analysis when creating roadmaps, scoping, and sequencing our projects. We can only speculate about how things would turn out if we chose different projects.
Causal analysis
Causal analysis uses controlled experiments to see both sides of a decision (i.e., a treatment and control group). This lets us make much stronger claims about the efficacy of a treatment than just predictive analysis.
Causal analysis is most well known in medicine for clinical drug trials. By comparing the outcomes of subjects in the control and treatment groups, we can be fairly certain that differences in outcomes were caused by the treatment.
In software development, we use causal analysis when we do A/B tests. Is the user more likely to click the button if we present information this way, or that way?
Causal analysis is not generalizable. Are the subjects in an experiment representative of the broader population? You need to re-do the experiment to find out. Say an e-commerce company finds that adding a big flashing “NEW” icon next to products increases sales. How long does that effect last? You’d have to do another A/B test to find out. Would it work as well for a different brand? Gotta do another A/B test.
Causal analysis is clever in that the mechanism by which the treatment works is irrelevant to the experiment. It doesn’t let us say why a new drug works, or why users click the button more; just that we have some degree of confidence that the treatment causes some desirable outcome.
Mechanistic analysis
Mechanistic analysis supports the strongest claims about why and how a system works.
It requires a thorough understanding of the mechanisms that make the system work. You can repeatedly apply the well-modeled system to new situations. It relies well-known scientific theories that are taken as axioms, and uses deductive reasoning to derive useful applications.
Some examples: When 19th century inventors sent electrical current through carbon filaments and discovered that the filaments glowed, they discovered a physical properties that we can use to produce artificial light. Such a system is enduring: every carbon filament glows in predictable ways when you run electricity through it.
When rocket scientists calculate how much fuel they’ll need to launch a payload into orbit, or when civil engineers analyze whether a bridge design will support a given load, they use mechanistic analysis. Then answer is the solution to a mathematical equation.
They don’t build many bridges, then drive cars over them to see if they stay up. That kind of predictive analysis is insufficient for the challenge.
In software engineering, mechanistic analysis is best used when testing and debugging. If we can reliably reproduce a bug, or reliably verify the correctness of a system, it’s strong evidence that we’ve isolated a property of our system, and we’re well on our way to knowing how to modify the system to fix the bug.
Mechanistic analysis requires the most effort, but yields the highest confidence in our understanding of a system.
Conclusion
Enumerating these 4 types of analysis from low effort / weak claims to high effort / strong claims has helped me coach teams on a pragmatic depth of understanding. To decide if we should invest effort in a product change, we’ll use predictive analysis. To learn exactly how much the product change improved the user experience, we’ll use causal analysis. And to ensure we thoroughly fix any bugs along the way, we’ll use mechanistic analysis.
