Thoughts on scientific product development

I enjoyed reading a post today from Laserlike (Mike Speiser), on Scientific product development.

By embracing a scientific approach to product development, not only will your business have a much higher probability of success, but it will also be a more fun and creative place to work. Nothing kills innovation like the fear of failure. And nothing leads to failure like a process that resembles astrology more than it does astronomy.

There are two concepts I want to delve into a little more. The first is his version of the Code/Data/Learn feedback loop.

He focuses on having a clear idea of the problem you are trying to solve, running experiments, and then: "Did the results of your test match what you expected? If not, kill the feature and start over." My experience is this kind of thinking can run into trouble. The two extremes our brains tend to swing to are either "I know it's right, just do it" or, if things don't go as planned, "screw it, just give up." The goal of iterative development is to give us guard rails so we don't veer off to either extreme. When an experiment doesn't go as planned, that's the time to learn. Why didn't reality conform to our expectations? Unpacking our assumptions usually leads to important insights that should be plowed into a next edition of the feature. You can't do science and be defensive at the same time. If the product people on your team think they have to get every experiment right on the first try, there's no chance you're going to iterate and learn. So don't just kill the feature - iterate. Only when you've tried everything you can think of, and you're not learning any more, is it time to bring out the axe. Keep split-testing, but keep this iron rule: if it doesn't change customer behavior, it's not a feature. Kill it.



A second idea is that less is more in product design:

A key tenet of the philosophy is that uncluttered products with fewer, better features are preferred to similar products with more features. I agree with the less is more product development approach, but for a different reason.

The reason I like less is more as an approach is that it allows for a more scientific approach to product development. By starting a new product off with as few features as possible (1?), you can be incredibly scientific. With 10 features in a single release, you may spend more time trying to figure out what is working and what isn’t working than it took to build the thing in the first place. As you incrementally experiment with your product, you can observe the impact of a particular feature one at a time and adjust accordingly.

This is a great articulation of the principle that working in small batches allows problems to become instantly localized. The smaller the unit of a release is, the more you can learn about it, because the number of variables that have changed is so few. It's worth doing this not just for features, but for any changes you make to your product. For example, trying to make server software more scalable. Our instinct is sometimes to do giant big-bang rewrites. These rarely work on the first try - usually small mistakes wind up blowing away all the benefit of the macro change. If we can find ways to break the change down into tiny pieces, we can find and revert the small mistakes as they are introduced.

When I've worked to get everyone to build in small batches, I would see this pattern in action. Lots of engineers are busy checking in and deploying their work. Someone has managed to convince themselves that they have to do their big architecture change in one fell swoop. So they work on their own branch for a while, and then try to check in their massive changeset. All day long, I keep noticing them "bouncing" their changes against production: constantly checking it in, realizing something's not right, and reverting. What's not right? First, the tests don't pass. Then it turns out theirs some integration problem because their long-lived branch diverged too much. Then, during deployment, the cluster immune system rejects the change... etc. Meanwhile, a dozen other engineers are getting annoyed, because this constant bouncing is making it harder for them to work (they can't deploy changes while tests or the cluster are in red-alert mode). When we think that working solo, on a branch, in large batches is more efficient, this is the kind of waste we forget about. Sure, we kept that one engineer busy while they toiled away on their own, but did that optmize the whole team's efforts?

Working in a scientific and iterative way is not about cold, calculating facts. On the contrary - creativity being constantly challenged by reality is beautiful. This is a point practitioners of science are always trying to make. My experience is that this way of working is liberating. We don't have to sit around and argue about who's opinion is correct. We can just blitz through new, creative ideas, testing to see what's real. Now that is fun.