Lean Startup at Scale
Guest post by Lisa Regan, writer for The Lean Startup Conference.
As Lean Startup methods have been used now for a number of years, we’ve become increasingly interested in how companies use them to sustain growth. That is, once you’re no longer a small company and you have some success, how do you execute and continue to grow through innovation? Next Tuesday, October 22 at 10a PT, we’ll take a look at this advanced entrepreneurship question. In a webcast conversation, Lean Startup for Growing Companies, Eric Ries will talk with two Lean Startup Conference speakers: Wyatt Jenkins, VP of Product at Shutterstock, a stock photo site that has become one of the world’s largest two-sided marketplaces, and has expanded since its founding in 2003 from a single development team to 12 cross-functional teams spanning all phases of product development; and Ari Gesher, a senior engineer at Palantir Technologies, which specializes in data-mining software for a diverse set of problems across verticals, including disaster recovery work recognized by the Clinton Global Initiative, and has grown since its founding in 2004 to over 1,000 employees and undisclosed revenues reported to be approaching $1B today. The webcast is free with registration and will include a live Q&A with attendees.
Below are excerpts from conversations we had with Ari and Wyatt about their growing companies and about some of their suggestions for staying with Lean Startup as you expand:
When we asked Ari to talk about how he addressed some of the challenges of growth at Palantir, he gave the example of developing iterative cycles that could accommodate a scaling company:
Ari: At Palantir, we've had to tailor our software development process over time to deal with the scale of the team and the scale of our core software products. Each plateau of scale has required adjustments--changing or dropping an old process that wasn't working or creating a new process to deal with novel challenges. One good example is the way in which we've adjusted the length of different phases of our agile sprints. We don't follow a set agile methodology, but rather follow a more home-grown, minimal version of various approaches. We work in prototypically four-week iterations, with quality engineers and software developers working in close collaboration.
It wasn’t always this way. Palantir is a deep technical play and we had a lot of code to write just to fill out the product vision that we had already validated with potential customers; it took us two straight years of development to go from early prototypes to software that could be used in production. In these halcyon days, we weren't using iterative cycles as there was often nothing that could be tested for long periods of time (aside from unit testing, of course). During this period, the Palantir Gotham team grew from five developers to around 35.
This finally bit us after a four month stint of development blew through its testing schedule by a factor of four: two scheduled weeks turned into two months before the product reached stability.
So what was going on? Software development is all about managing complexity and the bigger and more mature the codebase gets, the more complex it gets. The interconnectedness had come to a head, such that new code was much more likely to disrupt existing code than ever before. As we started looking at bug counts something else became clear: we could now create bugs faster than we could find them.
We had finally hit the point where we needed to impose clear process - process whose main goal was to ensure that software stayed stable. But we couldn't have identified this without having clear metrics (that high bug count) to assess our development process. The result was a new process of four-week iterative cycles all about throttling new code. Here's the simplest form of that cycle:
The new process let us manage the velocity of change in the codebase low, keeping the resistance manageable.
More important, the iteration framework gave us something like a meta-process: we could try new ideas about how to manage development process and measure them against historical data to see what could further optimize the process.
Note that the iteration template above is our prototypical iteration. What started as a four-week cycle has since expanded to five-week cycle — adding a second week of regression to pay down technical debt. For the final iteration that turns into an external release, we push out to a six-week cycle, adding an additional week of testing: end-of-release testing.
We asked Wyatt about his experience with testing at Shutterstock, and the best ways to approach it in a growing company:
Wyatt: The concept that your "big idea" is nothing but a hypothesis until we test it is a part of the Shutterstock culture. We learned some hard lessons via A/B testing, but realized quickly that the pace at which we could perform tests and the systems for reading tests had to be both excellent and easily accessible by many different groups in the organization. Because of this (and because we believe our data is a competitive advantage that we don't want to share with third parties), we chose to build many tools ourselves. At this point, we have our own open-sourced click tracking "lil brother," an internal data visualization tool built on Rickshaw, as well as an A/B testing platform called Absinthe. We pride ourselves on the speed of testing hypotheses and reading those tests. Shutterstock is in a competitive market, but we have the most traffic and the most usage, meaning that we can run more tests (achieve significance sooner) and learn faster than our competitors.
The upshot of this is that speed matters if you hope to build, measure and learn faster than your competition. As Shutterstock has grown, there are a few key elements to our continued development speed:
Wyatt: We've employed a number of systems in the organization that keep all of us close to the customer. As we've grown, we now have a great qualitative research team dedicated helping us stay close. There are 5-10 customers in our office (or remote) per week for developers, product owners and marketers to speak to and validate learning. The important aspect of scaling customer development is to build it into the process and make it so easy to put your idea in front of a customer that everybody does it. Skip the focus groups--they don't work and they take too long to set up. Create a steady stream of customer input that anyone can dip into.
Ari also talked about the changes in the information flow as a company grows, in his case thinking of it moving in a variety of directions--between the company and its customers, certainly, but also between the company and its own internal teams, or between parts of the company itself:
Ari: One of the biggest effects of scale has to do with internal information flows. For example, a small team of four people starting work on a product has it easy. By just sitting in one room, they can have amazing shared situational awareness. There's two things at work here: as new information comes into that room, it's as easy as an offhand comment or a lunch conversation to share. The second thing is that it has no history - everyone on the team is starting from a place of zero knowledge and accumulating context as it arrives.
I joined Palantir when it was one room and fifteen people--the above model was still pretty functional. We all knew a lot about what was going on. Things like shared meals helped us stay in sync such that the knowledge about everything from the state of the product to the outcome of our last meeting with potential customers was pervasive.
Another interesting feature of those early years: most of what we needed to know was outside the company.
Growth changed all that. We've been building the company for almost ten years now and we have three major locations in the United States, as well as about half-a-dozen overseas offices. Over a thousand people work here now.
Most of the information that most of the people need to do their jobs is actually generated inside the company now. We have a long history and so new employees have to spend a long time getting up to speed on the why, what, and how of everything that we do. As a result, we've had to designs process, protocols, and infrastructure to make sure that critical information flows to the right people in a timely manner. We've had to design and implement training programs to help on-board people to our culture and technology. We have a blog that's internal-only to capture important stories for prosperity. We have a dizzying array of email lists and careful protocols about which lists get copied to make sure we can maintain a shared situational awareness that can only hope to approach what we had when we were in one room. We have an internal directory that lets people tag themselves with the things they know about--often learning is about finding who knows the answer and can explain the full why (not just the what) of something.
And none of that addresses exactly how the company as a whole learns from the world. There is now an immense flow of information coming in from the world, about how our product is working (and not working), about how our software is solving customer problems.
There are two teams that handle the bulk of the learning that comes in from the field: Support and Product Navigation. Both of these teams are collating, distilling, and turning into knowledge the information flowing from the field.
Unlike most support teams, the [Palantir] Support Team is not contacted by end users but instead by our people in the field. Our model is to place Forward Deployed Engineers (FDEs) on customer sites to add with integration of data, customization of our platforms and applications to a specific task, and training of the customer's analysts who will be the end-users of the system. If an FDE runs into trouble with the product or a novel situation that we did not anticipate, they contact the Support Team, who handles both resolving the situation, communication with the product team and documenting the knowledge gained in this situation so it can be avoided in the future.
The Product Navigators are responsible for understanding the use cases that are covered by our products--the gaps, what new features we need, what's working, what's not. This is not bug tracking but more along the lines of customer development and guidance to the product team on what to build next. They collate, distill, and prioritize information coming in from the FDEs and product instrumentation about how well the product is actually solving customer problems, what features are in use (or that users don't understand). This is a part of what many other organizations would call product management, but we decouple the learning portion of that discipline from the design portion (handled by product designers and engineers based on the knowledge created by the Product Navigation team) of product management.
Both of these teams were created to handle the sheer scale of information coming in from the field as our customer base as grown from zero to where it is today.
--
Register today for our free webcast to join Eric, Ari, and on October 22 at 10a PT. For even more Lean Startup learning, register for The Lean Startup Conference, December 9 – 11 in San Francisco.
As Lean Startup methods have been used now for a number of years, we’ve become increasingly interested in how companies use them to sustain growth. That is, once you’re no longer a small company and you have some success, how do you execute and continue to grow through innovation? Next Tuesday, October 22 at 10a PT, we’ll take a look at this advanced entrepreneurship question. In a webcast conversation, Lean Startup for Growing Companies, Eric Ries will talk with two Lean Startup Conference speakers: Wyatt Jenkins, VP of Product at Shutterstock, a stock photo site that has become one of the world’s largest two-sided marketplaces, and has expanded since its founding in 2003 from a single development team to 12 cross-functional teams spanning all phases of product development; and Ari Gesher, a senior engineer at Palantir Technologies, which specializes in data-mining software for a diverse set of problems across verticals, including disaster recovery work recognized by the Clinton Global Initiative, and has grown since its founding in 2004 to over 1,000 employees and undisclosed revenues reported to be approaching $1B today. The webcast is free with registration and will include a live Q&A with attendees.
Below are excerpts from conversations we had with Ari and Wyatt about their growing companies and about some of their suggestions for staying with Lean Startup as you expand:
When we asked Ari to talk about how he addressed some of the challenges of growth at Palantir, he gave the example of developing iterative cycles that could accommodate a scaling company:
Ari: At Palantir, we've had to tailor our software development process over time to deal with the scale of the team and the scale of our core software products. Each plateau of scale has required adjustments--changing or dropping an old process that wasn't working or creating a new process to deal with novel challenges. One good example is the way in which we've adjusted the length of different phases of our agile sprints. We don't follow a set agile methodology, but rather follow a more home-grown, minimal version of various approaches. We work in prototypically four-week iterations, with quality engineers and software developers working in close collaboration.
It wasn’t always this way. Palantir is a deep technical play and we had a lot of code to write just to fill out the product vision that we had already validated with potential customers; it took us two straight years of development to go from early prototypes to software that could be used in production. In these halcyon days, we weren't using iterative cycles as there was often nothing that could be tested for long periods of time (aside from unit testing, of course). During this period, the Palantir Gotham team grew from five developers to around 35.
This finally bit us after a four month stint of development blew through its testing schedule by a factor of four: two scheduled weeks turned into two months before the product reached stability.
So what was going on? Software development is all about managing complexity and the bigger and more mature the codebase gets, the more complex it gets. The interconnectedness had come to a head, such that new code was much more likely to disrupt existing code than ever before. As we started looking at bug counts something else became clear: we could now create bugs faster than we could find them.
We had finally hit the point where we needed to impose clear process - process whose main goal was to ensure that software stayed stable. But we couldn't have identified this without having clear metrics (that high bug count) to assess our development process. The result was a new process of four-week iterative cycles all about throttling new code. Here's the simplest form of that cycle:
- Week -1 - Planning/End-of-Cycle - Software engineers are planning: writing specifications, doing light prototyping, and experimentation. During this week, quality engineers are attending planning meetings but also doing end-of-cycle testing of the previous iteration - the final testing on an internal release before it's deployed to our dog-fooding systems.
- Week 0 - New Feature 1 - Software engineers are busy building brand new features. Quality engineers are writing up test plans based on the specifications and creating test data (if necessary). By mid-week, the software engineers have to deliver something testable so the quality engineers can start testing and (of course) filing bugs.
- Week 1 - New Feature 2 - Development and testing continue together. By the end of the week, the new code should be stable, with all filed bugs fixed.
- Week 2 - Regression - Software engineers start paying off technical debt by attacking a queue of existing bugs. Quality engineers make full regression passes through the software, looking for old functionality that may have been impacted by the new changes this iteration.
- Week 3 - Planning/End-of-Cycle - See Week -1.
The new process let us manage the velocity of change in the codebase low, keeping the resistance manageable.
More important, the iteration framework gave us something like a meta-process: we could try new ideas about how to manage development process and measure them against historical data to see what could further optimize the process.
Note that the iteration template above is our prototypical iteration. What started as a four-week cycle has since expanded to five-week cycle — adding a second week of regression to pay down technical debt. For the final iteration that turns into an external release, we push out to a six-week cycle, adding an additional week of testing: end-of-release testing.
We asked Wyatt about his experience with testing at Shutterstock, and the best ways to approach it in a growing company:
Wyatt: The concept that your "big idea" is nothing but a hypothesis until we test it is a part of the Shutterstock culture. We learned some hard lessons via A/B testing, but realized quickly that the pace at which we could perform tests and the systems for reading tests had to be both excellent and easily accessible by many different groups in the organization. Because of this (and because we believe our data is a competitive advantage that we don't want to share with third parties), we chose to build many tools ourselves. At this point, we have our own open-sourced click tracking "lil brother," an internal data visualization tool built on Rickshaw, as well as an A/B testing platform called Absinthe. We pride ourselves on the speed of testing hypotheses and reading those tests. Shutterstock is in a competitive market, but we have the most traffic and the most usage, meaning that we can run more tests (achieve significance sooner) and learn faster than our competitors.
The upshot of this is that speed matters if you hope to build, measure and learn faster than your competition. As Shutterstock has grown, there are a few key elements to our continued development speed:
- Small, autonomous teams: The more a team can do on their own, the faster they can go. The hand-offs between teams are (mostly) eliminated, and close-working autonomy creates a good startup vibe as well.
- Continuous deployment: A key component of speed is to keep pushing out work. This has kept us lean as well—we don't have release trains, and code generally goes live to the site every day of the week.
- Don't get religious about process – just continuously improve. You need to strike a balance between process and problem-solving. You don’t want to get so committed to a particular process that you can’t adapt to problems as they actually present themselves. So, depending on which team you are talking to at Shutterstock, we may be Lean Startup or Agile or Kanban or some other method depending on the type of problem that team is designed to solve. But that doesn’t mean that we don’t take these ideas seriously. You want to be flexible enough to change your process across teams as you scale--but as a rule of thumb we question every new bit of process someone tries to add because process is easy to add and very difficult to remove once it's in place. For that reason, it’s important to go with methods, like Lean Startup, that have proven results for the kind of problem that team is trying to address.
Wyatt: We've employed a number of systems in the organization that keep all of us close to the customer. As we've grown, we now have a great qualitative research team dedicated helping us stay close. There are 5-10 customers in our office (or remote) per week for developers, product owners and marketers to speak to and validate learning. The important aspect of scaling customer development is to build it into the process and make it so easy to put your idea in front of a customer that everybody does it. Skip the focus groups--they don't work and they take too long to set up. Create a steady stream of customer input that anyone can dip into.
Ari also talked about the changes in the information flow as a company grows, in his case thinking of it moving in a variety of directions--between the company and its customers, certainly, but also between the company and its own internal teams, or between parts of the company itself:
Ari: One of the biggest effects of scale has to do with internal information flows. For example, a small team of four people starting work on a product has it easy. By just sitting in one room, they can have amazing shared situational awareness. There's two things at work here: as new information comes into that room, it's as easy as an offhand comment or a lunch conversation to share. The second thing is that it has no history - everyone on the team is starting from a place of zero knowledge and accumulating context as it arrives.
I joined Palantir when it was one room and fifteen people--the above model was still pretty functional. We all knew a lot about what was going on. Things like shared meals helped us stay in sync such that the knowledge about everything from the state of the product to the outcome of our last meeting with potential customers was pervasive.
Another interesting feature of those early years: most of what we needed to know was outside the company.
Growth changed all that. We've been building the company for almost ten years now and we have three major locations in the United States, as well as about half-a-dozen overseas offices. Over a thousand people work here now.
Most of the information that most of the people need to do their jobs is actually generated inside the company now. We have a long history and so new employees have to spend a long time getting up to speed on the why, what, and how of everything that we do. As a result, we've had to designs process, protocols, and infrastructure to make sure that critical information flows to the right people in a timely manner. We've had to design and implement training programs to help on-board people to our culture and technology. We have a blog that's internal-only to capture important stories for prosperity. We have a dizzying array of email lists and careful protocols about which lists get copied to make sure we can maintain a shared situational awareness that can only hope to approach what we had when we were in one room. We have an internal directory that lets people tag themselves with the things they know about--often learning is about finding who knows the answer and can explain the full why (not just the what) of something.
And none of that addresses exactly how the company as a whole learns from the world. There is now an immense flow of information coming in from the world, about how our product is working (and not working), about how our software is solving customer problems.
There are two teams that handle the bulk of the learning that comes in from the field: Support and Product Navigation. Both of these teams are collating, distilling, and turning into knowledge the information flowing from the field.
Unlike most support teams, the [Palantir] Support Team is not contacted by end users but instead by our people in the field. Our model is to place Forward Deployed Engineers (FDEs) on customer sites to add with integration of data, customization of our platforms and applications to a specific task, and training of the customer's analysts who will be the end-users of the system. If an FDE runs into trouble with the product or a novel situation that we did not anticipate, they contact the Support Team, who handles both resolving the situation, communication with the product team and documenting the knowledge gained in this situation so it can be avoided in the future.
The Product Navigators are responsible for understanding the use cases that are covered by our products--the gaps, what new features we need, what's working, what's not. This is not bug tracking but more along the lines of customer development and guidance to the product team on what to build next. They collate, distill, and prioritize information coming in from the FDEs and product instrumentation about how well the product is actually solving customer problems, what features are in use (or that users don't understand). This is a part of what many other organizations would call product management, but we decouple the learning portion of that discipline from the design portion (handled by product designers and engineers based on the knowledge created by the Product Navigation team) of product management.
Both of these teams were created to handle the sheer scale of information coming in from the field as our customer base as grown from zero to where it is today.
--
Register today for our free webcast to join Eric, Ari, and on October 22 at 10a PT. For even more Lean Startup learning, register for The Lean Startup Conference, December 9 – 11 in San Francisco.
0 comments:
welcome to my blog. please write some comment about this article ^_^