Case Study: Using an LOI to get customer feedback on a minimum viable product
How much work should you do on a new product before involving customers? If you subscribe to the theory of the minimum viable product, the answer is: only enough to get meaningful feedback from early adopters. Sometimes the best way to do this is to put up a public beta and drive a limited amount of traffic to it. But other times, the right way to learn is actually to show a product prototype to customers one-on-one. This is especially useful in situations, like most B2B businesses, where the total number of customers is likely to be small.
This case study illustrates one company’s attempt to do customer development by testing their vision with customers before writing a single line of code. In the process, they learned a lot by asking initial prospects to sign a non-binding letter of intent to buy the software. As you’ll see, this quickly separated the serious early adopters from everyone else. Mainstream customers don’t have enough motivation to buy an early product, and so building in response to their feedback is futile.
Along the way, this case study raises interesting ethical issues. The lean startup methodology is based on enlisting customers as allies, which requires honesty and integrity. If you deceive customers by showing them screenshots of a product that is “in-development” but for which you have written no code, are you lying to them? And, if so, will that deception come back to haunt you later? Read on and judge for yourself.
The following was written an actual lean startup practitioner. It was originally posted anonymously to the Lean Startup Circle mailing list, and then further developed on the Lean Startup Wiki’s Case Studies section. If you’re interested in writing a future case study, or commenting/contributing to one, please join the mailing list or head on over to the wiki. What follows is a brief introduction by me, the case study itself, and then some Q&A led by LSC creator Rich Collins. Disclaimer: claims and opinions expressed by the authors of case studies are theirs alone; I can’t take credit or responsibility. – Eric Ries
In April of 2009 my partner and I had an idea for a web app, a B2C platform that we are selling as SaaS [software-as-a-service]. We decided from the get-go that, while we clearly saw the benefits and necessity of our concept, we would remain fiercely skeptical of our own ideas and implement the customer development process to vet the idea, market, customers etc, before writing a single line of code.
My partner was especially adamant about this as he had spent the last 6 months in a cave writing a monster, feature-rich web app for the financial sector that a potential client had promised to buy, but backed out at the last second. They then tried to shop the app around, and found no takers. Thousands of lines of code, all for naught -- as is usually the case without a customer development process. (See Throwing away working code for more on this unfortunate phenomenon. -Eric)
We made a few pencil drawings of what the app would look like which we then gave to a graphic designer. With that, the graphic designer created a Photoshop image. We had him create what we called our "screenshots" (which suggests that an app actually existed at the time) and had him wrap them in one of these freely available PS Browser Templates. Now armed, with 4 "screenshots" and a story, we approached our target market, some of which was through warm introductions, and some, very literally, was through simple cold-calling.
Once we secured a meeting, we told our potential customers that we were actively developing our web app (implying that code was being written) and wanted to get potential user input into the development process early on. Looking at paper print-outs of our "screenshots", no one could tell that this was simply a printout of a PSD, and not a live app sitting on a server somewhere. We walked them through what we thought would be the major application of our product. Most people were quite receptive and encouraging. What proved to be very interesting was that we quickly observed a bimodal distribution with regards to understanding the problem and our proposed solution:
On the third visit, we pressed those who saw merit in the idea to sign a legally non-binding Letter of Intent. Namely, that they agree to use it free of charge if we deliver it to them and it is capable of X, Y and Z. And not only do they agree to use it, but that they intend to purchase if by Y date at X price if it meets their needs.
By the way, this LOI was not written in legalese. Three quarters of it was simple everyday English. In fact, we customer dev-ed the LOI itself. The first time, we asked a client to sign it before we had even written it. When they agreed to sign it, we quickly whipped it up while sitting in a coffee shop and emailed it off to them. This would help us separate the wheat from the chaff when it came to determining interest and commercial viability. Once we had two LOIs signed and in-hand, we actually began to write code.
We also implicitly used the LOIs for price structure and price discovery - which we are still working on. We backed into prices from all sorts of angles, estimating the time-cost of equivalent functionality, competitive offerings, other tools we were potentially displacing -- but in the end, we lobbed a few numbers at them and waited to see if they flinched.
Customer A got X price, Customer B got X + Y price, and so on. So far, our customers have never mentioned price as an objection, which suggests to me that at this point we are very much underpriced. The LOI was also useful as we leveraged it by approaching the competitor of one of those who signed by simply letting them know that their competitor will be using our app. They returned our cold intro email within 8 mins.
We have two customers that have balked at signing LOIs, but want to use our product. This has been somewhat of a quandary for us. When we decided to go the LOI route, we thought that we would not bend and that we would only service those customers who would sign the LOI. In the end, we decided that these two customers were large enough to help us with exposure, provide good usage data and worth the risk of them wasting our time. Time will tell if this theory proves correct.
Right now, the app itself is pretty ugly, a bit buggy and slow -- and doesn't even do a lot. It is borderline embarrassing. Don't get me wrong, it does the few necessary things. BUT it definitely does NOT have the super-duper-hyper-ultra-cool Web 2.0 spit and polish about it. Interestingly enough, our ratio of positive comments to negative comments from actual users is about 10 to 1. One of our first customers had a disastrous launch with it, yet, has signed on to try it again (granted, they did get it for free and we did offer it for free for this next time). But they didn't hesitate to try it again. I thought we would have to plead, beg and beseech. But for them, it was a no-brainer. So, we have to be doing something right.
Our feature set is very limited and being developed almost strictly from user input. While I personally have all sorts of super-duper-hyper-ultra-cool Web 2.0 ideas --- we are holding ourselves back, and forcing ourselves to wait for multiple, explicit and overlapping user requests. We have seen our competitors whose feature sets are very rich, to say the least, but we think in some cases, are as over-engineered as they are feature-rich.
Only time and the market will tell if they are innovative and we are slow, lazy pigs or they have gotten ahead of themselves/the market and our minimalist solution will be better received.
Rich Collins, founder of the Lean Startup Circle, responded to the poster with some Q&A.
LSC: What is your response to some of the people on Hacker News that questioned the ethics of taking this approach?
Some of the commenters have some good points. It definitely explores ethical boundaries. However, I don't think we indulged in any zero-sum game type deception. By that, I mean our intentional fuzziness about the state of development did not cause harm in any manner to our prospective clients. In fact, just us showing up at their offices and talking about our screenshots benefited our prospective clients tremendously as:
"Oh, this is just a Photoshop file? Well, come back to us when you are further along." which defeats the whole purpose of getting face time for Customer Development!
When you tell them, the app is in development (and it was, even before coding, we were spending a lot of time on what we wanted and didn't want, how it would look, use cases ‚ etc) the prospects are interested in providing input and shaping the product. They need to feel and see some momentum.
LSC: Your use of a non-binding letter of intent was another interesting tactic. Did the customers that signed it end up paying for your product?
Yes and no. We had a dispute with one signee and couldn't convert them. However, we successfully converted others. I should also mention that there was one client who refused to sign an LOI, but we are in the process of converting them.
The LOI was designed to give us hard, non-bullshit-able feedback instantly. Too often people will affirm your idea so that you (or they) can save face, which BTW is a form of well-intentioned and socially acceptable deception. This is why, IMHO, friends, wives, and significant others are probably not good people to talk to about your idea. At the end of the day, no one knows if the idea is any good. The market will tell you.
LSC: Would you respond to a few selected Hacker News comments?
"If I were one of your prospects, I would never sign a letter of intent based on drawings only. I'd make you come back later with something, anything I could play with ... Come back when you have something real to show. Until then you're no different from any other poser."
I myself probably would never sign an LOI on screenshots only. However, our customers did a lot of stuff that I would never do. Lesson learned: I am not my customer. We think differently. We solve our problems differently. We have different needs and wants. Repeat after me: You are not your customer.
LSC: And one more: "Except the LOIs in this case are utterly meaningless. I've been on the customer side of LOIs that were signed on request, knowing that it obligated us to nothing."
Wrong. We got instantaneous feedback on the validity of the idea and started our sales process concurrently. While legally non-binding, customers who have signed an LOI are a lot less likely to disappear or make themselves hard to get a hold of. LOIs, while clearly not as good as signed sales contract, do have meaning and are valuable. I encourage B2B startups to keep them in their customer development arsenal.
Special thanks to Rich Collins, the Lean Startup Circle practitioners, and to everyone who has contributed to the Case Studies on the wiki. And thanks to these entrepreneurs for sharing their story. Have a case study you’d like to share? Head on over to the Lean Startup Wiki.
This case study illustrates one company’s attempt to do customer development by testing their vision with customers before writing a single line of code. In the process, they learned a lot by asking initial prospects to sign a non-binding letter of intent to buy the software. As you’ll see, this quickly separated the serious early adopters from everyone else. Mainstream customers don’t have enough motivation to buy an early product, and so building in response to their feedback is futile.
Along the way, this case study raises interesting ethical issues. The lean startup methodology is based on enlisting customers as allies, which requires honesty and integrity. If you deceive customers by showing them screenshots of a product that is “in-development” but for which you have written no code, are you lying to them? And, if so, will that deception come back to haunt you later? Read on and judge for yourself.
The following was written an actual lean startup practitioner. It was originally posted anonymously to the Lean Startup Circle mailing list, and then further developed on the Lean Startup Wiki’s Case Studies section. If you’re interested in writing a future case study, or commenting/contributing to one, please join the mailing list or head on over to the wiki. What follows is a brief introduction by me, the case study itself, and then some Q&A led by LSC creator Rich Collins. Disclaimer: claims and opinions expressed by the authors of case studies are theirs alone; I can’t take credit or responsibility. – Eric Ries
In April of 2009 my partner and I had an idea for a web app, a B2C platform that we are selling as SaaS [software-as-a-service]. We decided from the get-go that, while we clearly saw the benefits and necessity of our concept, we would remain fiercely skeptical of our own ideas and implement the customer development process to vet the idea, market, customers etc, before writing a single line of code.
My partner was especially adamant about this as he had spent the last 6 months in a cave writing a monster, feature-rich web app for the financial sector that a potential client had promised to buy, but backed out at the last second. They then tried to shop the app around, and found no takers. Thousands of lines of code, all for naught -- as is usually the case without a customer development process. (See Throwing away working code
We made a few pencil drawings of what the app would look like which we then gave to a graphic designer. With that, the graphic designer created a Photoshop image. We had him create what we called our "screenshots" (which suggests that an app actually existed at the time) and had him wrap them in one of these freely available PS Browser Templates. Now armed, with 4 "screenshots" and a story, we approached our target market, some of which was through warm introductions, and some, very literally, was through simple cold-calling.
Once we secured a meeting, we told our potential customers that we were actively developing our web app (implying that code was being written) and wanted to get potential user input into the development process early on. Looking at paper print-outs of our "screenshots", no one could tell that this was simply a printout of a PSD, and not a live app sitting on a server somewhere. We walked them through what we thought would be the major application of our product. Most people were quite receptive and encouraging. What proved to be very interesting was that we quickly observed a bimodal distribution with regards to understanding the problem and our proposed solution:
- people either became very excited and started telling us what we should do, what features it needed and how to run with this, or
- they didn't think there was a real problem here, much less a needed solution.
On the third visit, we pressed those who saw merit in the idea to sign a legally non-binding Letter of Intent. Namely, that they agree to use it free of charge if we deliver it to them and it is capable of X, Y and Z. And not only do they agree to use it, but that they intend to purchase if by Y date at X price if it meets their needs.
By the way, this LOI was not written in legalese. Three quarters of it was simple everyday English. In fact, we customer dev-ed the LOI itself. The first time, we asked a client to sign it before we had even written it. When they agreed to sign it, we quickly whipped it up while sitting in a coffee shop and emailed it off to them. This would help us separate the wheat from the chaff when it came to determining interest and commercial viability. Once we had two LOIs signed and in-hand, we actually began to write code.
We also implicitly used the LOIs for price structure and price discovery - which we are still working on. We backed into prices from all sorts of angles, estimating the time-cost of equivalent functionality, competitive offerings, other tools we were potentially displacing -- but in the end, we lobbed a few numbers at them and waited to see if they flinched.
Customer A got X price, Customer B got X + Y price, and so on. So far, our customers have never mentioned price as an objection, which suggests to me that at this point we are very much underpriced. The LOI was also useful as we leveraged it by approaching the competitor of one of those who signed by simply letting them know that their competitor will be using our app. They returned our cold intro email within 8 mins.
We have two customers that have balked at signing LOIs, but want to use our product. This has been somewhat of a quandary for us. When we decided to go the LOI route, we thought that we would not bend and that we would only service those customers who would sign the LOI. In the end, we decided that these two customers were large enough to help us with exposure, provide good usage data and worth the risk of them wasting our time. Time will tell if this theory proves correct.
Right now, the app itself is pretty ugly, a bit buggy and slow -- and doesn't even do a lot. It is borderline embarrassing. Don't get me wrong, it does the few necessary things. BUT it definitely does NOT have the super-duper-hyper-ultra-cool Web 2.0 spit and polish about it. Interestingly enough, our ratio of positive comments to negative comments from actual users is about 10 to 1. One of our first customers had a disastrous launch with it, yet, has signed on to try it again (granted, they did get it for free and we did offer it for free for this next time). But they didn't hesitate to try it again. I thought we would have to plead, beg and beseech. But for them, it was a no-brainer. So, we have to be doing something right.
Our feature set is very limited and being developed almost strictly from user input. While I personally have all sorts of super-duper-hyper-ultra-cool Web 2.0 ideas --- we are holding ourselves back, and forcing ourselves to wait for multiple, explicit and overlapping user requests. We have seen our competitors whose feature sets are very rich, to say the least, but we think in some cases, are as over-engineered as they are feature-rich.
Only time and the market will tell if they are innovative and we are slow, lazy pigs or they have gotten ahead of themselves/the market and our minimalist solution will be better received.
Rich Collins, founder of the Lean Startup Circle, responded to the poster with some Q&A.
LSC: What is your response to some of the people on Hacker News that questioned the ethics of taking this approach?
Some of the commenters have some good points. It definitely explores ethical boundaries. However, I don't think we indulged in any zero-sum game type deception. By that, I mean our intentional fuzziness about the state of development did not cause harm in any manner to our prospective clients. In fact, just us showing up at their offices and talking about our screenshots benefited our prospective clients tremendously as:
- Those clients who had never even entertained the functionality we were proposing gained significant knowledge.
- With that knowledge, they could (and did) Google our competition and start exploring the space and current offerings.
"Oh, this is just a Photoshop file? Well, come back to us when you are further along." which defeats the whole purpose of getting face time for Customer Development!
When you tell them, the app is in development (and it was, even before coding, we were spending a lot of time on what we wanted and didn't want, how it would look, use cases ‚ etc) the prospects are interested in providing input and shaping the product. They need to feel and see some momentum.
LSC: Your use of a non-binding letter of intent was another interesting tactic. Did the customers that signed it end up paying for your product?
Yes and no. We had a dispute with one signee and couldn't convert them. However, we successfully converted others. I should also mention that there was one client who refused to sign an LOI, but we are in the process of converting them.
The LOI was designed to give us hard, non-bullshit-able feedback instantly. Too often people will affirm your idea so that you (or they) can save face, which BTW is a form of well-intentioned and socially acceptable deception. This is why, IMHO, friends, wives, and significant others are probably not good people to talk to about your idea. At the end of the day, no one knows if the idea is any good. The market will tell you.
LSC: Would you respond to a few selected Hacker News comments?
"If I were one of your prospects, I would never sign a letter of intent based on drawings only. I'd make you come back later with something, anything I could play with ... Come back when you have something real to show. Until then you're no different from any other poser."
I myself probably would never sign an LOI on screenshots only. However, our customers did a lot of stuff that I would never do. Lesson learned: I am not my customer. We think differently. We solve our problems differently. We have different needs and wants. Repeat after me: You are not your customer.
LSC: And one more:
Wrong. We got instantaneous feedback on the validity of the idea and started our sales process concurrently. While legally non-binding, customers who have signed an LOI are a lot less likely to disappear or make themselves hard to get a hold of. LOIs, while clearly not as good as signed sales contract, do have meaning and are valuable. I encourage B2B startups to keep them in their customer development arsenal.
Special thanks to Rich Collins, the Lean Startup Circle practitioners, and to everyone who has contributed to the Case Studies on the wiki. And thanks to these entrepreneurs for sharing their story. Have a case study you’d like to share? Head on over to the Lean Startup Wiki.
0 comments:
welcome to my blog. please write some comment about this article ^_^