Nidhal Guessoum and I were supposed to be in Istanbul right now for a meeting. But Iceland is really making life miserable for everyone. Really, what is the point of this particular volcanic island? Hawaii'an islands, at least have spectacular beaches - so we can tolerate its eruptions a bit more - but Iceland? C'mon. Plus, just to mess things up a bit more, they have given an unpronounceable name to the volcano: Eyjafjallajokull. Damn you Eyjafjallajokull!!
Back to the blog.
A new survey by the Pew Forum reveals sub-Saharan Africa to be very religious. The dominant religions are Christianity and Islam, but there is plenty of mixing with local religious traditions. Thus, witch doctors are in demand as much as regular religious prayers. The results from the survey are not entirely surprising - but the degree of religiosity is still quite striking:
The vast majority of people in many sub-Saharan African nations are deeply committed to the practices and major tenets of one or the other of the world's two largest religions, Christianity and Islam. Large majorities say they belong to one of these faiths, and, in sharp contrast with Europe and the United States, very few people are religiously unaffiliated. Despite the dominance of Christianity and Islam, traditional African religious beliefs and practices have not disappeared. Rather, they coexist with Islam and Christianity. Whether or not this entails some theological tension, it is a reality in people's lives: Large numbers of Africans actively participate in Christianity or Islam yet also believe in witchcraft, evil spirits, sacrifices to ancestors, traditional religious healers, reincarnation and other elements of traditional African religions.1
Here is a figure comparing religiosity in 19 sub-Saharan African countries and comparing it with data from various countries across the world (click on the image to see a less-blurry image - I couldn't fix it):
Here is a second figure about the belief in the protective power of sacrifices to spirits or ancestors - and we see a wide support here:
I don't know why places like Rwanda, Nigeria, Zambia, etc, have such lower numbers compared to the rest of sub-Saharan Africa, especially because they don't really stand out from religiosity numbers.
Fascinating numbers. I could not help but remember another Pew study that showed that people in the US were mixing up religions by picking and choosing various aspects of different religious traditions. In Africa, however, the mixing has happened because of strong local cultural elements encountering large expansionist religions. Will more globalization bring more religions into the mix in Africa?
Myth: Lean Startups are small bootstrapped startups.
Truth: There’s nothing wrong with raising venture capital. Many lean startups are ambitious and are able to deploy large amounts of capital. What differentiates them is their disciplined approach to determining when to spend money: after the fundamental elements of the business model have been empirically validated. Because lean startups focus on validating their riskiest assumptions first, they sometimes charge money for their product from day one – but not always.
Myth: Lean Startups replace vision with data or customer feedback.
Want to see more preview videos? Take a look at our new sneak-preview site, sponsored by KISSmetrics.
We've updated our list of simulcast locations (see below). Thanks to volunteer organizers around the world, the conference is now available on every continent (well, except Antarctica) and in almost fifty cities. Most of these events are free, but they do require that you RSVP. An up-to-date list of simulcast locations is always available at http://sllconf.com/streaming.
Our sponsors continue to support the conference as well as deserving entrepreneurs. Three premier early-stage venture investors have joined forces to sponsor the event: Baseline, Floodgate, and First Round Capital. In addition to supporting their portfolio companies in coming to the event, they also have taken the additional step of underwriting a limited number of discounted tickets that are available to the general public. The latest batch, sponsored by First Round, is available here.
Thanks again to all of our amazing sponsors, volunteers and staff who are working very hard to make this event a reality.
This is a guest post by Nidhal Guessoum (see his earlier posts here). Nidhal is an astrophysicist and Professor of Physics at American University of Sharjah.
Arab science educators like me were jolted, about eighteen months ago, when the results of TIMSS 2007 were announced. TIMSS (Trends in International Mathematics and Science Study) is an international, standardized test administered to students of Grades 4 and 8. A number of countries have participated in it since its inception in 1995; it is run every 3 or 4 years.
The results for 2007 showed dismal performances by almost all the Arab states that took part in it. In the Grade 4 tests, Algeria, Kuwait, Morocco, Qatar, Tunisia, and Yemen participated, in addition to Dubai (one of the UAE’s states). In the Grade 8 tests, in addition to these 7 countries, the following took part: Bahrain, Egypt, Jordan, Lebanon, Oman, Palestine, Saudi Arabia, and Syria. All Arab results were below average, sometimes far below. For example, in Grade 4 Science, the six participating full Arab states were all at the very bottom, and while Dubai/UAE had the best results among Arab states, it was still significantly below the international average. In Grade 8, seven of the worst ten countries were Arab ones, with Dubai/UAE again better than its brethren states and only slightly below the international average. In Mathematics, the situation was similarly bad...
These depressing results were a harsh reminder that while Arab states have made huge efforts and progress in general literacy and enrollment at all levels, the quality of education still leaves much to be desired. It was also interesting to note that when the results were broken by gender, practically all Arab/Muslim states (particularly in the Gulf, but also in Jordan and Palestine) girls outperformed boys by leaps and bounds. For example, in Qatar, girls in Grade 8 Science outscored the boys by 70 points (where the average score was 319)…
How far we have come, and yet how far we still have to go?
Figure 1- Percentages of female enrollment at Arab universities
I here wish to highlight the one aspect that is the least noticed and appreciated, the one which breaks stereotypes and preconceptions about the contemporary Arab society: higher education of women and female enrollment in science fields.
Figure 2- Percentages of female enrollment at universities in various world countries
As can already be seen from my remark above on TIMSS results of Arab students broken by gender, girls are fast achieving much higher performance than boys in most Arab countries. Below are histograms (from recent World Bank data) showing the percentages of female general enrollment at Arab universities (Figure 1), compared with universities in other world countries (Figure 2).
Figure 3- Percentages of females enrolled in Science fields
Even more interesting is the data shown in Figure 3: percentages of females enrolled in Science fields at Arab universities, compared with other countries. It is immediately striking that girls make up as much as 80 % of science students in some Arab countries! It is also interesting to note that the “traditional” Gulf states (Qatar, UAE, Kuwait, Bahrain, Saudi Arabia) are where girls are going to universities – and to science fields – in droves! This, in my view, shows that the matter is not related to religion or even tradition, but rather affluence. The richer a country is (at least in the Arab world), the more its female population will get to universities and outnumber the boys in Science fields…
Another example is the number of graduates in Science subjects in a country like Saudi Arabia: in 1999-2000, 372 graduates were awarded M. Sc. degrees in those fields, 39% of them were women; 52 graduates were granted Ph. D. degrees, 79% of them were women!
This is a guest post by Nidhal Guessoum (see his earlier posts here). Nidhal is an astrophysicist and Professor of Physics atAmerican University of Sharjah.
The Algerian High Islamic Council organized a conference on “Islam and the Rational Sciences, past and present” in Algiers (Algeria) from March 29 to 31. Most of the talks revolved around the scientific contributions of Muslim scientists during the Golden Age of the Islamic Civilization (roughly between the 10th and the 14th centuries). A few presenters attempted to examine the present state of affairs in the area of overlap or interaction between Science and Islam.
As I mentioned in a previous post, I had proposed to the organizers to (critically) review the current very popular idea of “scientific miraculousness of the Qur’an” (known as I`jaz); I was not sure at all whether they’d accept for me to discuss this topic, as my abstract was clearly stating that I was going to be very critical of that “theory”; to my surprise, the organizers not only welcomed it but gave it a prominent place in the program, putting it in the very first session, right after the official opening ceremony, which was attended by the minister of religious affairs and other dignitaries, and reported on by the state TV channel and dozens of media outlets.
As soon as I finished my presentation (unfortunately, all talks were given only 20 minutes), the next speaker (an established scholar of the mathematics of the Islamic era) whispered quick praises for my “bold” talk but predicted a strong reaction later, during the discussion period. He was not wrong. I was stunned by how much my 20 minutes of bullet points, condensed and pitched at a tone that I thought was mild and non-aggressive, how much those ideas and examples had struck a sensitive chord and ignited a wide range of reactions. These ranged from the meekly supportive to the strongly negative, including a warning to me not to “go down that path” (someone surreptiously pointed to Nasr H. Abu Zayd, the Egyptian scholar who was declared an apostate for his views on the “historicity” of the Qur’an and who has since lived in exile in Holland). Still a few attempted to engage me in respectful arguments, agreeing with me that much of what has been written on the “scientific miraculousness of the Qur’an” is wildly wrong, but insisting that some verses do contain “quite clear” references to facts that only now do we know to be true…
I was also surprised at the flood of requests for media interviews, with at least two long (30 and 60-minute) TV-guest appearances, a short sound bite for the evening news, a long leading-newspaper interview that appeared the day after next, and countless long recorded interviews with newspapers and magazines, to be published later. In one of those interviews, the reporter, not clearly registering my strong critique of the I`jaz theory, kept asking me for examples of “the latest scientific discoveries that we can indeed find in the Qur’an”… In another interview, the last question the reporter asked me was “how has the International Commission on Scientific Signs in the Qur'an and the Sunnah (here the English website, alongside the huge Arabic website and others in 8 additional languages) reacted to your critique?” When I said “I don’t think they have heard or read my critiques”, he replied “I’ll make sure they are aware of that; I am their representative in Algeria…”
All in all, I thought it was a highly successful appearance, certainly in terms of media impact, but also to some extent in the stir that I produced in the many minds that listened to me. At least now the general acceptance of that trend has been strongly shaken. Someone who used to fully accept that theory told me that I had “hit the brake” on the (fast) drive that the proponents of the “scientific miraculousness of the Qur’an” theory had been having… Read More »
Business Recorder - Pakistan's biggest financial daily - confirms that little knowledge, when combined with pre-existing religious beliefs, can be a source of a major embarrassment (hat tip to Muhammad Akbar Hussain and Zakir Thaver). This story also upsets me personally as the Business Recorder idiocy is related to astronomy. Okay, I don't want to make you wait any longer. Here is the headline: "Signs of the Day of Judgement: NASA confirms the possibility of sun rising from the West"
So first you notice that the story is about religion - rather than science. Second, there is a clear desire to seek an affirmation of a religious claim from science. Third, the source of authority regarding this lies in the West - in this case (and quite often) NASA (even though there is no mention of NASA in the story itself). Nidhal Guessoum recently attended a whole meeting dedicated to finding science in the Qur'an - and you can read his posts here and here.
But what is this bold claim, in a major newspaper, based upon? Well, as it turns out, it is based upon a gross misunderstanding of how planets of our solar system appear to move in the sky. Muslim astronomers, a thousand years ago, had a better understanding of astronomy than these editors of Business Recorder in the 21st century. Here is the claim:
ARTICLE (April 09 2010): The science of astronomy states that the speed of planet Mars has been decreasing in its course toward the eastern direction in the past few weeks to the level we notice the "waver" between the east and the west and on Wednesday the 30th of July the planet movement stopped going toward the eastern direction.
Then in the months of August and September...Mars changed its course in the opposite direction to the West- and that until the end of September which means the sun will rise now from the west on Mars!! And this weird phenomena of the opposite movement is called "Retrograde Motion" Most scientist state that all the planets will go through the same once at least and our planet Earth is one of them. Planet Earth will move in the opposite direction some day and the sun will rise from the west!!
This might occur soon and we are unaware! The rise of the sun from the west is mentioned in the hadith and this is the major sign of the day of judgement, most if not all, the minor signs have occurred. Wake up. Our beloved messenger Mohamed (Peace Be Upon Him) said: "One of the signs of the hour the sun will rise from the west, where no longer tauba (forgiveness) will be granted"
Where can I go and cry?? Okay. I'm taking a deep breath. Phhooooo.
So the starting point of the claim is correct, but rest is all garbage. Astronomers from antiquity have monitored the motion of planets and the stars. They noticed that planets, such as Mars or Jupiter, appear to move when compared with the background stars. For example, if we look at Mars night after night, we will see it change its location with respect to the background stars. The motion, however, is not constant in speed or in direction. For example, sometimes it may appear to slow down and then over the period of next few nights speed up again. In between, it may also appear to us to change the direction it was moving with respect to the background stars. Please note that it is only our perception with respect to the background that is changing.
Nothing is happening to the orbit of Mars!
This change in appearance is caused by the fact that the Earth is going around the Sun faster than Mars (because it is closer to the Sun). When we catch Mars from behind and then overtake it, it looks as if Mars has changed its direction. This is not so strange. You can see the same thing happen when one car overtakes another car on the road. If the slower car is ahead, you will see it first slow with respect to background trees or buildings, then almost stop - when you are parallel to it, and then you will see it going backwards - when you overtake it. The same happens to our view of the motion of Mars (for more, check out this excellent site for retrograde motion, including some nice and simple visuals. Also see this simple animation).
Now, it is quite straightforward to explain this retrograde motion in a sun-centered solar system. But it was a major challenge for Greeks and then later Muslim astronomers to explain it in the context of an geocentric cosmology. This was the primary reasons for the invention of epicycles - which were eventually discarded in the 16th century.
You must now be wondering this is all well and good. But the Business Recorder goes on to claim that "Most scientist state that all the planets will go through the same once at least and our planet Earth is one of them". Hmm...the first part is technically correct. It happens not once, but every time Earth overtakes Mars, Jupiter, Saturn, etc (or when Venus and Mercury overtakes the Earth), we see retrograde motion of planets. So it happens predictably and relatively frequently. What about the Earth? Well, you'll have to go to another planet to see the Earth do the same thing. For the editors of Business Recorder, I would recommend, Venus.
But Salman, you ask, what about the Sun rising from the West on Mars? Here is Business Recorder unleashing its shock & awe on any kind of reasoning. The appearance of the Sun rising and setting has to do with the rotation of the planet on its axis - and (generally) not the direction of its motion around the Sun. So, even if we accept the lunacy of the original claim, that will still not result in the Sun rising from the West!
To be fair to Business Recorder, many nonsense ideas show up in prestigious newspapers (NYT, couple of years ago, published an opinion piece by an astrologer, Washington Post provides a forum for (yikes!) Deepak Chopra, and the Huffington Post routinely publish New Age nonsense). But this article is a good example of why religion should not be mixed with science. The Business Record editors knew what answer they were looking for (i.e. a confirmation of their version of Islam) and did not even bother to check the validity, or even a remote plausibility, of any of the claims. (And if you hear about another Mars rumor - that Mars will be so close that it will appear to be as big as the Moon - please check out this).
Please please please. Keep religion away from science.
Read the Business Recorder article here and read more about retrograde motion here. And below find Sagan talking about retrograde motion, the idea of epicycles, religion, and idea of a sun-centered cosmology:
This looks like a 50s sci-fi movie or an episode of the original Twilight Zone. America has a black president and US and Russia have just signed a treaty to reduce nuclear weapons and appear to trust each other. Okay - so they only reduce their arsenals by 1/3rd (and retain enough to destroy the planet several times over), but still, this is still an excellent step in the right direction. This makes NPT relevant too - as one of the key condition was the reduction of nuclear weapons by states who have them. Much more work needs to be done, but lets take a moment to applaud this:
In a ceremony filled with flourish and the echoes of history, President Obama and President Dmitri A. Medvedev put aside the tensions of recent years to seal the New Start pact paring back their nuclear arsenals.
For a broader context, including that of Iran, check out this very sensible interview with Obama. Unlike Bush, it is refreshing to see Obama taking local political realities of Iran into account, especially when dealing with rhetoric (rather than calling them evil!). Also, he is right in saying that Iranian government is not crazy and, in fact, it will react in a pragmatic way. Lets see what happens at the nuclear summit in US this week. Here is the Obama interview:
Two quick things: Obama's response to Palin bait was perfect. Ignore her. Second, it was amusing to see Obama at one point ask George Stephanopoulos to formulate his question properly.
Next on the Twilight Zone: India and Pakistan sign a peace treaty, open their borders for free trade, and send all of their nuclear weapons into the Sun on a rocket jointly built by their respective national space agencies. Unfortunately, this ends up creating a radiation-charged Nuclear Man from the Sun...oh wait. This is from the awful Superman IV. What, you say? Was there a Superman IV? Unfortunately yes, and I even accidentally watched it once on TV. So much for the world peace.
Simulcasting in twenty cities worldwide leads today's news for the Startup Lessons Learned conference on April 23. Thanks to the generosity of new sponsors, we also have a handful of heavily-discounted tickets available.
Discounted Hotel Rate Traveling to SF for the conference? We have a special hotel rate of $189 a night at the Westin where the event is being held. However, this discount expires this Monday, April 12: be sure to reserve your room before then.
Streaming Our simulcast program has expanded to five continents (so far). We have a new section of the website devoted to remote viewing, which is located here. More locations are being added as we can confirm them. If you'd like to host a simulcast, please sign up here. A complete list of locations is below. Most events are free, but they all require that you RSVP in advance. Special thanks to our streaming sponsor Justin.tv for handling the actual broadcasts.
Sponsors We are enormously grateful to our latest conference sponsors: KISSmetrics, Microsoft BizSpark and Baseline Ventures. Microsoft will be making scholarships available to companies in their BizSpark program; stay tuned for details.
Discounted Tickets As part of their sponsorship, Baseline is offering discounted tickets to their portfolio companies. However, in an additional act of generosity, Baseline's Steve Anderson has asked us to make discounted tickets available to the general public as well. He's footing the bill for a block of half-price tickets which are available now on a first come, first served basis. You can get those tickets here, while they last. (And, if you feel so moved, you can leave Steve a thank-you message; he's @standers on Twitter.)
And, don't forget, full ticket scholarships are also available from IMVU. The application form is here.
Lean startups don’t optimize. At least, not in the traditional sense of trying to squeeze every tenth of a point out of a conversion metric or landing page. Instead, we try to accelerate with respect to validated learning about customers.
For example, I’m a big believer in split-testing. Many optimizers are in favor of split-testing, too: direct marketers, landing page and SEO experts -- heck even the Google Website Optimizer team. But our interest in the tactic of split-testing is only superficially similar.
Take the infamous “41 shades of blue” split-test. I understand and respect why optimizers want to do tests like that. There are often counter-intuitive changes in customer behavior that depend on little details. In fact, the curse of product development is that sometimes small things make a huge difference and sometimes huge things make no difference. Split-testing is great for figuring out which is which.
But what do you learn from the “41 shades of blue” test? You only learn which specific shade of blue customers are more likely to click on. And, in most such tests, the differences are quite small, which is why sample sizes have to be very large. In Google’s case, often in the millions of people. When people (ok, engineers) who have been trained in this model enter most startups, they quickly get confused. How can we do split-testing when we have only a pathetically small number of customers? What’s the point when the tests aren’t going to be statistically significant?
And they’re not the only ones. Some designers also hate optimizing (which is why the “41 shades of blue” test is so famous – a famous designer claims to have quit over it). I understand and respect that feeling, too. After you’ve spent months on a painstaking new design, who wants to be told what color blue to use? Split-testing a single element in an overall coherent design seems ludicrous. Even if it shows improvement in some micro metric, does that invalidate the overall design? After all, most coherent designs have a gestalt that is more than the sum of the parts – at least, that’s the theory. Split-testing seems fundamentally at odds with that approach.
But I’m not done with the complaints, yet. Optimizing sounds bad for visionary thinking. That’s why you hear so many people proclaim proudly that they never listen to customers. Customers can only tell you want they think they want, and tend to have a very near-term perspective. If you just build what they tell you, you generally wind up with a giant, incoherent mess. Our job as entrepreneurs is to invent the future, and any optimization technique – including split-testing, many design techniques, or even usability testing – can lead us astray. Sure, customers think they want something, but how do they know what they will want in the future?
You can always tell who has a math background in a startup, because they call this the local maximum problem. Those of us with a computer science background call it the hill-climbing algorithm. I’m sure other disciplines have their own names for it; even protozoans exhibit this behavior (it's called taxis). It goes like this: whenever you’re not sure what to do, try something small, at random, and see if that makes things a little bit better. If it does, keep doing more of that, and if it doesn’t, try something else random and start over. Imagine climbing a hill this way; it’d work with your eyes closed. Just keep seeking higher and higher terrain, and rotate a bit whenever you feel yourself going down. But what if you’re climbing a hill that is in front of a mountain? When you get to the top of the hill, there’s no small step you can take that will get you on the right path up the mountain. That’s the local maximum. All optimization techniques get stuck in this position.
Because this causes a lot of confusion, let me state this as unequivocally as I can. The Lean Startup methodology does not advocate using optimization techniques to make startup decisions. That’s right. You don’t have to listen to customers, you don’t have to split-test, and you are free to ignore any data you want. This isn’t kindergarten. You don’t get a gold star for listening to what customers say. You only get a gold star for achieving results.
What should you do instead? The general pattern is: have a strong vision, test that vision against reality, and then decide whether to pivot or persevere. Each part of that answer is complicated, and I’ve written extensively on the details of how to do each. What I want to convey here is how to respond to the objections I mentioned at the start. Each of those objections is wise, in its own way, and the common reaction – to just reject that thinking outright – is a bad idea. Instead, the Lean Startup offers ways to incorporate those people into an overall feedback loop of learning and discovery.
So when should we split-test? There’s nothing wrong with using split-testing, as part of the solution team, to do optimization. But that is not a substitute for testing big hypotheses. The right split-tests to run are ones that put big ideas to the test. For example, we could split-test what color to make the “Register Now” button. But how much do we learn from that? Let’s say that customers prefer one color over another? Then what? Instead, how about a test where we completely change the value proposition on the landing page?
I remember the first time we changed the landing page at IMVU from offering “avatar chat” to “3D instant messaging.” We didn’t expect much of a difference, but it dramatically changed customer behavior. That was evident in the metrics and in the in-person usability tests. It taught us some important things about our customers: that they had no idea what an avatar was, they had no idea why they would want one, and they thought “avatar chat” was something weird people would do. When we started using “3D instant messaging,” we validated our hypothesis that IM was an activity our customers understood and were interested in “doing better.” But we also invalidated a hypothesis that customers wanted an avatar; we had to learn a whole new way of explaining the benefits of avatar-mediated communication because our audience didn’t know what that word meant.
However, that is not the end of the story. If you go to IMVU’s website today, you won’t find any mention of “3D instant messaging.” That’s because those hypotheses were replaced by yet more, each of which was subject to this kind of macro-level testing. Over many years, we’ve learned a lot about what customers want. And we’ve validated that learning by being able to demonstrate that when we change the product as a result of that learning, the key macro metrics improve.
A good rule of thumb for split-testing is that even when we’re doing micro-level split-tests, we should always measure the macro. So even if you want to test a new button color, don’t measure the click-through rate on that button! Instead, ask yourself: “why do we care that customers click that button?” If it’s a “Register Now” button, it’s because we want customers to sign up and try the product. So let’s measure the percentage of customers who try the product. If the button color change doesn’t have an impact there – it’s too small, and should be reverted. Over time, this discipline helps us ignore the minor stuff and focus our energies on learning what will make a significant impact. (It also just so happens that this style of reporting is easier to implement; you can read more here)
Next, let’s take on the sample-size issue. Most of us learn about the samples sizes from things like political polling. In a large country, in order to figure out who will win an election with any kind of accuracy, you need to sample a large number of people. What most of us forget is that statistical significance is a function of both sample size and the magnitude of the underlying signal. Presidential elections are often decided by a few percentage points or less. When we’re optimizing, product development teams encounter similar situations. But when we’re learning, that’s the rare exception. Recall that the biggest source of waste in product development is building something nobody wants. In that case, you don’t need a very large sample.
Let me illustrate. I’ve previously documented that early-on in IMVU’s life, we made the mistake of building an IM add-on product instead of a standalone network. Believe me, I had to be dragged kicking and screaming to the realization that we’d made a mistake. Here’s how it went down. We would bring customers in for a usability test, and ask them to use the IM add-on functionality. The first one flat-out refused. I mean, here we are, paying them to be there, and they won’t use the product! (For now, I won’t go into the reasons why – if you want that level of detail, you can watch this interview.) I was the head of product development, so can you guess what my reaction was? It certainly wasn’t “ooh, let’s listen to this customer.” Hell no, “fire that customer! Get me a new one” was closer. After all, what is a sample size of one customer? Too small. Second customer: same result. Third, fourth, fifth: same. Now, what are the odds that five customers in a row refuse to use my product, and it’s just a matter of chance or small sample size? No chance. The product sucks – and that is a statistically significant result.
When we switch from an optimization mindset to a learning mindset, design gets more fun, too. It takes some getting used to for most designers, though. They are not generally used to having their designs evaluated by their real-world impact. Remember that plenty of design organizations and design schools give out awards for designing products that never get built. So don’t hold it against a classically trained designer if they find split-testing a little off-putting at first. The key is to get new designers integrated with a split-testing regimen as soon as possible. It’s a good deal: by testing to make sure (I often say “double check”) each design actually improves customers lives, startups can free designers to take much bigger risks. Want to try out a wacky, radical, highly simplified design? In a non-data-driven environment, this is usually impossible. There’s always that engineer in the back of the room with all the corner cases: “but how will customers find Feature X? What happens if we don’t explain in graphic detail how to use Feature Y?” Now these questions have an easy answer: we’ll measure and see. If the new design performs worse than the current design, we’ll iterate and try again. But if it performs better, we don’t need to keep arguing. We just keep iterating and learning. This kind of setup leads to a much less political and much less arbitrary design culture.
This same approach can also lead us out of the big incoherent mess problem. Teams that focus on optimizing can get stuck bolting on feature upon feature until the product becomes unusable. No one feature is to blame. I've made this mistake many times in my career, especially early on when I first began to understand the power of metrics. When that happens, the solution is to do a whole product pivot. "Whole product" is a term I learned from Bill Davidow's classic Marketing High Technology. A whole product is one that works for mainstream customers. Sometimes, a whole product is much bigger than a simple device - witness Apple's mastery of creating a whole ecosystem around each of their devices that make them much more useful than their competitors. But sometimes a whole product is much less - it requires removing unnecessary features and focusing on a single overriding value proposition. And these kinds of pivots are great opportunities for learning-style tests. It only requires the courage to test the new beautiful whole product design against the old crufty one head-to-head.
By now, I hope you’re already anticipating how to answer the visionary’s objections. We don’t split-test or talk to customers to decide if we should abandon our vision. Instead, we test to find out how to achieve the vision in the best possible way. Startup success requires getting many things right all at once: building a product that solves a customer problem, having that problem be an important one to a sufficient number of customers, having those customers be willing pay for it (in one of the four customer currencies), being able to reach those customers through one of the fundamental growth strategies, etc. When you read stories of successful startups in the popular and business press, you usually hear about how the founders anticipated several of these challenges in their initial vision. Unfortunately, startup success requires getting them all right. What the PR stories tend to leave out is that we can get attached to every part of our vision, even the dumb parts. Testing the parts simply gives us information that can help us refine the vision – like a sculptor removing just the right pieces of marble. There is tremendous art to knowing which pieces of the vision to test first. It is highly context-dependent, which is why different startups take dramatically different paths to success. Should you charge from day one, testing the revenue model first? Or should you focus on user engagement or virality? What about companies, like Siebel, that started with partner distribution first? There are no universally right answers to such questions. (For more on how to figure out which question applies in which context, see Business ecology and the four customer currencies.)
Systematically testing the assumptions that support the vision is called customer development, and it’s a parallel process to product development. And therein lies the most common source of confusion about whether startups should listen to customers. Even if a startup is doing user-centered design, or optimizing their product through split-testing, or conducting tons of surveys and usability tests, that’s no substitute for also doing customer development. It’s the difference between asking “how should we best solve this problem for these customers?” and “what problem should we be solving? and for which customer?” These two activities have to happen in parallel, forming a company-wide feedback loop. We call such companies built to learn. Their speed should be measured in validated learning about customers, not milestones, features, revenue, or even beautiful design. Again, not because those things aren’t important, but because their role in a startup is subservient to the company’s fundamental purpose: piercing the veil of extreme uncertainty that accompanies any disruptive innovation.
The Lean Startup methodology can’t guarantee you won’t find yourself in a local maximum. But it can guarantee that you’ll know about it when it happens. Even better, when it is time to pivot, you’ll have actual data that can help inform where you want to head next. The data doesn’t tell you what to do – that’s your job. The bad news: entrepreneurship requires judgment. The good news: when you make data-based decisions, you are training your judgment to get better over time. Read More »
applied to innovation. That process involves rapid hypothesis testing, 
" is a term I learned from Bill Davidow's classic 
