Our hosts for September were Heather of Randomosity and the Girl and Temperance of High on the Hog. (The pictures through out the post are from Judy of Culinary Escapades, Rayrena of I Love Happy Cows and Kat of A Good Appetite. Here is what Heather posted.

I have long been a fan of Chinese food and the recent Summer Olympics in Beijing have made me crave the delicious dishes even more. One dish that I have always loved to eat is dumplings. While I love dumplings at restaurants, I had never been able to make a decent looking or tasty dumpling at home. So my co-host, Temperance, and I thought this would be a perfect challenge for us in September.

Chinese dumpling or Jiaozi, generally consists of minced meat and chopped vegetables wrapped into a piece of dough. Popular meat fillings include ground pork, ground beef, ground chicken, shrimp, and even fish. Popular mixtures are pork with Chinese cabbage, lamb with spring onion, leeks with eggs, etc. Jiaozi are usually boiled or steamed. Jiaozi is a traditional dish for Chinese New Year's Eve. Family members gather together to make dumplings. (1)

The recipe we have chosen is simple but it allows for variation. If shrimp and turkey isn't your thing, pork and chicken are other options. Or you can go straight vegetable filling if you prefer. You can also pan fry the dumplings instead of steaming. I've tested the recipe both ways and each way is delicious.

Shrimp Dumplings
Prep Time: 20 min, Cook Time: 20 min, Makes 24

Dumplings
4 large shiitake mushrooms
3 scallions
1/2 garlic clove
8 ounces shrimp, peeled and deveined
6 ounces ground turkey
2 teaspoons soy sauce
1 teaspoon dark sesame oil
3 dashes of hot red-pepper sauce
24 dumpling wrappers (see recipe below)

Dipping Sauce
2 tablespoons soy sauce
2 teaspoons rice wine vinegar
1/2 teaspoon honey
1/2 teaspoon dark sesame oil
1/2 teaspoon minced fresh ginger or 1/8 teaspoon ground ginger

1. Coat a steamer basket with a non stick cooking spray and set aside. In a small saucepan, soak the mushrooms in boiling water to cover for 15 minutes, then drain. Remove and discard the stems; cut the caps into quarters.

2. In a food processor, combine the mushroom caps, scallions, and garlic and whirl until coarsely chopped. Add the shrimp and whirl until finely chopped. Transfer to a medium bowl and stir in the turkey, soy sauce, oil, and red-pepper sauce.

3. Place 1 tablespoon of the shrimp mixture in the center of each dumpling wrapper. Dampen the edges with water, the fold up the sides around the filling, pleating the edges. Place in the steamer basket, leaving 1/2 inch of space between the dumplings for the steam to circulate. Set over boiling water, cover, and steam for 15 minutes.

4. For the dipping sauce, in a small bowl, whisk together the soy sauce, vinegar, honey, oil, and ginger. Serve the dumplings hot with the dipping sauce.

Personalize it!
For a different flavor, use ground pork in place of the ground turkey. You can also drop a pinch of chopped scallions into the dipping sauce if you like

HOT WATER DOUGH:

4 cups all-purpose flour
1/2 teaspoon salt
1 1/2 to 1 3/4 cups boiling water

In a stainless steel bowl mix flour and salt. Slowly add hot water to flour in 1/4 cup increments. Mix with chopsticks until a ball is formed and the dough is not too hot to handle. On a floured surface, knead dough until it becomes a smooth, elastic ball. Place back in bowl and cover with a damp cloth. Allow to rest for at least 1 hour. Working on a floured surface with floured hands, roll out dough to form a long 'noodle', 1-inch in diameter. Cut 1/2-inch pieces and turn them over so the cut sides are facing up. Flatten with your palm and roll out thin using a rolling pin. The dumpling wrapper should end up about 3 inches in diameter. (2)

Source: Extraordinary Meals from Ordinary Ingredients, ©2007 The Reader's Digest Association, Inc.
(1) Wikipedia link for Dumplings
(2) Ming's Dumpling Recipe

Quotes From the Forum:

Although I do love the pre-packaged skins from Chao's... Less preparation and cooking time, more eating time! I am glad that I can say I tried it from scratch though, my yeye and mama L would be proud.
Jen of Delightful Delicacies

I subbed tofu for the shrimp and turkey, and they tasted great! I ate way too many of them, and was uncomfortably full for awhile, but they were worth it. Great pick!
Debyi of Healthy Vegan Kitchen

All and all these were quite a treat and easier than we expected. We made a little dim sum feast that night that also included chive pancakes and sesame beans.
Kat of A Good Appetite

Read More »

It's not always fun being small. When you have an infinitesimal number of customers, it can be embarrassing. Some might look at my tiny "5 readers" badge and laugh. But as long as your ego can take it, there are huge advantages to having a small number of customers.

Most importantly, you can get to know those few customers in a way that people with zillions of customers can't. You can talk to them on the phone. You can provide personalized support. You can find out what it would take for them to adopt your product, and then follow up a week later and see if they did. Same with finding out what it would take to get them to recommend your product to a friend. You can even meet the friend.

For companies in the early-adopter phase, you can play "the earlyvangelist game" whenever a customer turns out to be too mainstream for your product. Pick a similar product that they do use, and ask them "who was the first person you know who started using [social networking, mobile phones, plasma TV, instant messaging...]? can I talk to them?" If your subject is willing to answer, you can keep going, following the chain of early-adoption back to someone who is likely to want to early-adopt you.

That level of depth can help you build a strong mental picture of the people behind the numbers. It's enourmously helpful when you need to generate new ideas about what to do, or when you face a product problem you don't know how to solve.

(For example, we used to be baffled at IMVU by the significant minority of people who would download the software but never chat with anyone. It wasn't until we met a few of them in person that we realized that they were having plenty of fun dressing up their avatar and modeling clothes. They wanted to get their look just right before they showed it to anyone else - they would even pay money to do it. But all of our messaging and "helpful tutorials" were pushing them to chat way before they were ready. How annoying!)

And since I have a blog, I have a way to ask questions directly to you. If you have a minute, post your answers in a comment, or email me. Here's what I want to know:

1. First of all, the NPS question: On a scale of 1-10 (where 10 is most likely), how likely is it that you you would recommend this blog to a friend or colleague?
2. How did you hear about it?
3. What led you to become a subscriber, versus just reading an article and leaving like everybody else? (or, if you're not a subscriber, what would it take to convince you?)
   
4. What do you hope to see here in the future?

Thanks, you loyal few. I am grateful for your time and feedback.
Read More »

I have had the opportunity to meet a lot of iPhone-related companies lately. Many of them have really cool products shipping or about to be released, and I wholeheartedly agree with my friends at the iFund that the next generation of applications is going to be amazing.

I've also been playing around with the App Store. From a technical point of view, it's amazing. You just install app after app after app, and it just works. My home screen is a giant mess, because installing apps is just so much fun.

But from a customer experience point of view, I'm not yet sold. Figuring out which apps are going to be any good is almost impossible. Even with only a few months of development, third parties have crammed every single category in the store full of apps. Most of my time in the store is spent scrolling through endless lists. And what distinguishes a good app? I can't really tell. All I see is a name, an icon, a price, the developer's name, and a review star-rating. The reviews are all over the map. When I choose to read them, it seems totally random what I'll find. But even clicking through to see a screenshot and some reviews is incredibly time consuming, given the hundreds of apps in most categories. Most of the time, I have no idea if I'm going to like the app after I install it.

So how's a normal person to choose? I think this is a major challenge for companies that hope to build dominance in some category on the iPhone. Today, the fact that the store is open and has almost no barriers to entry is great for the companies I meet, because they can get their first versions in front of customers quickly, and start iterating fast. But if they are lucky enough to have success, the store is going to become a nightmare, because it will give all of their competitors easy access to their customers and an opportunity to compete with them on an even playing field. The app store is not set up to allow anyone to achieve a durable advantage.

Browsing the app store is an awful lot like shopping in a retail grocery store. You see row after row of tiny boxes, each vying for your attention. They can't present much information, unless you take the box off the shelf and look at it. They rely on impressions, branding and price to try and get you to do that. The store determines which products sit on which shelves, and which yours sits next to. Of course, for a few extra dollars, the right people can get their products on more shelves, or in premium locations, or in giant promotional stands.

Sound familiar? In a world where competition is based on brief looks in predefined categories, it's hard to just "build a better mousetrap" and hope for the best. This is what brand marketers and consumer packaged goods companies have been studying and refining for years: how to win the battle in your mind before you ever set foot in the store. Once you have come to think of Crest as the #1 toothpaste, and, more importantly, your toothpaste, it's unlikely you're going to pay attention to the other boxes on the shelf, no matter how shiny they are.

There are other models, in other distribution channels. On Facebook, viral distribution has proved decisive. Those companies who have learned to build apps that optimize the viral loop dominate in every category where they compete. Not many customers ever browse the app directory or search for specific apps - they don't have to, they find out about apps by being invited by a friend. If you sell an online service that solves a defined problem, you can compete in SEO or SEM. If your site is consistently ranked #1 for a given search term, you can make it very hard for someone else to compete for new customers. In other markets, he who controls the directory has the power, like Download.com in the world of windows shareware.

Word of mouth is a powerful force multiplier in all of these models. If everyone I know is using a specific product, in most markets that's a heavy influence. And in some markets it's decisive, because of well-known network effects (as happened with Microsoft, eBay, and many others). But if your product category doesn't have strong network effects, word of mouth alone is not usually enough to fend off a competitor who also has a quality product.

So what model will prevail on the iPhone? So far, I don't see any apps that have much in the way of viral distribution. Do any apps really cause my friends to sign up, as a natural side-effect of my using the app? I haven't found any yet. And I don't see much searching for apps going on. Do most people know what kind of app they want? And how can they tell the best app for a given search? For example, I did a search for "taxi" in the app store. I got 4 results, 3 free, one for $0.99. I downloaded and tried all four of them (because I had time to kill) - and I'm still not sure which one was the best. Back when I was staring at the search screen, it really was a crapshoot.

So unless someone cracks the code on one of these other models, I think we may revert to the retail model, where good positioning and good branding will win. When I'm scrolling through the endless list of games in that category, the icon that I've come to associate with "that company that makes amazing iPhone games" is going to get a disproportionate share of my attention. Does that mean existing brands have the advantage? I'm not sure. For a lot of brands, their iPhone products will run into the line-extension trap (see The 22 Immutable Laws of Marketing). That looked to me like what's happening with EA's iPhone offerings. So there is an opportunity to build new brands with attributes like "the most amazing mobile apps" but I think building a company around that strategy means really thinking through how to do it. Just bringing a good app to market isn't going to be enough.

So for those who are thinking of starting a new company to build iPhone apps, here's the question I would be pondering. After I've built my first successful app, and all kinds of competitors have copied me and have similar apps right next to mine in the store, how will I continue to get new customers? How will new customers know that my apps are superior?

Read More »

Noah Kagan has a great discussion of usability testing which can help get you over the "that's too hard" or "that's too expensive" fear.

At Facebook we never did testing or looked at analytics. At Mint, Aaron (CEO) was very very methodical and even flew in his dad who is a usability expert. We did surveys, user testing and psychological profiles. This was extremely useful in identifying the types of users we may have on the site and especially for seeing how people use the site. I never really did this before and was AMAZED how people use the site vs. what I expected. Most people know I am very practical or as my ex-gfs call it “cheap.” Anyways, here how our new start-up user tests.

His tips are both very practical and very effective - I've used craigslist, surveymonkey, and, yes, even cafes too. Usability testing is great for coming up with ideas about what to change in your product, but don't forget to split-test those ideas to make sure they work, too.

Read more at How to Usability Test your Site for Free | Noah Kagan's Okdork.com
Read More »

Split-testing is a core lean startup discipline, and it's one of those rare topics that comes up just as often in a technical context as in a business-oriented one when I'm talking to startups. In this post I hope to talk about how to do it well, in terms appropriate for both audiences.

First of all, why split-test? In my experience, the majority of changes we made to products have no effect at all on customer behavior. This can be hard news to accept, and it's one of the major reasons not to split-test. Who among us really wants to find out that our hard work is for nothing? Yet building something nobody wants is the ultimate form of waste, and the only way to get better at avoiding it is to get regular feedback. Split-testing is the best way I know to get that feedback.

My approach to split-testing is to try to make it easy in two ways: incredibly easy for the implementers to create the tests and incredibly easy for everyone to understand the results. The goal is to have split-testing be a continuous part of our development process, so much so that it is considered a completely routine part of developing a new feature. In fact, I've seen this approach work so well that it would be considered weird and kind of silly for anyone to ship a new feature without subjecting it to a split-test. That's when this approach can pay huge dividends.

Reports
Let's start with the reporting side of the equation. We want a simple report format that anyone can understand, and that is generic enough that the same report can be used for many different tests. I usually use a "funnel report" that looks like this:

	Control	Hypothesis A	Hypothesis B
Registered	1000 (100%)	1000 (100%)	500 (100%)
Downloaded	650 (65%)	750 (75%)	200 (40%)
Chatted	350 (35%)	350 (35%)	100 (20%)
Purchased	100 (10%)	100 (10%)	25 (5%)

In this case, you could run the report for any time period. The report is set up to show you what happened to customers who registered in that period (a so-called cohort analysis). For each cohort, we can learn what percentage of them did each action we care about. This report is set up to tell you about new customers specifically. You can do this for any sequence of actions, not just ones relating to new customers.

If you take a look at the dummy data above, you'll see that Hypothesis A is clearly better than Hypothesis B, because it beats out B in each stage of the funnel. But compared to control, it only beats it up through the "Chatted" stage. This kind of result is typical when you ship a redesign of some part of your product. The new design improved on the old one in several ways, but these improvements didn't translate all the way through the funnel. Usually, I think that means you've lost some good aspect of the old design. In other words, you're not done with your redesign yet. The designers might be telling you that the new design looks much better than the old one, and that's probably true. But it's worth conducting some more experiments to find a new design that beats the old one all the way through. In my previous job, this led us to confront the disappointing reality that sometimes customers actually prefer an uglier design to a pretty one. Without split-testing, your product tends to get prettier over time. With split-testing, it tends to get more effective.

One last note on reporting. Sometimes it makes sense to measure the micro-impact of a micro-change. For example, by making this button green, did more people click on it? But in my experience this is not useful most of the time. That green button was part of a customer flow, a series of actions you want customers to complete for some business reason. If it's part of a viral loop, it's probably trying to get them to invite more friends (on average). If it's part of an e-commerce site, it's probably trying to get them to buy more things. Whatever its purpose, try measuring it only at the level that you care about. Focus on the output metrics of that part of the product, and you make the problem a lot more clear. It's one of those situations where more data can impede learning.

I had the opportunity to pioneer this approach to funnel analysis at IMVU, where it became a core part of our customer development process. To promote this metrics discipline, we would present the full funnel to our board (and advisers) at the end of every development cycle. It was actually my co-founder Will Harvey who taught me to present this data in the simple format we've discussed in this post. And we were fortunate to have Steve Blank, the originator of customer development, on our board to keep us honest.

Code
To make split-testing pervasive, it has to be incredibly easy. With an online service, we can make it as easy to do a split-test as to not do one. Whenever you are developing a new feature, or modifying an existing feature, you already have a split-test situation. You have the product as it will exist (in your mind), and the product as it exists already. The only change you have to get used to as you start to code in this style, is to wrap your changes in a simple one-line condition. Here's what the one-line split-test looks like in pseudocode:

if( setup_experiment(...) == "control" ) {
    // do it the old way
} else {
   // do it the new way
}

The call to setup_experiment has to do all of the work, which for a web application involves a sequence something like this:

1. Check if this experiment exists. If not, make an entry in the experiments list that includes the hypotheses included in the parameters of this call.
2. Check if the currently logged-in user is part of this experiment already. If she is, return the name of the hypothesis she was exposed to before.
3. If the user is not part of this experiment yet, pick a hypothesis using the weightings passed in as parameters.
4. Make a note of which hypothesis this user was exposed to. In the case of a registered user, this could be part of their permanent data. In the case of a not-yet-registered user, you could record it in their session state (and translate it to their permanent state when they do register).
5. Return the name of the hypothesis chosen or assigned.

From the point of view of the caller of the function, they just pass in the name of the experiment and its various hypotheses. They don't have to worry about reporting, or assignment, or weighting, or, well, anything else. They just ask "which hypothesis should I show?" and get the answer back as a string. Here's what a more fleshed out example might look like in PHP:

$hypothesis = setup_experiment("FancyNewDesign1.2",
                            array(array("control", 50),
                                  array("design1", 50)));
if( $hypothesis == "control" ) {
    // do it the old way
} elseif( $hypothesis == "design1" ) {
   // do it the fancy new way
}

In this example, we have a simple even 50-50 split test between the way it was (called "control") and a new design (called "design1").

Now, it may be that these code examples have scared off our non-technical friends. But for those that persevere, I hope this will prove helpful as an example you can show to your technical team. Most of the time when I am talking to a mixed team with both technical and business backgrounds, the technical people start worrying that this approach will mean massive amounts of new work for them. But the discipline of split-testing should be just the opposite: a way to save massive amounts of time. (See Ideas. Code. Data. Implement. Measure. Learn for more on why this savings is so valuable)


Hypothesis testing vs hypothesis generation
I have sometimes opined that split-testing is the "gold standard" of customer feedback. This gets me into trouble, because it conjures up for some the idea that product development is simply a rote mechanical exercise of linear optimization. You just constantly test little micro-changes and follow a hill-climbing algorithm to build your product. This is not what I have in mind. Split-testing is ideal when you want to put your ideas to the test, to find out whether what you think is really what customers want. But where do those ideas come from in the first place? You need to make sure you don't get away from trying bold new things, using some combination of your vision and in-depth customer conversations to come up with the next idea to try. Split-testing doesn't have to be limited to micro-optimizations, either. You can use it to test out large changes as well as small. That's why it's important to keep the reporting focused on the macro statistics that you care about. Sometimes, small changes make a big difference. Other times, large changes make no difference at all. Split-testing can help you tell which is which.

Further reading
The best paper I have read on split-testing is "Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO" - it describes the techniques and rationale used for experiments at Amazon. One of the key lessons they emphasize is that, in the absence of data about what customers want, companies generally revert to the Highest Paid Person's Opinion (hence, HiPPO). But an even more important idea is that it's important to have the discipline to insist that any product change that doesn't change metrics in a positive direction should be reverted. Even if the change is "only neutral" and you really, really, really like it better, force yourself (and your team) to go back to the drawing board and try again. When you started working on that change, surely you had some idea in mind of what it would accomplish for your business. Check your assumptions, what went wrong? Why did customers like your change so much that they didn't change their behavior one iota?
Read More »

This article is incredibly depressing on the one hand (because it demonstrates that Google is now undeniably corrupt), but ignoring that, look at the numbers this guy was able to achieve in 6 months:Stuck in Google’s DoghouseFrom the article:According to the letter Mr. Savage submitted to the Justice Department, Google at first gave him nothing but encouragement, even naming Sourcetool its
Read More »

Frequency is more important than talking to the "right" customers, especially early on. You'll know when the person you're talking to is not a potential customer - they just won't understand what you're saying. In the very early days, the trick is to find anyone at all who can understand you when you are talking about your product.

In our first year at IMVU, we thought we were building a 3D avatar chat product. It was only when we asked random people we brought in for usability tests "who do you think of as our competitors?" that we learned different. As product people, we thought of competition in terms of features. So the natural comparison, we thought, would be to other 3D avatar based products, like The Sims and World of Warcraft. But the early customers all compared it to MySpace. This was 2004, and we had never even heard of MySpace, let alone had any understanding of social networking. It required hearing customers say it over and over again for us to take a serious look, and eventually to realize that social networking was core to our business.

Later, when the company was much larger, we had everyone on our engineering team agree to sit in on one usability test every month. It wasn't a huge time commitment, but it meant that every engineer was getting regular contact with an actual customer, which was invaluable. Most of the people building our product weren't themselves target customers. So there was simply no substitute for seeing actual customers with the product, live.

Today, when I talk to startup founders, the most common answer I get to the question "do you talk to your customers?" is something like "yes, I personally answer the customer support emails." That's certainly better than nothing, but it's not a good substitute for proactively reaching out. As Seth writes this week in Seth's Blog: Listening to the loud people, the most aggressive customers aren't necessarily the ones you want to hear from. For example, my experience with teenagers is that they are very reluctant to call or email asking for support, even when they have a severe problem. They just don't need another authority figure in their life.

Don't confuse passion with volume. The people who are the lifeblood of an early-stage startup are earlyvangelists. These are people who understand the vision of your company even before the product lives up to it, and, most importantly, will buy your product on that basis. In some situations, they are also the vocal minority who wants to reach out and get in your face when you do something wrong, but not always. If you're just getting negativity from someone, they are more likely a internet troll - not an earlyvangelist. (For more on earlyvangelists and why they are so important, see Steve Blank's The Four Steps to the Epiphany)

Here's the suggestion from Seth Godin I want to emphasize:

And here's one thing I'd do on a regular basis: Get a video camera or perhaps a copy machine and collect comments and feedback from the people who matter most to your business. Then show those comments to the boss and to your staff and to other customers. Do it regularly. The feedback you expose is the feedback you'll take to heart.

It's not enough to just look at the feedback that comes across your desk. You need to foster situations where you - and everyone you work with - is likely to see feedback that matters. Some techniques that I've found especially helpful:

1. Build your own tracking survey, using a methodology like Net Promoter Score (NPS) to identify and get a regular check-up from promoters (and to screen out detractors). As a nice side-effect, NPS gives you a very reliable report card on customer satisfaction.
   
2. Create a members-only forum where only qualified customers (perhaps, paying customers) can post. Let them connect with each other, but also with you. Treat these people as VIPs, and listen to what they have to say.
   
3. Establish a customer advisory board. Hand pick a dozen customers who "get" your vision. The way I have run these in the past (when I was dealing with extremely passionate customers) is to have them periodically produce a "state of your company" progress report. I would insist that this report be included in the materials at every board meeting, uncensored and unvarnished.

Read More »

How do you build a new product with constant customer feedback while simultaneously staying under the radar? Trying to answer that question at IMVU led me to discover Google AdWords and the world of search engine marketing.

SEM is a simple idea. You declare how much someone clicking an advertisement is worth to you, and then the search engine does its best to get you as many clicks as it can at that price. There's a lot of complexity that I'm leaving out, naturally, because I want to stay focused on the simple idea at the center of SEM: that you can pay peanuts to have people come to your website. In a mature company with a mature product, the goal is to pay for lots of people to come to your website. But I think the genius of Google's innovation is that it allows you to pay for just a few people. Think of it as micropayments for beta testers.

My first AdWords campaign was limited to five dollars a day, and we were buying clicks at five cents a click. That yields 100 clicks a day, every day. Probably if I had been an experienced marketer, I would have known that tiny volume to be insignificant, and I would have been embarrassed. Luckily, I didn't know any better. 100 clicks might not sound like very much, but look at it this way: it meant that every single day, 100 human beings were coming to our website and being offered our product.

We expected that many of those people would buy our product (that's why we charged from day one). But anyone who had done direct response marketing before would have known better. At first, zero people bought anything from us. We tried tinkering with the payment system. Still zero. Maybe the problem was helping people find the payment system. Still nothing. We kept working our way backwards, until we realized that nobody was even making it past the first landing page. Oops. Slowly, over time, we optimized (or eliminated) each step in the process of becoming a customer by giving us money. And one day a remarkable thing happened: we started making more than five dollars a day in revenue.

In the process, we had also built a simple cohort-based analytics system. Its simplicity made it effective - everyone in the company could use and understand it. It just answered this question: for any given time range, for the 100% of people who registered in that period, what percentage of them downloaded? chatted once? chatted five times? bought something? That simple funnel analysis became our scorecard, and helped us refine our product with constant customer input.

After we were making more than five dollars a day, we could take the profits and reinvest them in raising the budget. As we would find more keywords to bid on (always bidding the minimum five cents per click), sometimes the increase in volume would drive our funnel percentages down. When that happened, we'd stop raising the budget, and keep optimizing the product. In this way, we gradually built out a more and more mainstream product.

So how do you find those initial 100 clicks a day? We started by using features of our product as keywords, but this was of limited volume. Eventually Steve Blank, one of our early investors, suggested a technique that not only increased the number of clicks, but also started to use AdWords as a learning and discovery tool. We ran ad campaigns against every single product we could think of in an adjacent market space to ours. We tried obvious competitors as well as long-shots. Since we were only paying per click, it didn't cost us anything to cast a wide net. We would pretty much bid on any phrase that was "[name of competitive product] chat" and variations like that. And then we would use that simple analytics system I mentioned to monitor the conversion rates of customers from each campaign. Those rates gave us a map that told us a lot about our customers; insights that proved stable even when the company grew orders of magnitude bigger.

Only much later did I realize that this was an application of customer development to online marketing. It's now a technique I recommend for any web-based startup.
Read More »

Growing renewable audiences (a talk at O’Reilly Alphatech Ventures) | Futuristic Play by @Andrew_Chen

Non-sustainable:

In fact, I’ll describe press and blog traffic as “fool’s gold” because of the associated emotions that it brings. It’s easy to overestimate the impact of this kind of traffic because it just feels good to have your name and company featured. It strokes your ego. You might get a bunch of inbound emails from other press and partners, and all of these things can contribute to a feeling that you’re on your way to getting tons of traffic. Problem is, you inevitably become yesterday’s old news.

vs. sustainable:

Compare this to the renewable strategies, like viral marketing, SEO, widgets, and ads, which can scale into 10s of millions of users but are primarily centered around tough, non-user centric work. These are things that if you get right, you can optimize your way into a big, sustainable audience.

In an enterprise sales context, this is called a "repeatable and scalable sales process" - once you know how to do this, your company can graduate from early adopters and make an attempt at the mainstream.
Read More »

Marc's writing has been a huge influence on me in thinking through the consequences of the way the current generation of "digital natives" is educated. Are today's kids apathetic? He's argued that a kid who can't pay attention in class but can master the latest Halo in 14 straight hours doesn't have an attention problem, he or she has a boredom problem.

Among his ideas are that today's kids should be taught in class in a way that is relevant to their actual lives, which necessarily means allowing them to use technology. Why ban cell phones when we can take advantage of the fact that, in most classes, they are pervasive.

Apparently a school in Australia has taken his suggestion, and you can read all about it at Marc Prensky's Weblog: Cell Phones in Class -- A Huge Breakthrough.
Read More »

(This article is a draft - your comments are especially welcome as I think through these issues. Please leave feedback!)

I am convinced one of Joel Spolsky's lasting contributions to the field of managing software teams will turn out to be the Joel Test, a checklist of 12 essential practices that you could use to rate the effectiveness of a software product development team. He wrote it in 2000, and as far as I know has never updated it.

I have been thinking a lot about what a new version of this test would look like, given what I've seen work and not work in startups. Like many forms of progress, most of the items on the new test don't replace items from Joel's - they either supplement or extend the ideas on which the original is based.

Let's start with the original list:

1. Do you use source control? This is still an essential practice, especially on the web. There was a time when "web content" was considered "not code" and therefore not routinely source controlled. but I have not seen that dysfunction in any of the startups I advise, so hopefully it's behind us. Joel mentions that "CVS is fine" and so is Subversion, its successor. I know plenty of people who prefer more advanced source control system, but my belief is that many agile practices diminish the importance of advanced features like branching.
2. Can you make a build in one step? You'd better. But if you want to practice rapid deployment, you need to be able to deploy that build in one step as well. If you want to do continuous deployment, you'd better be able to certify that build too, which brings us to...
3. Do you make daily builds? Daily builds are giving way to true continuous integration, in which every checkin to the source control system is automatically run against the full battery of automated tests. At IMVU, our engineering team accumulated thousands upon thousands of tests, and we had a build cluster (using BuildBot) that ran them. We did our best to keep the runtime of the tests short (10 minutes was our target), and we always treated a failing test as a serious event (generally, you couldn't check in at all if a test was failing). For more on continuous deployment, see Just-in-time Scalability.
   
4. Do you have a bug database? Joel's Painless Bug Tracking is still the gold standard.
5. Do you fix bugs before writing code? Increasingly, we are paying better lip service to this idea. It's incredibly hard to do. Plus, as product development teams in lean startups become adept at learning-and-discovery (as opposed to just executing to spec), it's clear that some bugs shouldn't be fixed. See the discussion of defects later in this post for my thoughts on how to handle those.
   
6. Do you have an up-to-date schedule? This, along with #7 "Do you have a spec?" are the parts of the Joel Test I think are most out-of-date. It's not that the idea behind them is wrong, but I think agile team-building practices make scheduling per se much less important. In many startup situations, ask yourself "Do I really need to accurately know when this project will be done?" When the answer is no, we can cancel all the effort that goes into building schedules and focus on making progress evident. Everyone will be able to see how much of the product is done vs undone, and see the finish line either coming closer or receding into the distance. When it's receding, we rescope. There are several ways to make progress evident - the Scrum team model is my current favorite.
7. Do you have a spec? I think the new question needs to be "does the team have a clear objective?" If you have a true cross-functional team, empowered (a la Scrum) to do whatever it takes to succeed it's likely they will converge on the result quickly. You can keep the team focused on customer-centric results, rather than conformance to spec. Now, all well-run teams have some form of spec that they use internally, and Joel's advice on how to craft that spec is still relevant. But increasingly we can move to a world where teams are chartered to accomplish results instead of tasked with creating work on spec.
8. Do programmers have quiet working conditions? Joel is focused on the fact that in many environments, programmers are considered "just the hired help" akin to manual labor, and not treated properly. We always have to avoid that dysfunction - even the lean manufacturing greats realized that they couldn't afford to see their manual-labor workforce that way. I think we need to modify this question to "Do programmers have access to appropriate working conditions?" We want every knowledge worker to be able to retreat into a quiet haven whenever they need deep concentration. But it's not true that energized programmers primarily do solitary work; certainly that's not true of the great agile teams I've known. Instead, teams should have their own space, under their control, with the tools they need to do the job.
   
9. Do you use the best tools money can buy? Joel said it: "Top notch development teams don't torture their programmers." Amen.
   
10. Do you have testers? I think reality has changed here. To see why, take a look at Joel's Top Five (Wrong) Reasons You Don't Have Testers. Notice that none of those five reasons deals with TDD or automated testing, which have changed the game. Automated testing dramatically reduces the cost of certifying changes, because it removes all of the grunt work QA traditionally does in software. Imagine a world where your QA team never, ever worries about bug regressions. They just don't happen. All of their time is dedicated to finding novel reproduction paths for tricky issues. That's possible now, and it means that the historical ratio of QA to engineering is going to have to change (on the other hand, QA is now a lot more interesting of a job).
    
11. Do new candidates write code during their interview? Completely necessary. I would add, though, a further question: Do new employees write code on their first day? At IMVU, our rule was that a new engineer needed to push code to production on their first day. Occasionally, it'd have to be their second day. But if it languished until the third day, something was seriously wrong. This is a test of many key practices: do you have a mentoring system? Is your build environment difficult to set up? Are you afraid someone might be able to break your product without your automated defenses knowing about it?
12. Do you do hallway usability testing? I love Joel's approach to usability, and I still recommend his free online book on UI design. Some people interpret this to mean that you have to do your usability "right" the first time. I strongly disagree. Usability design is a highly iterative process, and the more customers who are involved (via in-person interview, split-test experiment, etc) the better.
    

Now let's take a look at some new questions:

Do you work in small batches? Just like in lean manufacturing, it's generally more efficient to drive down the batch size. I try to encourage engineers to check in anytime they have the software in a working state in their sandbox. This dramatically reduces the waste of integration risk. We rarely have code conflicts, since nobody gets out of sync for very long. And it's way easier to deploy small bits of code, since if something goes wrong, the problem is automatically localized and easy to revert.

Do you routinely split-test new features? I hope to write at a future date about how to build your application so that A/B tests are just as easy as not doing them.

Do you practice Five Why's? Joel himself has written about this topic, in the context of doing root cause analysis to provide excellent quality of service without SLAs. I'm not aware of anyone using this tool as extensively as we did at IMVU, where it became the key technique we used to drive infrastructure and quality improvements. Instead of deciding upfront what might go wrong, we use what actually went wrong to teach us what prevention tactics we need to do. Our version of this was to insist that, for every level of the problem that the post-mortem analysis uncovered, we'd take at lesat one corrective action. So if an employee pushed code that broke the site, we'd ask: why didn't our cluster immune system catch that? why didn't our automated tests catch it? why couldn't the engineer see the problem in their sandbox? why didn't they write better code? why weren't they trained adequately? And make at all five of those fixes.

Do you write tests before fixing bugs? If a bug is truly a defect, then it's something that we don't want to ever see again. Fixing the underlying problem in the code is nice, but we need to go further. We need to prevent that bug from ever recurring. Otherwise, the same blindspot that lead us to create the bug in the first place is likely to allow it happen again. This is the approach of test-driven-development (TDD). Even if you've developed for years without automated tests, this one practice is part of a remarkable feedback loop. As you write tests for the bugs you actually find and fix, you'll tend to spend far more time testing and refactoring the parts of the code that are slowing you down the most. As the code improves, you'll spend lest time testing. Pretty soon, you'll have forgotten that pesky impulse to do a ground-up rewrite.

Can you tell defects from polish? Bugs that slow you down are defects, and have to be fixed right away. However, bugs that are really problems with the experience design of your product should only be fixed if they are getting in the way of learning about customers. This is an incredibly hard distinction to understand, because we're so used to a model of product development teams as pure "execution to spec" machines. In that model, anything that the product owner/designer doesn't like is a bug, and Joel's right that we should always fix before moving on (else you pile up an infinite mound of debt). However, in the learning phase of a product's life, we're still trying to figure out what matters. If we deploy a half-done feature, and customers complain about some UI issues (or split-tests demonstrate them), we should refine and fix. But oftentimes, nobody cares. There are no customers for that feature, UI issues or no. In that case, you're better off throwing the code away, rather than fixing the UI. The hardest part is forcing yoursel fot make this decision binary: either continue to invest and polish or throw the code out. Don't leave it half-done and move on to new features; that's the fallacy Joel tried to warn us about in the first place.

Do your programmers understand the product they are building and how it relates to your company's strategy? How can they iterate and learn if they don't know what questions are being asked at the company's highest levels. At IMVU, we opened up our board meetings to the whole company, and invited all of our advisers to boot. Sometimes it put some serious heat on the management team, but it was well worth it because everyone walked out of that room feeling at a visceral level the challenges the company faced.

What other questions would you ask a brand-new startup about its product development practices? What answers would predict success?

Read More »

Smarticus — 10 things you could be doing to your code right now

A great checklist of techniques and tools for making your development more agile, written from a Rail perspective. Of the techniques he mentioned, I think four are fundamental and critical for any lean startup:

TDD (or the even more politely named TATFT)
Continuous integration
Automate your deployments
Collect statistics

The tools to help you do these things are getting better and better every day, but don't confuse tools with process. Whatever state your code or team is in, you can always start going faster. Or to borrow from a military context (John Boyd) "people-ideas-hardware, in that order."

Read More »

Is it too self-referential to post a blog entry about someone else's blog entry about how often to write a blog entry? I dunno. But Seth Godin is a great writer, so I don't see why I can't crib from him whenever. His post is ostensibly about how often to release new work (whether you're a blog writer, movie star, software team...) but it's really about how to manage your effort between what he calls the frontlist and backlist. Here's my favorite part:

If you've got a team, part of the team should obsess about the backlist, honing it, editing it and promoting it, while the rest work to generate (as opposed to promote) the frontlist.

The opportunity isn't to give into temptation and figure out how to recklessly and expensively market the frontlist. It is to adopt a long and slow and ultimately profitable strategy of marketing your ever-growing backlist.

I see startups struggle with this all the time. Life is so easy in the days before the "launch" - you just focus on building and polishing those new features. But then what? You have customers, they are using your product, and you are trying to help them. But you're also trying to build and polish new features. And fix the ones from before. And polish them more. And maybe even learn something along the way.

My career has been full of "student body right" moments, where the whole team is suddenly forced to change direction. Often, it's just a reaction to a deficit of frontlist or backlist work. The leadership art is to balance the needs of the present with the needs of the future. Seth's post doesn't tell us how to do that, but he at least clues us in to this essential idea: that we have to make sure to do it at all. I wish he'd mentioned it a little sooner...

Update: bonus thought from Dharmesh Shah's 8 Startup Insights Inspired By The Mega Mind of Seth Godin:

6. Beware The Need for Critical Mass

I’m going to lead with a quote from Seth on this one: “Failing for small audiences is a loud cue that you will fail even bigger with big audiences.” Too often, startup founders talk about how they are pushing to get to “critical mass” and how “economies of scale” are going to kick in. That’s all fine and dandy. I get it. I’ve been in the software industry for a long time. But, is it absolutely, positively necessary to get to some “critical mass” before your business starts to make any sense at all? Is that mass all that critical? Does it have to be?

Can’t you make some kind of business out of something that looks a bit like this:

Mass You Have < The Magical Mass That Is Critical

Why do so many startups have these mythical, magical numbers (“once we hit 1,000,000, users rainbows are going to spontaneously pop out of nowhere and magic fairy dust will fall out of the sky and make our financials look sooo much better”).

Read More »

I'm still alive and everyone is okay.

Actually, I've been working on a new project for work, which is exciting AND...remember that writing course I splurged about a while back? Well, it's started and it's intense.

But I promise I'll be back soon. In fact, in all this overload I've even started fleshing out a running blog...but one that's geared toward women and especially mom's.

And as soon as I publish this post, I'm going to re-read this earlier post. Because there I go again, opening my mouth.

So stay tuned.

Splurge-less

We have not been splurging one bit. But as the weather gets cooler and I grow more tired, I can't help but daydream about this little ensemble.
Read More »

I still remember the first time I switched to LAMP. I was building a new startup in 1999, and wanted to do it right. I had heard that all great companies built their applications on Oracle. So one of the first things we did was to hire an Oracle expert and get to work. Our paltry funding didn't allow us to buy expensive Sun or SGI boxes, but we had a pretty beefy intel-based box from Dell. We had just heard about Oracle's support for Linux, so we installed Red Hat and tried to install Oracle. We tried for weeks. I don't really remember why it didn't work. It wouldn't boot. It would boot and crash. We tried to learn how to create a schema, or really anything at all, and mostly failed. It felt like software that grownups were supposed to use, and we didn't qualify.

Meanwhile, we were building our app in PHP, using a generic DB driver and mysql, "for the time being." As the days turned into weeks, eventually the mysql version of our app became the only version, and at some point we just decided to give up on Oracle, fire the expert, and see what happened.

That startup didn't turn out so well, but not for lack of technology. It cost us a few hundred thousand dollars to get our app up and running, but none of that was dollars spent on software licenses or professional services. We just had our app support a few tens of thousands of customers, and it did well. Some combination of the dot-com crash and a just terrible business plan prevented us from having to take our scalability problems to the next level.

Looking back, that was a special moment. I can't really imagine how much it cost our "grownup" counterparts at other dot-com startups to get their first app up and running. My guess is many millions more than we spent.

Our open source counterparts who did solve the scale problem, had some serious hardware costs to deal with. So did I when I finally found myself building an app with real scalability, a few years later, but a combination of our just-in-time scalability technique and great open source scaling tools, made it manageable.

We're in a new wave of platform evolution. Now you can build an app at true web scale without incurring any software costs, or any fixed hardware costs. You don't need to invent a new architecture, and you don't need to even build your architecture up-front. You can turn your entire application infrastructure investment into a pay-as-you-go variable cost, and bring new products to market at speeds an order of magnitude faster than just 10 years ago.
Read More »

This site combines a good concept with a good name and good execution:Chickipedia.comIt has a nice traffic curve in Alexa:Chickipedia traffic
Read More »

(Update April, 2011: In September, 2008 I wrote the following post in which I published my thoughts on the term "lean startup" for the first time. In the interests of preserving that history, I have left the original post unchanged and unedited. To learn more about the progress of the the lean startup movement since 2008, click here.)

---

I've been thinking for some time about a term that could encapsulate trends that are changing the startup landscape. After some trial and error, I've settled on the Lean Startup. I like the term because of two connotations:

1. Lean in the sense of low-burn. Of course, many startups are capital efficient and generally frugal. But by taking advantage of open source, agile software, and iterative development, lean startups can operate with much less waste.
2. The lean startup is an application of Lean Thinking. I am heavily indebted to earlier theorists, and highly recommend the books Lean Thinking and Lean Software Development. I also owe a great debt to Kent Beck, whose Extreme Programming Explained: Embrace Change was my first introduction to this kind of thinking. (So far, I have found "lean startup" works better with the entrepreneurs I've talked to than "agile startup" or even "extreme startup.")

What are the characteristics of a lean startup? One that is powered by three drivers, each of which is a part of a major trend:

1. The use of platforms enabled by open source and free software. At the application-stack layer, I see LAMP + Danga as the most common combination. In recent years, we've also got great new options all up and down the stack, in particular things like Amazon EC2 and RightScale (none of which would be possible without the free software movement).
2. The application of agile development methodologies which dramatically reduce waste and unlock creativity in product development. (See Customer Development Engineering for my first stab at articulating the theory involved)
3. Ferocious customer-centric rapid iteration, as exemplified by the Customer Development process.

My belief is that these lean startups will achieve dramatically lower development costs, faster time to market, and higher quality products in the years to come. Whether they also lead to dramatically higher returns for investors is a question I'm looking forward to studying.
Read More »

Yesterday, I had the opportunity to guest lecture again in Steve Blank's entrepreneurship class at the Berkeley-Columbia executive MBA program. In addition to presenting the IMVU case, we tried for the first time to do an overview of a software engineering methodology that integrates practices from agile software development with Steve's method of Customer Development.

I've attempted to embed the relevant slides below. The basic idea is to extend agile, which excels in situations where the problem is known but the solution is unknown, into areas of even greater uncertainty, such as your typical startup. In a startup, both the problem and solution are unknown, and the key to success is building an integrated team that includes product development in the feedback loop with customers.

2008 09 06 Eric Ries Haas Columbia Customer Development Engineering

View SlideShare presentation or Upload your own.

As always, we had a great discussion with the students, which is helping refine how we talk about this. As usual, I'm heavy on the theory and not on the specifics, so I thought I'd share some additional thoughts that came up in the course of the classroom discussion.

1. Can this methodology be used for startups that are not exclusively about software? We talk about taking advantages of the incredible agility offered by modern web architecture for extremely rapid deployment, etc. What about a hardware business with some long-lead-time components?
   
   To be clear, I have never run a business with a hardware component, so I really can't say for sure. But I am confident that many of these ideas still apply. One major theory that has influenced the way I think about processes comes from Lean Manufacturing, where they use these same techniques to build cars. If you can build cars with it, I'm pretty sure you can use it to add agility and flexibility to any product development process.
   
   
2. What's an example of a situation where "a line of working code" is not a valid unit of progress?
   
   This is incredibly common in startups, because you often build features that nobody wants. We had lots of these examples at IMVU, my favorite is the literally thousands of lines of code we wrote for IM interoperability. This code worked pretty well, was under extensive test coverage, worked as specified, and was generally a masterpiece of amazing programming (if I do say so myself). Unfortunately, positioning our product as an "IM add-on" was a complete mistake. Customers found it confusing and it turned out to be at odds with our fundamental value proposition (which really requires an independant IM network). So we had to completely throw that code away, including all of its beatiful tests and specs. Talk about waste.
   
   
   
3. There were a lot of questions about outsourcing/offshoring and startups. It seems many startups these days are under a lot of pressure to outsource their development organization to save costs. I haven't had to work this model under those conditions, so I can't say anything definitive. I do have faith that, whatever situation you find yourself in, you can always find ways to increase the speed of iteration. I don't see any reason why having the team offshore is any more of a liability in this area than, say, having to do this work while selling through a channel (and hence, not having direct access to customers). Still, I'm interested in exploring this - some of the companies I work with as an advisor are tackling this problem as we speak.
   
   
   
4. Another question that always comes up when talking about customer development, is whether VC's and other financial backers are embracing this way of building companies. Of course, my own personal experience has been pretty positive, so I think the answer is yes. Still, I thought I'd share this email that happened to arrive during class. Names have, of course, been changed to protect the innocent:
   
   

   
   Hope you're well; I thought I'd relay a recent experience and thank you.
   I've been talking to the folks at [a very good VC firm] about helping them with a new venture ... Anyway, a partner was probing me about what I knew about low-burn marketing tactics, and I mentioned a book I read called "Four Steps to the…"
   It made me a HUGE hit, with the partner explaining that they "don't ramp companies like they used to, and have very little interest in marketing folks that don't know how to build companies in this new way."

Anyway, thanks to Steve and all of his students - it was a fun and thought-provoking experience..

Read More »

I've been incredibly impressed by Greasemonkey, the Firefox add-on that lets you easily extend the browser with simple Javascript. I've been even more impressed by the various Greasemonkey compilers out there, that let you turn a Greasemonkey script into a full-blown Firefox extension, for easy distribution.

I know some of those compilers are no longer available (some are hosted, others are not), so I took the liberty of putting up a copy of the PHP Greasemonkey Compiler. I've made some pretty minor modifications to it, mostly in the way of simple usability changes. As I make more, you'll find them (with source) at my Greasemonkey Compiler page.
Read More »

If you are trying to build a scalable LAMP service, it's always best to start with the original and still quite relevant presentation, from Brad Fitzpatrick when he was at LiveJournal. You can find the 2005 version here. You'll learn how they pioneered the use of a lot of open source tools at new levels of scale, and even created quite a few more, that are essential scaling aids. Three of my favorite:

• memcached - an in-memory object caching system. For tips on how to integrate it into your database and application layers, you can see the tail-end of my JIT Scalability talk.
• MogileFS - "Distributed (meta) file system. Spray files across cheap disks on your network. Pay less for storage. No proprietary on-disk file formats." Excellent if you happen to be building a site that, for example, receives and stores gigabytes of photo uploads on a regular basis.
• Perlbal - reverse proxy written in Perl. At IMVU, we used its relatively straightforward plugin system to implement a simple long-poll solution for chat.

If I was building something from scratch today, I'd probably use Gearman for asynchronous task processing, too.

Thanks, Danga (and everyone else who's contributed to these projects).
Read More »

I like theory too much. But hey, it's what helps me think about problems. This simple feedback loop has proven its worth to me time and again. It's inspired by the classic OODA Loop and is really just a simplified version of that concept, applied specifically to creating a software product development team.

There are three stages:

1. We start with ideas about what our product could be.
   
2. We're a software company, so what we do everyday is turn ideas into code.
3. Hopefully, we find out what happened when people use that code, creating data.

Giving rise to three verbs:

1. Implement (programming!) where we turn ideas into code the best way possible.
2. Measure what happened, as quickly as possible.
3. Learn from the data, letting it influence our ideas for the next iteration through the loop

So far, it's all obvious. What helped me is the insight from the Theory of Constraints that in a dynamic system anything that optimizes the sub-parts tends to sub-optimize the whole. Which is a fancy way of saying: focus on total time through the loop, not on the time of any individual activity.

Optimize speed through the whole loop. This sometimes steps on the favored opinions of functional specialists in any organization.

My personal favorite: "Code without data collection? Faster but..." Ever heard a programmer argue for ripping out all that pesky data monitoring code? It's slowing down the system, wasting resources, creating ugly code and uglier scaling problems. If we just stopped measuring, we could write code a hell of a lot faster.

If you have worked with a Professional Data Warehouse Expert, you might have seen: "Measure 10000 things? Comprehensive but..." No human being can learn from 10,000 graphs. It's overwhelming. To turn data into learning, you have to focus on the few key pieces of data the everyone agrees are important. And you have to get the decision makers and implementers to look at (and believe!) the data on a regular basis.

How about documentation that nobody reads? Reports that go unnoticed? Alerts that go off so often that they get ignored? Split-test experiments that go on forever? All of these are true waste, and they generally happen because somebody is optimizing for their particular part of the puzzle, not for the team as a whole.
Read More »

What does life feel like in the chasm? How do you plan for it? A growing startup with a well-run product team will have a history of steady progress. Incremental feature releases leading to correlated growth. The strength of the team determined the pace, and the best companies iterate and learn fastest.

Now imagine it just stops working. You execute just like before. You delight customers just like before. You listen and learn. You innovate as well as ever. Nothing seems to matter. In a subscription business, maybe your attrition starts matching your acquisition, balancing like magic. In an eyeballs business, you just can't seem to acquire or activate that next step-up of customers. Or your cost of customer acquisition just magically floats up to match your customer lifetime value.

We talk a good game about technology adoption curves and crossing the chasm, but most of us don't seem to recognize when it's happening. we don't talk enough about how it feels. Things just stop working. It's everybody's fault and nobody's too. The result: frustration, impotence, humiliation.

My two cents: talk about this beforehand. It's going to be hard to talk about in the midst of the malaise. Be ready to drive home this simple point: in a phase change the things that made you successful before don't work anymore.
Read More »

My favorite question to ask a software development team is "how do you do a release." The mechanical steps, the culture of how it's done, how they evaluate success, how they cope with failure. You can tell a lot about a company from their deployment flow.

How fast do they iterate? Certainly no faster than the time it takes to release. And most likely, the release overhead is considered "minor." Minor with respect to what? Usually, whatever is used as a development cycle. I haven't met anyone who would do a day-long deployment every day.

How long does the software sit un-deployed? Does it require weeks of QA?

How worried is the team about a bad release? Can they consistently produce quality code?

How painful are integrations? How often do bugs regress?

How tolerant is the company of waste? Software is generally always getting better or getting worse. Processes are always deteriorating or self-regulating.

Can they tell a good release from a bad one? To answer that question you need to know what the objective of the release was. Do the engineers understand what they are building? Do they measure? Do they believe the data?

These questions often produce fascinating answers. Of course, it's easy to sit outside any team and criticize its failures. That's a waste of time. You can use questions like these to learn about a team and its culture. Mechanical mistakes are the result of human mistakes. One person making a mistake may be a deranged lone wolf, but systematic mistakes are the result of systemic problems. And the same is true in reverse - a lone brilliant coder can build a great widget, but it takes a system of people working well together to produce consistently great results.

After an hour with a team talking dirty about deployment, you'll know.
Read More »

At my previous company, we pioneered an approach to building out our infrastructure that we called "Just-In-Time Scalability." We wanted an agile approach that would allow us to build our software architecture as we needed it, without downtime, but also without large amounts of up-front cost. After all, the worst kind of waste in software development is code to support a use case that never materializes. Scalable systems are no exception - if your assumptions about how many customers you'll have, or how they will behave are just a little bit wrong, you can wind up with a massive amount of wasted code.

Chris and I had the opportunity to present on our approach this past spring at the MySQL Conference. You can also download our presentation, "Just-In-Time Scalability: Agile Methods to Support Massive Growth."

Update: PDF version for "lazy linux users" and others who love freedom.
Read More »

You cannot control what you cannot see, and the hardest part of managing software projects is that the final product is so intangible. Automated tests make defects visible. In almost all Agile Development systems, thousands of automated tests are run against every change to the software. These tests are written by the engineers themselves, and they must all pass every time. Even one test failing is considered a “red alert” situation that requires all work to stop. This is equivalent of the andon cord in a Toyota production plant. It signals the presence of a defect in the assembly process and directs all attention to fixing it before work can continue. As the team gets better and making more changes without introducing new problems, the team can speed up. A failing test provides a natural feedback mechanism, slowing the team down when going too fast.

Read More »