One of my goals in writing posts about topics like continuous deployment is the hope that people will take those ideas and apply them to new situations - and then share what they learn with the rest of us. So I was excited to read a recent post about applying the concept of continuous deployment to that thickest-of-all-clients, the MMOG. Joe Ludwig goes through his current release process and determines that the current time to make and deploy a release is about seven and half hours, which is why his product is released about once a month. While that's actually quite speedy for a MMOG, Joe goes through the thought experiment of what it would take to do it much faster:

Programmer Joe � Continuous Deployment with Thick Clients
If it takes seven and a half hours to deploy a new build you obviously aren’t going to get more than one of them out in an 8 hour work day. Let’s forget for a moment that IMVU is able to do this in 15 minutes and pretend that our target is an hour. For now let’s assume that we will spend 10 minutes on building, 10 minutes on automated testing, 30 minutes on manual testing ...

In fact, if those 30 minutes of manual testing are your bottleneck and you can keep the pipeline full, you can push a fresh build every 30 minutes or 16 times a day. Forget entirely about pushing to live for a moment and consider what kind of impact that would have on your test server. Your team could focus on fixing issues that players on the test server find while those players are still online. Assuming a small change that you can make in half an hour, it would take only an hour from the start of the work on that fix to when it is visible to players. That pace is fast enough that it would be possible to run experiments with tuning values, prices of items, or even algorithms.Of course for any of this to work the entire organization needs to be arranged around responding to player feedback multiple times per day. The real advantage of a rapid deployment system is to make your change -> test -> respond loop faster.

This is a great example of lean startup thinking. Joe is getting clear about what steps in the current process actually deliver value to the company, the imagining a world in which those steps were emphasized and others minimized. Of course, as soon as you do that, you start to reap other benefits, too.

I'd like to add one extra thought to Joe's thought experiment. Let's start with a distinction between shipping new software to the customer, and changing the customer's experience. The idea is that often you can change the customer's experience without shipping them new software at all. This is one of the most powerful aspects of web architecture, and it often gets lost in other client-server programming paradigms.

From one point of view, web browsers are a horribly inefficient platform. We often send down complete instructions for rendering a whole page (or even a series of pages) in response to every single click. Worse, those instructions often kick off additional requests back and forth. It would be a lot more efficient to send down a compressed packet with the entire site's data and presentation in an optimized format. Then we could render the whole site with much less latency, bandwidth usage, and server cost.

Of course, the web doesn't work this way for good reasons. Its design goals aren't geared towards efficiency in terms of technical costs. Instead, it's focused on flexibility, readability, and ease of interoperability. For example, it's quite common that we don't know the exact set of assets a given customer is going to want to use. By deferring their selection until later in the process, we can give up a lot of bookkeeping (again trading off for considerable costs). As a nice side-effect, it's also an ideal platform for rapid changes, because you can "update the software" in real time without the end-user even needing to be aware of the changes.

Some conclude that this phenomenon is made possible because the web browser is a fully general-purpose rendering platform, and assume that it'd be impossible to do this in their app without creating that same level of generality. But I think it's more productive to think of this as a spectrum. You can always move logic changes a little further "upstream" closer to the source of the code that is flowing to customers. Incidentally, this is especially important for iPhone developers, who are barred by Apple Decree from embedding a programming language or interpreter in their app (but who are allowed to request structured data from the server).

For example, at IMVU we would often run split-test experiments that affected the behavior of our downloadable client. Although we had the ability to do new releases of the client on a daily basis (more on this in a moment), this was actually too slow for most of the experiments we wanted to run. Plus, having the customer be aware that a new feature is part of a new release actually affects the validity of the experiment. So we would often ship a client that had multiple versions of a feature baked into it, and have the client call home to find out which version to show to any given customer. This added to the code complexity, latency, and server cost of a given release, but it was more than paid back by our ability to tune and tweak the experimental branches in near-real-time.

Further upstream on the spectrum are features that can be parameterized. Common examples are random events which have some numeric weighting associated with them (and which can be tuned) or user interface elements that are composed of text or graphics. We tacked on a feature to the IMVU client that worked like this: whenever the client called home to report a data-warehousing event, we used to have a useless return field (the client doesn't care if the event was successfully recorded). We repurposed that field to optionally include some XML describing an on-screen dialog box. That meant we could notify some percentage of customers of something at any time, which was great for split-testing. A new feature would often be first "implemented" by a dialog box shown to a few percent of the userbase. We'd pay attention to how many clicked the embedded link to get an early read on how much they cared about the feature at all. Often, we'd do this before the feature existed at all, apologizing all the way.

There are plenty more techniques even further upstream. Eventually, you wind up with specialized state machines, interpeters or a full-fledged embedded platform. We eventually embedded the Flash interpreter into our process, so we could experiment with our UI more quickly.

In fact, we considered releases themselves to be a special case of this more general system. We had a structured automated release process. After all, the release itself was just a static file checked into our website source control. Every new candidate release was automatically shown to a small number of volunteers, who would be prompted to upgrade. The system would monitor their data and if it looked within norms gradually offer the release to a small number of new users (who had no prior expectation of how the product should work). It would carefully monitor their behavior, and especially their technical metrics, like crashes and freezes. If their data looked OK, we'd have the option to ramp up the number of customers bit by bit until finally all new users were given the new release and all existing users were promtped to upgrade. Although we'd generally do a prerelease every day, we wouldn't pull the trigger on a full release that often, because our upgrade path for existing users wasn't (yet) without cost. It also gave our manual QA team a chance to inspect the client before it was widely deployed, due to the lower level of test coverage we have on that part of the product (it's much harder).

In effect, every time we check in code to our client code base, we are kicking off another split-test experiment that asks: "is the business better off with this change in it than without it?" Because of the mechanics of our download process, this is answered a little slower than on the web. But that doesn't make it any less important to answer.

To return to the case of the thick-client MMOG, there are some additional design constraints. It's more risky to have different players using different versions of the software, because that might introduce gameplay fairness issues. I think Joe's idea of deploying to the test server is a great way around this, especially if there is a regular crew of players subjecting the test server to near-normal behavior. But I also think this could work in a lot of production scenarios. Take the case of determining the optimal spawn rate for a certain monster or treasure. Since this is a parameterized scenario, it should be possible to do time-based experiments. Change the value periodically, and have a process for measuring the subsequent behavior of customers who were subjected to each unique value. If you want to be really fancy, you can segregate the data for customers who saw multiple variations. I bet you'd be able to write a simple linear optimizer to answer a lot of design questions that way.

My experience is that once you have a tool like this, you start to use it more and more. Even better, you start to migrate your designs to be able to take ever-increasing advantage of it. At IMVU, we found ourselves constantly migrating functionality upstream, in order to get faster iteration. It was a nearly uncoscious process; we just felt that much more productive with rapid feedback.

So thanks for sharing Joe! Good luck with your thought experiment, and let us know if you ever decide to make those changes a reality.

Read More »

I wrote earlier about the issue of distribution advantage on the iPhone. As the gold rush drives thousands of apps onto the platform, it's getting harder and harder for new entrants to get oxygen. Take a look at The App Store after the gold rush - FierceDeveloper:

According to a recent BusinessWeek feature, the flood of new games, productivity tools and related iPhone software is making it difficult for the vast majority of apps to crack the consumer consciousness. A number of developers are slashing their prices to remain competitive, but it appears that the gold rush that followed on the heels of the App Store's July 10 grand opening is already over, and the get-rich-quick stories of developers like Steve Demeter--who reportedly raked in $250,000 in just two months for his iPhone game Trism--have already passed into coder lore.

The App Store is a channel for customer acquisition. As the channel gets more and more crowded, just launching an app in the store is getting worse and worse as a strategy for each new entrant. This is completely analogous to the situation elsewhere on the internet, where launching a new website, product, or service with PR is getting harder and harder. Customers and prospects are overwhelmed by the number of media and companies clamoring for their attention. If your launch is not immediately successful, you quickly fall into oblivion. On the App Store, the same dynamic is in play. If your app doesn't immediately make it into the Top 25 page, it's pretty hard to have any kind of durable growth.

So what can you do? I think it's helpful to think about two kinds of competition for distribution: acquisition competition and retention competition.

Acqusition competition is how new apps get new customers. On the web, we have many of these channels: SEM, SEO, world of mouth, PR and viral. On the iPhone, it seems that two are driving most of the installs: the "newest apps" RSS feed (which may be combined with PR) and a primitive form of SEO, when people search the App Store for a specific kind of app. Over time, we should get more channels that service the long tail of apps for which the current channels are not working. For example, if any of the mobile ad networks gets major traction, they may become a dominant way that people discover new apps.

Retention competition is how you get people to come back to your app. The primary place this competition is visible is on the home screen of the iPhone itself. But the real battle is in the mind of the people who have installed your application. What causes them to come back to your app, instead of spending their time doing something else? Are they turning on their phone specifically to get your app? Do they browse around looking for an app to pass time? Does your app solve a specific problem that they have? Do you have a way to notify them by SMS or email when something notable happens?

The reason I think it's important to think about retention competition when you are thinking about acquisition is that it strongly influences your acquisition options. If your app has incredibly strong retention, you will probably do very well with the current PR/new app system of acquisition. Why? Because you'll be able to leverage your strong retention to stay in the Top 25 list, which will lead to strong acquisition, in a nice positive feedback loop. If your app has strong word-of-mouth or viral components, your retention drives new acquisition, and it's not so important to have good placement in the store. If and when a good SEM solution shows up for iPhone, you may be able to use it to artificially drive your app into the Top 25, as a one-time event. Then, if your retention is good enough, you can stay there. Or if your lifetime value is high enough, you can just keep spending on SEM.

So if you have a new app that you are thinking of launching, what should you do? My advice: don't launch big. Don't do PR upfront, don't put out a press release. Figure out how to launch quietly, so you can find out what your retention and referral rates are going to be. If necessary, consider doing this under a different brand name than the one you are wedded to using. Having that data will let you pick an acquisition strategy that is appropriate for your app. It's like knowing the future.

Read More »

I have had the opportunity to meet a lot of iPhone-related companies lately. Many of them have really cool products shipping or about to be released, and I wholeheartedly agree with my friends at the iFund that the next generation of applications is going to be amazing.

I've also been playing around with the App Store. From a technical point of view, it's amazing. You just install app after app after app, and it just works. My home screen is a giant mess, because installing apps is just so much fun.

But from a customer experience point of view, I'm not yet sold. Figuring out which apps are going to be any good is almost impossible. Even with only a few months of development, third parties have crammed every single category in the store full of apps. Most of my time in the store is spent scrolling through endless lists. And what distinguishes a good app? I can't really tell. All I see is a name, an icon, a price, the developer's name, and a review star-rating. The reviews are all over the map. When I choose to read them, it seems totally random what I'll find. But even clicking through to see a screenshot and some reviews is incredibly time consuming, given the hundreds of apps in most categories. Most of the time, I have no idea if I'm going to like the app after I install it.

So how's a normal person to choose? I think this is a major challenge for companies that hope to build dominance in some category on the iPhone. Today, the fact that the store is open and has almost no barriers to entry is great for the companies I meet, because they can get their first versions in front of customers quickly, and start iterating fast. But if they are lucky enough to have success, the store is going to become a nightmare, because it will give all of their competitors easy access to their customers and an opportunity to compete with them on an even playing field. The app store is not set up to allow anyone to achieve a durable advantage.

Browsing the app store is an awful lot like shopping in a retail grocery store. You see row after row of tiny boxes, each vying for your attention. They can't present much information, unless you take the box off the shelf and look at it. They rely on impressions, branding and price to try and get you to do that. The store determines which products sit on which shelves, and which yours sits next to. Of course, for a few extra dollars, the right people can get their products on more shelves, or in premium locations, or in giant promotional stands.

Sound familiar? In a world where competition is based on brief looks in predefined categories, it's hard to just "build a better mousetrap" and hope for the best. This is what brand marketers and consumer packaged goods companies have been studying and refining for years: how to win the battle in your mind before you ever set foot in the store. Once you have come to think of Crest as the #1 toothpaste, and, more importantly, your toothpaste, it's unlikely you're going to pay attention to the other boxes on the shelf, no matter how shiny they are.

There are other models, in other distribution channels. On Facebook, viral distribution has proved decisive. Those companies who have learned to build apps that optimize the viral loop dominate in every category where they compete. Not many customers ever browse the app directory or search for specific apps - they don't have to, they find out about apps by being invited by a friend. If you sell an online service that solves a defined problem, you can compete in SEO or SEM. If your site is consistently ranked #1 for a given search term, you can make it very hard for someone else to compete for new customers. In other markets, he who controls the directory has the power, like Download.com in the world of windows shareware.

Word of mouth is a powerful force multiplier in all of these models. If everyone I know is using a specific product, in most markets that's a heavy influence. And in some markets it's decisive, because of well-known network effects (as happened with Microsoft, eBay, and many others). But if your product category doesn't have strong network effects, word of mouth alone is not usually enough to fend off a competitor who also has a quality product.

So what model will prevail on the iPhone? So far, I don't see any apps that have much in the way of viral distribution. Do any apps really cause my friends to sign up, as a natural side-effect of my using the app? I haven't found any yet. And I don't see much searching for apps going on. Do most people know what kind of app they want? And how can they tell the best app for a given search? For example, I did a search for "taxi" in the app store. I got 4 results, 3 free, one for $0.99. I downloaded and tried all four of them (because I had time to kill) - and I'm still not sure which one was the best. Back when I was staring at the search screen, it really was a crapshoot.

So unless someone cracks the code on one of these other models, I think we may revert to the retail model, where good positioning and good branding will win. When I'm scrolling through the endless list of games in that category, the icon that I've come to associate with "that company that makes amazing iPhone games" is going to get a disproportionate share of my attention. Does that mean existing brands have the advantage? I'm not sure. For a lot of brands, their iPhone products will run into the line-extension trap (see The 22 Immutable Laws of Marketing). That looked to me like what's happening with EA's iPhone offerings. So there is an opportunity to build new brands with attributes like "the most amazing mobile apps" but I think building a company around that strategy means really thinking through how to do it. Just bringing a good app to market isn't going to be enough.

So for those who are thinking of starting a new company to build iPhone apps, here's the question I would be pondering. After I've built my first successful app, and all kinds of competitors have copied me and have similar apps right next to mine in the store, how will I continue to get new customers? How will new customers know that my apps are superior?

Read More »