Solving the Pipeline Problem

This post was co-written by Sarah Milstein and Eric Ries, co-hosts of The LeanStartup Conference.

It’s well-known—and we ourselves have been publicly frustrated—that white men tend to dominate the speaker rosters for tech and entrepreneurship conferences, not to mention the portfolios of many entrepreneurship programs. Conference hosts, VC’s, and others often attribute this to a "pipeline problem," the idea that there simply aren’t enough qualified women or people of color who wanted to or were qualified to participate. So we were proud earlier this week to announce our program for The Lean Startup Conference, which comprises approximately 40% women and 25% people of color. We still have room to grow, but this is a significant improvement over last year’s conference, which had almost none of either. Our approach was deliberate, and we want to share it with you, in the hopes that you can replicate it for other conferences and for processes like hiring where equity is important.

We have argued previously that the pipeline problem may be caused by the selection process itself. If under-represented groups have a reasonable expectation of not being selected, it’s perfectly reasonable that they therefore don’t apply. After all, if you were thinking of submitting a proposal to present at a conference that had a record of not choosing people like you—if you weren’t sure that your proposal would be assessed on its merits or that you’d be welcome at the event—why would you spend the time applying? The problem compounds itself: because women and people of color are not often speakers, we’re less aware of them, and we’re less likely to think of them for our own events.

There’s a solution that addresses these issues: meritocratic selection. It’s not a game of quotas; it’s quite the opposite. Indeed, we picked the speakers we thought had the best stories and would be the most engaging presenters. We didn’t rule out any candidates for being white or men, and we didn’t favor women or people of color. Instead, we used a handful of principles to guide us: transparent process,  blind selection, proactive outreach and enlisting help. Here’s how they played out.

Transparent process. Over the summer, we made a big deal about how we wanted to find speakers based on merit (i.e., the great stories they had to share) rather than on their proximity to us. In August, we posted a call for speakers, asking people to apply. We explained that we were looking for speakers based on what they knew rather than who they knew. We also noted that in the past, Eric had mostly drawn from a pool of people he had worked with directly, which meant that in 2010 and 2011, this conference had had almost no speakers typically under-represented at tech events. We received nearly 200 applications, more than half from women and about a quarter from people of color. That included a notable number of black people, which we call out because we’ve received very few applications from black speakers at other conferences we’ve run. Similarly, quite a few men and women over the age of about 50 applied to speak. One person talked about the importance of representing people with disabilities. In addition to being pleased by the demographic range, we were stunned by the consistently high quality of the proposed talk ideas across groups, as the previous open calls we’ve run have brought in relatively few great speakers.

Here’s what resulted from that call for speakers: from the pool of applicants whom we chose for the December 3 conference program, more than half were women. From the pool of applicants whom we chose for the December 2 Ignite: Lean Startup program, more than a third were people of color. And we had to make some heartbreaking decisions to pass on a number of speakers just because we didn’t have enough time on stage for everyone. Taken together, this shows that when you have a broader group from which to draw, there’s a good distribution of speakers; not all—not even most—of the good speakers are white men. The common conference organizer’s argument that we don’t know any black people in tech or that women didn’t apply to speak just doesn’t hold up.

Did employing a transparent process make a difference? The August call for proposals described above was actually our second attempt to change the makeup of our applicant pool; the first attempt failed. In June, we posted our first call for speakers, asking people to nominate others they thought would be good speakers. We noted that we were particularly keen on learning about women and people of color who might be great speakers but weren’t on our radar yet, but we didn’t say anything about what had happened in the past or how we were trying to change it. We received about 35 nominations. Although some were very good, just about 10% were women and almost none were people of color. Every piece of data we have been able to gather on conferences says that 10% is the standard rate at which women will apply or be nominated and that very few people of color will be among these pools.

Let’s dwell on this for a moment. By using principles of meritocratic selection—i.e., being explicit about our desire to find great speakers whom we didn’t know, and by being honest about the process we’d used in the past—we created an atmosphere in which a much broader range of strong speakers felt invited to participate.

It’s also worth mentioning that we knew we’d hit on something important not just because we got such different results in the second round, but also because applicants told us so. We got a lot of comments like this:

“I LAUGH when you say, ‘under-represented at a tech conference,’ because had you not presented such a compelling invitation, I would have never even dreamed of applying for a position of a speaker.”

“The main reason I'm applying is because I have a huge amount of respect and admiration for your efforts to reach out to a new circle of contributors. Anything I can do to support you, whether it's speaking or just behind the scenes, is personally very worthwhile to me.”

“Thanks for opening this up to the non-famous. I think there are lots of great stories out there to be told.”

Blind review. It’s well documented that when people know the sex or race of a job applicant or the main character in a story, they generally assess the person’s qualifications and performance more favorably if the person is male or white. Because we all have internal biases, everybody does it, including women and people of color. We wanted to eliminate that bias as best we could. So we asked applicants to submit some written information, along with a video, and we made the first cut based on the write-ups, which we read without checking names or other identifying info. 

Did using blind review make a difference? By reading the applications, we quickly eliminated people who weren’t a fit for the conference because their topic clearly wasn’t on point for our audience. That was a small group of people, but the distribution was broad. Beyond that, blind review didn’t make a big difference. Why? We asked for relatively little info in writing, both because we wanted to hear directly from speakers (not their PR people) and because video much better represents the medium we’re trying to assess (often, people who can write a nice description of their talk can’t deliver the presentation well, and vice versa). So video was much more important to us than writing, but we couldn’t assess it blind. It’s worth considering whether there’s more we can do with this tool in the future.

Proactive outreach. A very common way that conferences build a program is to brainstorm speakers, come up with a bunch of familiar names, and then notice late in the game that you have almost no women or people of color (at which point, you might scramble to find some or just complain that you don’t know any). Our initial brainstorm wasn’t magically diverse; in fact, it included an overwhelming percentage of young white men. But we immediately—during the same meeting—started digging deeper to think of people we’d left out, and when we came up with less-top-of-mind candidates from under-represented groups, we reached out to them early and often. We kept that up throughout the whole cycle.

Did proactive outreach make a difference? As we noted earlier, Eric’s previous conferences rosters were based almost entirely on people who were top of mind, and they skewed heavily toward white and male.

Enlisting help. There were two primary ways we asked other people to help us. First, when we had a call for participation, we posted it to mailing lists with lots of women and/or people of color, we asked our friends to do the same, and we approached organizations like Women 2.0 to publish or cross-post our info on their blogs and Twitter accounts. Second, when we asked other people for speaker ideas or when we formed our few panels with moderators, we told them about our search for under-represented speakers, and we said that we’d appreciate their support.

Did enlisting help make a difference? It can be scary to ask for this kind of help, because you don’t want to suggest that you’re interested in diversity over quality. But we found that people were consistently, surprising open to working with us on this issue. And it did make a difference: When we forgot to mention to people that we were looking for speakers we didn’t already know, we almost always got back suggestions for more white men who’d be great presenters. When we included a note about our efforts to reach farther, we almost always got back suggestions for women and people of color who’d be great presenters. In addition, we heard regularly from people who’d found out about our call for speakers through Women 2.0 and other mailing lists we worked with.

--

Our four-pronged approached generated really encouraging, even exciting, results. We’re further encouraged by the fact that we didn’t execute this plan perfectly, so there’s still room for improvement. Here’s what we we’ll work on next time out.

Coordinating our approaches. As noted above, in addition to our public call for speakers, we also reached out to people we did already know in the Lean Startup community whom we thought would have really compelling stories. Although this group included some women and people of color, it had a big percentage of white men. Because we started this process before the public call, and because we didn’t know if our public call would work, we quickly filled a number of speaking slots with these great candidates from the group we did know, and we didn’t leave as much room as we could have for terrific speakers new to us. We’ll use a hybrid approach again, but next time, we’ll trust the open-call process and be more deliberate about the distribution of speakers.

Getting beyond the easy wins. Related to the issue above, it’s just plain easier to put people onstage when you know them personally or by reputation. There are a few reasons for this. First, like most conferences, we want to put well-known people onstage, because they’re a draw. So when somebody has a big name and is available, we we’re more likely to consider them for our program, even if their story is only tangential to our angle. For instance, we were interested in having some well-known entrepreneurs join us, and we were willing to let them talk about their experiences generally rather than have them focus on Lean Startup principles. Unsurprisingly, all of the people who fit into this category were white men; women and people of color with comparable stories but less prominence didn’t get this kind of consideration. Correspondingly, we had much stricter standards for people who were less well known, insisting that they have a direct Lean Startup story—and if we didn’t already know them, we were less trusting that their stories were really good examples of Lean Startup methods. (For the record, we had so many good Lean Startup speakers, we didn’t wind up taking anybody with a general entrepreneurship story. But we definitely considered a bunch of them.)

Additionally, we found that when we already knew a speaker, it was easier to find good slots for them in the program because we were more aware of how they were using Lean Startup principles. For instance, if we were looking for somebody from a hardware startup to tell a great pivot story, we might have had six candidates…but it just so happened that the one guy we knew from that group had an even better story about innovation accounting. Because we were familiar with his company generally, we knew his other areas of expertise, and it was therefore easier to put him in the program. It was harder to find the various angles that would work from people we didn’t know.

In other words, between knowing their stories less well and bringing our biases to bear in trusting them, we had a harder time green-lighting the people we didn’t know. Again, that’s human. But if we don’t make conscious decisions to account for it, we’ll never change the dynamics of these events. 

Finding people in other under-represented groups.Our program has almost no people of our parents’ generation, and we don’t think we have any speakers with disabilities. In the future, we can apply the approach above to find more great speakers from these groups.

--

Pulling together a conference program that highlights deservedly well-known people in your field *and* reveals compelling new stories is a challenge. But it’s one you can take on—and that, frankly, not enough conferences do.  As we’ve found, meritocratic selection isn’t just a phrase you can mull or a fantasy for the future. These tools, used thoughtfully and in combination, can help you achieve it: transparent process, blind selection, proactive outreach and enlisting help. And we hope this post will help spark this question: if you’re not using these tools, is your selection process susceptible to unconscious biases that could be making it work less well? Is it really as merit-based as it could be?