Continuing our series of blogs on pitfalls and best practices of win/loss, today we look at how to size a win/loss program. The original publication of this blog was here.
Ben: The last time you and I talked about win/loss pitfalls, we looked at the importance of sales buy-in to a win/loss program. Today’s topic is planning the right number of interviews—that “Goldilocks” number of not too many and not too few.
Ken: Yes, too often sales buy-in is unquestioned and that can lead to trouble down the road. At least nobody forgets to ask the question, “how many interviews should we do?” Still, it’s common that they try to do too many or end up not doing enough. So, two pitfalls in one!
Ben: Interesting. So, where do you find the minimum to be? What’s the smallest number of interviews worth doing?
Ken: Well, you can get a lot of value just by doing a handful of interviews. That could be a pilot project that proves the value of win/loss to the organization. Especially for those first interviews, the transcripts can be eye-opening. Clients are sometimes surprised that things they thought were their strengths and weaknesses didn’t matter nearly as much as they expected; they are winning and losing for other reasons. The main problem with doing too few interviews is that the findings won’t have enough credibility with your company’s management to make significant changes. Sure, a couple of interviews will be interesting and suggestive, but it’s unlikely that executives will change the company’s strategy or operations based on just a few customer interviews. You need enough to be representative of your market reality and the analysis must be forward-looking and done with enough rigor that it will support management investment decisions.
Ben: That sounds like you would potentially need hundreds of interviews.
Ken: No, nothing like that! At PSP, our rule of thumb is 16 interviews—8 wins and 8 losses—to get some statistical confidence in the top issues found in a win/loss study. We use Student’s T Test to determine statistical confidence in the results. It shows whether a competitive issue that appears in interviews is statistically significant or merely anecdotal. That’s what’s important. Then, you can present the findings to senior management and be clear about what they represent. So, findings won’t be merely anecdotal. You can take action based on them.
Ben: If you’re planning a program, then, should you go for 16 per quarter?
Ken: Yes, 16 per quarter is that “Goldilocks” number and it’s a reasonable number to do in one quarter if you have enough competitive deals to support it. We will typically get one interview done for every 3-5 contacts we get from a client. So, to do 16 interviews, we will need from 50-80 contacts. The ratio varies by market, where some are harder because the purchases are more sensitive, and people don’t want to talk about them. And usually non-blind or client-introduced interviews require fewer contacts, all other things being equal.
Ben: What would happen if you went slower? Say, 8 or 4 interviews per quarter?
Ken: Sure, if you don’t have enough competitive deals or budget, then you can reduce the rate to 8 per quarter with the understanding that it will take two quarters to get the statistical confidence of 16 interviews. You can theoretically slow that down to 4 interviews a quarter, but if you drag it out too much then the interviews won’t represent a consistent snapshot of the market anymore. You are changing, your competitors are changing. Not so much in one quarter, but over a year, quite a bit. So, you want to have a good sample in a reasonable amount of time. That’s why I like to do an analysis on the interviews done at least every six months if not every quarter.
Ben: How about the other problem—doing too many interviews. Is that really a problem?
Ken: In the case of too many, it’s not that you’re trying to do a needlessly large number of interviews. No, I’ve never seen that problem in practice. Rather, people can sign themselves up for doing more interviews than they can practically deliver in a reasonable amount of time. It can be frustrating to set out to do 16 interviews per quarter and then not find enough competitive deals to get them done. I’d rather set expectations on doing 8 and delivering 8 than coming up short.
Ben: That was my experience. We grew the win/loss program organically, adding regions and products over time. It was a process for us. It got easier as the organization warmed up to the idea of win/loss. We ended up doing hundreds of interviews, but we didn’t do it overnight.
Ken: If you have enough deals so that you can do all 16 from one sales region and you have good buy-in from that sales organization, then it’s very doable. But if you have a large, complex global sales organization, then going global right from day one can be a tall order. It can be better to run an 8-interview pilot in one region for the first quarter, and then expand. Much of this depends on the relationship of the people running the win/loss program and the salespeople in the field. The stronger that relationship, the more ambitious you can be in rolling out a win/loss program. But as in so many things, keeping things simple and starting small with the vision and funding to expand to global scale can be a wiser approach.
Ben: So, it can come back to the buy-in from the sales force?
Ken: Absolutely. That’s the number one thing. It can take time for a win/loss program to take root with the sales organization. Reps may have participated in similar sounding programs in the past and gotten no personal benefit, or they may be leery of having people outside the organization calling their customers. Or they are just busy selling and don’t prioritize helping the win/loss program. It’s very common. If you build up the win/loss program from a realistic number of interviews, you can develop a good reputation with the sales force as being a genuinely useful contributor to their success. This will increase the cooperation you get and make it easier to do more interviews. It usually takes 2-3 quarters to get that going in any given region of the sales force, in my experience. Certainly, coming into a new region after establishing the program elsewhere makes it MUCH easier.