Why calculating win probability is deceptive and a waste of time

Win probability is the likelihood that you'll win your pursuit. It would be so nice to be able to predict the probability of winning a bid. It would be really nice to know what percentage your chances are. It would so help with resource allocation and making decisions. But there are just two problems with expressing win probability as a percentage:

1. None of the algorithms that make the attempt to calculate your percentage chance of winning have statistically significant data to base their calculations on. They have no basis to claim accuracy, despite claims to the contrary. How many leads, comparing apples to apples, have you run through your algorithm and correlated with winning? Dozens? It’s probably not enough to establish statistical significance for a single variable let alone all the factors that could impact award. You can get more data by considering more companies, but then you also decrease the likelihood of having an apples-to-apples comparison. Even within the same company it’s hard to compare apples to apples when the customer, offering, evaluation criteria, and other circumstances can be so different. You don’t have enough data for it to average out. A vendor with data from many, many contractors have a lot of data that isn't relevant to your business creating an average estimate that is an interesting benchmark, but not a predictor of your win probability.
2. All win probability algorithms that attempt to calculate a percentage are guesses piled on top of guesses. Since the chances of the customer accepting your proposal are not numerically calculable, we use proxies. Instead of quantified events we use indicators and guess at some numerical value to weight them with. Some of these indicators are quite subjective. What is your level of customer intimacy? What past performance score will you get? What are your strengths and weaknesses? If you try to quantify these, you at best have guesses. But what weight will you give each of them? How much do your indicators matter when compared to each other? Which will impact the customer's decision more? Converting the indicator into a number with a guess and using a guess for the weight means multiplying your guesses as well as your margin of error. Garbage in, exponential garbage out.

Combine a statistically unreliable result with a huge margin of error and you get something not worth considering. Using guesses as input for guesses with no statistical significance can’t be made scientific. This remains true even if you use the word “probability” and assign it a number.

Does anyone ever go back and compare their predicted win probability with their win rate to see if it’s accurate? I've never seen anyone do this. Think about what it means regarding the reliability of win probability percentages. Think about what it would take to do it. Think about what it would take to reconcile the differences. Garbage in, garbage out, and everyone knows it but pretends differently.

By expressing win probability as a percentage, you may actually reduce your ability to guess your win probability accurately. It’s not just that you have a fake probability. You potentially have a misleading probability. Honest garbage in, unreconcilable garbage out. If your numbers are not accurate, the decisions you are making on those numbers will not be accurate either. Instead of having a data-driven culture, you have a culture that is based on cooking the books.

What you really need isn't a number

You really don’t need a number to use win probability as a decision support tool. Sure, it’s nice to allocate your resources by percentages. And maybe a guess is the only way to do that. But when it comes to making decisions, you don’t need win probability to be expressed as a percentage. What you need are the indicators. You need good quality indicators, so that when you guess it’s based on the best quality input possible. You can add up your indicators any way you want to provide a better guess and express it as a score, color, adjective or anything other than a number and not be misleading.

What things impact your win rate the most? 

Start with what you think impacts the likelihood of winning your pursuit. Go ahead and guess. Guess a lot. Collect as many potential indicators as you can possibly think of. Include both the good and the bad. Then track those indicators across all of your bids. You might not have enough bids to achieve statistical significance, but using some data to test your beliefs about win probability is better than just going on your beliefs alone. And maybe you can refine them over time by collecting more data. Maybe you can even approach statistical significance if you can build a history that includes hundreds of bids.

You'll find that some are better indicators than others. You will likely find some of the results to be counter-intuitive and not at all what you expected. Industry rules of thumb aren't. Knowing when this is true is a competitive advantage. Don’t trust people who say they know what it takes to win, especially when it's based on experience at other companies. Your customers, your relationship with them, the nature of your offering, your ability to turn information into a winning proposal, and your circumstances, add up to a unique context. I have seen the way hundreds of companies conduct their pursuits. Most of them are guessing. Some have convinced themselves that they are experts even though they are guessing. 

Put effort into finding indicators that are objective, so that the results aren’t as influenced by wishful thinking, misapplied incentives, and the convenience of the moment. And use a little logic. Knowing the customer for a long time can have zero impact on your likelihood of winning something new. But having an information advantage, calculated by the number of questions you can answer, is a potential indicator. 

In fact, if you only had to choose one indicator, having an information advantage would be a great one to base your guesses on. But still, confirm that by correlating it with your win rate. Just look out for apples, oranges, and statistical significance. And laugh at win probability numbers.

But what do you tell finance?

The head of finance needs a reliable basis of estimate for future wins in order to be able to deliver reliable financial projections. You do not have to present win probability as a percentage in order to accomplish this. If you can show that certain indicators correspond with a certain percentage ranges. Quintiles (20/40/60/80%) might be sufficient. Or even a red/yellow/green scale that converts to 25/50/75%. The more historical data you have, the more precise you can make the ranges. This is far more reliable than an "algorithm" that calculates a percentage based on guesses multiplied by guesses.

What does win probability tell you about people

What people trust tells you something of their judgment. For example, do they trust the algorithm they invented because they trust their own judgment? And do they expect you to trust it even though they can't prove it based on historical data? Do they like having a win probability based on their judgment without even having an algorithm because it's subject to manipulation? Do they make decisions based on subjective win probability mumbo-jumbo or do they make decisions on well defined, objective indicators? Are they data-driven, but ignoring statistical significance? Predicting the future with as many variables involved as win probability is beyond human capability. It's beyond AI capability, although AI might do a great job at surfacing the things that impact win probability if you have the data to feed it. How people approach predicting the future tells you a lot about their judgment and trustworthiness.

PS:
I considered the following alternatives for the title of this article:
The ugly truth about win probability that no one talks about
Lies, damn lies, and win probability
Your carefully calculated win probability is wrong
Calculating win probability is like intentionally following a mirage
Your win probability comes with the certainty of being wrong
Don't count on your win probability
What are the chances that your win probability is correct?