Today we will try to assess how difficult it is to make forecasts in the absence of data for analysis. An example for us, in particular, is the 2017 boxing match between Floyd Mayweather and Conor McGregor.
That fight was actually unprecedented, since McGregor, a recognized MMA fighter, entered the boxing ring for the first time, and even against Mayweather himself, the unsurpassed champion. This circumstance gave fertile ground for controversy and speculation, but it was a huge headache for anyone who tried to predict the result, since predictions, as you know, rely on the past, which forms our idea of the future. So what can players do in such unique circumstances? In our opinion, they can refer to the approach developed by the Nobel laureate in nuclear physics.
Forecasting is a quantitative science. The more historical data you have, the more likely you are to create an accurate picture – with a particular system or model – of what the future might look like. When someone draws on existing knowledge to determine its impact on a particular event from a dataset, they can make more accurate predictions than existing ideas. Good examples are weather forecasts or outright bets on the English Football Premier League.
However, when there is no data, it all comes down to qualitative analysis – building a reasoned argument about what might happen. This may not seem like a better way than getting your finger wet and poking it into the air to determine the direction of the wind, but here science helps us in the form of Fermi’s method.
Enrico Fermi was a famous physicist, considered the architect of the nuclear age. He received the Nobel Prize in Physics in 1938 and was the creator of the world’s first nuclear reactor. He is also remembered for his approach to guesswork for quantifying something that is deemed impossible to reasonably calculate given the limited information available.
When teaching this approach, for example, he asked his students questions such as: “How many piano tuners are there in Chicago?” This is not a trick question. Try to spend a few minutes pondering the question and sketch out a master plan based on a series of graded subqueries that will ultimately lead you to a reasonable answer to the main question.
If you create yourself the following subtasks (or follow a similar logic, but with your own slightly different questions), you will get a good idea of the answer:
- How many pianos are there in Chicago?
- How often is the piano tuned each year?
- How long does it take to set up the piano?
- How many hours per year does the average piano tuner work?
Using your guesswork for the first three questions, you can calculate how many hours of piano tuning work is spent in Chicago each year, and divided by the number of hours one piano tuner spends in a year. At the end of the day, you can get a reasonable guess as the answer to your main question. Of course, to answer questions one, two, and three, you must break them down into additional sub-problems.
Therefore, for the first question, you will need to guess the population of Chicago using any knowledge about the population of other US metropolitan areas. You can assume the figure will be somewhere between 2-2.5 million inhabitants (actually 2.7 million in 2016).
Next, you need to figure out what percentage of people have a piano at home: the rule of thumb is that the ratio can be one in 100 (that is, only about 25,000 if you use the upper hypothetical bound for the population). Then you add the tools you use for bars, clubs, schools, etc; so the number can be doubled to 50,000.
Answers to questions two and three are simple intuition, unless, of course, you have subject knowledge. The piano is tuned about once a year and takes about two hours. The answer to the fourth question may be based on your own experience or based on average employment rates when working five days a week on standard vacations.
So, if you assume that there are 50,000 pianos in total in Chicago that need to be tuned once a year, and this process takes two hours for each, that means 100,000 hours of tuning. Divide that by the 1,600 hours spent on average per year of tuning, and you have 62.5 piano tuners in town.
There is no unequivocal answer as to how many there really are, although the analysis of the Yellow Pages directory showed the figure 83, so since we got somewhere between 55 and 70, this can be considered quite close to the exact result.
It is not worth giving too much importance to the accuracy of our answer in this example: the approach we have chosen is much more important. This kind of thinking encourages accurate predictions in the absence of data, like betting on Mayweather vs. McGregor.
Super Forecasters – Sound Assessment Project
The Fermi Method is explored in Super Forecasting: The Art and Science of Forecasting, an excellent book by Philip Tetlock and Dan Gardner. The book explores the development of the science of forecasting, using the Sound Assessment Project (GJP) as a background.
For four years, Tetlock has been inviting “20,000 intellectually curious laymen” to join his project and predict the outcome of a wide variety of geopolitical puzzles. His team was part of the broader Intelligence Advanced Research Projects Activity, an agency within the US intelligence services, aimed at improving their forecasting standards for critical political and economic events that directly affect national interests.
IARPA has created a forecasting tournament with five teams led by leading scientists in the field, including the GJP and its amateurs. Over the course of five years, IARPA has posed about five hundred questions, and the answers to them had to be sent every day at the same time.
Forecast accuracy was measured using Brier scores . Forecasts were evaluated by summing the difference between the confidence level of the forecasts and the actual result (squared). The requirement that a forecast must be made with an indication of your confidence in it is equally rewarding or punishing for that confidence and is a great way to correctly identify tipsters.
Bookmakers do not make predictions; they simply offer a measure of the likelihood that something will happen, presented as odds. In this regard, Pinnacle BC is on safer ground with well-established sports that follow a fixed set of rules and have good reliable and accessible historical data.
Then, based on our knowledge of the incident, we can build models and get a proper idea of the likelihood of future outcomes in terms of initial odds.
However, in order to attract new customers and increase popularity among existing users, we need to go beyond this basic offering. In this way, we are discovering a world of new or unremarkable sports that may be incomplete with previous match data (the best examples are esports, specials and political elections).
Elections are held so infrequently and under such different circumstances that historical data is of very little value. Polls are unreliable for a variety of reasons, while hoping for news is also a minefield. This means that bookmakers are not strong in political predictions.
Another great example of this problem is horse racing (note that Pinnacle does not offer horse racing bets, but this knowledge is useful for general development).
The two-year-old race, which features first-time horses (along with those who have already taken part, but have not won anything), are a perfect example of a bet that is mostly devoid of solid foundation. To make matters worse, the races are short, with almost no room for error, as horses can behave unpredictably when they first enter the racetrack.
How can you predict the ability of a horse that has never raced in a race where success (from the trainer’s or owner’s point of view) would simply be to provide your pet with a positive racing experience?
- Ask a series of deductive questions that are similar to the piano tuner problem.
- How good is the reputation of the stud farm? How successful is the breeder?
- What about a coach? What is his performance with the winners among the horses participating for the first time in the races, and at the same distance?
- What about the same jockey stats?
These questions can allow you to make an educated guess about the odds of horses – ideally by aggregating them into an overall ranking – and using Brier’s metrics to accurately gauge your level of market confidence.
Thus, these seemingly insoluble problems represent additional opportunities for players, since the bookmakers are in the same boat with you – albeit with a very important advantage. We do not have a model or approach that can be 100% trusted, so we will rely on our experience and knowledge, as well as Fermi’s approach.
10 Commandments for Good Forecasting
The challenges faced by the GJP project are no different from those faced by gamblers and bookmakers as they move away from traditional sports markets into the world of exotic betting, which brings us back to the Mayweather vs. McGregor fight. We have a reasonable understanding of both boxing and MMA fights, but a boxer versus a mixed martial artist is essentially a Fermi problem.
The good news here is that based on the findings of the GJP, there are some very practical findings that have been experimentally proven to raise the baseline of prediction for hobbyists.
Tetlock actually came up with 10 Commandments of Good Forecasting based on the experience of the GJP. Using randomized trials, Tetlock found that those who read his studies containing these principles increased their Brier score by 10%. This may be enough to get you long term profitability as a player.
- Focus on the problems where your hard work is more likely to pay off, ignoring the obvious and the intractable. There is little chance that you will discover something about the Premier League that the market has not yet accounted for. Find a sweet spot where it is realistic to assume that value can be found with reasonable time and effort.
- Break big problems down into smaller ones. For example, the question “Who will win the fight between Mayweather and McGregor?” – to easier questions like “What could be McGregor’s boxing training?”, “What are the motives of the opponents?”, “In what style will McGregor fight, and what is the probability of Mayweather’s success against this style?”, and so on. Assign value and confidence values to your answers.
- Balance of both sides. With regard to the Mayweather vs. McGregor fight, you need to go beyond just boxing or MMA. McGregor has a ton of fans in the MMA community who will no doubt support him here, but is their subjective perspective valuable when betting? Likewise, how much do boxing bettors know about MMA? Try to balance both points of view.
- Balance between underestimating and overestimating new information. Much depends on your experience and weighing the value of the information source. There will be a lot of judgment online about this fight, so take the time to find out the best sources of information.
- Challenge your prejudices. If you’ve only bet on boxing before and can’t imagine anything but a victory for Mayweather, force yourself to think about any scenarios in which he might lose, and vice versa.
- Translate guesswork into likelihood. An experienced forecaster will have broader views than simply “Mayweather is synonymous with victory” or “McGregor has no chance.” Your approach should reflect a more nuanced assessment as measured by probability, not rhetoric.
- Learn to balance overconfidence in your assumptions and underestimation. This means striking a balance between procrastinating to a point of inaction and the moment you go all-in without a balanced judgment.
- Analyze failures and successes with equal care. A very unwise approach is not to take responsibility for your own mistakes. Similarly, you can make the right decisions and still get the wrong result, and vice versa.
- Show your best in others and let others show their best in you. This refers to the team nature of the GJP, so it will only be appropriate if you work as part of a group or, perhaps, you are very active on social networks and are ready to share the results of your work, as well as accept and give constructive criticism.
- Improvement comes only from putting your good intentions into practice. If you see betting as a leisure activity, don’t expect to win in the long run. If you are not satisfied with this prediction, accept the fact that you will have to devote time and effort to place your bets in a systematic and structured manner.