Predictions about everything from global politics to the weather are not hard to come by. You find them on news channels, in bestselling books, and among friends and family. But most of these predictions have one thing in common: After the event, no one thinks to formally measure how accurate they were. This lack of measurement means that you have no sense of how accurate any particular source usually is.<\/strong> Without that baseline, how do you know who to listen to the next time you need to make a decision<\/a>?<\/p>\n\n\n\n Given how important accurate predictions are, it\u2019s surprising that we have no standard way of measuring forecast accuracy. Instead, forecasters in popular media deliver their predictions with so much confidence that we take them at their word, and by the time the events they predict happen (or don\u2019t), the news cycle has moved on. The loudest voice is often the most convincing one, regardless of how accurate they are.<\/strong> <\/p>\n\n\n\n (Meteorologists are an exception to this rule. They use data to continually update weather forecasts, and they compare the actual weather to their predictions after the fact to measure forecast accuracy and get insight into what they may have missed.)<\/p>\n\n\n\n Given all the ways our brains can work against us, forecasting accurately is incredibly difficult. But determining whether a forecast is accurate in the first place presents difficulties of its own. A forecast judged by different standards than the forecaster intended will be deemed a failure, even if it\u2019s not<\/strong>. This is the case for one of the most famous forecasting flops of all time: the 2007 claim by Steve Ballmer, then-CEO of Microsoft, that there was \u201cno chance\u201d that Apple\u2019s iPhone would get \u201cany significant market share.” <\/p>\n\n\n\n In hindsight, this prediction looks spectacularly wrong, and it often tops lists of \u201cWorst Tech Predictions Of All Time.” But judging Ballmer\u2019s prediction is more difficult than it seems. What did he mean by \u201csignificant\u201d? And was he referring to the US market or the global market? The smartphone market<\/a> or the mobile phone market as a whole? <\/p>\n\n\n\n These questions matter, because the answers lead us to very different conclusions. Judged against the US smartphone market (where the iPhone commands 42% of the market share), Ballmer is laughably wrong. But in the global mobile phone market (not just smartphones), that number falls to 6%\u2014far from significant. <\/p>\n\n\n\n Although Ballmer\u2019s infamous iPhone forecast seems <\/em>clear at first, it\u2019s actually ambiguous. The nature of language means that certain words can be interpreted differently by different people, and forecasts tend to be full of these words (like \u201csignificant,\u201d \u201clikely,\u201d and \u201cslight\u201d). Think about a forecast that claims a particular result \u201cmay happen.” Like the doomsday prediction, there\u2019s technically no way to discredit this forecast\u2014if something \u201cmay\u201d happen, it\u2019s also implied that it may not. Either way, the forecaster is correct. But the forecast itself is useless for making decisions. <\/p>\n\n\n\n \u201cLikely\u201d is another word that often pops up in forecasts and presents similar problems. If a forecaster claims an event is \u201clikely\u201d to happen and then it doesn\u2019t\u2014was the forecaster wrong? Our gut reaction is to say yes, but that\u2019s incorrect. Think of it this way: If you reach into a bag that you know contains twenty red balls and one blue ball, you could correctly claim that it\u2019s \u201cmost likely\u201d you\u2019ll draw a red ball. If you happen to draw the lone blue ball, your claim is still correct\u2014you just happened to get an unlikely result. <\/p>\n\n\n\n Lack of timelines is another common problem in popular forecasts. If someone says \u201cthe world will end tomorrow,” that has a clear end date\u2014tomorrow, if the world has not ended, we can safely say they were wrong. But if someone says \u201cthe world will end,” any arguments to the contrary can be met with \u201cjust wait and see.” The lack of a time frame means that no matter how much time passes, they can\u2019t be proven wrong. <\/strong><\/p>\n\n\n\n If the example of the red and blue balls brought back memories of math textbooks, there\u2019s a reason for that. Probability is one of the biggest obstacles to judging forecast accuracy. <\/strong>Calculating the probability of pulling a blue ball out of a bag is fairly easy\u2014even if you don\u2019t know any probability formulas, you can just keep blindly pulling a ball out of the bag, recording its color, then putting it back and repeating the process. After enough trials, it would be easy to say which color ball you\u2019re most likely to draw and about how much more likely you are to draw that color than the other. <\/p>\n\n\n\n However, attaching an accurate number to the probability of a real-world event is almost impossible. To do so, we\u2019d need to be able to rerun history over and over again, accounting for all the different possible outcomes of a given scenario. This means that for most events that forecasters are concerned with, it is impossible to know for sure that there is a specific probability of the event happening. Therefore, any probability attached to an event in a forecast is only the forecaster\u2019s best guess, not an objective fact. <\/strong>This can be misleading, but it doesn\u2019t mean that estimated probabilities are useless. <\/p>\n\n\n\n In fact, using numerical probability estimates in forecasts is critical. In the 1950s, the CIA<\/a> forecasting team discovered this after delivering a report forecasting the likelihood of the Soviet Union invading Yugoslavia. The report concluded that an attack was a \u201cserious possibility.” When a State Department official later asked the director of the forecasting team what they meant by \u201cserious possibility\u201d in terms of odds, he estimated the odds at 65 to 35, much higher than how the State Department had interpreted it. <\/p>\n\n\n\n This miscommunication was understandably alarming. The director of the forecasting team, Sherman Kent, took the problem back to his team and asked them each to put a number on \u201cserious possibility.” Though they had all collectively approved of the phrasing in the official report, every single team member assigned a different numerical value to those words. <\/strong>Kent was horrified: Not only were the forecasters not on the same page, but their forecasts were being used to inform foreign policy. If their reports were misunderstood, there could be global consequences. <\/p>\n\n\n\n That claim may sound dramatic, but it\u2019s exactly what happened in 1961 when President Kennedy commissioned the Joint Chiefs of Staff to report on his plan to invade Cuba. The final report predicted a \u201cfair chance\u201d of success, and the government went ahead with what became the Bay of Pigs disaster. After the fact, it was clarified that \u201cfair chance\u201d meant three to one odds against<\/em> success, but President Kennedy interpreted the phrase more positively and acted accordingly. <\/p>\n\n\n\n In the aftermath of the failed Bay of Pigs invasion, Sherman Kent proposed a universal standard for official forecasts that would eliminate ambiguity by assigning numerical probabilities to particular words. He created the chart below:<\/p>\n\n\n\n The table above would make it difficult to misinterpret a forecast but was rejected outright by the intelligence community, who felt that expressing probabilities numerically was crude and misleading. They feared readers would fall into the common trap of interpreting numbers to mean something is <\/em>X percentage likely to happen, not that the forecaster believes<\/em> that to be the likelihood. <\/p>\n\n\n\n That distinction matters, since it affects not just what we do with a prediction but how we judge the person who made it. What probability percentages mean and what people think they mean are entirely different. For example, if a meteorologist correctly predicts a 70% chance of rain, it means that if we were able to replay that day hundreds of times, it would rain in 70% of those replays.<\/p>\n\n\n\n But that\u2019s not how we typically read weather forecasts. Instead, we fall for what the authors call the \u201cwrong-side-of-maybe fallacy,” where we interpret any prediction higher than 50% to mean something <\/strong>will<\/em><\/strong> happen and anything lower than 50% to mean it won\u2019t. <\/strong>So if the meteorologist predicts a 70% chance of rain on a day where it does not rain, we think she was wrong, and consequently, that she must not be very good at her job. <\/p>\n\n\n\n In spite of those risks, meteorology has embraced the clarity of numbers, and most of us are now accustomed to seeing weather forecasts in terms of percentages. But avoiding baseless negative judgment is a major reason forecasters in other fields prefer vague language like \u201cserious possibility.” <\/p>\n\n\n\n How can we measure the overall accuracy of a particular forecaster? For singular events, there really isn\u2019t an accurate way\u2014even with modern technology, replaying history to see every possible outcome is a power still reserved for fictional heroes like Dr. Strange. Instead, we rely on aggregates. <\/p>\n\n\n\n Let\u2019s imagine that the meteorologist in the above example predicts the weather every day for several years, racking up hundreds of total predictions. While we still can\u2019t say how accurate her forecast is for any specific day, we can <\/em>figure out how accurate she is in general through a process called calibration. <\/p>\n\n\n\n Let\u2019s say the meteorologist predicted a 70% chance of rain in 100 of her daily forecasts. If it actually did rain on 70 of those days, her forecasting is perfectly calibrated. In other words, a given event happens 70% of the time that she says there is a 70% chance of that event happening. <\/p>\n\n\n\n Visually, you can represent calibration with a line graph, with \u201cforecasted percentage\u201d on the X-axis and \u201cpercentage correct\u201d on the Y-axis. Perfect calibration is an exact diagonal line<\/strong>, like on the graph below:<\/p>\n\n\n\n Obviously, most forecasters aren\u2019t spot-on every time. The same graph setup can help us judge an individual forecaster\u2019s accuracy by plotting each of her predictions on the graph, calculating the curved trend line for all those points, and comparing that line to the perfect diagonal line. If the forecaster is under-confident (and chronically underestimates probabilities), her curve will be far over the line; If she\u2019s overconfident, the curve will be far under the line. Calibration is helpful, but it\u2019s not the only important measure for evaluating forecasts. A forecaster who always predicts probabilities near the level of chance (50%) will be fairly well-calibrated, but the information isn\u2019t helpful\u2014it\u2019s the mathematical equivalent of a shrug. Stronger forecasters are accurate outside the range of chance\u2014they\u2019re willing to assign much higher or lower odds to a particular event, despite the increased risk of being wrong. We can measure this using resolution<\/strong>. Forecasters with higher resolution are more impressive than cautious forecasters who are equally well-calibrated. You can see this on the graphs below.<\/p>\n\n\n\n Combining measures of calibration and resolution gives us a concrete way to evaluate forecaster accuracy. These measures are combined into a single number, called a Brier score<\/strong>. Brier scores express the difference between a forecast and what really happened. Scores range between 0 and 2, where zero is an absolutely perfect forecast and two is a forecast that is wrong in every possible way. Random guessing, over time, produces a score of .5. <\/p>\n\n\n\n A forecaster\u2019s Brier score is only meaningful in the context of the types of forecasts they make. For example, if a forecaster predicts the weather in Phoenix, Arizona to be \u201chot and sunny\u201d every day for the month of June, their Brier score is likely to be almost zero, since Phoenix summers are notoriously hot and sunny. This is an impressive score but says very little about the forecaster\u2019s skill because it took very little thoughtful consideration. <\/p>\n\n\n\n Brier scores also give us a way to compare one forecaster to another\u2014we can say that a forecaster with an overall Brier score of .2 is a more accurate forecaster than someone with a score of .4. But context is important here, too, because Brier scores don\u2019t account for the difficulty of each prediction.<\/strong> Comparing weather forecasters using their Brier scores is helpful, but it\u2019s not fair to compare the Phoenix forecaster to a forecaster in a less stable climate like Missouri. Even if the Missouri forecaster\u2019s score is slightly higher (and thus less accurate), earning that score in unpredictable circumstances is still much more impressive than a better score in Phoenix.<\/p>\n\n\n\n Brier scores measure forecast accuracy against what really happened. They\u2019re a great way to measure the performance of individual forecasters, but there\u2019s a caveat\u2014they don\u2019t rule out the possibility that someone with a stellar Brier score is just an incredibly lucky guesser. <\/em>To do that, we need a way to compare forecasters\u2019 performance to each other<\/em> over time\u2014If someone outperforms other forecasters year after year, we can confidently say their success comes down to skill; If they score above average one year and below average the next, it\u2019s possible that initial success was just beginner\u2019s luck. <\/p>\n\n\n\n Tracking each forecaster\u2019s performance compared to the group <\/strong>reveals how much of the superforecasters\u2019 success comes down to luck and how much is real skill. <\/strong>To understand the role of luck<\/a> (or chance) in forecasting, we need to understand randomness. Skills and traits that are normally distributed in a population can be plotted on a classic bell curve<\/a>. <\/p>\n\n\n\nEvaluating Forecast Accuracy<\/h3>\n\n\n\n
Some Forecasts Are Too Vague to Judge<\/strong><\/h3>\n\n\n\n
Probabilities Are Useful Estimates, Not Facts<\/strong><\/h3>\n\n\n\n
Certainty<\/strong><\/td> Word<\/strong><\/td><\/tr> 100%<\/td> Certain<\/td><\/tr> 87-99%<\/td> Almost certain<\/td><\/tr> 63-86% <\/td> Probable<\/td><\/tr> 40-63% <\/td> Chances about even<\/td><\/tr> 20-39% <\/td> Probably not<\/td><\/tr> 1-19% <\/td> Almost certainly not<\/td><\/tr> 0%<\/td> Impossible<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n Using Numbers to Evaluate Forecast<\/strong> Accuracy<\/h3>\n\n\n\n
![](\"https:\/\/lh6.googleusercontent.com\/K0FJso-TJPJxxLC59wGiE0RPCLVQYv1gBD4WvBh6MAyOA_w23GNEfxvlO5rezmAfiGBRQNEF_ovYDkDaJ9o3_2onuI8T4zU0FypPRAXn9NJ6cpbep3eiH77fXC6ZJR1w9nXa-j0b\")
![\"\"\/](\"https:\/\/lh3.googleusercontent.com\/zAnIJjtKBRZ1ST5sbDl3ttwoEr47CMQ33OiS74NBrFJZkiNcvygSwMBqp46Tixg3eyb8hCATpYbX33FJr9AYDAk8ckxXW1_Sx8nAYXXkVdIKiMvLaW5IwQlndJtpAQCosGB8gOHh\")
![\"\"\/](\"https:\/\/lh3.googleusercontent.com\/stDVKcDfiTDtgGw_UkTi-mMlI8nTH3OmA7AzSBtnWjkG0BdqhEur6Nc1l3tEDQIg4ZqKLGIMcw9uyZNwHSM3uPicZgrwLJaH3fa0vXfyhinosT_jnA29LMQanVZQd5o1AM3FTTQW\")
![\"\"\/](\"https:\/\/lh3.googleusercontent.com\/qr39fE2d-232eLttLclDRNc4I3CofPSGi1zKpdeBnWOm8oCWWm-HXG2CYIGX6EWK3EtOsMpvhvjBjGMTW9fSnX8lxpNsVUS6Glh_zkHHAdbrF7EQ5BNdvkLsKIADIjOl8dFOgREj\")
![\"\"\/](\"https:\/\/lh4.googleusercontent.com\/TD2o1UEwyy5XDANQa5XSpNZQ-RHHZ0vxjgDKh6z7v4jTL_4kd5Ssyq-JqJa_9PEjeaEi657nCBnD2YFUoK1yZ_5MYNDudPoRZYsWguThBaZ_Xz4wfiWm67H6jknUx5TRV8hlmwxZ\")
Interpreting Brier Scores<\/strong><\/h4>\n\n\n\n
Skill vs. Luck<\/strong><\/h3>\n\n\n\n
![\"\"\/](\"https:\/\/lh6.googleusercontent.com\/Xj4CYUPP5ewouo5aj7fqzVeJpuqkpRleoLQQr77EhrHhP6DyG7LuO2Mp5LQ-_bxQ94sDeOgabSu53I9S2GI-lVA6K1YpMEwh1fdremXuhygyq4b1SvIBXVuIoNrewMIFm-Yhaeos\")
<\/figure>\n\n\n\n![\"\"\/](\"https:\/\/lh6.googleusercontent.com\/a5nNZ_gP5Hs2XVKSpv5zbnlFDwt_eOQVh23IWYDuZEtPHqIN0CMMW_2wtn4dVmhm8ebE6ICS6_exqs2JDzXRtwVMdmBLJ7lYECvV2G96qS0S2Y4qnMj_mgmJYBeUwKjg-pTCEwNM\")
<\/figure>\n\n\n\n