PDF Summary:Superforecasting, by Philip E. Tetlock

Every day, you make predictions, like when traffic will be heaviest on your route to work or whether the value of a company’s stock will increase or decrease. We use the information from these predictions to guide our decisions. But how can we ensure that we’re making the best predictions possible?

In Superforecasting, authors Philip Tetlock and Dan Gardner explore that question by analyzing the strategies of 'superforecasters,' volunteer analysts who predict the likelihood of global events with impressive accuracy. In this guide, we’ll explore the traits and tactics that make superforecasters so “super” and how you can use them to improve your own predictions. We’ll also compare the authors’ approach to predicting the future to those of other influential thinkers like Nassim Nicholas Taleb.

(continued)...

Trait 2: They Generate Multiple Perspectives

Superforecasters also rely on aggregated judgment (aggregation is the process of combining data from multiple sources). Tetlock and Gardner argue that aggregation is a powerful tool for forecasters because the aggregated judgment of a group of people is usually more accurate than the judgment of an average member of the group.

Superforecasters use aggregation by pulling from many sources and using many tools to produce an answer, despite being just one person. This skill doesn’t come naturally to most people—we struggle to even recognize that there are other perspectives beyond our own “tip-of-the-nose” view, let alone fully consider those ideas. This is part of what sets superforecasters apart.

How to Generate New Perspectives Like a Superforecaster

To think like a superforecaster, you need to look beyond the tip-of-your-nose view and find new ways of viewing a problem. Here are some tips to get you in the right mindset:

• Reverse-engineer the problem. For example, if you’re asked to predict the likelihood of the U.S. raising the federal minimum wage to $15 per hour, pretend it’s already happened and work backwards from there. What would have had to change to lead to a change in the minimum wage?

• Broaden your horizons. Commit to reading books and following news sources outside of your comfort zone. The more diverse perspectives you expose yourself to, the easier it will be to approach a particular problem from a different vantage point.

• Try attacking your own conclusions. This is similar to the technique of “negative empiricism” that Nassim Nicholas Taleb describes in The Black Swan, which involves deliberately searching for evidence that will disprove your argument. For example, for the minimum wage question, if you ultimately conclude that the U.S. federal government will raise the minimum wage, go back and look for evidence that they won’t.

Trait 3: They Think in Probabilities

According to the authors, superforecasters are probabilistic thinkers. This goes beyond just phrasing their forecasts in terms of probability percentages. In situations where most of us think in terms of black and white, superforecasters think in shades of grey.

Most people’s mental “probability dial” has three distinct settings: yes, no, and maybe. By contrast, probabilistic thinkers have an unlimited number of settings. They’re more likely to answer questions in terms of percentages rather than “yes” or “no.” And this is not just in the realm of forecasting—this is how superforecasters normally think and speak in their everyday lives.

(Shortform note: Superforecasters’ emphasis on probabilistic thinking may help to explain the gender gap among superforecasters, who tend to be male. Young children perform about the same on tests of probabilistic thinking regardless of gender—however, by age 10, boys tend to outperform girls, a pattern that holds true for many other math skills. This could be because, as Sheryl Sandberg argues in Lean In, social norms discourage girls from pursuing math and science.)

Trait 4: They Think From the Outside In

Tetlock and Gardner argue that when superforecasters first encounter a question, they begin by looking at the wide perspective of that question before accounting for the specifics (in Thinking, Fast and Slow, Daniel Kahneman calls that wider perspective the “outside view”). Compare this to the “inside view,” which describes the particular details of a situation.

For example, imagine someone tells you about their physician friend, Dr. Jones, and asks you to estimate the likelihood that Dr. Jones is a pediatrician. If you start with the inside view, you’ll analyze the specifics of Dr. Jones’s life and personality and make predictions based on what you find. The trouble is, specifics can often lead us to make random and extreme guesses. If we’re told that Dr. Jones loves children and worked at a summer camp for sick children during college, we might say it’s 80% likely that Dr. Jones is a pediatrician. On the other hand, if we’re told that Dr. Jones is a very serious, reserved person and has no plans to become a parent, we might swing to the other extreme and guess 2%.

In contrast, if you start with the outside view, you’ll ignore any details about the specific person. Instead, you’d try to answer the question “What percentage of doctors specialize in pediatrics overall?” This gives you a base rate from which to calibrate your prediction, which is more likely to lead to an accurate forecast than if you begin with a random “inside view”-inspired guess.

Master the Outside View With a Premortem

In Thinking, Fast and Slow, Daniel Kahneman advises using a “premortem” analysis to avoid the dangers of inside-out thinking. A premortem analysis is a mental exercise in which you imagine that whatever you’re working on (be it a project or a forecast) has already come to fruition—and was a complete disaster. Your goal is to come up with as many reasons as possible to explain this hypothetical “failure.”

This approach is helpful because, by nature, the inside view makes a situation feel “special” because it predisposes you to focus on what makes the situation unique. That feeling can make it more difficult to notice biases in your answer because you might assume the current situation won’t abide by the usual “rules.” For example, most newlyweds probably don’t expect to ever get divorced, despite the 40-50% divorce rate. That’s because, from the inside, the relationship feels “special” or distinct from the relationships that ended in divorce.

The premortem technique can help you reorient to the outside view because assuming your answer is incorrect will likely force you to recognize that the specifics of this situation aren’t as important as the base rate. For example, if you’re predicting whether a startup will succeed, it’s tempting to take the inside view and make your forecast based on the business model or the founder’s previous business experience. However, if you try a premortem analysis, it will be easy to come up with reasons the company failed given that the failure rate for startups is roughly 90%. That sobering statistic can help remind you that even if the inside view looks like a recipe for success, the odds are stacked so strongly against new businesses that failure is much more likely.

Trait 5: They Have a Growth Mindset

Forecasting involves quite a bit of failure because forecasters are asked to predict the unpredictable. While no one enjoys being wrong, the authors argue that superforecasters are more likely than regular forecasters to see their failures as an opportunity to learn and improve. Educational psychologists call this a “growth mindset.” People with a growth mindset believe that talent and intelligence can be developed through learning and practice.

The idea behind the growth mindset seems intuitive, but in practice, the authors report that most of us gravitate towards a “fixed mindset” instead. The fixed mindset tells us that talent and intelligence are traits we’re born with, so practice can only strengthen the natural abilities that are already there.

Grow Your Own Growth Mindset

Like many other superforecaster skills, a growth mindset isn’t an inborn trait—it can be grown and developed with practice. In Mindset, psychologist Carol Dweck lays out a few concrete tips to help you transition from a fixed mindset to a growth mindset.

• First, acknowledge and accept your fixed mindset. Most people call on both mindsets in different scenarios, so having a predominantly fixed mindset doesn’t make you a bad or inferior person. It’s just another way of thinking, and if you want to develop a growth mindset, you absolutely can.

• Next, take note of the situations that trigger your fixed mindset. For example, you might notice yourself slipping into a fixed mindset when you’re overwhelmed by a demanding task or when a colleague gets promoted over you.

• Think of your fixed mindset as a separate persona and give it a name. That way, when you catch yourself thinking, “I should give up; I’m just not talented at this,” you can say, “Oh, that’s just Rigid Rita acting up.” Assigning those thoughts to a separate persona will help you remember that you don’t have to believe those fixed-mindset thoughts; you can choose to respond with a growth mindset instead.

• Finally, when you notice your fixed mindset persona taking over, remind that persona that you are capable of growth and that risk and effort are necessary parts of that.

Trait 6: They’re Intellectually Humble

According to the authors, superforecasting requires the humility to admit when you don’t know the answer and to acknowledge that bias might cloud your judgment. This is called intellectual humility, which is an acknowledgment of the power of randomness. It involves admitting that some things are impossible to predict or control, regardless of your skill.

Professional poker player Annie Duke describes this as the difference between “humility in the face of the game” and “humility in the face of your opponents.” In other words, Duke’s long record of success indicates that she is an exceptionally talented poker player and is probably more skilled than most of her opponents. But all of Duke’s skill and experience doesn’t mean she will automatically win every game or that she is even capable of fully understanding every possible intricacy. Like superforecasters, her skills allow her to beat her opponents but not the game itself.

To Foster Humility, Understand the Role of Luck in Success

Annie Duke’s distinction between “humility in the face of the game” and “humility in the face of your opponents” reflects author Nassim Taleb’s views on luck and success. In Fooled by Randomness, Taleb argues that, while skill is a good predictor of moderate success, luck is a better predictor of wild success.

Similarly, Duke understands that winning at poker requires a certain degree of luck; if she were extremely skilled but terribly unlucky, she’d be able to carve out a decent record, but she certainly wouldn’t be the champion player she is now. Therefore, Duke is able to remain humble because she understands that no matter how well she plays, she’s always one streak of bad luck away from a loss.

Trait 7: They’re Team Players

In forecasting tournaments, superforecasters work in teams to create forecasts. According to the authors, one feature of successful teams is the way they freely share resources with each other. Psychologist Adam Grant calls people who give more than they receive “givers.” He compares them to “matchers” (who give and take in equal measure) and “takers” (who take more than they give). Grant found that givers tend to be more successful than matchers or takers. Tetlock and Gardner argue that successful superforecasting teams tend to be stacked with givers.

Superforecasting Teams Are Optimally Distinct

In his book Give and Take, Adam Grant offers another clue as to what makes superforecasting teams so prone to generosity. Grant argues that we’re more motivated to help people who are part of our own social and identity groups (which can be anything from immediate family to school classmates to fellow football fans). Additionally, the more unique the group is compared to the dominant culture, the more inclined members are to help one another (this is called “optimal distinctiveness”). Participating in forecasting tournaments is a rare hobby, and superforecasters are a unique subgroup of forecasters; that uniqueness might strengthen their group identity and make them even more likely to share resources with one another.

Part 3: Can Forecasting Solve the Most Important Questions?

According to the authors, the field of forecasting is facing an important challenge: Namely, the idea that the questions people really care about and need to answer are typically too big for a forecaster to even attempt. For example, a solid superforecaster can predict the likelihood that China will begin closing any of its hundreds of coal plants (which experts say could help the country meet its environmental goals), but they can’t answer the real question people are asking: “Will we be able to prevent the most devastating effects of climate change?”

This is a valid criticism—luckily, Tetlock and Gardner argue that we can get around it by breaking big questions like “Will things turn out okay?” into a host of smaller questions that superforecasters can answer. This is called Bayesian question clustering. The answers to these questions contribute a small piece of the overall answer. Cumulatively, the answers to those small questions can approximate an answer to the bigger question.

For example, if we ask enough questions about factors that could contribute to worsening climate change, we know that the more “yes” answers we get to the small questions (for example, whether sea levels will rise by more than one millimeter in the next year, or whether the United States government will invest more money in solar energy), the more likely the answer to the big question is also a “yes.”

(Shortform note: This technique may help to answer a common critique of forecasting: that it is an example of the “streetlight effect,” or the equivalent of looking for your lost keys under the streetlight—even if that’s not where you lost them—because that’s where the light is best. This is related to black swan thinking—whatever future events you can predict (metaphorically shine a light on) won’t matter because the only truly important events are, by definition, unpredictable. To see the utility of Bayesian question clustering, we can change the metaphor a bit: If a forecaster searches for multiple puzzle pieces under the streetlight as opposed to a single set of keys, they may find enough pieces to at least see the gist of the whole puzzle—even if half the pieces are still lost in the dark.)