{"id":26321,"date":"2021-02-19T09:03:02","date_gmt":"2021-02-19T13:03:02","guid":{"rendered":"https:\/\/www.shortform.com\/blog\/?p=26321"},"modified":"2021-02-24T19:59:48","modified_gmt":"2021-02-24T23:59:48","slug":"iarpa","status":"publish","type":"post","link":"https:\/\/www.shortform.com\/blog\/iarpa\/","title":{"rendered":"IARPA Forecasting Tournament Results"},"content":{"rendered":"\n<p>What is IARPA? When was it founded and what was its purpose?<\/p>\n\n\n\n<p>The Intelligence Advanced Research Activity (IARPA) is an intelligence arm operating under the Office of the Director of National Intelligence. From 2011 to 2015, IARPA ran a forecasting tournament to identify forecasting professionals and gain insight into their methods.<\/p>\n\n\n\n<p>Read about the IARPA forecasting tournament.<\/p>\n\n\n\n<!--more-->\n\n\n\n<h2 class=\"wp-block-heading\">IARPA: Intelligence Advanced Research Activity <\/h2>\n\n\n\n<p>In 2006, the IC created a research arm called the Intelligence Advanced Research Projects Activity (IARPA). But IARPA quickly hit a snag: Useful research requires data, and they had no way to measure forecaster accuracy or track their methods.&nbsp;<\/p>\n\n\n\n<p>To address this, IARPA officials approached Tetlock and his partner, Barbara Mellers, for help creating a forecasting tournament that would identify superforecasters and give researchers insight into their methods. Unlike the earlier EPJ tournament, forecasters would make predictions about events months into the future, not years, since forecasting accurately more than a year out is almost impossible.<\/p>\n\n\n\n<p>The IARPA tournament was a bold move. If tournament results showed that a group of volunteers with no access to classified information made more accurate predictions than professional analysts, those analysts\u2019 careers would be on the line. That type of bureaucratic boat-rocking is rare, even in fields with such high stakes.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Forecasting <strong>Tournament Results<\/strong><\/h3>\n\n\n\n<p>The IARPA tournament was designed for researchers to glean insight into the statistical superforecasting methods. Armed with a much larger sample size, they could finally evaluate forecaster accuracy <em>and<\/em> compare forecasters to each other. Over the first two years of the tournament, Tetlock identified more \u201cfoxes\u201d who scored better than 98% of the group. He dubbed them \u201csuperforecasters.\u201d&nbsp;<\/p>\n\n\n\n<p>In addition to identifying superforecasters, data from the multi-year IARPA tournament made it possible to track superforecasters\u2019 performance over time and measure their regression to the mean. Surprisingly, superforecasters\u2019 skills not only held up from one year to the next, they actually <em>improved<\/em>.&nbsp;<\/p>\n\n\n\n<p>Remember, for a population this size, we expect to see regression to the mean, which would mean people in the top 2% would do slightly worse than the previous year, not better. The lack of any regression at all suggests that something is skewing the data. The authors think this happened because forecasters who did exceptionally well in year one were given the \u201csuper\u201d designation and put on teams of other superforecasters for year two. It\u2019s possible that this recognition provided <a href=\"https:\/\/www.shortform.com\/blog\/a-sense-of-accomplishment\/\">a sense of accomplishment<\/a> that inspired forecasters to work even harder\u2014in the second year, superforecasters raised their individual scores enough to offset the usual regression.&nbsp;<strong>Over time, though, even skilled performers show some regression.<\/strong> After several years in the tournament, researchers determined the year-to-year correlation for individual scores is .65. In other words, the scores of about 30% of superforecasters regress toward the mean each year, while the other 70% remain \u201csuper.&#8221; If forecasting accuracy were purely a matter of luck, we\u2019d expect 100% of superforecasters to regress to the mean over time. So we can confidently say skill plays a role for superforecasters\u2014but what kind of skill?<\/p>\n","protected":false},"excerpt":{"rendered":"<p>What is IARPA? When was it founded and what was its purpose? The Intelligence Advanced Research Activity (IARPA) is an intelligence arm operating under the Office of the Director of National Intelligence. From 2011 to 2015, IARPA ran a forecasting tournament to identify forecasting professionals and gain insight into their methods. Read about the IARPA forecasting tournament.<\/p>\n","protected":false},"author":7,"featured_media":26322,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[39,25],"tags":[203],"class_list":["post-26321","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-history","category-statistics","tag-superforecasting","","tg-column-two"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v24.3 (Yoast SEO v24.3) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>IARPA Forecasting Tournament Results - Shortform Books<\/title>\n<meta name=\"description\" content=\"From 2011 to 2015, IARPA ran a forecasting tournament to gain insight into superforecasters&#039; approach. Read about the IARPA findings.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.shortform.com\/blog\/iarpa\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"IARPA Forecasting Tournament Results\" \/>\n<meta property=\"og:description\" content=\"From 2011 to 2015, IARPA ran a forecasting tournament to gain insight into superforecasters&#039; approach. Read about the IARPA findings.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.shortform.com\/blog\/iarpa\/\" \/>\n<meta property=\"og:site_name\" content=\"Shortform Books\" \/>\n<meta property=\"article:published_time\" content=\"2021-02-19T13:03:02+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-02-24T23:59:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/s3.amazonaws.com\/wordpress.shortform.com\/blog\/wp-content\/uploads\/2021\/02\/analysis-statistics-office-forecasting.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1220\" \/>\n\t<meta property=\"og:image:height\" content=\"650\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Darya Sinusoid\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Darya Sinusoid\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.shortform.com\/blog\/iarpa\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/iarpa\/\"},\"author\":{\"name\":\"Darya Sinusoid\",\"@id\":\"https:\/\/www.shortform.com\/blog\/#\/schema\/person\/0421cce75bc249b11e2517b3a91f9c46\"},\"headline\":\"IARPA Forecasting Tournament Results\",\"datePublished\":\"2021-02-19T13:03:02+00:00\",\"dateModified\":\"2021-02-24T23:59:48+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/iarpa\/\"},\"wordCount\":505,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/iarpa\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2021\/02\/analysis-statistics-office-forecasting.jpg\",\"keywords\":[\"Superforecasting\"],\"articleSection\":[\"History\",\"Statistics\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.shortform.com\/blog\/iarpa\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.shortform.com\/blog\/iarpa\/\",\"url\":\"https:\/\/www.shortform.com\/blog\/iarpa\/\",\"name\":\"IARPA Forecasting Tournament Results - Shortform Books\",\"isPartOf\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/iarpa\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/iarpa\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2021\/02\/analysis-statistics-office-forecasting.jpg\",\"datePublished\":\"2021-02-19T13:03:02+00:00\",\"dateModified\":\"2021-02-24T23:59:48+00:00\",\"description\":\"From 2011 to 2015, IARPA ran a forecasting tournament to gain insight into superforecasters' approach. Read about the IARPA findings.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/iarpa\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.shortform.com\/blog\/iarpa\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.shortform.com\/blog\/iarpa\/#primaryimage\",\"url\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2021\/02\/analysis-statistics-office-forecasting.jpg\",\"contentUrl\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2021\/02\/analysis-statistics-office-forecasting.jpg\",\"width\":1220,\"height\":650},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.shortform.com\/blog\/iarpa\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.shortform.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"IARPA Forecasting Tournament Results\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.shortform.com\/blog\/#website\",\"url\":\"https:\/\/www.shortform.com\/blog\/\",\"name\":\"Shortform Books\",\"description\":\"The World&#039;s Best Book Summaries\",\"publisher\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.shortform.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.shortform.com\/blog\/#organization\",\"name\":\"Shortform Books\",\"url\":\"https:\/\/www.shortform.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.shortform.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2019\/06\/logo-equilateral-with-text-no-bg.png\",\"contentUrl\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2019\/06\/logo-equilateral-with-text-no-bg.png\",\"width\":500,\"height\":74,\"caption\":\"Shortform Books\"},\"image\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.shortform.com\/blog\/#\/schema\/person\/0421cce75bc249b11e2517b3a91f9c46\",\"name\":\"Darya Sinusoid\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.shortform.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2021\/07\/Untitled-design-1.png\",\"contentUrl\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2021\/07\/Untitled-design-1.png\",\"caption\":\"Darya Sinusoid\"},\"description\":\"Darya\u2019s love for reading started with fantasy novels (The LOTR trilogy is still her all-time-favorite). Growing up, however, she found herself transitioning to non-fiction, psychological, and self-help books. She has a degree in Psychology and a deep passion for the subject. She likes reading research-informed books that distill the workings of the human brain\/mind\/consciousness and thinking of ways to apply the insights to her own life. Some of her favorites include Thinking, Fast and Slow, How We Decide, and The Wisdom of the Enneagram.\",\"url\":\"https:\/\/www.shortform.com\/blog\/author\/darya\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"IARPA Forecasting Tournament Results - Shortform Books","description":"From 2011 to 2015, IARPA ran a forecasting tournament to gain insight into superforecasters' approach. Read about the IARPA findings.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.shortform.com\/blog\/iarpa\/","og_locale":"en_US","og_type":"article","og_title":"IARPA Forecasting Tournament Results","og_description":"From 2011 to 2015, IARPA ran a forecasting tournament to gain insight into superforecasters' approach. Read about the IARPA findings.","og_url":"https:\/\/www.shortform.com\/blog\/iarpa\/","og_site_name":"Shortform Books","article_published_time":"2021-02-19T13:03:02+00:00","article_modified_time":"2021-02-24T23:59:48+00:00","og_image":[{"width":1220,"height":650,"url":"https:\/\/s3.amazonaws.com\/wordpress.shortform.com\/blog\/wp-content\/uploads\/2021\/02\/analysis-statistics-office-forecasting.jpg","type":"image\/jpeg"}],"author":"Darya Sinusoid","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Darya Sinusoid","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.shortform.com\/blog\/iarpa\/#article","isPartOf":{"@id":"https:\/\/www.shortform.com\/blog\/iarpa\/"},"author":{"name":"Darya Sinusoid","@id":"https:\/\/www.shortform.com\/blog\/#\/schema\/person\/0421cce75bc249b11e2517b3a91f9c46"},"headline":"IARPA Forecasting Tournament Results","datePublished":"2021-02-19T13:03:02+00:00","dateModified":"2021-02-24T23:59:48+00:00","mainEntityOfPage":{"@id":"https:\/\/www.shortform.com\/blog\/iarpa\/"},"wordCount":505,"commentCount":0,"publisher":{"@id":"https:\/\/www.shortform.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.shortform.com\/blog\/iarpa\/#primaryimage"},"thumbnailUrl":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2021\/02\/analysis-statistics-office-forecasting.jpg","keywords":["Superforecasting"],"articleSection":["History","Statistics"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.shortform.com\/blog\/iarpa\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.shortform.com\/blog\/iarpa\/","url":"https:\/\/www.shortform.com\/blog\/iarpa\/","name":"IARPA Forecasting Tournament Results - Shortform Books","isPartOf":{"@id":"https:\/\/www.shortform.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.shortform.com\/blog\/iarpa\/#primaryimage"},"image":{"@id":"https:\/\/www.shortform.com\/blog\/iarpa\/#primaryimage"},"thumbnailUrl":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2021\/02\/analysis-statistics-office-forecasting.jpg","datePublished":"2021-02-19T13:03:02+00:00","dateModified":"2021-02-24T23:59:48+00:00","description":"From 2011 to 2015, IARPA ran a forecasting tournament to gain insight into superforecasters' approach. Read about the IARPA findings.","breadcrumb":{"@id":"https:\/\/www.shortform.com\/blog\/iarpa\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.shortform.com\/blog\/iarpa\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.shortform.com\/blog\/iarpa\/#primaryimage","url":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2021\/02\/analysis-statistics-office-forecasting.jpg","contentUrl":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2021\/02\/analysis-statistics-office-forecasting.jpg","width":1220,"height":650},{"@type":"BreadcrumbList","@id":"https:\/\/www.shortform.com\/blog\/iarpa\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.shortform.com\/blog\/"},{"@type":"ListItem","position":2,"name":"IARPA Forecasting Tournament Results"}]},{"@type":"WebSite","@id":"https:\/\/www.shortform.com\/blog\/#website","url":"https:\/\/www.shortform.com\/blog\/","name":"Shortform Books","description":"The World&#039;s Best Book Summaries","publisher":{"@id":"https:\/\/www.shortform.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.shortform.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.shortform.com\/blog\/#organization","name":"Shortform Books","url":"https:\/\/www.shortform.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.shortform.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2019\/06\/logo-equilateral-with-text-no-bg.png","contentUrl":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2019\/06\/logo-equilateral-with-text-no-bg.png","width":500,"height":74,"caption":"Shortform Books"},"image":{"@id":"https:\/\/www.shortform.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.shortform.com\/blog\/#\/schema\/person\/0421cce75bc249b11e2517b3a91f9c46","name":"Darya Sinusoid","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.shortform.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2021\/07\/Untitled-design-1.png","contentUrl":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2021\/07\/Untitled-design-1.png","caption":"Darya Sinusoid"},"description":"Darya\u2019s love for reading started with fantasy novels (The LOTR trilogy is still her all-time-favorite). Growing up, however, she found herself transitioning to non-fiction, psychological, and self-help books. She has a degree in Psychology and a deep passion for the subject. She likes reading research-informed books that distill the workings of the human brain\/mind\/consciousness and thinking of ways to apply the insights to her own life. Some of her favorites include Thinking, Fast and Slow, How We Decide, and The Wisdom of the Enneagram.","url":"https:\/\/www.shortform.com\/blog\/author\/darya\/"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2021\/02\/analysis-statistics-office-forecasting.jpg","_links":{"self":[{"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/posts\/26321","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/comments?post=26321"}],"version-history":[{"count":4,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/posts\/26321\/revisions"}],"predecessor-version":[{"id":27029,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/posts\/26321\/revisions\/27029"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/media\/26322"}],"wp:attachment":[{"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/media?parent=26321"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/categories?post=26321"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/tags?post=26321"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}