{"id":7905,"date":"2020-03-14T11:42:00","date_gmt":"2020-03-14T15:42:00","guid":{"rendered":"https:\/\/www.shortform.com\/blog\/?p=7905"},"modified":"2022-04-12T22:17:49","modified_gmt":"2022-04-13T02:17:49","slug":"what-is-split-testing-ab-testing","status":"publish","type":"post","link":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/","title":{"rendered":"What Is A\/B Testing? (The Lean Startup)"},"content":{"rendered":"\n

So you want to answer your key hypothesis about your product or company. How do you get the right data? Through split testing or A\/B testing. Learn the key principles of Split Testing from Lean Startup here.<\/p>\n\n\n\n\n\n\n\n

Intro to Split Testing<\/h2>\n\n\n\n

Let\u2019s say you develop a new feature to your product and you release it to all your users. Suddenly your metrics improve. But how do you know seasonal effects aren\u2019t at play \u2013 that the users who joined later aren\u2019t just naturally more engaged? Or that you got a burst of users from an unexpected news article?<\/p>\n\n\n\n

An A\/B test or split test avoids bias by splitting users into seeing two different versions of your product. <\/strong>By analyzing the metric resulting from both groups, you get strong quantitative evidence about which version users like more.<\/p>\n\n\n\n

For example, let\u2019s say you have a landing page MVP listing your potential features and a signup form. You\u2019re not sure which of two features your users will like more. So you set up an A\/B test \u2013 half of visitors see feature A on your landing page; the other half sees feature B<\/strong>. You measure the difference in signup rate. If feature A gets a 5% signup rate, but feature B gets 2%, this is evidence that your users may prefer feature A!<\/p>\n\n\n\n

A\/B testing should be applied at critical stages of your progress.<\/strong> You can test variations in your marketing pages, different signup and payment processes, different product interfaces, even turn features on and off in your product.<\/p>\n\n\n\n

As described, one serious benefit to A\/B split testing is reducing time confounds<\/strong>. Since the same group of users are randomly assigned to two groups at once, you don\u2019t need to worry that an earlier group of users is different from a later group.<\/p>\n\n\n\n

Another benefit of A\/B split testing is that it lowers politics<\/strong>. You don\u2019t have to squabble with your team over which features are better \u2013 you can put it to the test with an MVP. SPlit A\/B testing also lets you assign credit where it\u2019s due<\/strong>. If you launch a bunch of marketing and product changes at once, and your metrics improve, who\u2019s responsible? Without independent experiments, it\u2019s difficult to tell. But by running separate A\/B tests, you would be able to see that marketing\u2019s redesigned pages caused the signup boost, not the new product feature<\/a>.<\/p>\n\n\n\n

Finally, split testing lets you gauge the real effect your work is having on users. The product team may obsess over features that users absolutely don\u2019t care about.<\/p>\n\n\n\n

Useful Reporting in Split Testing<\/h2>\n\n\n\n

To make A\/B split testing work, you need to have discipline around analyzing your experiments and planning the next step. Often this means producing reports, which Eric Ries believes should fit 3 A\u2019s:<\/p>\n\n\n\n

Actionable<\/strong>: correctly designed experiments will show a clear cause and effect, and the right metrics will show whether you\u2019re really making progress. From here, you can iterate through the Built-Measure-Learn loop.<\/p>\n\n\n\n

Accessible<\/strong>: simplify the metrics and help people understand what they mean and why they\u2019re important. Consider making metrics publicly viewable by any member of your team.<\/p>\n\n\n\n

Auditable<\/strong>: people should be able to dig into the raw data and trace how the metrics are compiled. If someone doesn\u2019t like the result of an experiment, she may be tempted to question technicalities of the data. You need to be able to prove that the analysis is faultless.<\/p>\n\n\n\n

Now when you\u2019re facing the data, you need to decide what to do. Is there still promise in your direction, and should you keep trying to iterate through the Build-Measure-Learn<\/a> loop? Or have the metrics come back so disappointing so often that it\u2019s time to change your strategy entirely \u2013 to pivot?<\/p>\n","protected":false},"excerpt":{"rendered":"

So you want to answer your key hypothesis about your product or company. How do you get the right data? Through split testing or A\/B testing. Learn the key principles of Split Testing from Lean Startup here.<\/p>\n","protected":false},"author":2,"featured_media":7879,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[45,79],"tags":[78],"class_list":["post-7905","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-business","category-entrepreneurship","tag-lean-startup","","tg-column-two"],"yoast_head":"\nWhat Is A\/B Testing? (The Lean Startup) - Shortform Books<\/title>\n<meta name=\"description\" content=\"Use split testing\u2014also known as A\/B testing\u2014to see which approach works better. Learn the principles of A\/B testing from The Lean Startup.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What Is A\/B Testing? (The Lean Startup)\" \/>\n<meta property=\"og:description\" content=\"Use split testing\u2014also known as A\/B testing\u2014to see which approach works better. Learn the principles of A\/B testing from The Lean Startup.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/\" \/>\n<meta property=\"og:site_name\" content=\"Shortform Books\" \/>\n<meta property=\"article:published_time\" content=\"2020-03-14T15:42:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-04-13T02:17:49+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/s3.amazonaws.com\/wordpress.shortform.com\/blog\/wp-content\/uploads\/2020\/03\/leanstartup-split.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"647\" \/>\n\t<meta property=\"og:image:height\" content=\"379\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Allen Cheng\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Allen Cheng\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/\"},\"author\":{\"name\":\"Allen Cheng\",\"@id\":\"https:\/\/www.shortform.com\/blog\/#\/schema\/person\/e9c1cecc340502045489d4c4e6f4a877\"},\"headline\":\"What Is A\/B Testing? (The Lean Startup)\",\"datePublished\":\"2020-03-14T15:42:00+00:00\",\"dateModified\":\"2022-04-13T02:17:49+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/\"},\"wordCount\":648,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2020\/03\/leanstartup-split.jpg\",\"keywords\":[\"The Lean Startup\"],\"articleSection\":[\"Business\",\"Entrepreneurship\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/\",\"url\":\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/\",\"name\":\"What Is A\/B Testing? (The Lean Startup) - Shortform Books\",\"isPartOf\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2020\/03\/leanstartup-split.jpg\",\"datePublished\":\"2020-03-14T15:42:00+00:00\",\"dateModified\":\"2022-04-13T02:17:49+00:00\",\"description\":\"Use split testing\u2014also known as A\/B testing\u2014to see which approach works better. Learn the principles of A\/B testing from The Lean Startup.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#primaryimage\",\"url\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2020\/03\/leanstartup-split.jpg\",\"contentUrl\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2020\/03\/leanstartup-split.jpg\",\"width\":647,\"height\":379},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.shortform.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What Is A\/B Testing? (The Lean Startup)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.shortform.com\/blog\/#website\",\"url\":\"https:\/\/www.shortform.com\/blog\/\",\"name\":\"Shortform Books\",\"description\":\"The World's Best Book Summaries\",\"publisher\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.shortform.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.shortform.com\/blog\/#organization\",\"name\":\"Shortform Books\",\"url\":\"https:\/\/www.shortform.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.shortform.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2019\/06\/logo-equilateral-with-text-no-bg.png\",\"contentUrl\":\"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2019\/06\/logo-equilateral-with-text-no-bg.png\",\"width\":500,\"height\":74,\"caption\":\"Shortform Books\"},\"image\":{\"@id\":\"https:\/\/www.shortform.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.shortform.com\/blog\/#\/schema\/person\/e9c1cecc340502045489d4c4e6f4a877\",\"name\":\"Allen Cheng\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.shortform.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/658f486a7308ee9a8cffe118baa58e29b0ef902cb29203b65e033a7c49f06a2a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/658f486a7308ee9a8cffe118baa58e29b0ef902cb29203b65e033a7c49f06a2a?s=96&d=mm&r=g\",\"caption\":\"Allen Cheng\"},\"description\":\"Allen Cheng is the founder of Shortform. He has a passion for non-fiction books (having read 200+ and counting) and is on a mission to make the world's best ideas more accessible to everyone. He reads broadly, covering a wide range of subjects including finance, management, health, and society. Allen graduated from Harvard University summa cum laude and attended medical training at the MD\/PhD program at Harvard and MIT. Before Shortform, he co-founded PrepScholar, an online education company.\",\"sameAs\":[\"https:\/\/www.shortform.com\"],\"url\":\"https:\/\/www.shortform.com\/blog\/author\/allen\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"What Is A\/B Testing? (The Lean Startup) - Shortform Books","description":"Use split testing\u2014also known as A\/B testing\u2014to see which approach works better. Learn the principles of A\/B testing from The Lean Startup.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/","og_locale":"en_US","og_type":"article","og_title":"What Is A\/B Testing? (The Lean Startup)","og_description":"Use split testing\u2014also known as A\/B testing\u2014to see which approach works better. Learn the principles of A\/B testing from The Lean Startup.","og_url":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/","og_site_name":"Shortform Books","article_published_time":"2020-03-14T15:42:00+00:00","article_modified_time":"2022-04-13T02:17:49+00:00","og_image":[{"width":647,"height":379,"url":"https:\/\/s3.amazonaws.com\/wordpress.shortform.com\/blog\/wp-content\/uploads\/2020\/03\/leanstartup-split.jpg","type":"image\/jpeg"}],"author":"Allen Cheng","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Allen Cheng","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#article","isPartOf":{"@id":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/"},"author":{"name":"Allen Cheng","@id":"https:\/\/www.shortform.com\/blog\/#\/schema\/person\/e9c1cecc340502045489d4c4e6f4a877"},"headline":"What Is A\/B Testing? (The Lean Startup)","datePublished":"2020-03-14T15:42:00+00:00","dateModified":"2022-04-13T02:17:49+00:00","mainEntityOfPage":{"@id":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/"},"wordCount":648,"commentCount":0,"publisher":{"@id":"https:\/\/www.shortform.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#primaryimage"},"thumbnailUrl":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2020\/03\/leanstartup-split.jpg","keywords":["The Lean Startup"],"articleSection":["Business","Entrepreneurship"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/","url":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/","name":"What Is A\/B Testing? (The Lean Startup) - Shortform Books","isPartOf":{"@id":"https:\/\/www.shortform.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#primaryimage"},"image":{"@id":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#primaryimage"},"thumbnailUrl":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2020\/03\/leanstartup-split.jpg","datePublished":"2020-03-14T15:42:00+00:00","dateModified":"2022-04-13T02:17:49+00:00","description":"Use split testing\u2014also known as A\/B testing\u2014to see which approach works better. Learn the principles of A\/B testing from The Lean Startup.","breadcrumb":{"@id":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#primaryimage","url":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2020\/03\/leanstartup-split.jpg","contentUrl":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2020\/03\/leanstartup-split.jpg","width":647,"height":379},{"@type":"BreadcrumbList","@id":"https:\/\/www.shortform.com\/blog\/what-is-split-testing-ab-testing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.shortform.com\/blog\/"},{"@type":"ListItem","position":2,"name":"What Is A\/B Testing? (The Lean Startup)"}]},{"@type":"WebSite","@id":"https:\/\/www.shortform.com\/blog\/#website","url":"https:\/\/www.shortform.com\/blog\/","name":"Shortform Books","description":"The World's Best Book Summaries","publisher":{"@id":"https:\/\/www.shortform.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.shortform.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.shortform.com\/blog\/#organization","name":"Shortform Books","url":"https:\/\/www.shortform.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.shortform.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2019\/06\/logo-equilateral-with-text-no-bg.png","contentUrl":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2019\/06\/logo-equilateral-with-text-no-bg.png","width":500,"height":74,"caption":"Shortform Books"},"image":{"@id":"https:\/\/www.shortform.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.shortform.com\/blog\/#\/schema\/person\/e9c1cecc340502045489d4c4e6f4a877","name":"Allen Cheng","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.shortform.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/658f486a7308ee9a8cffe118baa58e29b0ef902cb29203b65e033a7c49f06a2a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/658f486a7308ee9a8cffe118baa58e29b0ef902cb29203b65e033a7c49f06a2a?s=96&d=mm&r=g","caption":"Allen Cheng"},"description":"Allen Cheng is the founder of Shortform. He has a passion for non-fiction books (having read 200+ and counting) and is on a mission to make the world's best ideas more accessible to everyone. He reads broadly, covering a wide range of subjects including finance, management, health, and society. Allen graduated from Harvard University summa cum laude and attended medical training at the MD\/PhD program at Harvard and MIT. Before Shortform, he co-founded PrepScholar, an online education company.","sameAs":["https:\/\/www.shortform.com"],"url":"https:\/\/www.shortform.com\/blog\/author\/allen\/"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/www.shortform.com\/blog\/wp-content\/uploads\/2020\/03\/leanstartup-split.jpg","_links":{"self":[{"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/posts\/7905","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/comments?post=7905"}],"version-history":[{"count":2,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/posts\/7905\/revisions"}],"predecessor-version":[{"id":64324,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/posts\/7905\/revisions\/64324"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/media\/7879"}],"wp:attachment":[{"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/media?parent=7905"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/categories?post=7905"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.shortform.com\/blog\/wp-json\/wp\/v2\/tags?post=7905"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}