Why Innovations Fail: The Top 3 Problems to Avoid

This article is an excerpt from the Shortform book guide to "The Master Guides: Business Innovation" by Shortform. Shortform has the world's best summaries and analyses of books you should be reading.

Like this article? Sign up for a free trial here.

Why do innovation projects fail? Why do businesses fail to succeed in the market?

Innovative products and services can reshape markets and propel companies from obscurity to wild success. However, most innovation projects fail: Many don’t even make it to market, and of those that do, the majority never generate the exponential growth and massive profits their developers were hoping for.

Here are the reasons why innovations fail, according to multiple experts.

Why Do Innovation Projects Fail?

Let’s look at the issue of why so many innovations fail. Many business consultants and other experts have written books to answer this question, and most of their answers can be grouped into three general approaches:

Some contend that innovative products fail when they’re not innovative enough.
Others blame problematic market forces.
And still others argue that innovation projects fail because of inadequate project management practices, pointing to the unique challenges of managing innovation projects.

1. Insufficient Innovation

Some experts argue that the high failure rate of innovation projects is, in essence, largely an illusion, since most of these projects lack true innovation. Recall that to be truly innovative a new product must be both different and better. Not all product developments fit these parameters: A company might produce something that isn’t clearly differentiated from existing alternatives, or a product with differences that don’t offer meaningful advantages to customers. It stands to reason that these new products wouldn’t succeed in the market.

In this section we’ll examine how different authors address the problem of insufficient innovation, focusing mostly on evaluating whether potential ventures are innovative enough to succeed and the different types of innovation to consider.

The Ten Types of Innovation

In Ten Types of Innovation, Larry Keeley, cofounder of the Doblin Innovation Firm, and his Doblin colleagues Ryan Pikkel, Brian Quinn, and Helen Walters assert that all innovations can be classified into 10 standard categories. They observe that most successful disruptive innovations (that is, most of the new products and services that reshape markets and generate substantial growth for the company) incorporate innovations in at least five of the 10 categories.

The key to success, then, is to understand what types of innovation your new offering incorporates and, if necessary, innovate in additional categories. The 10 categories that Keeley et al. identify are:

Core product innovations.
Interfacing product innovations.
Capability innovations.
Organizational innovations.
Relational innovations.
Revenue stream innovations.
Distribution innovations.
Support innovations.
Branding innovations.
Gratification innovations.

Blue Ocean Strategy

Keeley et al. argue that innovating in enough categories is crucial for the success of disruptive innovation projects. Similarly, in Blue Ocean Strategy, W. Chan Kim and Renée Mauborgne also advise you to innovate in multiple areas, arguing that your ultimate goal should be to create not just a new product but a new and uncontested market that your product will dominate. This offers the best route to success, because if you try to merely compete with established companies in existing markets, you’ll end up with low profit margins and minimal opportunities for growth.

To spark ideas for innovative products, Kim and Mauborgne recommend considering your product from several angles:

Consider all the different products that your customers could use to achieve the same goal through different means. For example, rock-climbing gear and video games are different in form and function, but they fulfill the same basic goal of connecting with friends in a thrilling environment. Ask yourself if you could combine desirable features from different alternatives to provide unique value.
Consider what characteristics motivate your customers to buy higher-quality products at higher prices versus lower quality at lower prices. Ask yourself if you can create a unique offering by focusing on the characteristics they’re willing to pay extra for and eliminating the others.
Identify underserved people in the purchasing chain. If the end user and the person with purchasing power are not one and the same, the end user may have pain points, or persistent problems, you can address to provide unique value.
Consider what happens before, during, and after your product is used. Kim and Mauborgne note that products are rarely used in isolation. Ask yourself whether there’s friction between any of these handoffs that you could alleviate to create unique value. It might be as simple as bundling complementary products with yours.
Kim and Mauborgne observe that most industries tend to gravitate toward either functionality or emotion. Consider making an emotional industry more functional by stripping away unnecessary extras, or making a functional industry more emotional.
Consider the needs of non-customers. Ask yourself why some people don’t buy your current product or products in your industry. Can you develop a new offering to solve the problem that’s keeping them away?

Then, to help you visualize whether a new product idea provides unique value, Kim and Mauborgne introduce the blue ocean strategy chart. On the horizontal axis, you list the attributes of your product that your customers care about. The vertical axis represents how much each attribute is present in a product. You plot the value of each characteristic on the graph as a point, and connect the points to create a strategy curve, both for your product (or prospective new offering) and its leading alternatives. The result might look something like this:

Kim and Mauborgne point out that if your product’s path closely follows another product (like Product X and Product Y in our example graph) then you’re not providing unique value. You want your strategy curve to diverge significantly from every other company’s.

Zero to One

In Zero to One, Peter Thiel echoes Kim and Mauborgne’s assertion that innovation is the key to avoiding direct competition, which is the key to growth and profit. He argues that first-of-a-kind products create de facto monopolies, which are not only good for your company, but also have the potential to benefit society. Thiel says you need five things to successfully launch a disruptive innovation startup:

Revolutionary technology or opportunity: Making an incremental improvement to an existing technology is generally not a great opportunity. You need a breakthrough that provides fundamentally new capabilities, or, at the very least, increases existing capabilities by a factor of 10. You also need to be the only one pursuing it. Otherwise, competition will consume all your profits.
Strategic timing: Even if you’re the first to invent a revolutionary technology, there needs to be widespread interest in solving a problem that the technology can solve for it to be a commercial success.
A great team: For an innovative business venture to succeed, you need managers and employees who are motivated and unified by the venture’s unique mission. Thiel advises that you choose people whose technical and interpersonal skills complement each other, establish clearly defined roles and responsibilities, and don’t overpay your CEO.
Effective distribution: Planning how to market and distribute your product is an integral part of designing it. For your venture to be viable, the cost of acquiring a customer (through advertising or other means) must be less than the total profits you’ll make from that customer—ideally no more than 30% of it.
Enduring value: You need to be the first to introduce your product in order to build a monopoly that will generate long-term profits, but moving first is just a means to an end, not an end in itself. Your product must be something that people will still want to buy in years to come and that’s difficult for other companies to copy.

2. Problematic Market Dynamics

Some experts have sought to explain why so many innovation projects fail by examining market mechanisms that work against innovators. Geoffrey Moore’s Crossing the Chasm and Clayton Christensen’s The Innovator’s Dilemma offer two such explanations.

Crossing the Chasm

Moore primarily addresses the challenges that would face a small startup company that has already made some kind of technological breakthrough and is looking to turn its breakthrough into a commercially viable product. He notes that this is hard to do because of the psychographics (the combination of psychology and demographics that determines customers’ purchasing behavior) of mainstream market customers.

Moore explains that as a new technology matures, different psychographic categories of customers adopt it at different stages of maturity and for different reasons. When you introduce an innovative technology, your first customers buy it either because they just love to try out new technology or they hope to gain a strategic advantage for their own business by adopting the technology early. These customers make up what Moore calls the “early market,” and they only represent about a sixth of the total market population.

The mainstream market (the other five-sixths of your potential customers) is made up of people who are more risk-averse. They prefer to buy products only from companies who’ve established their credibility as market leaders. They typically won’t buy a new product until they see other people using it and benefiting from it. And they generally don’t count your early-market customers as credible reviewers—they want to see other mainstream customers vouching for it before they buy. So mainstream customers won’t buy your product until other mainstream customers have already bought it and found it useful.

Moore calls this catch-22 the “chasm” between the early market and the mainstream market. Many innovative products enjoy brief success in the early market, but once the early market is saturated, sales stagnate or drop off. They end up failing because they can’t get past the chasm and break into the mainstream market.

The Innovator’s Dilemma

While Moore aims his advice at small startups (though his strategy for crossing the chasm is applicable to any company introducing an innovative product), Christensen primarily addresses established companies.

He observes that large companies often struggle with developing innovative products, especially disruptive innovations, despite the fact that they have more resources than startups do. He argues that this is because established companies, operating in large markets and needing large volumes of sales to survive, generally focus on pleasing their existing customers. They don’t seek out alternative customers and smaller markets because these groups are initially too small to provide the volume of sales they need. But disruptive innovations almost always start out in small markets, appealing to different customers than tried-and-true products do. Thus larger companies, which aren’t incentivized to pursue small markets, end up missing or ignoring opportunities that smaller companies more readily pick up on.

Christensen’s observations corroborate Moore’s, since both an early market and a niche mainstream market are small. And the fact that existing customers Christensen describes are rarely interested in disruptive innovations reflects the same risk-averse mentality in mainstream customers that Moore discusses.

Christensen also notes another reason established companies struggle with innovation: In most cases, producing a disruptive innovation requires a complete set of production tools and a new network of suppliers and distributors. This comes naturally to a startup, but it’s hard for an established company to justify the cost of retooling its production lines and overhauling its network of suppliers.

3. Inadequate or Inappropriate Project Management

Finally, some experts argue that the primary reason innovation projects fail is that they aren’t managed correctly. Innovation projects present unique challenges when it comes to project management. After all, innovation involves overturning conventions and doing things that have never been done before. How can you plan, schedule, budget, and manage tasks that don’t follow conventions or past precedents? In this section, we’ll discuss how different authors recommend managing innovation projects.

The Lean Startup

In The Lean Startup, Eric Ries points out that the biggest challenge in managing an innovation project (especially one that takes the form of a new startup company) is the uncertainty involved in doing something new. His strategy for overcoming this challenge is basically an adaptation of the scientific method, which he uses to gather data that eliminates uncertainties as efficiently as possible.

First, formulate a hypothesis. What do you believe about your product or your customers that’s vital to your business? Then, build the minimal product necessary to test your hypothesis. Observe the behavior of your prototype or of users who interact with it. Then analyze the data and reflect—how far off was your hypothesis? What do you need to change about your strategy? Should you actually change your entire direction? Based on your conclusions, update your hypothesis or create a new one and repeat the process.

The faster you move through this loop, the faster you’ll learn and the sooner you’ll have enough information to plan and achieve success. (And the less likely the project will be to fail due to financial backers pulling out because they’re not seeing their investment pay off.)

101 Design Methods

In 101 Design Methods, Vijay Kumar argues the first step to building a company that can consistently innovate successfully is to believe that it’s possible to plan and manage innovation projects. If you don’t believe that, you’ll probably end up either shying away from innovation projects entirely or letting them run without enough structure to make them successful.

His approach to managing innovation consists of breaking down the innovation process into seven standard tasks that you can plan, schedule, and manage without sacrificing the flexibility that you need on innovation projects:

Get a clear view of the big picture. Establish the general direction and goals of the project.
Research your operating environment. Identify your unknowns and plan out how to get the information you need.
Research your stakeholders. Make sure you understand their motivations and pain points.
Distill the information you gathered in tasks 2 and 3 into broadly applicable principles that can guide your decision-making.
Brainstorm ideas that could make up parts of the solution. Relating this to the authors’ 10 types of innovation, each idea would probably be a way to implement one of the 10 types.
Determine the combination of ideas from task 5 that gives you the best complete solution. This is where you would combine several types of innovation as the authors recommend.
Create a detailed plan for implementing the final solution, including things like budgets, timelines, and marketing tactics.

While the seven tasks follow a logical progression, understand that sometimes you have to revisit earlier tasks later in the project or cycle through multiple iterations of the whole sequence. As your team completes each task, take stock of the project and decide which task you need to do next. Then plan out how you’ll do the next task (including applicable schedules, budgets, and so forth).

Kumar suggests you keep track of where your project has been and where it’s going using a graphic map that shows the seven tasks and charts your path through them. The map also shows a pair of axes indicating how the focus of the different tasks varies from theoretical to practical and from learning about what already exists to creating new things. Here’s an adaptation of Kumar’s innovation task map, showing a project where you revisited task 2 after first doing task 4, revisited task 6 after first doing task 7, and end up doing two iterations of the full sequence of tasks:

It’s worth noting that while Kumar’s approach to innovation project management is different from Ries’s, the two are not mutually exclusive. On the contrary, Kumar strongly recommends using prototypes for learning and evaluation purposes during tasks 5, 6, and 7. Furthermore, his discussion of prototyping parallels Ries’s description of MVPs in his emphasis on building only what you need for the task and prototyping your solutions as early as possible.

Why Innovations Fail: The Top 3 Problems to Avoid