This article is an excerpt from the Shortform book guide to "Bad Science" by Ben Goldacre. Shortform has the world's best summaries and analyses of books you should be reading.
Like this article? Sign up for a free trial here.
What is data manipulation in research? What are methods of data manipulation you should avoid?
Unfortunately, some researchers manipulate data to fit their hypothesis. In Bad Science, Ben Goldacre discusses the ethics of altering data in an unprofessional way.
Check out how to spot data manipulation in research.
Bad Research Methods
Goldacre discusses the data manipulation in research methods people use to alter their results and gives you the tools to evaluate research methods yourself. Goldacre stresses that it’s important to understand and evaluate scientific literature so you can make informed healthcare decisions.
(Shortform note: Goldacre stresses the importance of improving your own science literacy. However, some studies argue that because low science literacy is a systemic problem, it requires systemic solutions. These studies suggest that the best way to combat scientific misinformation is by increasing the quality of K-12 science education rather than asking adults to educate themselves.)
Scientific trials should be evaluated on their methods, not their results. As we’ll show, by using manipulative methods, it’s possible to engineer whatever result you want. These kinds of manipulations often happen because of the economic pressure to produce and market new treatments. According to Goldacre, most medical research is funded by pharmaceutical companies, who in turn pressure researchers to produce results that help them get their drugs to market.
(Shortform note: The situation Goldacre describes, in which the majority of medical research is funded by companies with vested interests, may only be escalating. Experts note that in recent years, government and nonprofit medical research funding has declined dramatically. Since Goldacre authored Bad Science in 2008, the research landscape has become even more dominated by corporate interests.)
There are a few main ways researchers manipulate results. These include insufficient experimental controls, inadequate blinding, and inadequate randomization.
1. Experimental Controls Shape Results
Goldacre explains that scientific studies are designed to test one treatment at a time, without any influence from other factors. To do this, good researchers make use of experimental controls, which are elements of a study’s design that are put in place to prevent specific factors from influencing results. By contrast, some researchers use weak experimental controls to manipulate their results.
All scientific experiments involve a control subject and a test subject. Ideally, your control and test subjects should be identical in every way except for the variable you’d like to test—in the medical field, this variable is usually a medical treatment.
Spotting Bad Experimental Controls
Because controlling variables can be a complex process that includes compromise, you should carefully evaluate the studies you read. Studies that show a high level of detail and thorough commitment to controlling as many variables as possible are likely to be trustworthy. The best studies are aware of their own flaws and may even go so far as to suggest future methods that would improve those flaws.
By contrast, if a study doesn’t spend much time discussing how variables were controlled, Goldacre says you should take the study with a grain of salt. Its findings might be meaningful, but they might just as easily be the result of a variable the researchers conveniently left out.
2. Blinding and the Placebo Effect
Furthermore, Goldacre argues that some researchers manipulate their studies by allowing participant and researcher expectations to skew results due to the placebo effect.
The placebo effect is a phenomenon in which the strength of a medical treatment is affected by both the patient’s beliefs about the treatment and the doctor’s beliefs about the treatment. Because of the placebo effect, when you believe you’re receiving a more effective treatment, your health improves more than when you believe you’re getting an ineffective treatment, even if the treatments are identical. Similarly, when doctors believe they’re giving an effective treatment, their patients tend to improve more than when they believe they’re giving an ineffective treatment.
According to Goldacre, good studies make use of a technique called blinding to prevent the placebo effect from influencing their results. Blinding is the practice of keeping participants and administrators unaware of which treatment each participant receives. By keeping participants and those administering treatments in the dark, blinding ensures that no group of participants will be more influenced by the placebo effect than any other group.
Spotting Bad Blinding
Examining a study’s blinding practices can help you determine whether that study is trustworthy. Specifically, in trustworthy studies, all treatments should be administered in the same way to help ensure that participants all have the same expectations for their treatments. For example, if one group receives an experimental drug in the form of an injection to the arm, the group receiving the control drug should be given that drug by an arm injection, too. Furthermore, treatments should not be given by the researchers who designed the study. As you read a study, check to see if the study’s authors brought in outsiders to administer treatments. If they didn’t, their blinding practices may not have been thorough enough.
On the other hand, if a study’s description of its blinding methods is short, vague, or absent, take it as a red flag. By withholding this information, the study’s authors deny you the tools you need to evaluate their claims.
3. Randomization and Researcher Bias
In addition to weak blinding, Goldacre describes how unscrupulous researchers manipulate results by strategically sorting participants into different groups. For instance, if you wanted to make your new drug seem more effective than it is, you could decide to give your drug to the patients who were healthiest and most likely to improve anyway, while giving the standard treatment to the patients in the worst condition.
Spotting Bad Randomization
There are a few different shady randomization practices to look out for when evaluating research. Specifically, Goldacre is skeptical of the practice of assigning treatments based on the order that participants sign up for the trial. In a system like this, the first 10 participants to sign up might get treatment A, while the second group of 10 all get treatment B, and so on. While on the surface, this procedure might seem random, it leaves a lot of room for recruiters to influence results. For example, if you know that the next patient you recruit will be placed into the group testing your new treatment, you might choose a patient who you think is likely to get well anyway—when they do, it’ll appear as though your treatment worked.
Goldacre notes that some researchers manipulate their studies by randomizing patients based on factors that more subtly affect results. For instance, suppose you come up with a randomization strategy that sorts participants into groups based on ZIP code. While this seems random, some ZIP codes might contain more affluent neighborhoods, where residents can generally afford better health care. If these more affluent patients were all sorted into the same treatment group, it might make that treatment seem more effective than it really is.
———End of Preview———
Like what you just read? Read the rest of the world's best book summary and analysis of Ben Goldacre's "Bad Science" at Shortform.
Here's what you'll find in our full Bad Science summary:
- The strategies researchers, corporations, and journalists use to mislead the public
- The tools you need to identify and call out shady science when you see it
- Why media outlets have an ethical obligation to publish retractions
