Leland Teschler's Editorial: Why you can't always believe published research

Jan. 10, 2008
Take with a grain of salt any conclusions coming from scientific research: A lot of them are likely to be wrong.

That’s the premise put forth by John P.A. Ioannidis, a medical researcher in Greece and adjunct professor at Tufts University School of Medicine. Ioannidis has modeled what can happen when unconscious biases creep into hypothesis testing. The field he had in mind was medical research, but others commenting on his work see parallels in areas ranging from brain mapping to parapsychology.

Some of Ioannidis’ findings sound like common sense about relationships claimed to be statistically significant. For example, studies with small sample sizes are less likely to yield correct results, as are studies of observable effects that are relatively small. And studies confirming consequences for which there is already a lot of solid evidence are probably valid.

But some problems are not as obvious. For example, too much flexibility in defining or analyzing outcomes can potentially transform negative results into positive ones. And it’s a good idea to question graded scales that researchers invent themselves. Of course, fuzzy definitions are less of an issue where the outcome is inarguable (such as death).

There are similar difficulties in new fields where analysis techniques are still being hammered out. Researchers can be tempted to report only their “best” results rather than a sea of data that is inconclusive. Ioannidis says there’s evidence that researchers manipulate outcomes and only report them selectively even in randomized trials.

Two of Ioannidis’ conclusions are interesting because they have implications beyond medicine. He says the greater the financial interests and prejudices in a scientific field, the less likely research findings are true. In the same regard, hot fields chased by numerous scientific teams are less likely to yield correct research findings.

His explanation for these effects is that prestigious investigators can have their own biases that prejudice their actions. For example, they can have financial or other interests that may lead them to shoot down peer reviews of findings that refute their own. And there can be a bandwagon effect when many teams of investigators pursue the same field. The imperative is to publish ahead of the competition, with a priority put on disseminating the most impressive “positive” results. So researchers can be predisposed to confirm the initial idea rather than to find the truth.

Clearly, hot fields are not confined just to medical research. Two other areas that immediately come to mind are global warming and alternative energy. Cheerleaders on both sides of these issues tend to haul out peer-reviewed research findings and wave them at each other as though warding off vampires with a crucifix.

Those tempted to use peer-reviewed research this way would do well to realize the scientific ground on which they are standing may not be as solid as they think. As Ioannidis has said, “It is impossible to know with 100% certainty what the truth is in any research question.”

— Leland Teschler, Editor

Sponsored Recommendations

A closer look at modern design considerations for food and beverage

April 9, 2024
With new and changing safety and hygiene regulations at top of mind, its easy to understand how other crucial aspects of machine design can get pushed aside. Our whitepaper explores...

Condition Monitoring for Energy and Utilities Assets

April 9, 2024
Condition monitoring is an essential element of asset management in the energy and utilities industry. The American oil and gas, water and wastewater, and electrical grid sectors...

Strategizing for sustainable success in material handling and packaging

April 9, 2024
Download our visual factory brochure to explore how, together, we can fully optimize your industrial operations for ongoing success in material handling and packaging. As your...

Fueling the Future of Commercial EV Charging Infrastructure

April 9, 2024
Miguel Gudino, an Associate Application Engineer at RS, addresses various EV charging challenges and opportunities, ranging from charging station design strategies to the advanced...

Voice your opinion!

To join the conversation, and become an exclusive member of Machine Design, create an account today!