IESE Insight
Trust the data? A large-scale study finds results often differ when many researchers analyze the same data
An international study in Nature with nearly 500 researchers highlights the risk of relying on a single analytical approach.
When data are on the table, we tend to assume that rigorous and reliable analysis can settle a debate.
In recent years, research in social and behavioral sciences has adopted tools specifically designed to bolster that credibility: registered studies, replication studies, reproducibility checks, and so on. All of these aim to reduce bias and chance findings.
But research published in Nature and featuring the work of nearly 500 independent analysts, including IESE Business School’s Sebastian Hafenbrädl, suggests that we shouldn’t be so credulous. Results can still vary substantially, even when the same data are put to use answering the same question, by different people.
Generally, scientific research consists of an individual or a team analyzing a dataset and publishing the results. Peer reviews assess the methodology, but that methodology reflects a series of choices made by the research team. These include how the data are cleaned, which statistical models are used, which software is chosen, and how the results are interpreted. The differences that these choices can throw up are known as analytic variability, and this is the first study to measure them systemically at that scale.
The Nature study conducted 504 re-analyses of data from 100 previously published studies across the social and behavioral sciences, with the participation of 457 independent researchers. Researchers were given the same research question and dataset, but how they chose to approach the task was left up to them. Each study was re-analyzed by at least five researchers.
The results were striking. While the broad claims of the original studies were supported in about 3 out of 4 cases (74% of 504 re-analyses carried out), this didn’t represent scientific consensus across the board. Levels of uncertainty and statistical estimates differed. In assessing if all analysts of the same study coincided in their findings, the percentage fell to 1 in 3 (34%). One thing is for the majority to agree; and another, for nobody to disagree.
Neither experience nor volume of data offers a definitive solution
The study found that analysts with greater statistical expertise did not agree with one another any more than less experienced analysts. In studies with very large samples, analytical uncertainty persisted just as much.
Where differences were observed was in the type of study design: observational studies with their greater complexity of data showed more variation than more tightly controlled experimental studies.
One message is clear: Data do not speak for themselves, and one analysis should rarely be treated as the final word. Analytical results depend not only on the quality of the data but also on the decisions made during analysis.
Toward a more robust evidence-based culture
For practitioners who must make evidence-based decisions, the lesson to internalize is that you can never be too cautious in relying on a single analysis. The authors have two proposals to gain greater certainty before implementing a strategy:
- Multiverse method testing. Systematically explore multiple combinations of analytical decisions to verify whether the finding holds up.
- Multiple analysts. Commission the analysis to several independent teams. If they agree, the basis for decision-making is solid; if they disagree, an area of uncertainty has been identified before it impacts the decision.
While we often seek easy yet data-driven answers in times of uncertainty, this study is a timely reminder that empirical evidence is often not as straightforward as it appears. The scientific process is complex and layered, and acknowledging alternatives is both transparent and realistic. Important decisions are better informed by looking at multiple plausible analyses, understanding uncertainty, and asking how robust a finding is across different reasonable approaches.
ALSO OF INTEREST:
Good deeds, bad optics: telling people about your charity can backfire
