Studies You Should Know Why Preregistration Is No Stop to Poor Science
The Question
Psychology’s replication crisis prompted a wave of proposed remedies. Among the most celebrated has been preregistration: researchers publicly commit to their hypotheses, methods, and analyses before collecting data, making it harder to engage in the questionable practices that produced so many fragile findings. Preregistration badges have been adopted by journals as signals of scientific rigour. Klonsky (2024) argues that this solution is not a solution at all. It is the same mistake, repeated.
Campbell’s Law
The paper’s central organising idea is Campbell’s Law, a principle from social science that states: the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it was intended to monitor.
Applied to science, the argument runs as follows. Useful tools, such as hypotheses, p-values, and multi-study designs, were originally valued because they were associated with good science. Over time, they came to be treated as indicators of good science in themselves, and therefore as goals to be achieved rather than means to an end. Once that shift happened, the distortions followed: HARKing, p-hacking, and the exploitation of researcher degrees of freedom all emerged as rational responses to a system that rewarded the appearance of rigour rather than rigour itself (Klonsky, 2024).
The Preregistration Problem
Klonsky’s argument is that preregistration is now undergoing exactly the same transformation. It began as a genuinely useful tool for reducing certain kinds of bias. It is rapidly becoming an indicator of strong science and a goal in itself. And once it becomes a goal in itself, the distortions will follow just as predictably as they did before (Klonsky, 2024).
There is already evidence this is happening. Papers seeking preregistration badges routinely violate the rules and spirit of preregistration, deviating from stated analyses, selectively reporting outcomes, and treating the badge as a credential to be obtained rather than a commitment to be honoured (Klonsky, 2024). The badge confers trust; the behaviour underneath does not always warrant it.
The Specific Harms
Klonsky identifies four ways in which preregistration mandates are likely to harm rather than help psychological science.
First, they may discourage optimal scientific practice. Science at its best involves genuine exploration, following unexpected findings, revising hypotheses in response to data, and learning from what the data actually show. Strict preregistration regimes can pathologise this process, treating legitimate scientific flexibility as misconduct and forcing researchers to choose between honest reporting and institutional reward (Klonsky, 2024).
Second, they are likely to exacerbate the file drawer problem. If preregistered studies that produce null results are still less likely to be published, preregistration adds a new layer of difficulty: the existence of a registered protocol makes an unpublished null result more visible as a failure, increasing pressure to find something publishable or to quietly abandon the study (Klonsky, 2024).
Third, they will encourage PRARKing: preregistering after results are known. Just as HARKing emerged as a response to the pressure to have confirmed hypotheses, PRARKing is already emerging as a response to the pressure to have preregistered studies. Researchers run their analyses, then file a preregistration that happens to match what they found (Klonsky, 2024). The badge is obtained; the problem is unchanged.
Fourth, and perhaps most seriously, preregistration badges create false trust in fragile findings. If readers and reviewers treat the badge as a guarantee of rigour, they may scrutinise preregistered studies less carefully than unregistered ones, even when the preregistration itself was inadequate or violated. The badge becomes a shield (Klonsky, 2024).
What Klonsky Recommends Instead
The paper does not argue against preregistration as a tool. It argues against its canonisation as the arbiter of scientific quality. Klonsky (2024) suggests that multiple design features contribute to replicability, including adequate sample sizes, valid measurement, robustness checks, and preregistration, but that none of these should be treated as definitive proof of rigour on its own.
The most important solution, Klonsky argues, is sociocultural rather than procedural. Psychology needs to build a field that genuinely values and rewards robust science, rather than one that rewards the performance of robust science. Badges, mandates, and checklists cannot substitute for a culture that takes replication seriously and stops incentivising flashy but fragile findings at every level from graduate training to journal editorial practice (Klonsky, 2024).
Why It Matters
The replication crisis revealed that psychology had a deep structural problem: a system of incentives that rewarded the wrong things. The response has been, in large part, to add new requirements and new credentials. Klonsky’s paper is a serious argument that this approach misunderstands the problem. The issue was never that researchers lacked the right boxes to tick. It was that the boxes themselves became the point. Adding more boxes will not change that dynamic; it will only give it new forms to inhabit.
Excerpt
Preregistration was meant to fix psychology’s replication crisis by stopping researchers from moving goalposts after the fact. A 2024 paper argues it is already becoming the next goalpost, and that the underlying problem runs far deeper than any badge or mandate can reach.
Reference
Klonsky, E. D. (2024). Campbell’s law explains the replication crisis: Pre-registration badges are history repeating. Assessment. Advance online publication. https://doi.org/10.1177/10731911241253430
