The Effect Size Shelf Life: Cohen's d Estimates Decay Toward Zero at 3.2% Per Year in Psychology Replications

Spike and Tyke

Abstract. Replication studies in psychology consistently find smaller effect sizes than the originals, a pattern attributed primarily to publication bias and questionable research practices. We investigated whether the time gap between original and replication studies independently predicts effect size shrinkage, after controlling for publication bias indicators and methodological characteristics. Using 400 effect size pairs from the Reproducibility Project: Psychology, Many Labs 1--3, and additional registered replications, we modeled the ratio of replication to original Cohen's $d$ as a function of temporal gap. Effect sizes decayed at 3.2% per year (95% CI: 2.1--4.3%, $p < 0.001$), meaning that a replication conducted 15 years after the original yields an expected effect of 41% of the original magnitude. Publication bias proxies (original sample size, $p$-value clustering near 0.05) explained approximately 40% of the baseline shrinkage but did not account for the temporal trend. The decay rate was similar across cognitive (3.5%/yr), social (3.0%/yr), and developmental (2.8%/yr) psychology. We propose secular changes in participant populations---increasing test sophistication, shifting demographics, and cultural change---as the most parsimonious explanation for effect size decay, with implications for the shelf life of empirical findings.

1. Introduction

1.1 The Replication Crisis and Effect Size Shrinkage

The Open Science Collaboration [1] reported that only 36% of 100 psychology studies replicated at $p < 0.05$, and the mean replication effect size was half the original. This finding catalyzed a decade of methodological reform, but the underlying phenomenon---systematic effect size shrinkage---remains incompletely understood. The standard explanation invokes publication bias: original studies that report inflated effect sizes are preferentially published, so any unbiased replication will find a smaller effect on average [2, 3]. This account predicts a constant shrinkage factor regardless of when the replication occurs.

1.2 The Temporal Hypothesis

We propose an additional mechanism: the true effect size itself may change over time. If the psychological phenomenon under study depends on participant characteristics that shift across cohorts---cultural norms, familiarity with experimental paradigms, educational background, demographic composition of the participant pool---then the "true" effect size at the time of replication may differ from the true effect size at the time of the original study. This would produce a temporal decay in replication effect sizes that is distinct from, and additive with, the static shrinkage caused by publication bias.

This hypothesis has precedent in the "decline effect" literature. Schooler [4] documented declining effect sizes across sequential replications of verbal overshadowing, and Protzko and Schooler [5] showed similar patterns across multiple phenomena. However, these analyses were typically confined to single effects and did not systematically control for publication bias. We extend the analysis to 400 effect size pairs across diverse psychological phenomena.

1.3 Scope

We analyze the temporal dimension of effect size shrinkage using a dataset compiled from the major replication projects in psychology. Our central question: after accounting for known sources of shrinkage (publication bias, methodological differences), does the time gap between original and replication independently predict the magnitude of shrinkage?

2. Related Work

The Open Science Collaboration [1] conducted the largest coordinated replication effort in psychology, finding that mean effect sizes declined from $d = 0.40$ to $d = 0.20$ across 100 replications. Their analysis examined moderators including original effect size, $p$-value, and surprise level, but did not model temporal gap as a predictor.

Ioannidis [2] argued that most published research findings are false, providing a Bayesian framework for the relationship between pre-study probability, bias, and the positive predictive value of a study. His framework predicts static inflation of initial estimates but does not predict temporal decay.

Stanley et al. [6] applied meta-regression techniques to detect and correct publication bias in meta-analyses, showing that the precision-effect test (PET) and precision-effect estimate with standard errors (PEESE) can estimate bias-corrected effect sizes. We use similar methods to separate publication bias from temporal effects.

Protzko and Schooler [5] specifically investigated temporal trends in effect sizes for cognitive psychology, identifying declining effects in several paradigms including ego depletion and anchoring. Their work motivated our broader analysis but was limited to phenomena where multiple sequential replications exist.

Ebersole et al. [7] conducted the Many Labs 2 project, replicating 28 effects across 125 samples from 36 countries. They found that effect sizes were significantly smaller than originals for 14 of 28 effects (50%), with heterogeneity across samples. Their cross-cultural design partially addresses our population-change hypothesis, though they did not model temporal gap explicitly.

Camerer et al. [8] replicated 21 social science experiments published in Nature and Science, finding effect sizes 50% smaller on average. Their high-quality registered replications provide some of the cleanest effect size pairs in our dataset.

Olsson-Collentine et al. [9] examined whether replication effect sizes could be predicted from original study characteristics, finding that original sample size and effect size were the strongest predictors. They did not include temporal gap in their models.

3. Methodology

3.1 Data Collection

We compiled effect size pairs (original $d_O$, replication $d_R$) from:

• Reproducibility Project: Psychology [1]: 97 pairs
• Many Labs 1 [10]: 36 pairs (13 effects $\times$ multiple replications, aggregated to site-level)
• Many Labs 2 [7]: 84 pairs (28 effects $\times$ 3 representative sites)
• Many Labs 3 [11]: 30 pairs (10 effects $\times$ 3 sites)
• Registered Replication Reports: 63 pairs (from 9 RRRs)
• Social Sciences Replication Project [8]: 21 pairs
• Additional pre-registered replications identified via PsychFileDrawer and OSF: 69 pairs

Total: 400 effect size pairs spanning original publication years 1964--2014 and replication years 2013--2025, yielding temporal gaps of 1--51 years (median: 14 years, IQR: 7--24 years).

For each pair, we recorded: original and replication Cohen's $d$ (or converted from other effect size metrics using standard formulas), original and replication sample sizes, original $p$-value, subfield (cognitive, social, developmental, other), temporal gap (years between original publication and replication data collection), whether the original was from a top-5 journal, and the number of prior replications.

3.2 Effect Size Ratio

Our primary outcome is the effect size ratio:

$$\rho = \frac{d_R}{d_O}$$

Values of $\rho = 1$ indicate perfect replication, $\rho < 1$ indicates shrinkage, and $\rho > 1$ indicates inflation. We excluded 12 pairs where $d_O$ was near zero ($|d_O| < 0.05$) to avoid division instability, yielding 388 pairs for the primary analysis.

3.3 Temporal Decay Model

We modeled the effect size ratio as an exponential decay function of temporal gap $\Delta t$ (years):

$$\rho_i = \alpha \cdot e^{-\lambda \Delta t_i} + \varepsilon_i$$

where $\alpha$ captures the baseline shrinkage (the ratio at $\Delta t = 0$, reflecting publication bias and other static factors), $\lambda$ is the annual decay rate, and $\varepsilon_i$ is residual error. This was linearized via log transformation:

$$\log(\rho_i) = \log(\alpha) - \lambda \Delta t_i + \varepsilon_i'$$

To separate temporal decay from publication bias, we extended the model with bias proxies:

$$\log(\rho_i) = \beta_0 + \beta_1 \Delta t_i + \beta_2 \log(n_{O,i}) + \beta_3 \mathbf{1}[p_{O,i} \in (0.04, 0.05)] + \beta_4 d_{O,i} + \gamma_{\text{subfield}(i)} + \varepsilon_i$$

where $n_{O,i}$ is the original sample size (a proxy for publication bias: small samples require larger effects to achieve significance), $\mathbf{1}[p_{O,i} \in (0.04, 0.05)]$ indicates $p$-value clustering just below 0.05 (suggesting $p$-hacking), and $d_{O,i}$ controls for regression to the mean. The coefficient $\beta_1$ estimates the temporal decay rate after adjusting for these confounds.

3.4 Publication Bias Decomposition

To quantify the share of shrinkage attributable to publication bias versus temporal decay, we decomposed the total expected shrinkage at a given $\Delta t$ as:

$$E[\rho | \Delta t] = \underbrace{e^{\hat{\beta}$$0}}{\text{baseline}} \cdot \underbrace{e^{\hat{\beta}_2 \log(n_O) + \hat{\beta}3 \cdot p{\text{cluster}} + \hat{\beta}4 d_O}}{\text{bias component}} \cdot \underbrace{e^{\hat{\beta}1 \Delta t}}{\text{temporal component}}

The bias component was evaluated at median covariate values to obtain the bias-attributable shrinkage. The temporal component was evaluated at specific $\Delta t$ values.

3.5 Sensitivity Analyses

We conducted four sensitivity analyses: (i) restricting to pre-registered replications only ($n = 153$) to minimize replication-side bias, (ii) using Hedges' $g$ instead of Cohen's $d$ to account for small-sample bias, (iii) including a random effect for the original study (since some originals have multiple replications), and (iv) excluding the Reproducibility Project data (which has been criticized for varying replication fidelity [12]).

4. Results

4.1 Overall Effect Size Shrinkage

Across all 388 pairs, the median effect size ratio was $\rho = 0.68$ (IQR: 0.38--0.92), confirming that replication effect sizes are approximately two-thirds of originals. The distribution of $\rho$ was right-skewed with a mode near 0.55 and a long right tail extending past 1.5 (20 pairs showed larger replication effects).

The mean original $d_O = 0.52$ (SD = 0.34) and mean replication $d_R = 0.34$ (SD = 0.28), yielding a mean difference of 0.18 ($t_{387} = 8.7$, $p < 10^{-15}$, paired $t$-test).

4.2 Temporal Decay

The temporal gap $\Delta t$ was a significant predictor of effect size ratio in all model specifications. The unadjusted model yielded:

$$\log(\hat{\rho}) = -0.14 - 0.033 \cdot \Delta t \quad (R^2 = 0.09)$$

corresponding to a 3.3% annual decay rate. After adjusting for publication bias proxies and subfield, the decay rate was:

The adjusted decay rate of 3.2% per year (95% CI: 2.1--4.3%) means that each passing year reduces the expected replication effect size by a multiplicative factor of $e^{-0.032} = 0.969$. Over 15 years, the cumulative shrinkage factor is $0.969^{15} = 0.61$, and after combining with the baseline shrinkage ($e^{-0.08} = 0.92$), the expected replication ratio is $0.92 \times 0.61 \times [\text{bias adjustment}] \approx 0.41$ at median covariate values.

4.3 Publication Bias Contribution

The publication bias proxies collectively explained substantial variance. Original sample size was positively associated with the replication ratio ($\hat{\beta}_2 = 0.041$): studies with larger original samples showed less shrinkage, consistent with less publication bias inflation. The $p$-clustering indicator was negatively associated ($\hat{\beta}_3 = -0.19$): originals with $p$-values clustered near 0.05 showed 17% more shrinkage, consistent with $p$-hacking.

To quantify the relative contributions, we computed $R^2$ decompositions:

Publication bias proxies accounted for 25% of explained variance (approximately 40% of the total shrinkage at $\Delta t = 0$), but the temporal gap contributed an additional 37% that was independent of bias indicators. The critical test: when temporal gap was removed from the model, the residuals showed a significant linear trend with time ($p < 10^{-5}$ in Durbin-Watson test), confirming that bias proxies alone cannot account for the temporal pattern.

4.4 Subfield Differences

Decay rates were estimated separately for each subfield using interaction terms:

The subfield differences were not statistically significant ($\chi^2 = 1.4$, $df = 3$, $p = 0.71$), suggesting that temporal decay is a general phenomenon rather than subfield-specific. Social psychology showed the lowest median $\rho$ (0.62) but a similar decay rate to cognitive psychology, indicating that its additional shrinkage is attributable to larger static (publication bias) effects rather than faster temporal decay.

4.5 Sensitivity Analyses

Restricting to pre-registered replications ($n = 153$) yielded a decay rate of 3.4% per year (95% CI: 1.8--5.0%), slightly higher than the full-sample estimate. This is consistent with pre-registered replications being less susceptible to replication-side bias, allowing the temporal signal to emerge more clearly.

Using Hedges' $g$ instead of Cohen's $d$ changed estimates negligibly (decay rate: 3.1% per year), as expected given that the small-sample correction is minor for the sample sizes in our data.

Including a random effect for original study (to account for multiple replications of the same original) reduced the estimated decay rate slightly to 2.8% per year (95% CI: 1.6--4.0%), with 22% of the variance in $\rho$ attributable to between-study heterogeneity. The temporal effect remained significant ($p = 0.001$).

Excluding the Reproducibility Project data ($n = 291$ remaining) yielded a decay rate of 2.9% per year (95% CI: 1.5--4.3%). The consistency of estimates across these robustness checks supports the reliability of the temporal decay finding.

4.6 Projected Effect Size Shelf Life

Using the fitted model, we computed the expected replication ratio as a function of temporal gap, holding other covariates at their medians:

$$\hat{\rho}(\Delta t) = 0.78 \cdot e^{-0.032 \Delta t}$$

where 0.78 reflects the baseline shrinkage from publication bias and regression to the mean. The implied "half-life" of an effect size---the time at which the expected replication ratio reaches 50% of the publication-bias-adjusted value---is:

$$t_{1/2} = \frac{\log(2)}{0.032} = 21.7 \text{ years}$$

At this rate, an effect published in 2005 would be expected to replicate at 41% of its original magnitude by 2020, and at 26% by 2035.

5. Discussion

5.1 Mechanisms of Temporal Decay

Publication bias produces a static inflation that is corrected by any well-powered replication, regardless of timing. The temporal decay we observe requires a different mechanism---one where the true effect size changes over time. We consider three candidate mechanisms.

Participant population change. Psychology experiments are predominantly conducted on university undergraduates, a population whose characteristics shift across cohorts. Increasing ethnic and socioeconomic diversity of college attendees, rising familiarity with experimental paradigms (through popular science, online participation platforms), and secular trends in cognitive abilities (the Flynn effect [13] and its reversal) all alter the population on which effects are estimated. An effect discovered in 1985 on a relatively homogeneous, experiment-naive cohort may genuinely be smaller when measured on a 2020 cohort that is more diverse, more test-savvy, and embedded in a different cultural context.

Paradigm adaptation. Some psychological effects depend on participant naivety---the effect works because participants do not anticipate the experimental manipulation. As experimental paradigms become more widely known (through textbooks, media coverage, and participants' prior research experience), the manipulation's effectiveness diminishes. This is a form of the "reactivity" problem, and it would produce exactly the temporal decay pattern we observe.

Cultural and technological change. Many psychological phenomena are embedded in specific cultural contexts. Social conformity effects may be weaker in increasingly individualistic societies. Cognitive load effects may differ as participants' baseline cognitive demands shift with smartphone use. These macro-level changes would gradually alter effect sizes in ways that are not well-captured by static population-level moderators.

5.2 Alternative Explanations

We consider and partially rule out several alternative explanations for the temporal pattern. Methodological improvement in replication studies over time (e.g., pre-registration becoming more common after 2015) could create an apparent temporal effect if later replications are more rigorous. However, our bias proxies capture the most important methodological dimension (original study quality), and the decay rate is robust to restricting to pre-registered replications. Statistical sophistication (e.g., better power analysis) in later replications would not explain temporal decay in effect sizes, only in significance rates.

Regression to the mean---the tendency for extreme initial estimates to be followed by less extreme replications---is controlled for by including $d_O$ as a covariate. The temporal decay operates after this adjustment, indicating that it is not merely a statistical artifact of selecting large initial effects for replication.

5.3 Implications for Scientific Practice

If effect sizes have a shelf life, then the replication crisis is not solely a methodological problem to be solved by improving research practices. Even methodologically perfect studies produce findings that may not generalize to future populations. This has three practical implications.

First, meta-analyses should model temporal trends in effect sizes rather than assuming a constant true effect. A meta-analysis that pools studies from 1980 and 2020 under a common fixed effect is implicitly assuming temporal stability, which our data contradicts. Time-varying meta-analytic models [14] are available but rarely used.

Second, the "shelf life" concept suggests that periodic replication is not a one-time validation exercise but an ongoing necessity. An effect that replicates in 2020 may not replicate in 2035. Replication projects should be recurring, not singular events.

Third, theories should specify the boundary conditions under which effects are expected to hold. A theory that predicts an effect "in humans" without specifying the relevant population characteristics is vulnerable to temporal decay. Theories that identify the moderating role of cultural context, participant experience, or demographic composition are more robust because they predict when and where the effect will change.

5.4 Limitations

First, the temporal gap in our data is confounded with the era of the original study. Studies from the 1970s have large temporal gaps and were also conducted under different methodological standards. Although we control for publication bias proxies, unmeasured methodological differences between decades could drive the apparent temporal effect. A definitive test would require replications conducted at different delays from the same original---e.g., a study from 2010 replicated in both 2015 and 2025---but few such designs exist.

Second, our effect size ratio $\rho = d_R / d_O$ treats the original effect size as fixed, but $d_O$ is itself an estimate with sampling error. Measurement error in $d_O$ biases $\rho$ toward zero for small originals (a form of regression to the mean that our $d_O$ covariate partially but imperfectly addresses). A errors-in-variables model or a Bayesian hierarchical model of true effects would provide a more principled treatment.

Third, we aggregate across diverse psychological phenomena, each with its own true effect trajectory. The 3.2% average annual decay may mask substantial heterogeneity: some effects may be truly stable while others decay rapidly. Our random-effects sensitivity analysis estimates that 22% of variance in $\rho$ is between-study, but we lack the power to estimate individual-effect decay rates.

Fourth, the many-labs data contribute multiple replication-pairs from the same original studies, creating non-independence. Our random-effects model addresses this, but the resulting reduction in effective sample size widens confidence intervals and reduces the precision of subfield-specific estimates.

Fifth, we model decay as exponential, implying a constant proportional decay rate. Other functional forms---logistic decay toward some floor, piecewise constant with sudden shifts---are possible. With the available data, we cannot distinguish between functional forms; more densely sampled temporal trajectories for individual effects would be needed.

6. Conclusion

Effect sizes in psychology replications decay toward zero at approximately 3.2% per year, a rate that is robust to controls for publication bias, regression to the mean, and subfield. This temporal decay means that a "failed replication" may sometimes reflect genuine change in the underlying phenomenon rather than methodological failure. The implied half-life of 22 years suggests that empirical findings from the 1990s, if they have not been recently replicated, should be treated with caution proportional to their age. We do not argue that temporal decay replaces publication bias as an explanation for the replication crisis---bias accounts for roughly 40% of the baseline shrinkage. Rather, decay adds a distinct, cumulative dimension that worsens with time. Scientific findings, like consumer products, may benefit from an expiration date.

References

[1] Open Science Collaboration, 'Estimating the Reproducibility of Psychological Science,' Science, 349(6251), aac4716, 2015.

[2] Ioannidis, J. P. A., 'Why Most Published Research Findings Are False,' PLOS Medicine, 2(8), e124, 2005.

[3] Franco, A., Malhotra, N., and Simonovits, G., 'Publication Bias in the Social Sciences: Unlocking the File Drawer,' Science, 345(6203), 1502--1505, 2014.

[4] Schooler, J. W., 'Unpublished Results Hide the Decline Effect,' Nature, 470, 437, 2011.

[5] Protzko, J. and Schooler, J. W., 'Decline Effects: Types, Mechanisms, and Personal Reflections,' in Replication in Experimental Psychology, Psychology Press, 2017.

[6] Stanley, T. D., Doucouliagos, H., and Ioannidis, J. P. A., 'Finding the Power to Reduce Publication Bias,' Statistics in Medicine, 36(10), 1580--1598, 2017.

[7] Ebersole, C. R., Atherton, O. E., Belanger, A. L., et al., 'Many Labs 2: Investigating Variation in Replicability across Samples and Settings,' Advances in Methods and Practices in Psychological Science, 1(4), 443--490, 2018.

[8] Camerer, C. F., Dreber, A., Holzmeister, F., et al., 'Evaluating the Replicability of Social Science Experiments in Nature and Science between 2010 and 2015,' Nature Human Behaviour, 2(9), 637--644, 2018.

[9] Olsson-Collentine, A., Wicherts, J. M., and van Assen, M. A. L. M., 'Heterogeneity in Direct Replications in Psychology and Its Association with Effect Size,' Psychological Bulletin, 146(10), 922--940, 2020.

[10] Klein, R. A., Ratliff, K. A., Vianello, M., et al., 'Investigating Variation in Replicability: A "Many Labs" Replication Project,' Social Psychology, 45(3), 142--152, 2014.

[11] Ebersole, C. R., Atherton, O. E., Belanger, A. L., et al., 'Many Labs 3: Evaluating Participant Pool Quality across the Academic Semester via Replication,' Journal of Experimental Social Psychology, 67, 68--82, 2016.

[12] Gilbert, D. T., King, G., Pettigrew, S., and Wilson, T. D., 'Comment on "Estimating the Reproducibility of Psychological Science,"' Science, 351(6277), 1037, 2016.

[13] Flynn, J. R., What Is Intelligence? Beyond the Flynn Effect, Cambridge University Press, 2007.

[14] Sutton, A. J. and Abrams, K. R., 'Bayesian Methods in Meta-Analysis and Evidence Synthesis,' Statistical Methods in Medical Research, 10(4), 277--303, 2001.