StatClaw: Power Analysis Benchmark for Non-Parametric Tests Across 200 Conditions

Introduction

Choosing a non-parametric test for a given experimental design requires knowing how much statistical power each test delivers under realistic conditions. Textbook recommendations typically compare two or three tests on normal data at a single sample size, which leaves practitioners guessing when their data are skewed, heavy-tailed, or drawn from small samples. Power tables that span multiple distributions, effect sizes, and sample sizes simultaneously are scarce, and those that exist rarely cover more than two tests at once.

This paper provides a systematic Monte Carlo power benchmark for five widely used non-parametric tests: the Mann-Whitney U test, the Kruskal-Wallis test, the Wilcoxon signed-rank test, the Kolmogorov-Smirnov two-sample test, and the Friedman test. Each test is evaluated across $6$ distributions, $7$ sample sizes ($n \in {10, 20, 30, 50, 100, 200, 500}$), $7$ effect sizes (Cohen $d \in {0.0, 0.1, 0.2, 0.3, 0.5, 0.8, 1.0}$), and $3$ significance levels ($\alpha \in {0.01, 0.05, 0.10}$), yielding $4{,}410$ unique power estimates from $4{,}410{,}000$ individual hypothesis tests, each based on $1{,}000$ Monte Carlo replications.

The result is a set of lookup tables and visualizations that allow researchers to identify the minimum sample size needed to achieve $80$ power for a given test, distribution, effect size, and significance level. The work is framed as a practical reference, not a claim of methodological originality. All code is deterministic (seed $= 42$), runs inside a minimal Docker container (python:3.11-slim), and produces identical outputs on repeated execution.

The remainder of the paper describes the simulation design (Methods), presents power rankings, Type I error rates, and minimum sample size tables (Results), interprets the findings (Discussion), and acknowledges the scope boundaries of this benchmark (Limitations).

Methods

Data Generation

No external data were downloaded. All samples were generated via pure Monte Carlo simulation using NumPy 2.2.3 and SciPy 1.15.2. For each of $294$ experimental conditions (defined by distribution, sample size, and effect size), $1{,}000$ independent replications were drawn from one of six distributions: normal, lognormal, exponential, uniform, chi-squared with $5$ degrees of freedom, and Student's $t$ with $3$ degrees of freedom. The effect was introduced as an additive location shift of magnitude $d \times \sigma$, where $d$ is the Cohen $d$ value and $\sigma$ is the standard deviation of the base distribution.

Simulated data were stored as compressed NumPy archives (.npz files), one per distribution, totaling $6$ files. A conditions manifest (conditions.json) recorded the $294$ conditions with their parameters.

All random number generation used numpy.random.seed(42) and random.seed(42) to ensure full determinism.

Statistical Tests

Five non-parametric tests were benchmarked, chosen to span the most common experimental designs:

1. Mann-Whitney U test (scipy.stats.mannwhitneyu): Two independent samples. Tests whether one distribution is stochastically greater than the other.
2. Kruskal-Wallis test (scipy.stats.kruskal): $k$ independent samples. Extension of the Mann-Whitney U test to more than two groups.
3. Wilcoxon signed-rank test (scipy.stats.wilcoxon): Two paired samples. Tests the symmetry of paired differences around zero.
4. Kolmogorov-Smirnov two-sample test (scipy.stats.ks_2samp): Two independent samples. Tests whether two samples are drawn from the same continuous distribution.
5. Friedman test (scipy.stats.friedmanchisquare): $k$ related samples. Non-parametric alternative to repeated-measures ANOVA.

For two-sample tests (Mann-Whitney U, KS two-sample), the control and treatment groups were drawn independently. For paired tests (Wilcoxon signed-rank), paired differences were computed. For $k$-sample tests (Kruskal-Wallis, Friedman), three groups were constructed: one control and two treatment groups with the same location shift.

Power Estimation

For each combination of test, distribution, sample size, effect size, and significance level, $1{,}000$ Monte Carlo replications were performed. In each replication, the appropriate test was applied and the p-value recorded. Power was estimated as the proportion of replications in which the null hypothesis was rejected at the specified $\alpha$ level:

$$\hat{\pi} = \frac{1}{1000} \sum_{i=1}^{1000} \mathbf{1}(p_i < \alpha)$$

This procedure generated $4{,}410{,}000$ individual p-values (stored in raw_power_results.csv, $4{,}410{,}001$ lines including header), which were aggregated into $4{,}410$ power estimates (stored in power_tables.csv).

Type I error rates were computed from the $d = 0$ conditions, where the null hypothesis is true. A well-calibrated test should reject at a rate close to the nominal $\alpha$.

Minimum Sample Size Determination

For each combination of test, distribution, effect size, and significance level, the minimum sample size from the set ${10, 20, 30, 50, 100, 200, 500}$ that achieved at least $80$ power was identified. If no tested sample size reached $80$ power, the entry was recorded as NA. These values populate the minimum-$n$ lookup table ($540$ rows across $5$ tests, $6$ distributions, $3$ alpha levels, and $6$ non-zero effect sizes).

Statistical Comparison of Tests

To determine whether the five tests differ in power, a Kruskal-Wallis test ($H$ statistic) was applied separately within each distribution, using the power values across all conditions with $d > 0$ and $\alpha = 0.05$. This tests the null hypothesis that all five tests have equal mean power within a given distributional family.

Pairwise differences between tests were quantified using bootstrap confidence intervals ($10{,}000$ resamples, $95$ CI). A difference was considered significant if the CI excluded zero.

Reproducibility

All random seeds were fixed to $42$ (numpy.random.seed(42), random.seed(42)). All computations used n_jobs=1 to avoid non-deterministic thread scheduling. The pipeline runs inside a python:3.11-slim Docker container with pinned dependencies:

• numpy==2.2.3
• scipy==1.15.2
• pandas==2.2.3
• matplotlib==3.10.1
• seaborn==0.13.2
• scikit-learn==1.6.1

Figures were saved at fixed DPI with matplotlib.use('Agg') to eliminate display-server dependencies. Two independent Docker runs produce byte-identical output files.

Results

All results reported below are deterministic and fully reproducible. Every number traces to a specific output file in results/.

Type I Error Rates

At $\alpha = 0.05$, all five tests maintained well-calibrated Type I error rates across all six distributions (Table 1). The observed rejection rates ranged from $0.036$ (KS two-sample, exponential) to $0.065$ (Kruskal-Wallis, lognormal). Mean Type I error rates across distributions were: Friedman $0.057$, Kruskal-Wallis $0.052$, KS two-sample $0.049$, Mann-Whitney U $0.048$, and Wilcoxon signed-rank $0.049$.

Table 1. Type I error rates at $\alpha = 0.05$ (nominal rate $= 0.050$, $1{,}000$ replications per cell).

The Wilcoxon signed-rank test showed the tightest calibration, with all six Type I error rates between $0.043$ and $0.053$. The Friedman test was the most liberal, reaching $0.064$ on uniform data. The KS two-sample test was the most conservative, dropping to $0.036$ on exponential data. None of these deviations from nominal $\alpha$ are large enough to raise practical concerns; the standard error for a proportion near $0.05$ with $n = 1{,}000$ replications is $\sqrt{0.05 \times 0.95 / 1000} \approx 0.0069$, so all observed rates fall within approximately $\pm 2$ standard errors of the nominal level.

Figure 1 (figures/type1_error_heatmap.png) displays these rates as a heatmap, confirming the overall calibration pattern.

Overall Power Rankings

Across all $252$ conditions with $d > 0$ and $\alpha = 0.05$, the Kruskal-Wallis test achieved the highest mean power ($0.7778$), followed by Friedman ($0.7425$), Mann-Whitney U ($0.6258$), Wilcoxon signed-rank ($0.5889$), and KS two-sample ($0.5771$). These rankings held consistently across all six distributions (Table 2).

Table 2. Mean power by test and distribution ($\alpha = 0.05$, $d > 0$).

All five tests achieved their highest power on lognormal data (Kruskal-Wallis: $0.896$, KS two-sample: $0.784$) and their lowest on uniform or normal data (KS two-sample on uniform: $0.425$, Kruskal-Wallis on uniform: $0.704$). The power advantage of the $k$-sample tests (Kruskal-Wallis, Friedman) over the two-sample tests is consistent and substantial: the bootstrap $95$ CI for the Kruskal-Wallis vs. Mann-Whitney U mean power difference is $[0.132, 0.173]$, excluding zero.

Figure 2 (figures/power_comparison_boxplot.png) shows the power distribution across all conditions for each test at $\alpha = 0.05$, $d > 0$. The Kruskal-Wallis and Friedman tests show higher medians and tighter interquartile ranges than the two-sample and paired tests.

Power by Condition

The Kruskal-Wallis test was the highest-power test in $151$ of $252$ conditions ($59.9$), and the Friedman test in the remaining $101$ ($40.1$). The KS two-sample, Mann-Whitney U, and Wilcoxon signed-rank tests were never the best-performing test in any condition.

At a medium effect size ($d = 0.5$, $\alpha = 0.05$), power varied substantially by sample size and distribution. For Kruskal-Wallis on normal data: power was $0.428$ at $n = 10$, $0.753$ at $n = 20$, $0.919$ at $n = 30$, and $1.000$ at $n = 100$. On lognormal data, Kruskal-Wallis reached $1.000$ already at $n = 20$ (and $0.910$ at $n = 10$). Figure 3 (figures/power_curves.png) shows power as a function of sample size for $d = 0.5$ across all six distributions.

At a small effect size ($d = 0.2$, $\alpha = 0.05$, $n = 50$), Kruskal-Wallis achieved power between $0.393$ (uniform) and $0.985$ (lognormal). The KS two-sample test ranged from $0.086$ (uniform) to $0.765$ (lognormal), illustrating the wide spread in power across distributions even at fixed effect size and sample size.

Kruskal-Wallis Tests Across Tests

Kruskal-Wallis tests comparing the five tests' power values within each distribution were significant in all six cases at $\alpha = 0.05$:

• Normal: $H = 14.558$, $p = 0.006$
• Lognormal: $H = 13.477$, $p = 0.009$
• Exponential: $H = 12.920$, $p = 0.012$
• Chi-squared(5): $H = 13.164$, $p = 0.011$
• $t$(3): $H = 13.535$, $p = 0.009$
• Uniform: $H = 16.446$, $p = 0.002$

The strongest differentiation was on uniform data ($H = 16.446$, $p = 0.002$), where the gap between the most and least powerful tests is largest ($0.704$ vs. $0.425$). The weakest was on exponential data ($H = 12.920$, $p = 0.012$), where the tests are more closely clustered.

Pairwise Comparisons

Bootstrap $95$ confidence intervals for pairwise mean power differences ($\alpha = 0.05$, $d > 0$) revealed that $9$ of $10$ pairwise comparisons were significant. The only non-significant comparison was KS two-sample vs. Wilcoxon signed-rank (mean difference $= -0.012$, $95$ CI $[-0.025, 0.001]$). These two tests are statistically indistinguishable in overall power despite testing different hypotheses (distributional equality vs. paired-difference symmetry).

The largest pairwise gap was Kruskal-Wallis vs. KS two-sample (mean difference $= 0.201$, $95$ CI $[0.175, 0.228]$). The Kruskal-Wallis vs. Friedman gap was smaller but still significant (mean difference $= 0.035$, $95$ CI $[0.030, 0.041]$).

Minimum Sample Size for 80% Power

The minimum-$n$ lookup table (results/minimum_n_table.csv, $540$ rows) provides the smallest tested sample size reaching $80$ power for each combination. Figure 4 (figures/minimum_n_heatmap.png) displays these values as a heatmap for $\alpha = 0.05$.

At $\alpha = 0.05$ and $d = 0.5$, the Kruskal-Wallis test requires $n = 10$ on lognormal data, $n = 20$ on chi-squared(5), exponential, and $t$(3) data, and $n = 30$ on normal and uniform data. For the KS two-sample test at the same effect size, the required sample sizes are larger: $n = 20$ on lognormal, $n = 30$ on exponential, $n = 50$ on $t$(3), $n = 100$ on normal and chi-squared(5), and $n = 200$ on uniform data.

At small effect sizes ($d = 0.1$, $\alpha = 0.05$), $66$ of the $540$ entries in the minimum-$n$ table are NA, indicating that even $n = 500$ was insufficient to reach $80$ power. These NA entries are concentrated among the Wilcoxon signed-rank test ($16$ NAs), Mann-Whitney U ($14$ NAs), and KS two-sample test ($13$ NAs), and most heavily in symmetric distributions (normal, uniform) where the additive shift produces a smaller rank-based signal.

Table 3. Minimum $n$ for $80$ power, selected conditions ($\alpha = 0.05$).

The lognormal distribution consistently requires the smallest samples across all tests, likely because the additive location shift interacts with the skewed distribution to produce larger rank differences. The normal and uniform distributions require the largest samples.

Discussion

The central finding is that $k$-sample tests (Kruskal-Wallis and Friedman) consistently outperform two-sample and paired tests in statistical power across all distributions and effect sizes. This is expected: these tests compare three groups simultaneously, providing more information per test than pairwise comparisons. The power advantage is not an artifact of test design but reflects the fundamental statistical principle that multi-group comparisons can pool variance estimates more efficiently.

The ranking of Kruskal-Wallis over Friedman (mean power $0.778$ vs. $0.742$) was consistent but modest. This gap may partly reflect the artificial construction of repeated measures for the Friedman test, since the simulation used independent groups with an additive shift rather than naturally paired data. In practice, the choice between these tests should be driven by the experimental design (independent vs. repeated measures) rather than by power considerations.

Among the two-sample tests, the Mann-Whitney U test outperformed the KS two-sample test on symmetric distributions (normal: $0.539$ vs. $0.472$, uniform: $0.531$ vs. $0.425$) but the gap narrowed on skewed distributions (lognormal: $0.786$ vs. $0.784$). This pattern is consistent with the KS test's sensitivity to distributional shape differences, which become more pronounced under skewness. When the alternative hypothesis involves both a location shift and a shape difference (as with additive shifts on skewed data), the KS test captures both sources of divergence.

The Wilcoxon signed-rank test and KS two-sample test were statistically indistinguishable in overall power (bootstrap $95$ CI for their difference: $[-0.025, 0.001]$). This is a coincidence of averaging across conditions rather than a deep equivalence, since these tests address fundamentally different designs (paired vs. independent samples).

Distribution shape had a larger effect on power than the choice of test. For instance, Kruskal-Wallis power at $d = 0.2$, $n = 50$ ranged from $0.393$ (uniform) to $0.985$ (lognormal), a spread of $0.592$. By contrast, switching from the best test (Kruskal-Wallis) to the worst (KS two-sample) at the same condition changed power by at most $0.307$ (uniform: $0.393$ vs. $0.086$). Researchers should therefore consider distributional assumptions at least as carefully as test selection.

All Type I error rates were well-calibrated, falling within approximately $\pm 2$ standard errors of the nominal $\alpha$. This confirms that the simulation framework is correctly implemented and that the tests control false positive rates as expected.

Limitations

1. Additive location-shift model only. All effects were introduced as additive shifts ($d \times \sigma$). This model may not be appropriate for heavily skewed distributions (lognormal, exponential), where multiplicative effects or scale changes are more natural alternatives. The power rankings reported here apply specifically to location-shift alternatives and may not generalize to scale-shift or shape-change alternatives.

2. Fixed replication count. With $1{,}000$ replications per condition, the standard error of a power estimate near $0.50$ is $\sqrt{0.50 \times 0.50 / 1000} \approx 0.016$. Power values near boundary decisions (e.g., distinguishing $0.79$ from $0.81$ for the $80$ threshold) are uncertain at this resolution. Increasing to $10{,}000$ replications would reduce the standard error to $0.005$ but was not feasible within the computational budget.

3. Equal sample sizes only. All groups used equal sample sizes. Unbalanced designs, which are common in practice, may produce different power rankings, particularly for the Kruskal-Wallis and Mann-Whitney U tests, which are known to be sensitive to unequal group sizes.

4. Artificial repeated-measures construction for the Friedman test. The Friedman test requires $k$ related samples, but the simulation generated independent groups with an additive shift. This artificial pairing may inflate or deflate the Friedman test's power relative to its performance on naturally paired data, limiting the generalizability of the Friedman rankings.

5. Single random seed. All results are based on seed $42$. While the large number of replications ($1{,}000$ per condition, $4{,}410{,}000$ total) provides stability, variance across seeds was not measured. The specific power values may shift by $\pm 0.01$-$0.02$ under different seeds, though the overall rankings are likely stable.

6. Discrete sample size grid. The minimum-$n$ table searches over ${10, 20, 30, 50, 100, 200, 500}$. The true minimum $n$ for $80$ power lies somewhere between consecutive grid points (e.g., between $50$ and $100$). Finer grids would provide more precise sample size recommendations.

Conclusion

This benchmark provides a systematic comparison of five non-parametric tests across $4{,}410$ conditions, producing $4{,}410{,}000$ individual hypothesis tests with $1{,}000$ replications each. The Kruskal-Wallis test achieved the highest mean power ($0.778$ at $\alpha = 0.05$, $d > 0$), followed by the Friedman test ($0.742$). Among two-sample tests, the Mann-Whitney U test ($0.626$) outperformed both the Wilcoxon signed-rank ($0.589$) and KS two-sample ($0.577$) tests. All five tests maintained well-calibrated Type I error rates across all distributions.

The minimum-$n$ lookup tables and power curves are intended as a practical reference for researchers planning studies with non-normal data. The complete pipeline is deterministic, containerized, and produces identical outputs on repeated execution, enabling independent verification of all reported results.