A Calibrated Claim-Stability Benchmark for Single-Cell RNA-seq Workflows

Problem and Thesis

Most single-cell RNA-seq workflows are reproducible at the container, file, or script level, but they are not calibrated at the level that matters scientifically: whether the biological claims survive reasonable perturbations. The central contribution of this repository is therefore no longer a PBMC3k workflow note. It is a benchmark that asks whether a workflow can distinguish robust from fragile biological conclusions across multiple public single-cell lanes.

The benchmark is built around three ideas:

1. Claim stability is the primary endpoint. The benchmark asks whether canonical biological claims degrade under lane-compatible negative controls.
2. External validity is lane-specific. Orthogonal or label-backed metrics are necessary, but they should be used as gates only where they are biologically appropriate.
3. Reference-based calibration should be statistical, not rhetorical. Bootstrap confidence intervals and empirical-null permutation p-values quantify how far observed concordance sits above chance.

This is a claim-stability calibration contribution for scRNA-seq workflows, not a firstness claim about scRNA benchmark automation in general. The relevant novelty is the biological-claim endpoint, the frozen control-compatibility policy, and the calibration of lane-specific external evidence.

Benchmark Design

Freeze Contract

Each benchmark lane is normalized into a common freeze contract:

• canonical_input.h5ad
• dataset_manifest.json
• freeze_audit.json
• source_provenance.md
• benchmark_protocol.json

The runtime pipeline consumes only canonical_input.h5ad. Freeze metadata records the source identifier, source URL, download or publication provenance, SHA256, retained metadata columns, and observed matrix shape.

The benchmarked workflow itself is a Scanpy-centered canonical pipeline: highly variable gene flagging, PCA, nearest-neighbor graph construction, Leiden clustering over frozen resolution sweeps, UMAP for visualization, and marker-based cluster annotation.

Lane Taxonomy

The current submitted artifact bundle is built from six active lanes. A deferred Tabula Sapiens extension remains outside this evidence set.

Primary active lanes:

• pbmc3k: easy sanity lane for canonical PBMC lineage recovery
• kang_ifnb: donor and stimulation lane based on the Kang et al. interferon-beta PBMC dataset
• pbmcsca: cross-technology PBMC lane derived from the Ding et al. comparison panel
• citeseq_pbmc: orthogonal-modality lane with paired protein-backed labels
• multiome_pbmc: orthogonal-modality PBMC lane with paired chromatin-derived reference labels

Active supplementary lane:

• pancreas_integration: cross-study pancreas stress lane on the harmonized scIB panel

Deferred lane outside the submitted evidence set:

• tabula_sapiens_subset: atlas-stress extension for later expansion

Claim Families

The benchmark evaluates config-driven claim families rather than raw cluster IDs:

• lineage_presence
• structure_preservation
• stability_consistency
• protein_concordance where protein-backed labels exist

This design lets the benchmark score whether the workflow still supports the intended biological conclusion even when exact partitions or embeddings move.

Claim-Score Definition

For a lane with claim-family set F, each family f contains a finite set of Boolean checks D_f. The family score is the fraction of those checks that pass:

s_f = (1 / |D_f|) * sum_{d in D_f} I[d passes]

The lane-level claim score is the weighted mean of family scores:

S = (sum_{f in F} w_f * s_f) / (sum_{f in F} w_f)

A run is marked overall_status = passed only if both conditions hold:

1. S >= tau_overall
2. every family clears its own frozen threshold s_f >= tau_f

The primary endpoint compares canonical and compatible-control claim scores through margins m_c = S_canonical - S_control. A lane passes claim-score control separation only if every compatible control clears its configured degradation threshold delta_c. In the current active six-lane bundle, all configured active-lane controls use delta_c = 0.05.

The lane-specific weights and overall thresholds are frozen before evaluation:

• pbmc3k: uniform five-family weights 0.20 each, overall threshold 0.90
• kang_ifnb: lineage 0.30, composition 0.20, structure 0.25, stability 0.25, overall threshold 0.80
• pbmcsca, multiome_pbmc, and pancreas_integration: lineage 0.35, structure 0.35, stability 0.30, overall threshold 0.80
• citeseq_pbmc: lineage 0.25, structure 0.25, stability 0.20, protein 0.30, overall threshold 0.80

The paper bundle now also emits these parameters as a generated artifact table at outputs/benchmark_live_20260411_ceiling/paper_tables/claim_score_definition_table.md.

Negative Controls

The current control panel includes lane-compatible perturbations:

• overcluster
• undercluster
• hvg_truncation
• marker_shuffle
• protein_shuffle for the CITE-seq lane
• chromatin_shuffle for the multiome lane

Compatibility is explicit. The primary claim-score endpoint is evaluated only against claim-compatible controls; modality-label shuffles are reserved for the matching orthogonal metric. In the April 11 bundle this means protein_shuffle and chromatin_shuffle do not count against the primary claim-score pass/fail endpoint, while marker_shuffle and generic pipeline overrides do.

The CITE-seq lane intentionally uses a smaller claim-compatible panel than the generic PBMC lanes. Its frozen primary endpoint keeps overcluster and marker_shuffle as claim-compatible controls because the lane's main external test is already the protein-backed agreement gate; protein_shuffle probes that orthogonal gate directly, while transcript-only undercluster and hvg_truncation are not part of the frozen CITE primary claim panel.

Gating Versus Diagnostic Metrics

The benchmark does not treat every external metric as a universal gate.

• pbmc3k: no external gate
• kang_ifnb: ARI, NMI, and majority purity are diagnostic only
• pbmcsca: ARI, NMI, and majority purity are diagnostic only
• citeseq_pbmc: protein_agreement_overall is the gating external metric; ARI, NMI, and majority purity are diagnostic
• multiome_pbmc: chromatin_agreement_overall remains diagnostic in the current bundle because compatible-control separation is too weak for gate promotion
• pancreas_integration: ARI, NMI, and majority purity are diagnostic only

This policy is deliberate. The live artifact set shows that coarse reference-label concordance does not behave as a reliable gate for every lane-control combination, especially in donor-condition and cross-technology settings.

Statistical Calibration

The checked-in April 11, 2026 live bundle now uses one paper-depth calibration policy throughout:

• 2000 bootstrap replicates for canonical reference-backed metrics
• 10000 empirical-null label permutations for canonical reference-backed metrics
• 2000 compatible-control margin bootstraps and 10000 sign-flip null draws for the primary claim score
• 95% percentile confidence intervals

For reference-backed metrics, the null is defined by permuting reference labels against preserved predicted labels and cluster assignments. Control separation for those metrics is then summarized from the observed control rows rather than from separate control-run bootstraps. For the primary endpoint, claim-score uncertainty is summarized from compatible-control margins rather than from cell-level resampling.

Per-lane sign-flip p-values are retained in the artifact bundle for completeness, but they are discretized by construction: with k compatible controls, the minimum attainable p-value is 1 / 2^k. In the current bundle that means lanes with four compatible controls cannot go below 1/16, and the CITE-seq lane with two compatible controls cannot go below 1/4. We therefore interpret sign-flip support primarily at the pooled benchmark level, where the combined control count yields a useful null resolution.

Frozen Evaluation Policy

The central policy surfaces were frozen before the April 11, 2026 evaluation pass. In particular, lane inclusion, claim-compatible controls, modality-only controls, external gate assignments, comparator families, and canonical selection bounds were fixed in config before the final bundle was regenerated.

Comparator coverage was frozen alongside this table: PBMC-family lanes carry CellTypist, SingleR, and Azimuth; multiome adds Azimuth ATAC; pancreas ranking claims currently rely on SingleR, while same-bundle pancreas CellTypist runs are retained but reported as inconclusive_ontology_mismatch below the mapped-fraction floor. The artifact bundle emits the full generated frozen-policy surface at outputs/benchmark_live_20260411_ceiling/paper_tables/frozen_policy_table.md.

Datasets

PBMC3k

PBMC3k remains the easiest positive-control lane. It is useful for proving that the benchmark can pass a well-behaved canonical PBMC analysis, but it is not the paper’s headline contribution.

Kang IFN-beta PBMC

The Kang lane tests whether a workflow preserves donor and interferon-response structure rather than merely coarse lineage labels. This is the lane where condition-signal diagnostics matter most.

PBMCSCA

The PBMCSCA lane tests robustness across technologies. In the current artifact bundle it uses a published canonical H5AD derived from the SCP424 matrix-plus-metadata source bundle and pinned by SHA256. The live canonical policy is intentionally lower-resolution than the original draft because the benchmark must reward stable cross-technology claims, not overfit fine label granularity that the lane does not support cleanly.

CITE-seq PBMC

The CITE-seq lane adds orthogonal protein evidence. Here exact protein-backed agreement is a legitimate gate, while transcript-only concordance metrics remain diagnostic.

Multiome PBMC

The multiome lane adds paired chromatin-derived reference labels built from gene-activity annotation on the matched ATAC modality. In the current bundle it strengthens the claim-score benchmark as a fifth primary lane, but chromatin agreement remains diagnostic because its compatible-control separation is positive but too small for gate promotion.

Pancreas Integration

The pancreas lane is an active supplementary stress lane built from the harmonized scIB panel. It broadens the paper beyond PBMC-only evidence at the claim-score level, while its label-backed metrics remain diagnostic in v1. Its canonical resolution sweep is intentionally much lower than the PBMC lanes because the cross-study endocrine and exocrine major-label structure separates at substantially coarser Leiden resolutions than PBMC immune subsets.

Primary Results

Canonical Pass Status

In the April 11, 2026 live artifact bundle, all six active canonical lanes passed claim evaluation:

• pbmc3k: overall score 1.000
• kang_ifnb: overall score 0.917
• pbmcsca: overall score 1.000
• pancreas_integration: overall score 0.912
• citeseq_pbmc: overall score 0.915
• multiome_pbmc: overall score 1.000

All six also passed no-rerun semantic verification. The pancreas canonical selection now chooses a low-resolution run that satisfies the major-label claim set inside the configured 6-12 cluster bound rather than the earlier 31-cluster path. Across the six active lanes, the mean canonical overall score was 0.957.

Claim-Score Control Separation

The primary benchmark endpoint is claim-score degradation under compatible negative controls. All active workflow lanes cleared that endpoint in the refreshed bundle:

• pbmc3k: 4/4 compatible controls degraded
• kang_ifnb: 4/4 compatible controls degraded
• pbmcsca: 4/4 compatible controls degraded
• pancreas_integration: 4/4 compatible controls degraded
• citeseq_pbmc: 2/2 compatible claim controls degraded, with protein_shuffle excluded from the primary endpoint
• multiome_pbmc: 4/4 compatible claim controls degraded, with chromatin_shuffle excluded from the primary endpoint

At the benchmark level, claim-score control separation therefore passed across the full six-lane active set. Across the five calibration-core lanes, the mean compatible-control margin was 0.338 with 95% bootstrap CI 0.298-0.380 and sign-flip empirical p-value approximately 1e-4.

The headline statistics use two distinct denominators on purpose. The active-set pass count is 6/6 lanes because it includes the supplementary pancreas lane. The benchmark-level calibrated claim-margin summary uses only the five calibration-core lanes (pbmc3k, kang_ifnb, pbmcsca, citeseq_pbmc, multiome_pbmc), because pancreas is a supplementary stress lane rather than part of the calibration-core headline set.

External Validity

Canonical external-validity summaries were:

• kang_ifnb: ARI 0.469, NMI 0.562, majority purity 0.717
• pbmcsca: ARI 0.404, NMI 0.518, majority purity 0.699
• pancreas_integration: ARI 0.303, NMI 0.485, majority purity 0.651
• citeseq_pbmc: ARI 0.484, NMI 0.626, majority purity 0.907, exact protein agreement 0.588, compatibility-aware protein agreement 0.713, broad-label protein agreement 0.646
• multiome_pbmc: ARI 0.161, NMI 0.254, majority purity 0.679, chromatin agreement 0.479

Only the CITE-seq protein-backed metric is used as a gate in the current bundle. Its control separation remained strong: canonical exact protein agreement 0.588 versus mean compatible-control agreement 0.163, with mean separation margin 0.425 and all 3/3 compatible controls ranked below canonical.

The multiome lane now passes the primary claim endpoint, but its orthogonal chromatin metric is not promoted. Canonical chromatin agreement is 0.479, yet its compatible-control mean separation margin is only 0.016, only half of compatible controls rank below canonical, and its empirical-null p-value is 1.0. In this bundle, chromatin evidence therefore remains diagnostic rather than gating. Its low transcript-to-reference ARI and NMI are expected in this setting because transcript-derived labels are being compared against chromatin-derived reference labels, so bridge noise enters before any orthogonal gate-promotion decision is made.

Kang, PBMCSCA, and pancreas external label-backed metrics are all statistically above their empirical nulls on canonical runs, but they do not separate cleanly enough under the current control panels to serve as hard gates. They remain diagnostic.

Naive Uniform Gate Ablation

The lane-specific gate policy is not a retrospective convenience rule. Under a naive ablation that promotes every available external metric to a hard gate, 0/5 externally evaluated lanes would pass control separation in the current bundle:

• kang_ifnb would fail because NMI and majority_purity do not separate cleanly even though ARI does
• pbmcsca would fail because ARI, NMI, and majority_purity all remain unstable under compatible controls
• pancreas_integration would fail because all three label-backed metrics degrade under stress controls
• citeseq_pbmc would fail because ARI and NMI fail even though the protein-backed gate passes strongly
• multiome_pbmc would fail because both transcript label metrics and chromatin agreement remain too weak for gate promotion

That ablation is exactly why the paper freezes a lane-specific gate/diagnostic policy instead of pretending every external metric is a universal benchmark gate. The generated ablation table is emitted at outputs/benchmark_live_20260411_ceiling/paper_tables/uniform_gate_ablation_table.md.

Comparator Surfaces

The refreshed canonical comparator bundle now includes four live method families in the same artifact set:

• CellTypist across all six active lanes
• SingleR across all six active lanes
• Azimuth RNA across the five PBMC-family primary lanes
• Azimuth ATAC on multiome_pbmc

The cross-method picture is heterogeneous rather than rhetorical.

• CellTypist mapped confidently on the five PBMC-family lanes, with mapped fractions between 0.897 and 0.998, but remained inconclusive on pancreas at 0.524, below the locked 0.70 floor. Its claim-score control separation still passed on all five primary lanes, but every primary canonical comparator run already failed claim evaluation at baseline, with canonical scores ranging from 0.400 to 0.5625, and CITE exact protein agreement collapsed to 0.0316. This is the clearest negative result in the current bundle: a transcript-only reference mapper can be consistently degraded by controls while still failing the biologically grounded endpoint before perturbation.
• SingleR mapped conclusively on all six active lanes, including pancreas at mapped fraction 0.989. It passed comparator control separation on pbmc3k, pbmcsca, citeseq_pbmc, and multiome_pbmc, but failed on kang_ifnb with mean margin 0.00625 and on pancreas with mean margin -0.0802. Pancreas is therefore no longer excluded from ranking claims because of ontology mismatch; it is now a real comparator lane whose control behavior is simply weak.
• Azimuth RNA mapped conclusively on all five PBMC-family primary lanes, all at mapped fraction 1.000. It passed comparator control separation on pbmc3k, kang_ifnb, citeseq_pbmc, and multiome_pbmc, while pbmcsca remained borderline with mean margin 0.0481 and only 3/4 degrading controls. On the orthogonal lanes, Azimuth RNA achieved CITE exact protein agreement 0.8318 and multiome chromatin agreement 0.5816, both higher than the workflow’s own transcript-derived concordance summaries.
• Azimuth ATAC landed as a real multiome-only comparator on the frozen 10x fragments and exact canonical barcode set. Its canonical mapped fraction was 1.000, its comparator claim score was 0.9417, and its chromatin agreement was 0.5761. Claim-score control separation passed across the multiome control panel. However, the bridge/chromatin agreement itself only moved under chromatin_shuffle in this slice, so we keep it as comparator evidence rather than claiming a new hard orthogonal gate.

Bootstrap Intervals and Empirical Nulls

Canonical reference-backed metrics remained well above the empirical null for Kang, PBMCSCA, pancreas, and the CITE-seq gate, with empirical p-values approximately 1e-4 for those canonical runs. The refreshed 95% bootstrap intervals were:

• kang_ifnb ARI 0.463-0.475, NMI 0.557-0.567
• pbmcsca ARI 0.394-0.414, NMI 0.511-0.525
• pancreas_integration ARI 0.296-0.310, NMI 0.480-0.491
• citeseq_pbmc ARI 0.461-0.509, NMI 0.604-0.655, protein agreement 0.566-0.610
• multiome_pbmc ARI 0.130-0.197, NMI 0.220-0.300

The primary claim-score endpoint was calibrated separately from compatible-control margins. Its benchmark-level 95% interval was 0.298-0.380 around an observed mean margin of 0.338. Multiome chromatin agreement did not exceed its empirical null under the current diagnostic path.

Case Studies and Weak Spots

Kang: Why ARI and NMI Are Diagnostic Only

The Kang lane is fundamentally about donor and interferon-response biology, not about treating coarse cell-type labels as immutable cluster ground truth. The live condition-signal analysis makes this clear. Under the overcluster control, interferon-signal retention falls to 0.627 of canonical in CD4 T cells and 0.522 in CD8 T cells, while the undercluster control stays near canonical (0.998 and 1.012, respectively). This is a more biologically informative failure mode than insisting that ARI or NMI must always act as the primary gate.

CITE-seq: FCGR3A Mono Versus NK

The main weak spot in the CITE-seq lane is not hidden. FCGR3A Mono has exact agreement 0.000, and its dominant predicted label is NK in 89.4% of protein-backed reference cells. This is why the paper must report both exact and compatibility-aware summaries. The lane still clears its main gate because overall protein-backed agreement separates canonical from controls, but the FCGR3A Mono versus NK axis remains a diagnostic ambiguity that should be discussed explicitly rather than buried.

PBMCSCA: Why Reference Metrics Are Diagnostic Only

PBMCSCA still succeeds on the benchmark’s primary endpoint: the canonical run passed claim evaluation, and all 4/4 negative controls degraded the claim score. However, ARI and NMI do not separate cleanly under the current cross-technology control panel; some compatible controls improve coarse label concordance. That makes those metrics useful diagnostics but poor hard gates for this lane in the current version of the benchmark.

Reproducible Artifact Set

The paper-ready live bundle is pinned under outputs/benchmark_live_20260411_ceiling/ and includes:

• benchmark_summary.json
• control_separation.json
• external_validity_summary.json
• bootstrap_intervals.csv
• empirical_null_summary.json
• comparators/comparator_summary.json
• comparators/comparator_scores.csv
• paper_tables/*.md
• paper_tables/claim_score_definition_table.md
• paper_tables/frozen_policy_table.md
• paper_tables/uniform_gate_ablation_table.md
• paper_figures/*.png
• paper_figures/figure_data/*.csv
• paper_figures/release_manifest.json

The release manifest records source URLs, SHA256 values, benchmark commands, and artifact hashes so that the figures and tables are tied to one exact live bundle.

Discussion

The current six-lane bundle supports a clearer methodological conclusion than the original draft. Claim-score degradation is the most reliable benchmark endpoint across heterogeneous lanes, while external evidence must stay lane-specific. CITE-seq shows the positive case: a genuinely orthogonal modality can support a hard gate. Kang, PBMCSCA, pancreas, and multiome show the negative case: coarse transcript-reference concordance or noisy bridge labels can be informative without being trustworthy as universal gates.

The comparator layer is now scientifically useful rather than decorative. No single external method dominates the benchmark. CellTypist is the strongest example of why this benchmark exists: it shows stable control-separation behavior on the five primary lanes, yet its canonical runs already fail claim evaluation and it collapses on the CITE protein-backed metric. SingleR is stronger on pancreas and CITE but still fails Kang and pancreas control separation. Azimuth RNA is strongest on the PBMC-family lanes and improves CITE orthogonal agreement substantially, yet it remains borderline on PBMCSCA. The benchmark is therefore distinguishing method behavior, not just workflow self-consistency.

The calibration outputs add another layer of discipline. The benchmark-level claim-margin CI excludes zero, the sign-flip null supports non-random separation, and the emitted sensitivity curve shows the expected monotone increase in scaled degradation as perturbation strength rises. That curve is especially informative for near-boundary lanes such as Kang, where weaker observed margins only cross the 0.05 degradation threshold under moderate or stronger synthetic corruption, matching the lane’s smaller live compatible-control margins.

Multiome is also more informative as a diagnostic non-promotion than it would be as a forced second gate. The lane passes the primary endpoint and supports same-bundle chromatin-backed comparisons, but the current gene-activity and bridge-style orthogonal paths do not separate cleanly enough under compatible controls to justify promotion. Reporting that empirical non-promotion is more credible than silently upgrading chromatin evidence to match the narrative.

Limitations

This version is materially stronger than the original PBMC3k-only note, but it is not yet the absolute ceiling.

• multiome_pbmc and pancreas_integration are now active, but multiome chromatin evidence remains diagnostic rather than promoted to a hard gate
• same-bundle comparator evidence now spans CellTypist, SingleR, Azimuth RNA, and Azimuth ATAC, but pancreas still has only one conclusive comparator (SingleR) while CellTypist remains inconclusive_ontology_mismatch
• no external comparator dominates the benchmark: CellTypist fails canonical claim evaluation on all five primary lanes, SingleR fails control separation on Kang and pancreas, and Azimuth RNA fails control separation on PBMCSCA
• tabula_sapiens_subset remains planned
• the CITE FCGR3A Mono ambiguity remains diagnostic rather than fully resolved
• PBMCSCA external label-backed metrics are not yet strong enough to serve as hard gates
• the active set is still PBMC-heavy despite the addition of pancreas

These are real limits and should be stated plainly.

Conclusion

The strongest current version of this repository is no longer “we ran Scanpy on PBMC3k.” It is that a six-lane, same-bundle benchmark with four live comparator families can distinguish robust from fragile single-cell conclusions by combining calibrated claim-score control separation, empirically justified lane-specific external gates, and explicit cross-method failure analysis. The most important negative result is not a crashed workflow but a stable-but-biologically-failing comparator: CellTypist remains degradable under controls while its canonical runs already miss the claim endpoint and the CITE protein-backed signal. Together with the 0/5 naive uniform-gate ablation, that makes the benchmark’s main contribution methodological rather than rhetorical.