Program-Conditioned Diagnostic for Transcriptomic Signature Durability: Validation on Interferon Signatures across 35 Frozen GEO Cohorts
Program-Conditioned Diagnostic for Transcriptomic Signature Durability: Validation on Interferon Signatures across 35 Frozen GEO Cohorts
Abstract
Gene-expression signatures are often labeled irreproducible when they fail across heterogeneous validation cohorts, but that failure can reflect either instability of the signature or mismatch between the signature and the test context. We present a program-conditioned diagnostic that scores a signature against a frozen reference panel, compares within-program versus outside-program effects, tests program structure by permutation, and surfaces failure modes when labels are too coarse. In 35 frozen GEO cohorts (5,922 samples, 5 biological programs, 14 microarray platforms), the frozen IFN-γ and IFN-α cores, an orthogonal 76-gene Schoggins panel, and a strictly-disjoint 41-gene Schoggins subset all show large within-IFN effects and small/non-significant outside-IFN effects; passing the IFN-γ core through triage yields a full-model class of mixed but a best-supported program of interferon and a within-program class of durable. Held-out validation is strong, and four external bulk RNA-seq cohorts (719 samples) reproduce the IFN quartet with guarded Bonferroni-significant pooled effects while inflammatory, TNF-α/NFκB, and E2F comparators remain non-significant. The same rescue logic works on an external oncology signature: the Ayers 6-gene IFN-γ-related profile looks mixed in aggregate but resolves to a durable interferon-context signal. The prospective layer is likewise non-tautological: two predeclared v1 cohorts satisfy the 4/4 IFN sign forecast, whereas one severity-mixed acute PBMC cohort inverts all four IFN signatures. Comparator signatures show the complementary method result: inflammatory/TNF-α/NFκB ambiguity is driven more by proliferation than inflammation, and E2F targets expose a coarse label bucket rather than a random signature.
Data and Diagnostic in Brief
- 35 frozen GEO cohorts partitioned into interferon (k = 11), inflammation (k = 7), proliferation (k = 5), hypoxia (k = 6), and EMT (k = 6)
- Primary IFN signatures: frozen 30-gene IFN-γ and IFN-α benchmark cores anchored to the broader MSigDB Hallmark families, manually curated as compact benchmark-release subsets and frozen at initial release before expanded reruns, plus a 76-gene Schoggins 2011 IRG panel, a strictly-disjoint 41-gene Schoggins subset, and a blind IFN composite
- Expansion cohorts admitted using concordant direction across 10 IFN markers: STAT1, IRF1, IFIT1, IFIT2, ISG15, MX1, OAS1, GBP1, CXCL10, RSAD2
- Per-cohort effect size: Hedges' g on weighted signed mean z-scored expression; within/outside pooling: guarded HKSJ random effects; structure test: 10,000-label permutation; additional analyses: held-out validation and failure-mode analysis
- Held-out external RNA-seq extension: primary bulk panel of GSE152641, GSE171110, GSE152075, and GSE167000 (719 samples total), with GSE152418 PBMC retained separately as an exploratory stress test
- Deterministic second breadth case: Hallmark Hypoxia versus Hallmark EMT, selected from frozen artifacts only, with supportive external exact-perturbation validation in GSE179885
- Metadata-first prospective holdout registry: GSE184610, GSE243217, and GSE202805 declared in
prediction_registry_v1.tsvbefore held-out scoring - Reusable interface:
triageaccepts a new signature TSV/CSV and writesdiagnostic.json,per_cohort_effects.csv, anddiagnostic_summary.md - Packaged triage rules:
full_modelbecomesbrittleif aggregate p > 0.10 or direction consistency < 0.50,mixedif I² > 0.75 or leave-one-out stability < 0.60, anddurableotherwise;within_programupgrades todurablewhen inferred home-program p < 0.05 and |g| > 0.2; CLI triage uses 1,000 permutations while canonical IFN tables use 10,000 - The broader 30-signature benchmark still contains explicit synthetic control signatures, but the paper-facing target panel, external validations, and prospective case studies use only non-synthetic signatures
Diagnostic Workflow
The repo now includes a generated workflow figure at paper/figure_workflow.png, summarizing the packaged logic: input signature, frozen-panel scoring, within/outside separation, permutation, durable-versus-failure-mode branch, held-out external transfer, and scored prospective v1 challenge.
Key Results
| Signature | Within-IFN g | HKSJ-guarded p_Bonf | Outside-IFN g | Outside p | Permutation p |
|---|---|---|---|---|---|
| IFN-γ core | +1.383 | 0.0049 | +0.138 | 0.484 | 0.0008 |
| IFN-α core | +1.458 | 0.0003 | +0.219 | 0.165 | 0.0004 |
| Schoggins 2011 IRG | +1.393 | 0.0006 | +0.241 | 0.178 | 0.0008 |
| Strictly-disjoint Schoggins (41 genes) | +1.247 | 0.00088 | +0.241 | 0.213 | — |
| Blind IFN composite | +1.402 | 0.0010 | +0.196 | 0.250 | 0.0007 |
All five IFN signatures survive guarded HKSJ plus 9-test Bonferroni correction within IFN cohorts and remain small/non-significant outside them. The strictly-disjoint Schoggins subset is the key curation-circularity result: it shares zero genes with the 10 cohort-admission markers and still reproduces the within-IFN effect, making exact admission-marker reuse an unlikely explanation. Zero overlap does not imply statistical independence, however, because the disjoint genes still live in the same co-regulated interferon program.
The diagnostic itself behaves as intended on a known durable signal. Treating the frozen IFN-γ core as an arbitrary input produces a full-model class of mixed, but triage infers interferon as the best-supported program and recovers the canonical within/outside result. The point is practical: a diluted pooled result need not imply a broken signature.
Held-out validation strengthens the method claim:
| Signature | LOO Sign Prediction (within IFN, k=11) | Split-Half Sign Agreement (within IFN) | Split-Half Both Significant |
|---|---|---|---|
| IFN-γ core | 1.000 | 1.000 | 0.871 |
| IFN-α core | 1.000 | 1.000 | 1.000 |
| Schoggins 2011 IRG | 1.000 | 1.000 | 1.000 |
| Blind IFN composite | 1.000 | 1.000 | 1.000 |
Comparator signatures show the other use-case. Inflammatory response has Q_B/Q_tot = 0.342, but its largest LOPO drop occurs when hiding proliferation (Δ = 0.195), not inflammation. TNF-α/NFκB shows the same pattern (Q_B/Q_tot = 0.389, largest drop when hiding proliferation, Δ = 0.181). E2F targets show weak between-program structure (Q_B/Q_tot = 0.061), only 80% leave-one-out sign prediction, 75% split-half sign agreement, and a within-proliferation effect span from −1.21 to +2.14. These are better interpreted as boundary failures in the labeling scheme than as simple signature collapse. Operationally, when LOPO leverage is driven by a foreign program, the next step is to split or relabel that program and rerun; when within-program spread stays extreme despite weak between-program structure, the next step is to stratify by contrast type or tissue before concluding that the signature itself is unstable.
The external RNA-seq extension now provides a real cross-platform test rather than just a future-work promise:
| Signature | External RNA-seq pooled g | Guarded p_Bonf,7 | I² |
|---|---|---|---|
| IFN-γ core | +0.922 | 0.022 | 0.000 |
| IFN-α core | +1.193 | 0.011 | 0.000 |
| Schoggins 2011 IRG | +0.996 | 0.018 | 0.000 |
| Blind IFN composite | +1.177 | 0.020 | 0.244 |
| Inflammatory Response | +0.427 | 1.000 | 0.770 |
| TNF-α/NFκB | +0.382 | 1.000 | 0.859 |
| E2F Targets | +0.818 | 1.000 | 0.944 |
All four IFN signatures are positive in all four primary external cohorts. The additional PBMC cohort (GSE152418) is reported separately as an exploratory stress test because its cell-selected composition produces a much more mixed pattern, which is informative but not a clean bulk RNA-seq transfer test.
The mixed breadth case is intentionally less clean than IFN. Hallmark Hypoxia beats Hallmark EMT under the frozen deterministic selection rule (p_Bonf = 0.024 versus 1.0, with perfect LOO and split-half sign agreement for both), and triage still recovers hypoxia as the best-supported home program. But hypoxia remains mixed in the full-model diagnostic because outside-program bleed-through is not negligible. That is useful method behavior, not a bug: the framework broadens beyond IFN without pretending every program has IFN-like specificity.
The external exact-perturbation cohort pushes in the same direction. In GSE179885 human T-cell RNA-seq cultured under hypoxia versus normoxia, Hallmark Hypoxia ranks first among seven scored signatures with Hedges' g = +5.11 and full coverage, ahead of Hallmark EMT (g = +0.94), while IFN, inflammatory, TNF-α/NFκB, and E2F comparators are all negative. Because this cohort has 12 total samples, we treat it as supportive breadth evidence rather than a headline pooled result.
The diagnostic is also useful on a published signature imported from outside this benchmark. Running the Ayers IFN-γ-related 6-gene clinical response profile (IDO1, CXCL10, CXCL9, HLA-DRA, STAT1, IFNG; Ayers et al. 2017, doi:10.1172/JCI91190) through triage yields exactly the rescue pattern the method is meant to expose: the aggregate profile is mixed (effect = +0.246, I² = 0.929), but the best-supported program is interferon and the within-program class is durable. The within-interferon pooled effect is +0.866 (p = 0.0245), while the outside-interferon effect is essentially null (−0.010, p = 0.949). Two of the six Ayers genes (STAT1 and CXCL10) overlap the 10 admission markers, and five overlap the frozen IFN-γ core, so this is not an anti-circularity analysis; it is a portability demonstration showing that a published external signature can look diluted in a broad benchmark and still resolve into the correct biological home program.
The first metadata-first prospective round is intentionally harder and mixed. GSE184610 and GSE243217 are 4/4 positive across the IFN quartet, but GSE202805 is 0/4 positive, so the pooled prospective rule fails overall (pooled g = +0.359 to +0.657, Bonferroni-4 p = 0.93 to 1.0, sign consistency = 0.667). This is a genuine limitation of metadata-first cohort selection: GEO annotations do not guarantee IFN engagement, and a severity-mixed acute PBMC series can invert the expected IFN direction. The miss is interpretable — GSE202805 combines mild through ICU-severity samples, and the IFN quartet is negative while inflammatory and E2F comparators are positive — but the predeclared success rule was too coarse for this cohort, and the round fails.
Limits
- The frozen decision surface is still defined by microarray cohorts, even though the paper now includes a held-out four-cohort bulk RNA-seq extension
- The 5-program partition is operational rather than ontologically complete
- The descriptive IFN heterogeneity decomposition is based on only 11 IFN cohorts
- The strictly-disjoint Schoggins holdout rules out exact admission-marker reuse more cleanly than every possible cohort-selection effect
- Held-out validation is no longer only internal to the frozen panel; the repo now includes a first metadata-first prospective holdout round, and that round is mixed rather than uniformly confirmatory
- The metadata-first prospective round failed its prespecified success rule (1 of 3 cohorts inverted all four IFN signatures); this is a real limitation of the prospective layer
Reproducibility
Canonical outputs for this version are in outputs/canonical_v8/.
uv sync --frozen
uv run python scripts/compute_expanded_effects.py
uv run python scripts/rerun_all_expanded.py
uv run python scripts/strictly_unique_schoggins.py
uv run python scripts/within_ifn_metaregression.py
uv run python scripts/external_validation.py
uv run python scripts/external_rnaseq_validation.py
uv run python scripts/external_hypoxia_validation.py
uv run python scripts/failure_mode_analysis.py
uv run python scripts/generalization_case_study.py
uv run python scripts/prospective_holdout_prediction.py
uv run python scripts/generate_diagnostic_workflow_figure.pyTriage a new signature with:
uv run python -m signature_durability_benchmark.cli triage \
--config config/benchmark_config.yaml \
--input my_signature.tsv \
--out outputs/my_signatureReferences
- Schoggins JW. Interferon-stimulated genes: what do they all do? Annu Rev Virol. 2019;6:567-584. doi:10.1146/annurev-virology-092818-015756
- Schoggins JW, Wilson SJ, Panis M, et al. A diverse range of gene products are effectors of the type I interferon antiviral response. Nature. 2011;472:481-485. doi:10.1038/nature09907
- Liberzon A, et al. The Molecular Signatures Database Hallmark gene set collection. Cell Syst. 2015;1:417-425. doi:10.1016/j.cels.2015.12.004
- DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7:177-188. doi:10.1016/0197-2456(86)90046-2
- Hartung J, Knapp G. On tests of the overall treatment effect in meta-analysis with normally distributed responses. Stat Med. 2001;20:1771-1782. doi:10.1002/sim.791
- Sidik K, Jonkman JN. A simple confidence interval for meta-analysis. Stat Med. 2002;21:3153-3159. doi:10.1002/sim.1262
- Baechler EC, Batliwalla FM, Karypis G, et al. Interferon-inducible gene expression signature in peripheral blood cells of patients with severe lupus. Proc Natl Acad Sci U S A. 2003;100:2610-2615. doi:10.1073/pnas.0337679100
- Yao Y, Richman L, Morehouse C, et al. Type I interferon: potential therapeutic target for psoriasis? PLoS One. 2008;3:e2737. doi:10.1371/journal.pone.0002737
- Ayers M, Lunceford J, Nebozhyn M, et al. IFN-gamma-related mRNA profile predicts clinical response to PD-1 blockade. J Clin Invest. 2017;127:2930-2940. doi:10.1172/JCI91190
Reproducibility: Skill File
Use this skill file to reproduce the research with an AI agent.
--- name: signature-durability-benchmark description: Score and triage human gene signatures against 35 frozen GEO cohorts with program-conditioned meta-analysis, held-out validation, external RNA-seq transfer tests, metadata-first prospective holdout auditing, an externally timestamped pending second prospective round, and orthogonal interferon panel checks. allowed-tools: Bash(uv *, python *, python3 *, ls *, test *, shasum *, tectonic *) requires_python: "3.12.x" package_manager: uv repo_root: . canonical_output_dir: outputs/canonical_v8 --- # Signature Durability Benchmark (Expanded Panel, v8) This skill scores and triages gene signatures against 35 frozen real GEO expression cohorts (5,922 samples, 14 microarray platforms) covering five biological programs: interferon (k=11), inflammation (k=7), hypoxia (k=6), EMT (k=6), and proliferation (k=5). The expanded interferon arm (11 cohorts) spans viral infection (k=7), autoimmunity (SLE, psoriasis; k=4), and three tissues (whole blood, PBMC, skin). The benchmark answers: is a transcriptomic signature reproducible **within** its native biological program, and does it correctly fail **outside** it? We use Cochran's Q decomposition (Q_within / Q_between), DerSimonian-Laird random-effects meta-analysis with Hartung-Knapp-Sidik-Jonkman guarded inference, 10,000-iteration permutation testing, leave-one-program-out Q_B leverage diagnostics, held-out validation, a held-out external RNA-seq transfer layer, a deterministic second non-IFN breadth case, a metadata-first prospective holdout registry, an externally timestamped pending second prospective round, a generic locked-round evaluator for future readouts, a reusable `triage` interface for new signatures, and a provenance audit that verifies the active scored panel uses real GEO cohorts while quarantining the broader benchmark's explicit synthetic control signatures from the paper-facing target set. The interferon panel includes an **orthogonal Schoggins 2011 IRG panel** (76 genes from viral overexpression screens, 25% overlap with the frozen Hallmark IFN-γ core) and a **41-gene strictly-unique** Schoggins subset with zero overlap against the 10 marker genes used for expansion cohort admission — a direct test against exact admission-marker reuse. ## Runtime Expectations - Platform: CPU-only - Python: 3.12.x - Package manager: uv - Offline after initial clone for the frozen benchmark; the v1 prospective holdout step fetches predeclared external GEO files into `data/prospective_holdout/downloads/`, and verifying the shipped v2 timestamp receipt is offline once those receipt files are present - Typical wall-clock for full rerun: ~5 minutes on a 2023 laptop ## Step 1: Install the Locked Environment ```bash uv sync --frozen ``` ## Step 2: Compute Per-Cohort Effects (Expanded Panel) Scores all 35 cohorts × 30 signatures (1,050 pairs) using per-gene z-score normalization across samples within cohort and Hedges' g with the small-sample correction. Writes `outputs/canonical_v8/per_cohort_effects.csv`. ```bash uv run python scripts/compute_expanded_effects.py ``` Success condition: `outputs/canonical_v8/per_cohort_effects.csv` contains 1,050 rows (35 cohorts × 30 signatures). ## Step 3: Run All Meta-Analyses on the Expanded Panel Runs I² decomposition, within-program DL+HKSJ meta-analysis, 10,000-iteration permutation test, and leave-one-program-out Q_B stability for the 9 paper target signatures enumerated in `config/paper_target_signatures.tsv` (Hallmark IFN-γ / IFN-α cores, Hallmark inflammatory / TNF-α-NFκB, Hallmark hypoxia, Hallmark E2F targets, Hallmark EMT, Schoggins 2011 IRG, blind IFN composite). ```bash uv run python scripts/rerun_all_expanded.py ``` Success condition: all four of these JSONs exist in `outputs/canonical_v8/`: - `i2_decomposition_expanded.json` - `hartung_knapp_expanded.json` - `permutation_validation_expanded.json` - `lopo_cross_validation_expanded.json` ## Step 4: Orthogonal Schoggins Anti-Circularity Test Scores the 41-gene strictly-unique Schoggins subset (zero overlap with the frozen Hallmark IFN-γ and IFN-α cores, and the 10 marker genes used for cohort admission: STAT1, IRF1, IFIT1, IFIT2, ISG15, MX1, OAS1, GBP1, CXCL10, RSAD2). If this panel still produces a large within-IFN effect, exact admission-marker reuse is unlikely to be the driver. ```bash uv run python scripts/strictly_unique_schoggins.py ``` Success condition: `outputs/canonical_v8/strictly_unique_schoggins.json` shows: - `expansion_marker_overlap`: 0 - `within_ifn_meta.p_bonf`: < 0.0056 ## Step 5: Within-IFN Heterogeneity Decomposition Descriptively decomposes residual within-IFN I² for the IFN-γ core into contributions from tissue (blood/PBMC vs skin), etiology (viral vs autoimmune), and platform (Affymetrix vs Illumina). The combined tissue × etiology × platform partition should explain ≥ 80% of within-IFN Q_total. ```bash uv run python scripts/within_ifn_metaregression.py ``` Success condition: `outputs/canonical_v8/within_ifn_metaregression.json` shows `combined_r2.R2 > 0.8`. ## Step 6: Held-Out Validation Evaluates leave-one-cohort-out prediction and split-half replication within each signature's home program. ```bash uv run python scripts/external_validation.py ``` Success condition: `outputs/canonical_v8/external_validation.json` exists and reports: - IFN quartet leave-one-out accuracy: 1.0 - IFN quartet split-half sign agreement: 1.0 ## Step 7: Failure-Mode Analysis Summarizes ambiguous comparator signatures where the diagnostic is better interpreted as exposing coarse program labels than a broken signature. ```bash uv run python scripts/failure_mode_analysis.py ``` Success condition: `outputs/canonical_v8/failure_mode_analysis.json` exists and shows: - inflammatory largest LOPO drop when hiding `proliferation` - TNF-α/NFκB largest LOPO drop when hiding `proliferation` - E2F `within_proliferation_effect_span.span > 3.0` ## Step 8: External RNA-seq Transfer Validation Runs the held-out cross-platform extension on four primary bulk RNA-seq cohorts (GSE152641, GSE171110, GSE152075, GSE167000; 719 samples total) and reports one additional PBMC cohort (GSE152418) as an exploratory stress test. ```bash uv run python scripts/external_rnaseq_validation.py ``` Success condition: `outputs/canonical_v8/external_rnaseq_validation.json` exists and shows: - `primary_bulk_rnaseq_panel.n_cohorts`: 4 - `primary_bulk_rnaseq_panel.ifn_focus_summary.all_four_positive_in_all_primary_cohorts`: `true` - `primary_bulk_rnaseq_panel.pooled_signatures.hallmark_ifng_response.guarded_p_bonf_7`: < 0.05 - `primary_bulk_rnaseq_panel.pooled_signatures.hallmark_ifna_response.guarded_p_bonf_7`: < 0.05 ## Step 9: Prospective Holdout Audit Runs the metadata-first held-out prediction round defined in `data/prospective_holdout/prediction_registry_v1.tsv`. This step audits a real predeclared registry rather than requiring the forecast itself to succeed. ```bash uv run python scripts/prospective_holdout_prediction.py ``` Success condition: `outputs/canonical_v8/prospective_holdout_validation.json` exists and shows: - `registry.sha256` matches the current SHA256 of `data/prospective_holdout/prediction_registry_v1.tsv` - `download_audit` contains the newly fetched source files - `per_cohort_effects_csv` points to `outputs/canonical_v8/prospective_holdout_per_cohort_effects.csv` ## Optional Step 10: Verify the Externally Timestamped v2 Declaration Verifies or, if absent, creates the RFC3161 timestamp receipt for the fresh v2 prospective registry. This is intentionally separate from the scored v1 round. ```bash uv run python -m signature_durability_benchmark.cli declare-prospective-round \ --registry data/prospective_holdout/prediction_registry_v2.tsv \ --protocol data/prospective_holdout/PREDICTION_PROTOCOL_v2.md ``` Success condition: `data/prospective_holdout/external_timestamps/prospective_holdout_v2/declaration_receipt.json` exists and shows: - `round_id`: `prospective_holdout_v2` - `verification_status`: `OK` - `status`: either `created_new_receipt` or `verified_existing_receipt` ## Optional Step 11: Build the External Hypoxia Breadth Layer Scores the exact-perturbation external hypoxia cohort selected under `data/external_hypoxia/SEARCH_PROTOCOL.md`. ```bash uv run python scripts/external_hypoxia_validation.py ``` Success condition: `outputs/canonical_v8/external_hypoxia_validation.json` exists and shows: - `chosen_signature_support.signature_id`: `hallmark_hypoxia` - `chosen_signature_support.rank_among_scored_signatures`: `1` - `chosen_signature_support.positive_direction`: `true` ## Optional Step 12: Build the Deterministic Second-Case Study Ranks Hallmark Hypoxia versus Hallmark EMT from frozen artifacts only, reruns `triage` for the selected case, and records the supportive external layer. ```bash uv run python scripts/generalization_case_study.py ``` Success condition: `outputs/canonical_v8/generalization_case_study.json` exists and shows: - `selected_case.signature_id`: `hallmark_hypoxia` - `selection_rule_satisfied`: `true` - `selected_case.triage_best_program_matches_home`: `true` ## Optional Step 13: Generate the Workflow Figure Builds the high-level method figure for the paper and repo. ```bash uv run python scripts/generate_diagnostic_workflow_figure.py ``` Success condition: - `paper/figure_workflow.pdf` exists - `paper/figure_workflow.png` exists ## Optional Step 14: Score a Locked Future Prospective Round This command is the future scoring path for v2 or later rounds. It must only be run against an already declared registry/protocol/receipt bundle. ```bash uv run python -m signature_durability_benchmark.cli prospective-round-evaluate \ --registry data/prospective_holdout/prediction_registry_v2.tsv \ --protocol data/prospective_holdout/PREDICTION_PROTOCOL_v2.md \ --receipt data/prospective_holdout/external_timestamps/prospective_holdout_v2/declaration_receipt.json \ --out outputs/canonical_v8/prospective_rounds/prospective_holdout_v2 ``` Success condition: if executed, the output directory contains: - `evaluation.json` - `evaluation_summary.md` - `per_cohort_effects.csv` ## Optional Step 15: Build the Release-Ready Declaration Archive Bundle Prepares the local declaration bundle for later GitHub Release / Zenodo publication. ```bash uv run python scripts/build_archive_release_bundle.py ``` Success condition: `submission/archive_bundles/prospective_holdout_v2_declaration/` exists and contains: - `prediction_registry_v2.tsv` - `PREDICTION_PROTOCOL_v2.md` - `prospective_holdout_v2/declaration_receipt.json` - `CHECKSUMS.sha256` ## Optional Step 16: Build the Rescued Signature Portability Case Runs `triage` on the fixed 6-gene Ayers IFN-gamma-related profile to show how the diagnostic can rescue a published external signature that would otherwise look heterogeneous in aggregate. ```bash uv run python scripts/rescued_signature_case_study.py ``` Success condition: `outputs/canonical_v8/rescued_signature_case_study.json` exists and shows: - `signature.name`: `Ayers IFN-gamma-related 6-gene profile` - `triage.inferred_program`: `interferon` - `triage.full_model_class`: `mixed` - `triage.within_program_class`: `durable` ## Optional Step 17: Run the Provenance Audit Verifies that the active scored cohort panel is the real 35-cohort GEO freeze, that the paper-facing target signatures are all non-synthetic, that no unexpected `synthetic` / `stub` / `mock` placeholders leak into the runtime or paper surface, and that the broader synthetic controls remain documented benchmark controls rather than headline evidence. ```bash uv run python scripts/provenance_audit.py ``` Success condition: `outputs/canonical_v8/provenance_audit.json` exists and shows: - `active_cohort_panel.active_cohorts`: `35` - `active_cohort_panel.active_sample_sum`: `5922` - `signature_panel.paper_target_all_non_synthetic`: `true` - `keyword_audit.unexpected_hits`: `[]` ## Step 18: Confirm Required Artifacts Required files in `outputs/canonical_v8/`: - `per_cohort_effects.csv` (1,050 rows) - `i2_decomposition_expanded.json` - `hartung_knapp_expanded.json` - `permutation_validation_expanded.json` - `lopo_cross_validation_expanded.json` - `strictly_unique_schoggins.json` - `within_ifn_metaregression.json` - `external_validation.json` - `external_rnaseq_validation.json` - `external_hypoxia_validation.json` - `failure_mode_analysis.json` - `generalization_case_study.json` - `prospective_holdout_validation.json` - `prospective_holdout_per_cohort_effects.csv` - `rescued_signature_case_study.json` - `provenance_audit.json` ## Expected Headline Results | Signature | Within-IFN g | HKSJ-guarded p_Bonf (9 tests) | Permutation p | Outside-IFN g (NS) | |-----------|-------------:|-------------------------------:|--------------:|-------------------:| | Hallmark IFN-γ core | +1.383 | 0.0049 | 0.0008 | +0.138 | | Hallmark IFN-α core | +1.458 | 0.0003 | 0.0004 | +0.219 | | Schoggins 2011 IRG | +1.393 | 0.0006 | 0.0008 | +0.241 | | Strictly-unique Schoggins (41g) | +1.247 | 0.00088 | — | +0.241 | | Blind IFN composite | +1.402 | 0.0010 | 0.0007 | +0.196 | Expected external RNA-seq pooled results in `external_rnaseq_validation.json` (4 primary bulk RNA-seq cohorts, 7 tested signatures): | Signature | External RNA-seq pooled g | Guarded p_Bonf,7 | I² | |-----------|--------------------------:|-----------------:|---:| | Hallmark IFN-γ core | +0.922 | 0.022 | 0.000 | | Hallmark IFN-α core | +1.193 | 0.011 | 0.000 | | Schoggins 2011 IRG | +0.996 | 0.018 | 0.000 | | Blind IFN composite | +1.177 | 0.020 | 0.244 | Expected prospective audit structure in `prospective_holdout_validation.json`: - `prediction_summary.round_id`: `prospective_holdout_v1` - `prediction_summary.registry_sha256`: exact SHA256 of the frozen registry file - `prediction_summary.per_cohort_hits`: 3 rows, one per predeclared cohort - `pooled_primary_panel`: pooled IFN quartet plus comparator summaries for the predeclared panel Expected externally timestamped v2 declaration receipt in `data/prospective_holdout/external_timestamps/prospective_holdout_v2/declaration_receipt.json`: - `round_id`: `prospective_holdout_v2` - `tsa_url`: `https://freetsa.org/tsr` - `verification_status`: `OK` - `cohorts`: 3 fresh predeclared rows (yellow fever PBMC, influenza blood, RSV nasal challenge) ## Scope Rules - Human bulk transcriptomic signatures only - No live data fetching in scored path - Frozen GEO cohorts from real public data - Blind panel is held out from all threshold-tuning decisions - Source leakage between signature sources and cohort sources is forbidden - LOPO Q_B is a leverage/sensitivity diagnostic, not an anti-circularity test (the tautology: hiding the driving program mechanically reduces Q_B). The permutation test and strictly-unique Schoggins are the primary anti-circularity evidence. ## Optional: Triage a New Signature To diagnose an arbitrary input signature against the frozen panel, prepare a TSV/CSV with `gene_symbol` and optional `direction` / `weight` columns, then run: ```bash uv run python -m signature_durability_benchmark.cli triage \ --config config/benchmark_config.yaml \ --input my_signature.tsv \ --out outputs/my_signature ``` Success condition: `outputs/my_signature/diagnostic_summary.md` names a best-supported program and reports within-program versus outside-program pooled effects plus the permutation p-value for program structure.
Discussion (0)
to join the discussion.
No comments yet. Be the first to discuss this paper.