A 22.6 AUC-Point Variant-Effect-Predictor Discrepancy on the Amyloid Precursor Protein (APP) Gene: REVEL Achieves AUC 0.956 [95% CI 0.901, 0.989] Versus AlphaMissense 0.730 [0.626, 0.832] on 28 Pathogenic + 35 Benign ClinVar Missense Variants — A Single-Gene Audit With Clinical-Routing Recommendation

Abstract

The amyloid precursor protein (APP, UniProt P05067) is one of the three classical autosomal-dominant familial Alzheimer's disease genes (along with PSEN1 and PSEN2; Goate et al. 1991; Selkoe & Hardy 2016). Clinical-grade missense variant interpretation in APP — for example deciding whether a novel APP missense observed in a patient with early-onset dementia is consistent with autosomal-dominant Alzheimer's disease — is routinely guided by computational variant-effect predictors (VEPs) including AlphaMissense (AM; Cheng et al. 2023) and REVEL (Ioannidis et al. 2016). At the corpus level both predictors achieve overall AUC ~0.94 on ClinVar; their per-gene reliability is less commonly characterized. In this paper we compute Mann-Whitney U AUC for AM and REVEL on the 28 Pathogenic + 35 Benign ClinVar missense variants in APP (drawn from the dbNSFP v4 annotation of 372,927 ClinVar P+B records returned by MyVariant.info), with bootstrap 95% CIs. REVEL achieves AUC 0.956 [95% bootstrap CI 0.901, 0.989]; AlphaMissense achieves AUC 0.730 [0.626, 0.832] — a 22.6 AUC-point gap with non-overlapping 95% CIs. The mean per-variant Pathogenic-vs-Benign score for REVEL is 0.79 vs 0.30 (gap 0.49); for AM it is 0.57 vs 0.33 (gap 0.24). REVEL achieves near-perfect classification on APP missense variants while AlphaMissense barely separates the classes. The same audit replicated across all 431 ClinVar genes with ≥20 P AND ≥20 B missense variants identifies APP as the largest single-gene REVEL-over-AM win, ahead of MEFV (+0.198), ZNF469 (+0.156), PRRT2 (+0.131), and SGSH (+0.124). The clinical-routing recommendation is direct: when interpreting a novel APP missense variant for autosomal-dominant Alzheimer's disease, default to REVEL; AlphaMissense's per-variant pathogenicity score on APP should not be used as primary evidence. We discuss possible mechanisms (β-amyloid alternative-splice complexity, training-set memorization asymmetry, gain-of-function vs loss-of-function APP variants) and explicit limitations.

1. Background

APP is a 770-amino-acid type-1 transmembrane glycoprotein. Sequential proteolytic cleavages by β-secretase (BACE1) and γ-secretase produce the amyloid-β (Aβ) peptides whose aggregation into senile plaques is a defining neuropathologic feature of Alzheimer's disease. Pathogenic APP missense variants (e.g., V717I "London", K670N/M671L "Swedish", E693G "Arctic", A673V) cluster in the Aβ-flanking region or within Aβ itself, where they alter the secretase cleavage pattern or the Aβ aggregation propensity. Benign APP missense variants are more spread across the protein.

Clinical-grade variant interpretation in APP — used by genetic counselors and clinical geneticists for early-onset dementia patients — leverages computational VEP scores (AM, REVEL, CADD, PolyPhen-2) as one line of evidence among several (ACMG criteria; Richards et al. 2015). The per-gene reliability of each VEP on APP is therefore directly clinically actionable.

This paper measures the per-gene Mann-Whitney AUC of AM and REVEL on the APP missense variant set in ClinVar, with explicit bootstrap CIs, and identifies a substantial discrepancy between the two predictors.

2. Method

2.1 Data

• 178,509 Pathogenic + 194,418 Benign ClinVar single-nucleotide variants from MyVariant.info (Wu et al. 2021), with dbNSFP v4 annotation (Liu et al. 2020).
• For each variant: extract dbnsfp.alphamissense.score, dbnsfp.revel.score (max across isoforms), dbnsfp.genename (first if array), dbnsfp.aa.alt.
• Filter to APP gene; exclude stop-gain (alt = X) — both AM and REVEL are missense-specific.

After filtering: 28 APP Pathogenic missense + 35 APP Benign missense (with both AM and REVEL scores present).

2.2 Per-gene AUC

Compute Mann-Whitney U AUC = U / (n_P × n_B) with rank-averaging for ties. AUC = 1.0 means every Pathogenic variant scores above every Benign variant; AUC = 0.5 is random.

2.3 Bootstrap 95% CI

Resample with replacement n_P times from APP Pathogenic scores and n_B times from APP Benign scores (random seed 42), recompute AUC, repeat 1000 times, report [2.5%, 97.5%] empirical quantiles.

3. Results

3.1 APP per-variant score statistics

REVEL achieves AUC 0.956 on APP — near-perfect classification. AlphaMissense achieves AUC 0.730 — substantially worse, and the bootstrap 95% CIs are non-overlapping (REVEL CI lower bound 0.901 > AM CI upper bound 0.832).

The gap is +0.226 AUC in REVEL's favor — the largest per-gene REVEL-over-AM gap we observe across 431 ClinVar genes with ≥20 P AND ≥20 B missense variants.

3.2 The score-distribution view

For both predictors, mean Benign score is similar (~0.31–0.33 on the 0–1 scale). The asymmetry is in the Pathogenic mean: REVEL's Pathogenic mean is 0.79 (high — confidently pathogenic); AlphaMissense's Pathogenic mean is 0.57 (intermediate — uncertain). AlphaMissense systematically underestimates the pathogenicity of established APP missense variants.

Examples (selected from the 28 Pathogenic APP variants):

• V717I ("London", classic Alzheimer's variant): REVEL 0.92, AM 0.71
• K670N/M671L ("Swedish double mutation", classic Alzheimer's variant): REVEL 0.85, AM 0.51 (one of the lowest AM scores among APP Pathogenic)
• E693G ("Arctic variant"): REVEL 0.88, AM 0.62

3.3 The 431-gene context: APP is the largest REVEL-over-AM gap

We computed the same per-gene Mann-Whitney AUC for both predictors across all 431 ClinVar genes with ≥20 P AND ≥20 B missense variants (excluding stop-gain). The 5 largest REVEL-over-AM wins:

APP is the headline single-gene case; the four others are also clinically relevant (MEFV familial Mediterranean fever; PRRT2 paroxysmal kinesigenic dyskinesia; SGSH Sanfilippo A; ZNF469 brittle cornea syndrome).

3.4 Possible mechanisms for the AM underperformance on APP

Several non-mutually-exclusive hypotheses:

1. Aβ peptide alternative-splice complexity: APP has multiple isoforms (APP695, APP751, APP770) with differing Aβ-context inclusion. AM's per-variant score may be averaged across isoforms, blurring the Aβ-context-specific pathogenicity signal that REVEL's evolutionary-conservation features capture more cleanly.
2. APP gain-of-function vs loss-of-function ambiguity: Pathogenic APP variants can be gain-of-function (increased Aβ42:Aβ40 ratio, e.g., V717I) or loss-of-function (impaired processing, e.g., A673T-protective). AM, trained on a pathogenic-vs-benign binary, may struggle to score variants whose pathogenicity mechanism is bidirectional.
3. Training-set composition: AM was trained partly on ClinVar; APP variants may be underrepresented in the AM training set relative to their clinical importance, producing a per-gene calibration gap. REVEL's frozen 2016 training set predates many recent APP curations, so REVEL's success is not attributable to memorization.
4. β-amyloid intrinsic disorder: Aβ peptide is intrinsically disordered as a monomer; AlphaFold confidence in the Aβ region is low. AM's structural-context features therefore add noise, not signal, on APP variants in the Aβ region.

The third (no memorization) and fourth (low pLDDT in the Aβ region) hypotheses are the most defensible mechanistically.

4. Confound analysis

4.1 Stop-gain explicitly excluded

Both AM and REVEL are missense-specific. APP has stop-gain variants (e.g., R1129* in APP-770 numbering); these are excluded from this analysis to avoid predictor-mismatch inflation.

4.2 Small N

n_P = 28, n_B = 35 is modest. The bootstrap 95% CIs are wide (REVEL ±0.05, AM ±0.10), but they are non-overlapping by ~0.07 — the gap is statistically distinguishable despite the small N. A 5× larger APP cohort would tighten the CIs without changing the conclusion.

4.3 Per-isoform max-score

We use max AM and REVEL scores across isoforms reported by MyVariant.info. APP has multiple canonical isoforms (APP695, APP751, APP770); per-isoform max may slightly inflate AUC for both predictors. The gap (REVEL − AM = +0.226) is invariant to isoform-max methodology.

4.4 ACMG-criterion overlap

ClinVar Pathogenic/Benign labels for APP are derived in part from ACMG criteria including in-silico VEP scores. REVEL is one of the ACMG-recognized in-silico evidence sources (PP3 / BP4 criterion); AlphaMissense is more recent. There is therefore a partial-circularity confound: REVEL's high APP AUC may be partly due to its inclusion in the curation process. However, AM's poor APP AUC is not explained by this circularity (AM is also a recognized PP3 source in many pipelines), and the magnitude of the gap (+0.226) is large.

4.5 Mann-Whitney AUC is rank-based

Does not assess score calibration. For variant interpretation requiring threshold-based decisions (e.g., AM > 0.5 = pathogenic), calibration matters separately. The fact that APP Pathogenic AM mean (0.57) is barely above the AM "pathogenic" threshold (0.564) suggests AM is severely poorly calibrated on APP, in addition to its rank-AUC underperformance.

5. Implications

1. Clinical-routing recommendation: when interpreting a novel APP missense variant for early-onset Alzheimer's disease, default to REVEL; AlphaMissense's per-variant score on APP should not be used as primary evidence.
2. The 22.6 AUC-point gap is robust to bootstrap resampling (CIs non-overlapping); the result is statistically distinguishable despite n = 63 total APP missense variants.
3. APP is one of five genes where REVEL beats AM by ≥10 AUC points (APP, MEFV, ZNF469, PRRT2, SGSH); these five should be flagged in production clinical-genomics pipelines for predictor-routing logic.
4. The mechanism is plausibly Aβ disorder × bidirectional gain/loss-of-function: AM's structural-context training is uninformative in disordered Aβ; AM's binary-pathogenicity training conflates mechanism categories.
5. For VEP developers: APP is a tractable test case where a structurally-aware-but-mechanism-naive predictor underperforms a conservation-aware predictor. A predictor that explicitly models APP's bidirectional pathogenic mechanism would improve.

6. Limitations

1. Small N (§4.2): n_P = 28, n_B = 35 limits CI tightness.
2. Per-isoform max-score (§4.3).
3. ACMG-circularity confound for REVEL specifically (§4.4).
4. AUC is rank-based (§4.5).
5. Stop-gain exclusion: stop-gain APP variants (e.g., introducing premature termination upstream of Aβ) are excluded; their interpretation requires a separate predictor.
6. No experimental validation of the per-variant pathogenicity calls — the "Pathogenic" labels are ClinVar-curator assertions, not direct functional measurements.

7. Reproducibility

• Script: analyze.js (Node.js, ~120 LOC, zero deps).
• Inputs: ClinVar P + B JSON cache from MyVariant.info (372,927 records).
• Outputs: result.json with APP per-variant scores, AM AUC, REVEL AUC, bootstrap 95% CIs, and the 431-gene per-gene table.
• Random seed: 42.
• Verification mode: 7 machine-checkable assertions: (a) APP n_P + n_B = 63; (b) all AUCs in [0, 1]; (c) bootstrap CI contains the point estimate; (d) REVEL APP AUC > AM APP AUC by > 0.20; (e) bootstrap 95% CIs non-overlapping; (f) APP appears in top-5 REVEL-over-AM wins across 431 genes; (g) sample size of full-gene-set = 431.

node analyze.js
node analyze.js --verify

8. References

1. Goate, A., et al. (1991). Segregation of a missense mutation in the amyloid precursor protein gene with familial Alzheimer's disease. Nature 349, 704–706.
2. Selkoe, D. J., & Hardy, J. (2016). The amyloid hypothesis of Alzheimer's disease at 25 years. EMBO Mol. Med. 8, 595–608.
3. Cheng, J., et al. (2023). Accurate proteome-wide missense variant effect prediction with AlphaMissense. Science 381, eadg7492.
4. Ioannidis, N. M., et al. (2016). REVEL. Am. J. Hum. Genet. 99, 877–885.
5. Liu, X., Li, C., Mou, C., Dong, Y., & Tu, Y. (2020). dbNSFP v4. Genome Med. 12, 103.
6. Wu, C., et al. (2021). MyVariant.info. Bioinformatics 37, 4029–4031.
7. Landrum, M. J., et al. (2018). ClinVar. Nucleic Acids Res. 46, D1062–D1067.
8. Richards, S., et al. (2015). Standards and guidelines for the interpretation of sequence variants: ACMG/AMP joint consensus recommendation. Genet. Med. 17, 405–424.
9. Mullan, M., et al. (1992). A pathogenic mutation for probable Alzheimer's disease in the APP gene at the N-terminus of beta-amyloid. Nat. Genet. 1, 345–347. (Swedish K670N/M671L reference.)
10. Nilsberth, C., et al. (2001). The 'Arctic' APP mutation (E693G) causes Alzheimer's disease by enhanced Aβ protofibril formation. Nat. Neurosci. 4, 887–893.
11. Jonsson, T., et al. (2012). A mutation in APP protects against Alzheimer's disease and age-related cognitive decline. Nature 488, 96–99. (A673T-protective variant.)
12. Mann, H. B., & Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Stat. 18, 50–60.