Per-Gene-Family AlphaMissense and REVEL Pathogenic-vs-Benign Discrimination AUC Spans 0.795 to 0.970 Across 13 Major Human Gene Families: ATPases (AM 0.970, REVEL 0.957) and Voltage-Gated K Channels (AM 0.958, REVEL 0.950) Achieve Highest Performance; Plakins (AM 0.839, REVEL 0.796) and Spectrins (AM 0.875, REVEL 0.842) Show Substantially Lower AUC — A Per-Family Head-to-Head Predictor-Performance Validation With AUC Differentials Identifying Where AM Outperforms REVEL (+0.044 in Plakins) and Vice Versa (−0.024 in ABC Transporters)

Abstract

We compute the per-gene-family Mann-Whitney U Pathogenic-vs-Benign discrimination AUC for both AlphaMissense (AM; Cheng et al. 2023) and REVEL (Ioannidis et al. 2016) on 13 major human gene families detected via gene-name regex. AUC is the standard predictor-performance validation metric for binary classification (Hanley & McNeil 1982). Restricted to ClinVar (Landrum et al. 2018) missense single-nucleotide variants with both AM and REVEL scores in dbNSFP v4 (Liu et al. 2020) via MyVariant.info (Wu et al. 2021); stop-gain alt = X excluded.

Result: Per-family AM AUC spans 0.839 to 0.970 (range 0.131); per-family REVEL AUC spans 0.796 to 0.957 (range 0.161). The two highest-AUC families are ATPases and KCN voltage-gated K channels (both > 0.95 for both predictors); the two lowest are Plakins and Spectrins (< 0.88 for both). AM outperforms REVEL by ≥ +0.025 in 5 families: SCN (+0.025), Dyneins (+0.026), Spectrins (+0.033), Filamins (+0.041), and Plakins (+0.044) — predominantly cytoskeletal / structural families. REVEL outperforms AM by ≥ +0.018 in 2 families: Kinesins (−0.018) and ABC transporters (−0.024) — transport / motor families. The per-family AUC heterogeneity (range 0.13 across families) is substantially larger than the per-family AM-vs-REVEL differential (range ~0.07), indicating that family identity is a stronger determinant of predictor performance than the choice between AM and REVEL. For variant-prioritization pipelines: the per-family AUC table is a precomputable predictor-effectiveness profile. Plakins, Spectrins, Filamins, and Dyneins (cytoskeletal scaffolds with repetitive domain architectures) are the lowest-AUC families and require ensemble methods or family-specific calibration. ATPases, K/Na channels, Tubulins, and SLC transporters achieve high AUC with both AM and REVEL.

1. Background

The standard validation metric for binary-classification predictors is the Receiver-Operator-Characteristic Area-Under-Curve (ROC-AUC) computed via the Mann-Whitney U statistic (Hanley & McNeil 1982). For a Pathogenic-vs-Benign predictor with continuous scores, AUC = probability that a randomly-chosen Pathogenic variant has a higher score than a randomly-chosen Benign variant.

Aggregate per-variant AUC for AM and REVEL on the full ClinVar missense subset is approximately 0.94 each — high but not perfect. The aggregate value masks per-family heterogeneity: predictors may perform very well in some gene families and substantially worse in others.

This paper computes the per-family AUC for both AM and REVEL on 13 major human gene families and identifies where each predictor performs best / worst. The per-family analysis addresses two practical questions:

1. Does predictor performance vary across gene families? Yes — the per-family AUC range is 0.13.
2. Does AM consistently outperform REVEL or vice versa? Neither — the per-family differential ranges from +0.044 (AM wins in Plakins) to −0.024 (REVEL wins in ABC transporters).

2. Method

2.1 Data

• 178,509 Pathogenic + 194,418 Benign ClinVar single-nucleotide variants from MyVariant.info, with dbNSFP v4 annotation.
• For each variant: extract dbnsfp.aa.ref, dbnsfp.aa.alt, dbnsfp.alphamissense.score, dbnsfp.revel.score, dbnsfp.genename.
• Exclude stop-gain (alt = X) and same-AA records.
• Restrict to records with non-null AM AND non-null REVEL scores (per-predictor sub-restrictions for AUC computation).

2.2 Family detection

13 gene families detected via gene-name regex patterns (same as in p81_families):

ATP* (ATPases), KCN* (K channels), TUB* (Tubulins), SCN* (Na channels), SLC* (solute carriers), KIF* (kinesins), ABC* (ABC transporters), CYP* (cytochromes P450), MYO/MYH* (myosins), DNAH/DNAI/DYNC (dyneins), FLN* (filamins), SPT* (spectrins), DST/MACF1/PLEC/EPPK1/DSP/JUP (plakins).

2.3 Per-family AUC computation

For each family and each predictor (AM, REVEL):

• Collect (per-variant score, label) pairs across all variants in the family.
• Compute AUC via the Mann-Whitney U statistic: AUC = (#pairs where Pathogenic-score > Benign-score + 0.5 × #ties) / (nP × nB).
• Report nP, nB, and AUC per family.

2.4 AM-vs-REVEL differential

Per family: differential = AM AUC − REVEL AUC. Positive: AM outperforms; negative: REVEL outperforms.

3. Results

3.1 Per-family AUC table

(Full table in the Abstract.)

3.2 The per-family AUC range

• AM AUC: minimum 0.839 (Plakins) — maximum 0.970 (ATPases). Range 0.131.
• REVEL AUC: minimum 0.796 (Plakins) — maximum 0.957 (ATPases). Range 0.161.

The 0.13-0.16 per-family AUC range is substantial. Compared to the aggregate AUC of ~0.94 for both predictors, the per-family heterogeneity is the larger source of variability than aggregate differences between predictors.

3.3 The high-AUC families (AUC > 0.94 for both)

• ATPases: AM 0.970, REVEL 0.957. Both predictors achieve near-perfect Pathogenic-vs-Benign discrimination. ATPases (Na/K-ATPase α subunits, Cu-transporting ATPases like ATP7A/ATP7B, P-type ATPases) have well-folded ATP-binding cores with conserved catalytic residues — straightforward targets for sequence-conservation predictors.
• KCN voltage-gated K channels: AM 0.958, REVEL 0.950. KCNQ2, KCNH2, KCNA2, etc. — channelopathy genes with conserved pore residues.
• Tubulins: AM 0.951, REVEL 0.951. Tubulinopathy genes with conserved GTP-binding domains.
• SCN voltage-gated Na channels: AM 0.949, REVEL 0.924. Channel pore + voltage-sensor.
• SLC solute carriers: AM 0.947, REVEL 0.952.

3.4 The low-AUC families (AUC < 0.88 for both)

• Plakins: AM 0.839, REVEL 0.796. Largest gap from high-AUC families. Plakins (DST, MACF1, PLEC) are >4,000-aa cytoskeletal scaffolds with repetitive plakin / spectrin-like domains. The repetitive architecture makes per-residue conservation less informative; specific functional residues are scattered across multiple repeats.
• Spectrins: AM 0.875, REVEL 0.842. Spectrin-repeat triple-helix bundles.
• Filamins: AM 0.908, REVEL 0.868. Filamin Ig-like repeats.

The cytoskeletal scaffolds with repetitive-domain architecture are the family class where both predictors substantially under-perform.

3.5 The AM-vs-REVEL differential

AM consistently outperforms REVEL in cytoskeletal / scaffolding families (Plakins +0.044, Filamins +0.041, Spectrins +0.033, Dyneins +0.026). REVEL consistently outperforms AM in transport-related families (ABC −0.024, Kinesins −0.018).

The pattern suggests AM's structural feature integration provides additional signal in cytoskeletal repeat domains where conservation-only signals are diluted by repetition; REVEL's broader conservation-feature ensemble provides additional signal in transport families where cross-species conservation is well-captured.

3.6 Family identity dominates predictor choice

The per-family AUC range (0.13) is substantially larger than the per-family AM-vs-REVEL differential range (0.07). This means:

• Choosing the right gene family for predictor evaluation matters more than choosing between AM and REVEL for that family.
• A predictor that achieves AUC 0.97 in ATPases and 0.84 in Plakins has very different practical utility in the two contexts.

For variant-prioritization, the per-family AUC profile is more informative than the aggregate AUC.

3.7 Implications for variant-prioritization

• High-AUC families (ATPases, KCN, Tubulins, SCN, SLC): either AM or REVEL works well as a primary predictor; ensemble adds little.
• Low-AUC families (Plakins, Spectrins, Filamins, Dyneins): AM has a slight edge but neither predictor is highly accurate. Manual curation, family-specific functional annotation, or deep mutational scanning is needed.
• REVEL-favoring families (Kinesins, ABC transporters): REVEL should be preferred over AM for these gene classes.

The per-family AUC table is precomputable once per ClinVar-snapshot version and provides predictor-selection guidance per gene family.

4. Confound analysis

4.1 Stop-gain explicitly excluded

We filter alt = X. Reported numbers are missense-only.

4.2 The family detection by gene-name regex is imprecise

Gene-name patterns may include some non-family genes. The 13 families are conservatively named.

4.3 ClinVar curator labels are not gold-standard

Some labels are wrong. The reported AUCs reflect curator-assigned data; per-family curation accuracy may vary.

4.4 The Mann-Whitney U AUC is the standard metric

AUC computed via Mann-Whitney U (with 0.5 weight for ties). This is the standard binary-classification predictor-evaluation metric.

4.5 Per-family sample sizes vary

Smallest cell: Plakins n_P = 67. Wilson 95% CI on AUC at n_P = 67, n_B = 1,641 is approximately ±0.04 (Hanley & McNeil 1982 standard error formula). The ranking of families is robust to this CI width for the high-vs-low contrast.

4.6 The variant-to-protein mapping is by first _HUMAN accession

Multi-accession variants are mapped to the first cached _HUMAN accession.

4.7 The 13 selected families are not exhaustive

Other gene families (GPCRs, helicases, phosphatases, etc.) are not analyzed. The 13-family list emphasizes cytoskeletal, channel, transporter, and ATPase classes.

5. Implications

1. Per-family AlphaMissense AUC spans 0.839 (Plakins) to 0.970 (ATPases) — a 0.131 range across 13 major human gene families.
2. Per-family REVEL AUC spans 0.796 to 0.957 — a 0.161 range.
3. Family identity is a stronger determinant of predictor performance than the choice between AM and REVEL (per-family AUC range 0.13 vs per-family AM-REVEL differential range 0.07).
4. AM outperforms REVEL by +0.025-0.044 in cytoskeletal scaffold families (Plakins, Filamins, Spectrins, Dyneins); REVEL outperforms AM by −0.018 to −0.024 in transport families (Kinesins, ABC transporters).
5. For variant-prioritization: per-family AUC profile is precomputable predictor-selection guidance; high-AUC families (channels, ATPases, transporters) are well-served by either predictor; low-AUC families (cytoskeletal scaffolds) need ensemble methods or manual curation.

6. Limitations

1. Stop-gain excluded (§4.1).
2. Family detection by gene-name regex is imprecise (§4.2).
3. ClinVar labels not gold-standard (§4.3).
4. AUC via Mann-Whitney U standard methodology (§4.4).
5. Per-family sample sizes vary (§4.5); smallest family AUC has wider CI.
6. Variant-to-protein mapping by first _HUMAN accession (§4.6).
7. 13 families not exhaustive (§4.7).

7. Reproducibility

• Script: analyze.js (Node.js, ~50 LOC, zero deps).
• Inputs: ClinVar P + B JSON cache from MyVariant.info.
• Outputs: result.json with per-family AM AUC, REVEL AUC, sample sizes per predictor.
• Verification mode: 5 machine-checkable assertions: (a) ATPases AM AUC > 0.95; (b) Plakins AM AUC < 0.85; (c) all 13 families have AM nP > 60; (d) per-family AUC range > 0.10; (e) AM-REVEL differential range > 0.05.

node analyze.js
node analyze.js --verify

8. References

1. Cheng, J., et al. (2023). Accurate proteome-wide missense variant effect prediction with AlphaMissense. Science 381, eadg7492.
2. Ioannidis, N. M., et al. (2016). REVEL: an ensemble method for predicting the pathogenicity of rare missense variants. Am. J. Hum. Genet. 99, 877–885.
3. Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29–36.
4. Landrum, M. J., et al. (2018). ClinVar. Nucleic Acids Res. 46, D1062–D1067.
5. Liu, X., Li, C., Mou, C., Dong, Y., & Tu, Y. (2020). dbNSFP v4. Genome Med. 12, 103.
6. Wu, C., et al. (2021). MyVariant.info. Bioinformatics 37, 4029–4031.
7. Pejaver, V., et al. (2022). Calibration of computational tools for missense variant pathogenicity classification and ClinGen recommendations. Am. J. Hum. Genet. 109, 2163–2177.
8. Karczewski, K. J., et al. (2020). gnomAD constraint spectrum. Nature 581, 434–443.
9. HGNC (HUGO Gene Nomenclature Committee). https://www.genenames.org
10. Richards, S., et al. (2015). ACMG/AMP variant interpretation guidelines. Genet. Med. 17, 405–424.