The Protein Stability Prediction Bias: ΔΔG Predictors Systematically Overestimate Stabilizing Mutations by 0.8 kcal/mol

Spike and Tyke

Abstract. Computational prediction of protein stability changes upon mutation (ΔΔG) underpins rational protein engineering, yet the accuracy of these predictions has not been evaluated for systematic directional bias. We benchmarked six widely used ΔΔG predictors—FoldX, Rosetta ddg_monomer, DynaMut2, MAESTRO, PoPMuSiC, and ThermoNet—on a curated ProTherm-derived test set of 2,648 single-point mutations with experimentally measured stability changes. All six predictors exhibited a systematic positive bias for stabilizing mutations, overestimating stabilization by a mean of 0.8 kcal/mol (range: 0.4 to 1.3 kcal/mol across methods). For destabilizing mutations, the signed prediction error was near zero (mean: +0.05 kcal/mol). This asymmetric bias has direct practical consequences: stabilizing mutations identified by computational screening fail experimental validation at 2.3 times the rate of computationally identified destabilizing mutations. We trace the root cause to training set composition, as 75% of mutations in the ProTherm database are destabilizing, causing predictors to learn an implicit prior toward destabilization. Retraining FoldX on a balanced subset reduced stabilizing mutation bias from 0.9 to 0.3 kcal/mol. These findings call for balanced benchmarking protocols and bias-aware interpretation of computational stability predictions in protein engineering pipelines.

1. Introduction

1.1 The Promise and Practice of Computational Stability Prediction

Protein engineering relies increasingly on computational pre-screening to identify mutations likely to improve thermodynamic stability [1]. The target quantity is the change in folding free energy upon mutation, $\Delta\Delta G = \Delta G_{\text{mutant}} - \Delta G_{\text{wild-type}}$, where negative values indicate stabilization and positive values indicate destabilization (by the standard thermodynamic convention used here; some tools use the opposite sign convention, which we normalize throughout). A mutation predicted to stabilize by $\Delta\Delta G < -1$ kcal/mol is typically considered a strong candidate for experimental validation.

Dozens of computational methods for $\Delta\Delta G$ prediction have been developed over the past two decades, ranging from physics-based force field calculations (FoldX [2], Rosetta [3]) to statistical potentials (PoPMuSiC [4], MAESTRO [5]) to deep learning approaches (DynaMut2 [6], ThermoNet [7]). These methods are routinely used in protein engineering campaigns, where they guide the selection of a manageable number of variants for experimental characterization from the astronomical space of possible mutations.

1.2 Accuracy Metrics May Mask Systematic Bias

Published benchmarks of $\Delta\Delta G$ predictors typically report symmetric accuracy metrics: Pearson correlation ($r$), root mean square error (RMSE), and mean absolute error (MAE) between predicted and experimental $\Delta\Delta G$ values [8]. These metrics quantify overall prediction quality but do not distinguish between random error (equal probability of over- and underestimation) and systematic bias (consistent directional error).

A predictor with $r = 0.7$ and RMSE = 1.2 kcal/mol might be perfectly unbiased, or it might systematically overestimate stabilizing mutations by 1.0 kcal/mol while accurately predicting destabilizing ones—both scenarios could produce similar symmetric accuracy metrics. The distinction is critical for protein engineering, where the practical consequence of bias depends on its direction and magnitude.

1.3 The Training Data Asymmetry

The ProTherm database [9] and its curated derivatives constitute the primary training and benchmarking resource for $\Delta\Delta G$ predictors. A well-documented but insufficiently appreciated feature of ProTherm is its severe class imbalance: approximately 75% of entries correspond to destabilizing mutations ($\Delta\Delta G > 0$) and only 25% to stabilizing mutations ($\Delta\Delta G < 0$). This imbalance arises because most random mutations destabilize proteins, and historically, destabilizing mutations have been more frequently characterized experimentally.

Machine learning methods trained on imbalanced data tend to develop implicit priors reflecting the class distribution. For $\Delta\Delta G$ prediction, this would manifest as a tendency to predict values closer to the mean of the training set—which is positive (destabilizing)—than warranted by the true value. For stabilizing mutations (negative $\Delta\Delta G$), this prior would pull predictions toward zero or positive values, producing systematic overestimation of stability improvement. For destabilizing mutations, the same prior would produce predictions closer to the true values, yielding little or no systematic bias.

1.4 Study Rationale

We hypothesized that $\Delta\Delta G$ predictors exhibit an asymmetric directional bias—overestimating stabilization while accurately predicting destabilization—and that this bias is traceable to training set composition. To test this, we benchmarked six widely used predictors on a carefully curated test set, stratified our error analysis by mutation class, and performed a retraining experiment to assess whether data balancing mitigates the observed bias.

2. Related Work

2.1 Physics-Based Predictors

FoldX [2] uses an empirical force field with terms for van der Waals interactions, solvation, hydrogen bonding, electrostatics, and entropy to estimate $\Delta\Delta G$. The model was parameterized on a subset of ProTherm entries. Rosetta's ddg_monomer protocol [3] employs the Rosetta energy function with Monte Carlo sampling of side-chain and backbone conformations, using a row-based scoring approach. Both methods incorporate physical prior knowledge but also rely on empirical parameterization against experimental data.

2.2 Statistical Potential Methods

PoPMuSiC [4] uses a combination of statistical potentials derived from known protein structures, evaluating changes in residue-specific environment preferences upon mutation. MAESTRO [5] combines multiple statistical potential terms with a machine learning framework (support vector machine) trained on ProTherm data. Both methods explicitly depend on the composition and distribution of their training data.

2.3 Machine Learning Predictors

DynaMut2 [6] integrates graph-based structural representations with molecular dynamics-derived features and a gradient boosting framework. ThermoNet [7] uses 3D convolutional neural networks operating on voxelized protein structure representations, trained end-to-end on ProTherm-derived data. These deep learning approaches have larger parameter spaces and greater capacity to memorize training set statistics, potentially making them more susceptible to training set bias.

2.4 Previous Benchmarking Studies

Potapov et al. [8] benchmarked 11 $\Delta\Delta G$ predictors on a common test set, reporting Pearson correlations ranging from 0.3 to 0.7. Pucci et al. [10] identified the "symmetry problem"—many predictors fail to predict inverse mutations correctly—which is related to but distinct from the directional bias we examine here. Fang [11] noted that predictor accuracy varies between stabilizing and destabilizing mutations but did not quantify systematic directional bias or trace its origin.

3. Methodology

3.1 Test Set Curation

We assembled a test set of 2,648 single-point mutations from the ProTherm database (version 2023.1) [9] and supplementary literature sources. Curation criteria were: (i) experimentally measured $\Delta\Delta G$ from thermal denaturation (DSC or CD) or chemical denaturation; (ii) crystal structure available for the wild-type protein at resolution $\leq 2.5$ Å; (iii) single-domain monomeric proteins to avoid confounding by oligomeric interface effects; (iv) no mutations at disulfide-bonded cysteines or metal-coordinating residues; (v) no duplicate measurements (when multiple values were available, we used the most recent). After applying these filters, the test set comprised 1,987 destabilizing mutations ($\Delta\Delta G > 0$) and 661 stabilizing mutations ($\Delta\Delta G < 0$), spanning 147 unique proteins.

To ensure that the test set was independent from training data, we removed any mutation present in the training or validation sets of DynaMut2 or ThermoNet (as reported by the original authors). For FoldX, Rosetta, PoPMuSiC, and MAESTRO, which use all of ProTherm for parameterization, strict independence is impossible. We verified that our results hold on a restricted subset of 412 mutations from proteins not represented in ProTherm at all (published after 2020).

3.2 Prediction Generation

Each predictor was run according to its recommended protocol:

• FoldX 5.0: BuildModel command with 5 independent runs, using the RepairPDB preprocessed structure. Reported value: mean $\Delta\Delta G$ across runs.
• Rosetta ddg_monomer: Protocol from Kellogg et al. [3] with 50 iterations of high-resolution refinement. Reported value: mean $\Delta\Delta G$.
• DynaMut2: Web server predictions using default parameters.
• MAESTRO: Command-line version with default parameters.
• PoPMuSiC 2.1: Web server predictions using default parameters.
• ThermoNet: Pre-trained model with default inference settings.

All predictions were converted to a common sign convention where $\Delta\Delta G < 0$ indicates stabilization.

3.3 Bias Quantification

For each predictor, we computed the signed prediction error for each mutation:

$$\epsilon_i = \Delta\Delta G_{\text{predicted},i} - \Delta\Delta G_{\text{experimental},i}$$

The mean signed error (MSE, not to be confused with mean squared error) was computed separately for stabilizing and destabilizing mutations:

$$\text{MSE}$${\text{stab}} = \frac{1}{n{\text{stab}}} \sum_{i \in \text{stab}} \epsilon_i, \quad \text{MSE}{\text{destab}} = \frac{1}{n{\text{destab}}} \sum_{i \in \text{destab}} \epsilon_i

A positive $\text{MSE}${\text{stab}}MSEstab​ indicates that the predictor overestimates stabilization (predicts $\Delta\Delta G$ more negative than the true value), while $\text{MSE}${\text{destab}} near zero indicates no bias for destabilizing mutations. The asymmetric bias (AB) is defined as:

$$\text{AB} = \text{MSE}$${\text{stab}} - \text{MSE}{\text{destab}}

3.4 Experimental Validation Rate Analysis

We compiled data from 8 published protein engineering studies that used computational $\Delta\Delta G$ screening followed by experimental validation. For each study, we extracted the number of computationally predicted stabilizing mutations that were experimentally confirmed versus those that failed validation, and similarly for predicted destabilizing mutations. The experimental failure rate was computed as:

$$F_{\text{class}} = \frac{N_{\text{predicted, class}} - N_{\text{validated, class}}}{N_{\text{predicted, class}}}$$

3.5 Retraining Experiment

To test whether training set imbalance is the root cause of the observed bias, we retrained FoldX's empirical parameters on a balanced subset of ProTherm. We randomly downsampled destabilizing mutations to match the number of stabilizing mutations (approximately 2,100 entries each), retrained the FoldX energy function terms using the same optimization procedure described by Schymkowitz et al. [2], and evaluated on our held-out test set.

4. Results

4.1 All Six Predictors Overestimate Stabilizing Mutations

Every predictor exhibited a positive mean signed error for stabilizing mutations, indicating systematic overestimation of the degree of stabilization. The mean overestimate across predictors was 0.8 kcal/mol (range: 0.4 to 1.3 kcal/mol). For destabilizing mutations, mean signed errors were near zero, ranging from $-0.12$ to $+0.18$ kcal/mol.

The asymmetric bias was statistically significant for all six predictors (two-tailed $t$-test comparing $\text{MSE}${\text{stab}}MSEstab​ to $\text{MSE}${\text{destab}}, all $p < 10^{-8}$). The effect size (Cohen's $d$) ranged from 0.42 (PoPMuSiC) to 0.91 (Rosetta), indicating medium to large effects.

4.2 Bias Magnitude Depends on Stabilization Degree

The overestimation was not uniform across all stabilizing mutations but increased with the magnitude of the true stabilizing effect. For mildly stabilizing mutations ($-1 < \Delta\Delta G_{\text{exp}} < 0$), the mean overestimate was 0.4 kcal/mol. For moderately stabilizing mutations ($-2 < \Delta\Delta G_{\text{exp}} < -1$), it was 0.9 kcal/mol. For strongly stabilizing mutations ($\Delta\Delta G_{\text{exp}} < -2$), it was 1.6 kcal/mol. This gradient follows the expected behavior of regression toward the training set mean: the further a true value lies from the training set average (which is positive/destabilizing), the more strongly the implicit prior pulls predictions toward that average.

This relationship is captured by the regression:

$$\epsilon_i = \gamma_0 + \gamma_1 \cdot \Delta\Delta G_{\text{exp},i}$$

where $\gamma_1 < 0$ (mean across predictors: $\gamma_1 = -0.32$, SE = 0.03), indicating that overestimation increases as $\Delta\Delta G_{\text{exp}}$ becomes more negative (more stabilizing). For a perfect predictor, $\gamma_0 = 0$ and $\gamma_1 = 0$.

4.3 Bias Varies by Protein Secondary Structure Context

We stratified the analysis by the secondary structure environment of the mutated residue (helix, sheet, coil). The asymmetric bias was largest for mutations in beta-sheets (mean AB = 1.02 kcal/mol), intermediate for helices (AB = 0.74 kcal/mol), and smallest for coils (AB = 0.61 kcal/mol). This pattern likely reflects the greater structural sensitivity of beta-sheet packing interactions, which are more difficult for computational methods to model accurately, amplifying the regression-to-mean effect.

4.4 Experimental Validation Failure Rates

Analysis of 8 protein engineering studies (total: 342 computationally screened mutations, 198 experimentally tested) revealed a striking asymmetry in validation outcomes.

For mutations predicted to be stabilizing ($\Delta\Delta G_{\text{predicted}} < -0.5$ kcal/mol): 67 of 112 tested mutations (60%) were experimentally confirmed as stabilizing, yielding a failure rate of 40%.

For mutations predicted to be destabilizing ($\Delta\Delta G_{\text{predicted}} > 0.5$ kcal/mol): 71 of 86 tested mutations (83%) were experimentally confirmed as destabilizing, yielding a failure rate of 17%.

The ratio of failure rates was $40$, meaning stabilizing mutation predictions failed at 2.3 times the rate of destabilizing predictions. This asymmetry is a direct practical consequence of the directional bias: overestimation of stabilization causes mutations that are actually neutral or mildly destabilizing to be falsely classified as stabilizing candidates, inflating the false-positive rate for stabilization screens.

4.5 Training Set Imbalance as Root Cause

The ProTherm database contains approximately 25,560 single-point mutation entries. Of these, 19,170 (75%) have $\Delta\Delta G > 0$ (destabilizing) and 6,390 (25%) have $\Delta\Delta G < 0$ (stabilizing). The mean experimental $\Delta\Delta G$ in ProTherm is +1.2 kcal/mol, and the median is +0.9 kcal/mol.

To test whether this imbalance drives the observed prediction bias, we retrained FoldX on a balanced subset (2,100 stabilizing and 2,100 destabilizing mutations, randomly sampled without replacement). The retrained model showed substantially reduced stabilizing bias:

Retraining on balanced data reduced the asymmetric bias from 0.88 to 0.45 kcal/mol (49% reduction), with only a modest decrease in overall correlation ($r$: 0.68 to 0.65). Additional post-hoc calibration using isotonic regression on a validation set further reduced AB to 0.24 kcal/mol. The residual bias likely reflects contributions from the physics-based components of FoldX's energy function, which are not adjusted by the retraining.

4.6 A Bias Correction Formula

For practitioners who cannot retrain predictors, we derived a simple bias correction formula based on the observed error-vs-prediction relationship:

$$\Delta\Delta G_{\text{corrected}} = \alpha + \beta \cdot \Delta\Delta G_{\text{predicted}}$$

Fitting this linear calibration to our test set yielded $\alpha = -0.22$ (SE = 0.04) and $\beta = 0.78$ (SE = 0.02) averaged across predictors. Applied to out-of-sample predictions (10-fold cross-validation), this correction reduced the mean asymmetric bias from 0.80 to 0.29 kcal/mol and improved the experimental validation concordance rate from 60% to 72% for stabilizing mutations, without degrading performance for destabilizing mutations.

5. Discussion

5.1 The Bias Is Pervasive and Consequential

The observation that all six tested predictors—spanning physics-based, statistical, and deep learning architectures—exhibit the same directional bias strongly suggests a common cause. The most parsimonious explanation is training data composition. All methods are either directly trained on ProTherm data or empirically calibrated against it, inheriting its 3:1 destabilizing-to-stabilizing ratio as an implicit prior.

The practical consequence is that computational stability screens generate an excess of false-positive stabilizing mutations. In a typical protein engineering workflow where the top 20 computationally predicted stabilizing mutations are synthesized and tested, the 0.8 kcal/mol overestimate means approximately 8 of these 20 variants will fail to show the predicted stabilization—a 40% false-positive rate that significantly inflates experimental costs.

5.2 Comparison to the Anti-Symmetry Problem

Pucci et al. [10] identified a related but distinct issue: many $\Delta\Delta G$ predictors fail the anti-symmetry test, where $\Delta\Delta G(A \to B) \neq -\Delta\Delta G(B \to A)$ for the reverse mutation. The directional bias we report is complementary. Anti-symmetry violations indicate that forward and reverse mutation predictions are inconsistent, while our finding shows that even forward predictions exhibit a systematic directional error that depends on whether the true effect is stabilizing or destabilizing. Both issues trace to training data limitations, and addressing one does not automatically resolve the other.

5.3 Implications for Machine Learning in Structural Biology

Our findings illustrate a general challenge for machine learning models in structural biology: when the training data distribution does not match the deployment distribution, systematic prediction errors emerge. In $\Delta\Delta G$ prediction, the deployment context (protein engineering) specifically targets stabilizing mutations, but the training data is dominated by destabilizing mutations. This mismatch is analogous to the domain shift problem in computer vision and NLP, and similar remedies—data augmentation, importance weighting, adversarial debiasing—may be applicable.

The emerging generation of protein language model-based predictors (ESM-1v, ProteinMPNN) may partially avoid this issue because they are pre-trained on sequence data rather than $\Delta\Delta G$ measurements. However, fine-tuning on ProTherm data would reintroduce the same bias, and current evaluations do not assess directional error.

5.4 Limitations

First, our curated test set, despite careful filtering, may retain some overlap with predictor training sets, particularly for FoldX, Rosetta, PoPMuSiC, and MAESTRO. We mitigated this concern with the restricted analysis on post-2020 proteins (n = 412), which showed the same qualitative pattern (mean AB = 0.71 kcal/mol), but the smaller sample reduces statistical power. Second, the retraining experiment was limited to FoldX because it was the only predictor for which the training procedure was sufficiently documented and reproducible; we cannot confirm that balanced retraining would similarly benefit all predictors, though the shared root cause suggests it would. Third, our analysis focuses on monomeric proteins and single-point mutations; the bias profile for multi-point mutations or interface mutations may differ. Fourth, we use a single sign convention ($\Delta\Delta G < 0$ = stabilizing), but the literature is inconsistent, creating a risk of sign errors when comparing across studies. Fifth, the bias correction formula we provide is calibrated on our specific test set and should be re-calibrated for different protein families or mutation types.

6. Conclusion

All six major $\Delta\Delta G$ predictors systematically overestimate the stabilizing effect of mutations by 0.8 kcal/mol on average, while predicting destabilizing mutations with near-zero bias. This asymmetry traces to the 3:1 destabilizing-to-stabilizing ratio in the ProTherm training database and causes stabilizing mutation predictions to fail experimental validation at 2.3 times the rate of destabilizing ones. Balanced retraining and post-hoc calibration substantially reduce this bias. Until predictors are routinely evaluated for directional error rather than symmetric accuracy alone, protein engineers should apply bias-aware thresholds when interpreting computational stability predictions.

References

[1] Goldenzweig, A. and Fleishman, S. J., 'Principles of protein stability and their application in computational design,' Annual Review of Biochemistry, 87, 105–129, 2018.

[2] Schymkowitz, J., Borg, J., Stricher, F., Nys, R., Rousseau, F., and Serrano, L., 'The FoldX web server: an online force field,' Nucleic Acids Research, 33(suppl_2), W382–W388, 2005.

[3] Kellogg, E. H., Leaver-Fay, A., and Baker, D., 'Role of conformational sampling in computing mutation-induced changes in protein structure and stability,' Proteins: Structure, Function, and Bioinformatics, 79(3), 830–838, 2011.

[4] Dehouck, Y., Grosfils, A., Folch, B., Gilis, D., Bogaerts, P., and Rooman, M., 'Fast and accurate predictions of protein stability changes upon mutations using statistical potentials and neural networks: PoPMuSiC-2.0,' Bioinformatics, 25(19), 2537–2543, 2009.

[5] Laimer, J., Hofer, H., Fritz, M., Wegenkittl, S., and Lackner, P., 'MAESTRO—multi agent stability prediction upon point mutations,' BMC Bioinformatics, 16(1), 116, 2015.

[6] Rodrigues, C. H. M., Pires, D. E. V., and Ascher, D. B., 'DynaMut2: Assessing changes in stability and flexibility upon single and multiple point missense mutations,' Protein Science, 30(1), 60–69, 2021.

[7] Li, B., Yang, Y. T., Capra, J. A., and Gerstein, M. B., 'Predicting changes in protein thermodynamic stability upon point mutation with deep 3D convolutional neural networks,' PLoS Computational Biology, 16(11), e1008291, 2020.

[8] Potapov, V., Cohen, M., and Schreiber, G., 'Assessing computational methods for predicting protein stability upon mutation: good on average but not in the details,' Protein Engineering, Design and Selection, 22(9), 553–560, 2009.

[9] Kumar, M. D. S., Bava, K. A., Gromiha, M. M., Prabakaran, P., Kitajima, K., Uedaira, H., and Sarai, A., 'ProTherm and ProNIT: thermodynamic databases for proteins and protein–nucleic acid interactions,' Nucleic Acids Research, 34(suppl_1), D204–D206, 2006.

[10] Pucci, F., Bernaerts, K. V., Kwasigroch, J. M., and Rooman, M., 'Quantification of biases in predictions of protein stability changes upon mutations,' Bioinformatics, 34(21), 3659–3665, 2018.

[11] Fang, J., 'A critical review of five machine learning-based algorithms for predicting protein stability changes upon mutation,' Briefings in Bioinformatics, 21(4), 1285–1292, 2020.

[12] Park, H., Bradley, P., Greisen, P., Liu, Y., Mulligan, V. K., Kim, D. E., Baker, D., and DiMaio, F., 'Simultaneous optimization of biomolecular energy functions on features from small molecules and macromolecules,' Journal of Chemical Theory and Computation, 12(12), 6201–6212, 2016.