Minimax Regret Model Selection: When the Best Model for Any Task Is Never the Best Model for Every Task

Abstract

Model selection in machine learning implicitly assumes the practitioner knows which task the deployed system will face. In multi-task clinical settings—where the same diagnostic pipeline encounters heterogeneous patient populations—this assumption fails. We formalize model selection under task uncertainty as a decision-theoretic problem and apply Savage's minimax regret criterion to evaluate 24 prediction strategies (9 individual classifiers and 15 ensemble aggregation methods) across 5 clinical classification tasks derived from flow cytometry data. We construct the full 24×5 regret matrix, where regret measures the gap between each method's performance and the task-specific oracle. Our central finding is that the trimmed mean ensemble achieves a minimax regret of 0.007 AUROC, the lowest among all 24 methods, despite never being the single best method on 3 of 5 tasks. By contrast, AdaptiveSelect achieves the best raw performance on 2 tasks but suffers a minimax regret of 0.056—eight times higher—because it catastrophically underperforms on the remaining tasks. We connect this result to the forecast combination puzzle from economics, provide a formal analysis of robustness to task weighting, and propose an adaptive minimax framework that interpolates between minimax and Bayes-optimal selection as task uncertainty resolves. Our analysis demonstrates that decision theory provides a principled alternative to the common practice of selecting models by average performance, with particular relevance to clinical deployment where worst-case guarantees matter more than average-case optimality.

1. Introduction

Consider a clinical laboratory deploying an automated diagnostic system. The system will encounter patients with suspected myeloid neoplasms, lymphoid malignancies, erythroid abnormalities, mixed-phenotype leukemias, and cases requiring pan-leukemia screening. A natural question arises: which classification model should the laboratory deploy?

Standard machine learning practice offers a clear answer: evaluate candidate models on a held-out test set and select the one with the highest average performance. This approach is so deeply embedded in the field's culture that it rarely receives explicit justification. Yet it rests on an assumption that deserves scrutiny: that the distribution of future cases will match the distribution observed during evaluation.

In practice, this assumption is routinely violated. Hospital demographics shift. Referral patterns change. New clinical protocols alter the case mix. The system optimized for average performance across last year's distribution may perform poorly on next year's distribution—and the practitioner has no way of knowing which tasks will dominate.

This paper argues that model selection under task uncertainty is fundamentally a problem of decision theory, not statistical estimation. When the decision-maker does not know which task (or distribution over tasks) the deployed system will face, the question "which model is best?" is ill-posed unless accompanied by a criterion for what "best" means under uncertainty.

We examine three classical decision criteria:

1. Bayes optimality: Select the model with the highest expected performance under a known (or assumed) distribution over tasks. This is the implicit criterion behind "pick the model with the best average score."

2. Minimax: Select the model whose worst-case absolute performance is highest. This is extremely conservative and often selects mediocre models that avoid catastrophic failure.

3. Minimax regret (Savage's criterion): Select the model whose worst-case regret—the gap between its performance and what the task-specific best model achieves—is minimized. This is the criterion we advocate.

Minimax regret captures a subtle but crucial distinction: it does not penalize a model for facing a hard task (where all models perform poorly) but only for performing worse than it could have on that task. It is the decision-theoretic formalization of "never be too far from the best available option, regardless of what happens."

We apply this framework to a concrete empirical setting: 9 individual classifiers and 15 ensemble aggregation strategies evaluated on 5 clinical classification tasks derived from flow cytometry data. Our contributions are:

• A complete regret matrix for 24 methods across 5 tasks, revealing the structure of performance trade-offs that aggregate metrics obscure.
• Identification of the minimax-regret-optimal strategy: TrimmedMean ensemble, with a worst-case regret of 0.007 AUROC—meaning it is never more than 0.007 below the best available method on any task.
• A formal connection between minimax regret in model selection and the forecast combination puzzle from economics, where simple averages reliably outperform complex combination schemes.
• A robustness analysis showing that the minimax regret ranking is stable under a wide range of task-weighting schemes.
• An adaptive minimax framework that smoothly transitions from minimax (maximal uncertainty) to Bayes-optimal (known task distribution) selection as information about the deployment context accumulates.

The paper is organized as follows. Section 2 reviews decision-theoretic background. Section 3 formalizes the setup. Section 4 describes the empirical setting. Section 5 presents the regret matrix. Section 6 performs the minimax regret analysis. Section 7 connects to the forecast combination puzzle. Section 8 examines robustness to task weighting. Section 9 develops the adaptive minimax extension. Section 10 discusses limitations and concludes.

2. Decision-Theoretic Background

2.1 The Decision Problem

A decision problem consists of a set of actions (here, models to deploy), a set of states of nature (here, tasks the system will encounter), and a loss function mapping action-state pairs to real-valued outcomes. The challenge arises because the decision-maker must choose an action before observing the state.

Abraham Wald formalized statistical inference as a decision problem in his foundational work on statistical decision theory in the 1940s and 1950s. In Wald's framework, a decision rule maps observed data to actions, and decision rules are compared by their risk functions—the expected loss as a function of the unknown state. This framework unified hypothesis testing, estimation, and prediction under a single mathematical umbrella.

2.2 Minimax Decisions

The minimax criterion selects the action that minimizes the maximum possible loss across all states:

$$a^* = \arg\min_{a \in \mathcal{A}} \max_{s \in \mathcal{S}} L(a, s)$$

In our context, this translates to: select the model whose worst-case performance (across tasks) is highest. While appealingly conservative, pure minimax has a well-known deficiency: it treats absolute performance and relative performance identically. A model that achieves AUROC 0.70 on a task where the best available model achieves 0.71 is treated the same as one achieving 0.70 on a task where the best achieves 0.95. The first is nearly optimal; the second is catastrophically suboptimal.

2.3 Minimax Regret (Savage's Criterion)

Leonard Savage introduced the minimax regret criterion to address this limitation. Rather than minimizing worst-case loss, it minimizes worst-case regret, defined as the difference between the achieved outcome and the best achievable outcome in each state:

$$R(a, s) = L(a, s) - \min_{a' \in \mathcal{A}} L(a', s)$$

Or equivalently, in terms of a utility (performance) metric:

$$R(a, s) = \max_{a' \in \mathcal{A}} U(a', s) - U(a, s)$$

The minimax regret decision is then:

$$a^* = \arg\min_{a \in \mathcal{A}} \max_{s \in \mathcal{S}} R(a, s)$$

This criterion has several attractive properties:

1. Independence of irrelevant alternatives: Adding a dominated model to the candidate set does not change the minimax regret solution (unless the new model becomes the task-specific best on some task, thereby changing the regret definition).

2. Scale invariance to task difficulty: Regret measures the gap to the best available, not the absolute performance level. Hard tasks (where all models struggle) do not unduly influence the selection.

3. No distributional assumptions: Unlike Bayes optimality, minimax regret requires no prior over states. It provides guarantees for any task the system might encounter.

4. Moderate conservatism: Unlike pure minimax, it does not obsess over absolute worst-case performance. It asks only that you not deviate too far from the best option, whatever happens.

2.4 Bayes Optimality

The Bayesian approach places a prior distribution π over states and selects the action minimizing expected loss:

$$a^* = \arg\min_{a \in \mathcal{A}} \sum_{s \in \mathcal{S}} \pi(s) L(a, s)$$

When the prior is uniform (equal weight on all tasks), this reduces to selecting the model with the best average performance—exactly the standard ML practice. The Bayesian approach is optimal when the prior is correct but can perform arbitrarily badly when it is misspecified.

A deep result in decision theory connects these approaches: the minimax regret solution often coincides with the Bayes solution under the least favorable prior—the distribution over states that makes the decision problem hardest. This means that adopting minimax regret is equivalent to being Bayesian with a particular adversarial prior, providing an elegant bridge between the two paradigms.

2.5 Relationship Between Criteria

The three criteria form a spectrum of conservatism:

• Bayes (known prior): Optimistic—assumes you know the task distribution. Optimal when correct, brittle when wrong.
• Minimax regret: Moderate—hedges against the worst case but acknowledges that some tasks are inherently harder than others.
• Minimax (absolute): Pessimistic—optimizes for the absolute worst case, often leading to overly conservative choices.

In practice, the choice among these criteria should reflect the decision-maker's knowledge about the deployment environment. When task distributions are well-characterized (e.g., a stable hospital with years of historical data), Bayesian selection is appropriate. When task distributions are unknown or non-stationary (e.g., a new diagnostic system, a hospital with shifting demographics), minimax regret provides the most principled default.

3. Formal Setup

3.1 Notation

Let T = {t₁, t₂, ..., t_T} be a set of T classification tasks. Let M = {m₁, m₂, ..., m_M} be a set of M candidate prediction methods (individual models or ensemble strategies). Let U(m, t) ∈ [0, 1] denote the performance (AUROC) of method m on task t, estimated via cross-validation.

3.2 The Performance Matrix

The T × M performance matrix P has entries P[m, t] = U(m, t). Each row corresponds to a method, each column to a task. Standard model selection examines row means (average performance) or row minima (worst-case performance).

3.3 The Oracle and the Regret Matrix

For each task t, define the oracle performance:

$$U^*(t) = \max_{m \in \mathcal{M}} U(m, t)$$

This is the best achievable performance on task t by any method in the candidate set. The oracle is not a single method but a task-dependent selector.

The regret matrix R has entries:

$$R[m, t] = U^*(t) - U(m, t)$$

Regret is always non-negative. A regret of zero means method m is the best available for task t. A positive regret means there exists a better option for that task.

3.4 Decision Criteria in Matrix Form

Given the regret matrix R:

• Bayes-optimal (uniform prior): Select m minimizing the row mean of R[m, ·]. Equivalently, maximize the row mean of P[m, ·].

• Minimax regret: Select m minimizing the row maximum of R[m, ·]:

$$m^* = \arg\min_m \max_t R[m, t]$$

• Minimax (absolute): Select m maximizing the row minimum of P[m, ·]:

$$m^* = \arg\max_m \min_t P[m, t]$$

3.5 Properties of the Minimax Regret Solution

The minimax regret solution satisfies several desirable properties for model deployment:

Bounded suboptimality: For any task t, the performance deficit of the minimax regret solution is at most max_t R[m*, t]. This provides a uniform performance guarantee.

Robustness to task reweighting: If the task distribution changes, the worst-case regret remains bounded. Section 8 formalizes this.

Admissibility: If the minimax regret solution is unique, it is admissible—no other method dominates it across all tasks (assuming no ties in the regret matrix).

3.6 Dual Interpretation via Game Theory

The minimax regret problem has an elegant game-theoretic interpretation. Consider a two-player zero-sum game where:

• Player 1 (the modeler) chooses a method m.
• Player 2 (nature/adversary) chooses a task t.
• The payoff to the adversary is R[m, t].

The minimax regret solution for the modeler is the maximin strategy in this game. If we allow mixed strategies (randomized model selection), von Neumann's minimax theorem guarantees the existence of a saddle point, and the value of the game equals the minimax regret.

This dual interpretation reveals that the minimax regret solution is robust against an adversary who can choose the worst-case task after observing the modeler's choice—the strongest possible notion of robustness.

4. Empirical Setting

4.1 Clinical Classification Tasks

We evaluate our framework on five binary classification tasks derived from flow cytometry data in hematological diagnostics. Each task involves detecting a specific type of abnormal cell population from high-dimensional scatter and fluorescence measurements:

1. Myeloid blast detection (n=546; 134 positive, 412 negative): Identifying myeloid blast populations indicative of acute myeloid leukemia and related neoplasms. These blasts express characteristic myeloid markers (CD13, CD33, CD117, MPO) and are typically found in patients with suspected AML.

2. Lymphoid blast detection (n=546; 89 positive, 457 negative): Identifying lymphoid blast populations associated with acute lymphoblastic leukemia. Lymphoid blasts express B-lineage (CD19, CD10, CD22) or T-lineage (CD3, CD5, CD7) markers depending on subtype.

3. Erythroid lineage abnormalities (n=546; 47 positive, 499 negative): Detecting erythroid precursor abnormalities suggestive of erythroid-predominant leukemias or myelodysplastic syndromes. This is the most challenging task due to severe class imbalance and subtle phenotypic shifts.

4. Mixed-phenotype acute leukemia (n=546; 31 positive, 515 negative): Detecting cases where blast populations co-express markers from multiple lineages, a diagnostically challenging category with the most extreme class imbalance in our study.

5. Pan-leukemia screening (n=546; 218 positive, 328 negative): A broad screen for any type of blast population. This is the most balanced task and serves as a first-pass triage before lineage-specific classification.

These five tasks share the same patient cohort but define different classification targets, creating a natural multi-task setting. Crucially, the tasks vary in difficulty (class imbalance ranges from 40:60 to 6:94), discriminability (some lineages are immunophenotypically distinct, others overlap), and practical importance (pan-screening is common, MPAL detection is rare but clinically critical).

4.2 Feature Extraction

All tasks use the same feature representation: 128-dimensional embeddings extracted from a convolutional autoencoder pre-trained on flow cytometry scatter and fluorescence panels. This shared representation ensures that performance differences across tasks reflect genuine differences in task difficulty and classifier suitability, not differences in feature engineering.

4.3 Individual Classifiers (Base Models)

We evaluate nine individual classifiers spanning major algorithmic families:

These models were selected to provide diverse inductive biases. Linear models (LogReg, ElasticNet_LR) assume approximately linear decision boundaries. Kernel methods (SVM_RBF) can capture non-linear boundaries but are sensitive to kernel choice. Tree ensembles (RF_100, GBM_50) are non-parametric and handle interactions naturally. Instance-based methods (KNN_5, KNN_10) rely on local structure. The neural network (MLP_small) provides flexible non-linear capacity. NaiveBayes serves as a simple probabilistic baseline.

4.4 Ensemble Aggregation Strategies

Each ensemble method combines the probabilistic predictions (predicted class probabilities) of all 9 base models. We evaluate 15 aggregation strategies spanning three complexity tiers:

Simple aggregation (4 methods):

• SimpleAverage: Arithmetic mean of predicted probabilities.
• TrimmedMean: Trimmed mean discarding the single highest and lowest predictions (K=1).
• TrimmedMean_2: Trimmed mean with K=2 (discards top and bottom 2).
• MedianAggregation: Median of predicted probabilities.

Selection-based aggregation (5 methods):

• WeightedAverage: Weights proportional to individual cross-validated AUROC.
• TopK_K3: Average of top-3 models (by CV performance).
• TopK_K5: Average of top-5 models (by CV performance).
• TopK_Weighted_K3: Weighted average of top-3 models.
• TopK_Weighted_K5: Weighted average of top-5 models.

Learned aggregation (6 methods):

• AdaptiveSelect: Selects single best model per fold based on inner-CV.
• MetaStack_Ridge: Ridge regression on base model predictions.
• MetaStack_Lasso: Lasso regression on base model predictions.
• ElasticNet_a05: ElasticNet stacking (α=0.5).
• ElasticNet_a09: ElasticNet stacking (α=0.9, near-Lasso).
• NeuralMeta_MLP: MLP meta-learner on base model predictions.

4.5 Evaluation Protocol

All methods are evaluated using 5-fold stratified cross-validation repeated 10 times, with the average AUROC across repetitions serving as the point estimate. For ensemble methods, base model training and aggregation weight estimation use an inner cross-validation loop to prevent information leakage from the outer evaluation folds.

5. Computing the Regret Matrix

5.1 Raw Performance Matrix

We first present the complete performance matrix P[m, t] for all 24 methods across 5 tasks. Performance is measured in AUROC.

Table 1: Individual Classifier Performance (AUROC)

Table 2: Ensemble Method Performance (AUROC)

Several patterns emerge immediately. First, no single method dominates across all tasks. GBM_50 excels on Myeloid (0.920) and AnyBlast (0.943) but struggles on Mixed (0.762). RF_100 leads among individuals on Lymphoid (0.881), Erythroid (0.827), and Mixed (0.788) but falls behind on Myeloid. Among ensembles, AdaptiveSelect achieves the highest raw scores on Myeloid (0.925) and AnyBlast (0.948) but performs poorly on Mixed (0.756)—worse than every individual model except NaiveBayes.

Second, ensemble methods generally outperform individual classifiers. TrimmedMean exceeds every individual model on every task. However, the margin of improvement varies dramatically by task: TrimmedMean's advantage over the best individual is +0.020 on Mixed but essentially zero on Myeloid.

Third, the complexity-performance relationship is inverted. The simplest ensemble methods (SimpleAverage, TrimmedMean, MedianAggregation) consistently outperform the most complex ones (MetaStack_Lasso, ElasticNet_a09, NeuralMeta_MLP). This inversion is the ensemble analog of the forecast combination puzzle and will be analyzed in Section 7.

5.2 Per-Task Oracle

The per-task oracle—the best performance achievable by any method in the candidate set—defines the reference point for regret computation:

A striking feature: the oracle is not a single method. It requires AdaptiveSelect for Myeloid and AnyBlast, TrimmedMean for Lymphoid and Erythroid, and TrimmedMean_2 for Mixed. No method is the oracle on all tasks—this is the fundamental motivation for minimax regret.

5.3 The Full Regret Matrix

Table 3: Complete Regret Matrix R[m, t] = Oracle(t) − AUROC(m, t)

Individual Classifiers:

Ensemble Methods:

5.4 Structure of the Regret Matrix

The regret matrix reveals a clear anti-correlation pattern: methods that minimize regret on high-prevalence tasks (Myeloid, AnyBlast) tend to maximize regret on low-prevalence tasks (Erythroid, Mixed), and vice versa. This creates the fundamental tension that minimax regret resolves.

Specifically, the "Mixed" task (MPAL detection) is the discriminating axis of the regret matrix. Methods with low Mixed regret (TrimmedMean: 0.004, TrimmedMean_2: 0.000) achieve it by maintaining stable performance on this challenging, low-prevalence task. Methods with high Mixed regret (AdaptiveSelect: 0.056, NeuralMeta_MLP: 0.081) sacrifice Mixed performance to maximize performance on easier tasks.

This structure has a natural interpretation. Complex methods (AdaptiveSelect, NeuralMeta_MLP) have enough capacity to specialize, so they implicitly allocate their modeling budget toward tasks where improvement is easiest—high-prevalence, well-separated tasks. Simple methods (TrimmedMean, SimpleAverage) lack the capacity to specialize, forcing them to perform reasonably on all tasks. In the minimax regret framework, this capacity limitation becomes an advantage.

6. Minimax Regret Analysis

6.1 The Minimax Regret Ranking

Sorting all 24 methods by their maximum regret (the minimax regret criterion) yields:

6.2 Key Findings

Finding 1: TrimmedMean is the minimax regret optimal strategy. Its worst-case regret of 0.007 means that on any task, TrimmedMean is within 0.007 AUROC of the best available method. This is a remarkably tight bound—for practical purposes, TrimmedMean is nearly oracle-optimal on every task.

Finding 2: The minimax ranking inversely correlates with method complexity. The top 5 positions are held by simple aggregation methods (TrimmedMean, SimpleAverage, TrimmedMean_2, WeightedAverage, MedianAggregation). All 6 learned methods rank 15th or lower. The most complex method (NeuralMeta_MLP) ranks last.

Finding 3: Being the best on some tasks is costly. AdaptiveSelect achieves the oracle score on 2 of 5 tasks (zero regret on Myeloid and AnyBlast) but ranks 20th in minimax regret because its regret on Mixed (0.056) is eight times higher than TrimmedMean's maximum regret. Similarly, GBM_50 achieves zero regret on Myeloid and AnyBlast among individuals but ranks 17th overall.

Finding 4: The minimax and mean-regret rankings largely agree. TrimmedMean is also the mean-regret-optimal method (mean regret = 0.004). This is not always the case in decision theory—the minimax and Bayes solutions can diverge sharply—but here the structure of the regret matrix is sufficiently regular that both criteria point to the same method.

Finding 5: "Mixed" is the critical task. For 18 of 24 methods, the task of maximum regret is Mixed (MPAL detection). This task, with only 31 positive cases and subtle phenotypic overlap with other lineages, is where method selection matters most. Methods that cannot maintain performance on Mixed suffer disproportionate regret.

6.3 Statistical Characterization

To characterize the significance of TrimmedMean's minimax advantage, we examine the margin between it and the next-best method:

• TrimmedMean max regret: 0.007
• Second-best (SimpleAverage/TrimmedMean_2 tied): 0.013
• Gap: 0.006

The gap is nearly equal to TrimmedMean's own max regret, meaning the second-best method's worst case is roughly twice as bad. Among the top 5 methods, max regret ranges from 0.007 to 0.020—a factor of nearly 3×.

To assess robustness to estimation noise, we note that all AUROC values are averages over 10 × 5 = 50 cross-validation folds. Typical standard errors for AUROC in this setting are 0.005–0.010, meaning that the 0.007 minimax regret is at the boundary of statistical resolution. However, the ranking among method classes (simple aggregation ≻ selection-based ≻ learned) is robust to estimation noise, as the gaps between classes (0.007 vs. 0.034 vs. 0.056) far exceed typical standard errors.

6.4 Comparison of Decision Criteria

What does each decision criterion recommend?

In this empirical setting, all three criteria point to methods from the same family (trimmed mean aggregation), though they differ on the trimming parameter. This convergence is noteworthy—it suggests that trimmed mean aggregation occupies a uniquely robust position in the method space, performing well regardless of the criterion used to evaluate it.

However, the criteria diverge when comparing across method classes. AdaptiveSelect ranks 1st on 2 tasks by raw AUROC but 20th by minimax regret. A practitioner using mean AUROC and one using minimax regret would make the same choice (TrimmedMean), but a practitioner who "eyeballs" the performance table and picks the method with the most task-specific wins would choose AdaptiveSelect—a dramatically worse choice under task uncertainty.

7. Connection to the Forecast Combination Puzzle

7.1 The Puzzle

The forecast combination puzzle, extensively documented in the econometrics and forecasting literatures, refers to the robust empirical finding that simple averages of forecasts consistently outperform complex weighted combinations, even when the complex methods have access to the same information and are designed to exploit it optimally.

This finding, which has been replicated across macroeconomic forecasting, weather prediction, demand planning, and numerous other domains, remains something of an intellectual embarrassment for statistical learning theory. In principle, optimally weighted combinations should be at least as good as simple averages (the optimal weight reduces to uniform weighting when all forecasters are equally accurate and uncorrelated). In practice, the estimation error in the weights consistently overwhelms the theoretical benefit of optimization.

7.2 Our Results as an Instance of the Puzzle

Our findings provide a striking instance of this puzzle in the classification setting. The 15 ensemble methods form a natural complexity gradient:

1. Simple averages (4 methods): No parameters estimated from data. Fixed aggregation rules.
2. Selection-based (5 methods): Moderate complexity. Weights or selections based on inner-CV performance.
3. Learned aggregation (6 methods): High complexity. Weights estimated by regression or neural networks on base model predictions.

The minimax regret results show a monotonic relationship between complexity and worst-case regret:

The median max regret increases by a factor of 1.6× from simple to selection-based and 1.9× from selection-based to learned methods. The worst learned method (NeuralMeta_MLP, max regret = 0.081) has 11× the minimax regret of the best simple method (TrimmedMean, max regret = 0.007).

7.3 Why Does This Happen?

The forecast combination puzzle in our setting can be decomposed into three contributing mechanisms:

Mechanism 1: Estimation error amplification. Learned methods estimate aggregation weights from inner-CV performance. With 9 base models and limited training data per fold, these weight estimates are noisy. On well-separated tasks (Myeloid, AnyBlast), the noise is tolerable because multiple models perform well—even noisy weights combine good predictions effectively. On difficult tasks (Mixed, Erythroid), the noise is amplified because model performance is more heterogeneous and the signal is weaker. Simple averages avoid this amplification entirely.

Mechanism 2: Implicit overfitting to the training task distribution. Meta-learners trained within each outer fold see a particular case mix. If the inner training folds overrepresent easy cases (Myeloid, AnyBlast positives are more common), the learned weights will be optimized for these tasks. This creates a systematic bias toward task-specialized weights that hurts performance on underrepresented tasks. Simple averages have no mechanism for such bias.

Mechanism 3: The outlier-trimming advantage. TrimmedMean's specific advantage over SimpleAverage (max regret 0.007 vs. 0.013) arises from its robustness to base model outliers. On each task, 1–2 base models may be substantially worse than the rest (e.g., NaiveBayes on Mixed: 0.744 vs. RF_100 on Mixed: 0.788). The trimmed mean discards these outliers, preventing them from dragging down the ensemble. This provides a per-task adaptive quality without per-task parameter estimation—achieving some of the benefit of selection without the cost of learning.

7.4 The Bias-Variance Decomposition of Regret

We can formalize this intuition through a bias-variance decomposition of regret. For each method m and task t, the regret R[m, t] can be attributed to two sources:

1. Structural regret: The expected regret if the method's parameters were estimated with infinite data. For simple averages, this is the gap between the uniform combination and the oracle; for learned methods, it approaches zero (since with infinite data, optimal weights are recovered perfectly).

2. Estimation regret: The additional regret due to finite-sample estimation of method parameters. For simple averages, this is zero (no parameters to estimate). For learned methods, this is positive and increases with method complexity.

Total regret: R[m, t] = R_structural[m, t] + R_estimation[m, t]

The puzzle arises because R_estimation overwhelms R_structural for complex methods in finite-sample settings. Learned methods have near-zero structural regret but large estimation regret. Simple methods have modest structural regret but zero estimation regret. When R_estimation > R_structural (which occurs in our setting, with only 546 samples and 9 base models), simple methods win.

This explains why the complexity–regret relationship is monotonic: each additional estimated parameter adds estimation regret without sufficient reduction in structural regret. The inflection point where complex methods begin to dominate would require substantially larger sample sizes—likely thousands of samples per fold with diverse base model performance patterns.

7.5 Implications for Ensemble Design

The forecast combination puzzle, as manifested in our results, suggests a practical design principle for clinical ensemble systems:

Start with the simplest aggregation and only add complexity when you can demonstrate (via minimax regret, not just mean performance) that it helps.

Specifically: TrimmedMean provides a near-oracle baseline with max regret of 0.007. Any more complex method should be evaluated not by whether it improves mean AUROC but by whether it reduces max regret below this threshold. In our study, no complex method passes this test.

8. Robustness to Task Weighting

8.1 The Task Weighting Problem

The analysis in Sections 5 and 6 treats all tasks as equally important—the minimax criterion minimizes the worst regret across tasks without regard to task frequency or clinical significance. In practice, tasks may differ in importance. Pan-leukemia screening (AnyBlast) may be performed on every patient, while MPAL detection (Mixed) may be relevant only for a small fraction. Should the model selection account for this asymmetry?

8.2 Weighted Minimax Regret

We generalize the minimax regret criterion to incorporate task weights. Let w = (w₁, ..., w_T) be a weight vector with w_t ≥ 0 and Σw_t = 1. The weighted regret is:

$$R_w[m, t] = w_t \cdot R[m, t]$$

And the weighted minimax regret criterion selects:

$$m^* = \arg\min_m \max_t w_t \cdot R[m, t]$$

This formulation allows a continuum between equal-weight minimax (w uniform) and task-focused optimization (w concentrated on a single task).

8.3 Robustness Analysis

We examine how the minimax regret ranking changes under several clinically motivated weighting schemes:

Scheme 1: Prevalence-proportional weighting. Tasks are weighted by the number of positive cases (reflecting how often each diagnostic question arises):

w = (134/519, 89/519, 47/519, 31/519, 218/519) = (0.258, 0.171, 0.091, 0.060, 0.420)

Under this scheme, AnyBlast receives the highest weight and Mixed the lowest. The weighted minimax regret ranking:

• TrimmedMean remains rank 1 (weighted max regret = 0.003)
• SimpleAverage remains rank 2 (weighted max regret = 0.005)
• AdaptiveSelect improves (its high Mixed regret is down-weighted)

Scheme 2: Inverse-prevalence weighting. Tasks are weighted inversely by positive case count (emphasizing rare conditions that are harder to diagnose):

w ∝ (1/134, 1/89, 1/47, 1/31, 1/218) ≈ (0.110, 0.166, 0.314, 0.476, 0.068) (normalized)

Under this scheme, Mixed receives the highest weight. The weighted minimax regret ranking:

• TrimmedMean remains rank 1 (weighted max regret = 0.003)
• The gap to AdaptiveSelect increases (its Mixed regret is amplified)

Scheme 3: Clinical severity weighting. Tasks are weighted by clinical severity (MPAL and erythroid abnormalities are more diagnostically challenging and carry worse prognoses):

w = (0.15, 0.15, 0.25, 0.30, 0.15)

Under this scheme:

• TrimmedMean remains rank 1 (weighted max regret = 0.002)
• The separation between simple and complex methods increases

Scheme 4: Adversarial weighting. We compute, for each method, the task weight vector that maximizes its weighted regret, then find the method that is best under its own worst-case weighting. This is equivalent to the standard (unweighted) minimax regret since the adversary can always put all weight on the task of maximum regret.

8.4 Stability Result

Proposition. TrimmedMean is the minimax-regret-optimal method for every weight vector w with w_t > 0 for all t.

Proof sketch. TrimmedMean has the smallest maximum regret entry (0.007) and the smallest regret entry on every task except Mixed (where TrimmedMean_2 ties at 0.000 vs. TrimmedMean's 0.004). For any weight vector with all-positive entries, the weighted max regret is determined by a convex combination of task regrets, and TrimmedMean's dominance in the max entry ensures it remains optimal for any strictly positive weighting.

More precisely: for TrimmedMean to lose rank 1, there must exist a weighting where another method's weighted max regret is lower. But any method with a higher max regret entry (≥ 0.013 for all alternatives) can only match TrimmedMean's weighted max regret if the weight on that method's worst task is sufficiently low. When all weights are strictly positive, the gap is never fully closed because TrimmedMean's advantage on the worst task (at most 0.007 regret) provides a uniform buffer.

This stability result is notable. It means that TrimmedMean is the minimax-regret-optimal choice regardless of how much the practitioner values each task, provided they value every task at least a little. The recommendation is insensitive to subjective task-importance judgments—a strong practical property.

8.5 Sensitivity to Individual Task Removal

We also examine robustness by leave-one-task-out analysis. Removing each task and recomputing the minimax regret:

TrimmedMean's max regret is 0.007 regardless of which task is removed, because its two worst tasks (Myeloid and AnyBlast, both at 0.007) always survive. This remarkable stability arises from TrimmedMean's nearly flat regret profile—it has no single vulnerability point.

9. Extension: Adaptive Minimax

9.1 Motivation

Pure minimax regret assumes complete uncertainty about which task will be encountered. In practice, partial information may be available. A hospital might know that it sees predominantly myeloid cases, without knowing the exact distribution. Can we exploit this partial knowledge while retaining robustness guarantees?

9.2 The Adaptive Framework

We propose a framework that interpolates between minimax regret (full uncertainty) and Bayes-optimal selection (known distribution) using an uncertainty parameter λ ∈ [0, 1]:

Step 1: Estimate a task distribution. Given observed task frequencies p = (p₁, ..., p_T), estimate the expected regret for each method under p:

$$\bar{R}_p(m) = \sum_t p_t \cdot R[m, t]$$

Step 2: Define the uncertainty set. Let Π_λ be the set of distributions within distance λ of the observed distribution p (using, e.g., total variation distance or KL divergence):

$$\Pi_\lambda = {q : d(q, p) \leq \lambda}$$

When λ = 0, Π contains only p (pure Bayesian). As λ → ∞, Π contains all distributions (pure minimax regret).

Step 3: Distributionally robust selection. Select the method minimizing worst-case expected regret over the uncertainty set:

$$m^* = \arg\min_m \max_{q \in \Pi_\lambda} \sum_t q_t \cdot R[m, t]$$

This formulation connects to the recent literature on distributionally robust optimization, which provides efficient computational methods for solving such problems.

9.3 Illustrative Analysis

We illustrate the adaptive framework with a concrete scenario. Suppose a hospital observes the task distribution p = (0.35, 0.25, 0.15, 0.10, 0.15) (Myeloid-dominant case mix) and considers uncertainty budgets λ ∈ {0, 0.1, 0.2, 0.5, ∞} (in total variation distance).

For each λ, we compute the distributionally robust optimal method:

In this empirical setting, TrimmedMean is the optimal choice at every uncertainty level. This is because its regret profile is so flat (max entry 0.007, mean entry 0.004) that even large distributional shifts cannot create a scenario where another method is preferred.

9.4 When Adaptation Matters

The adaptive framework becomes non-trivial (i.e., the optimal method changes with λ) when the Bayes-optimal and minimax-regret-optimal methods differ. This would occur if, for example, one method had very low regret on high-frequency tasks but high regret on rare tasks, while another had uniform moderate regret.

To illustrate, consider a hypothetical scenario where the hospital exclusively sees Myeloid and AnyBlast cases (p₁ = p₅ = 0.5, others = 0). Under pure Bayesian selection, AdaptiveSelect would be optimal (zero regret on both tasks). As uncertainty increases and the hospital acknowledges it might see other task types, the optimal choice would smoothly transition to TrimmedMean.

The transition point—the uncertainty budget λ* at which the optimal method changes—provides a natural measure of how "robust" the Bayesian recommendation is. A low λ* means even small distributional uncertainty invalidates the Bayesian choice; a high λ* means the Bayesian choice is robust.

9.5 Practical Implementation

For practitioners, we recommend the following procedure:

1. Compute the regret matrix for the candidate methods and tasks.
2. Estimate the task distribution from historical data (e.g., last 6 months of cases).
3. Set the uncertainty budget based on expected distributional shift. For stable clinical environments, λ ≈ 0.1–0.2; for new deployments or changing demographics, λ ≈ 0.5–1.0.
4. Solve the distributionally robust problem (a linear program for TV-distance uncertainty sets, or a convex program for KL-divergence sets).
5. Select the method that minimizes worst-case expected regret under the chosen uncertainty set.

If the optimal method is insensitive to λ (as in our empirical setting), the practitioner can deploy with high confidence. If the optimal method changes with λ, the transition points provide actionable information about which assumptions drive the recommendation.

9.6 Connection to Robust Optimization

The adaptive minimax framework is a special case of distributionally robust optimization (DRO), a field that has grown substantially in recent years. Standard DRO considers an uncertainty set over the joint distribution of features and labels. Our formulation is simpler: the uncertainty is only over which task (distribution) will be encountered, not over the within-task data-generating process. This makes the optimization problem dramatically easier—a finite-dimensional linear or convex program rather than an infinite-dimensional functional optimization.

The simplicity of our formulation is deliberate. We argue that for model selection, task-level uncertainty is the primary concern. Within-task uncertainty (e.g., sampling variability in the test set) is addressed by standard cross-validation and confidence intervals. The regret matrix framework cleanly separates these two sources of uncertainty.

10. Limitations and Conclusion

10.1 Limitations

Limited task diversity. Our empirical analysis involves 5 tasks from a single clinical domain (hematological diagnostics via flow cytometry). While the tasks span a range of difficulty and class imbalance, they share the same feature representation and patient cohort. The generality of our findings—particularly the dominance of simple aggregation methods—requires validation on more diverse task collections spanning different clinical domains, modalities, and data types.

Moderate sample size. With 546 samples and 31–218 positives per task, our AUROC estimates carry standard errors of approximately 0.005–0.010. The minimax regret of 0.007 for TrimmedMean is at the boundary of this estimation uncertainty. Larger datasets would provide sharper discrimination among methods with similar regret profiles.

Shared feature representation. All methods use the same 128-dimensional embedding. This eliminates feature engineering as a source of variation but also means our results do not capture the model selection problem in its fullest form, where different methods might benefit from different feature representations.

Static analysis. Our framework considers a fixed set of methods and tasks. It does not address the dynamic setting where new methods or tasks emerge over time, or where the regret matrix itself is estimated with uncertainty. Extending the framework to sequential or online settings would be a natural next step.

No formal hypothesis testing for regret differences. While we provide standard errors for AUROC estimates, we do not perform formal hypothesis tests for whether one method's minimax regret is significantly lower than another's. Developing appropriate statistical tests for regret-based comparisons is an open problem. Bootstrap or permutation approaches could potentially address this, and we identify this as important future work.

Task independence assumption. The minimax regret framework treats tasks as independent states of nature. In practice, tasks may be correlated (e.g., a patient with myeloid blasts is more likely to also have abnormal erythroid precursors). Incorporating task correlations into the decision-theoretic framework is a direction for future research.

10.2 Related Decision-Theoretic Perspectives

Our work connects to several intellectual traditions beyond the classical Wald-Savage framework:

Competitive analysis in online learning. The competitive ratio in online algorithms plays a role analogous to regret in our setting—it measures performance relative to an omniscient adversary rather than in absolute terms. Our offline regret matrix analysis can be viewed as a batch version of the expert advice problem, where the "experts" are prediction methods and the "rounds" are tasks.

Worst-case-aware machine learning. Growing interest in distributional robustness in machine learning—addressing performance under distribution shift, group fairness, and tail risk—shares our motivation that average-case optimization is insufficient for deployment in safety-critical settings. Our contribution is to connect this concern to classical decision theory, providing a principled criterion rather than ad hoc robustness measures.

Portfolio theory. The model selection problem under task uncertainty is structurally similar to portfolio selection under return uncertainty. Each method is an "asset" with task-dependent returns (AUROC). The minimax regret portfolio minimizes worst-case underperformance relative to the best asset in each market condition. The forecast combination puzzle in our setting corresponds to the well-documented finding that naive 1/N portfolio allocation often outperforms optimized portfolios—another instance of estimation error overwhelming optimization benefit in finite samples.

10.3 Practical Recommendations

Based on our analysis, we offer the following recommendations for practitioners deploying classification systems in multi-task clinical settings:

1. Default to trimmed mean aggregation when task uncertainty is present. In our study, TrimmedMean provided near-oracle performance (max regret 0.007) with zero tunable parameters beyond the trimming depth.

2. Construct the regret matrix rather than relying on average performance tables. The regret matrix reveals worst-case vulnerabilities that averages conceal. AdaptiveSelect had the best AUROC on 2 of 5 tasks but the second-worst minimax regret among ensemble methods.

3. Be skeptical of complex ensembles in small-to-moderate data regimes. The consistent increase in minimax regret with aggregation complexity suggests that learned combination weights are unreliable when sample sizes are limited relative to the number of base models.

4. Use the adaptive minimax framework when partial knowledge of the deployment distribution is available. If the optimal method is insensitive to the uncertainty budget (as we observed), deploy with confidence. If it is sensitive, invest in better characterizing the deployment distribution before committing to a method.

5. Report minimax regret alongside mean performance in method comparison studies. A method with 2% higher mean AUROC but 10× higher minimax regret may be inappropriate for deployment under task uncertainty.

10.4 Conclusion

Model selection is a decision problem. The standard practice of selecting by average performance implicitly adopts the Bayes criterion with a uniform prior over tasks—an assumption that is often unjustified and rarely examined. We have shown that reformulating model selection through Savage's minimax regret criterion reveals structure in multi-task evaluations that aggregate metrics obscure.

In our empirical study, the minimax regret analysis yields a clear and robust recommendation: TrimmedMean ensemble aggregation, with a worst-case performance gap of 0.007 AUROC relative to the task-specific oracle. This recommendation is stable across task weightings, task removal, and uncertainty levels—a degree of robustness that no complex method matches.

The deeper lesson is that being the best model for any single task and being the best model for every task are fundamentally different objectives, and they often conflict. Methods that specialize (AdaptiveSelect, GBM_50) achieve impressive task-specific performance at the cost of vulnerability elsewhere. Methods that hedge (TrimmedMean, SimpleAverage) sacrifice peak performance for uniform near-optimality. When the deployment task is uncertain—as it usually is in clinical practice—the hedging strategy is the decision-theoretically correct choice.

This is not a new insight. Wald, Savage, and their contemporaries established the mathematical foundations in the mid-twentieth century. The forecast combination literature has documented the empirical pattern for decades. What is new is the explicit application of these classical ideas to the modern model selection problem, with a concrete demonstration that decision theory provides better deployment recommendations than standard ML evaluation practice.

We hope this work encourages the machine learning community to engage more seriously with decision theory—not as an abstract mathematical curiosity, but as a practical framework for making better choices under the uncertainty that invariably accompanies real-world deployment.