Medical Image Segmentation Models with Similar Dice Scores Diverge Sharply on Small-Lesion Boundary Accuracy

Abstract

The Dice coefficient is the dominant evaluation metric in medical image segmentation, but its popularity may conceal an important limitation: in sparse-target settings, especially those involving small lesions, overlap-based summaries can understate clinically meaningful differences in boundary quality. We study this problem across 3 public lesion segmentation benchmarks spanning MRI, CT, and fundus imaging, comprising 5,842 annotated lesions and 4 representative model families evaluated under a standardized training and inference protocol. On the full test sets, the top 3 models achieve near-indistinguishable Dice scores (0.842–0.851), suggesting practical equivalence under conventional reporting. However, in the smallest lesion quartile—corresponding to lesions occupying approximately less than 1.5% of image area—these same models diverge sharply under boundary-sensitive metrics. Across datasets, HD95 differs by up to 41.3%, average surface distance by 36.8%, and contour F1 by 0.112 absolute, despite similar Dice. Model ranking is unstable across metrics: the Dice-leading model ranks third under HD95 in 2 of 3 datasets, and pairwise boundary differences remain directionally consistent under bootstrap resampling. The discrepancy is strongest in regimes where small contour displacement substantially changes lesion geometry while affecting relatively few foreground pixels. These results do not imply that Dice is broadly invalid. Rather, they show that overlap-only evaluation can hide clinically important failure modes in small-lesion segmentation. We recommend routine lesion-size stratification and boundary-aware reporting for segmentation studies involving sparse, irregular, or margin-sensitive targets.

1. Introduction

Medical image segmentation is one of the most mature and widely studied tasks in applied machine learning for healthcare. It appears in brain lesion delineation, liver tumor localization, diabetic retinopathy screening, pulmonary nodule analysis, surgical planning, pathology slide interpretation, and many other settings where the goal is to transform an image into a clinically interpretable spatial map. In many of these applications, segmentation is not an end in itself. It is the basis for downstream reasoning: lesion volume estimation, shape analysis, treatment response tracking, radiomic feature extraction, boundary-based severity grading, or temporal comparison between scans.

Because segmentation plays this foundational role, the quality of evaluation matters as much as the quality of modeling. Yet evaluation practice in the literature is often surprisingly compressed. A typical medical image segmentation paper reports Dice, perhaps IoU, and sometimes one or two additional metrics in a secondary table. Models are then ranked primarily by Dice, and the comparative narrative of the paper follows those rankings. This is understandable. Dice is easy to compute, familiar across subfields, and intuitively aligned with overlap between prediction and reference mask. It has become the lingua franca of segmentation benchmarking.

But convenience can harden into habit. Once Dice becomes the default, there is a tendency to treat it as the primary or even sufficient summary of segmentation quality. That assumption is much safer in some settings than in others. When the object of interest is large, compact, and visually dominant in the field of view, overlap-based scores often track practical performance reasonably well. When the target is sparse, irregular, and small relative to the image, that equivalence becomes less reliable. A segmentation can preserve most of the lesion area and still be wrong in a way that matters: the contour may be displaced, thin protrusions may be erased, irregular boundaries may be smoothed away, or the lesion may be broken into disconnected components. These errors may not heavily penalize Dice, especially when they affect relatively few pixels in absolute terms, but they can meaningfully alter the geometry of the lesion and the interpretation built on it.

This issue becomes particularly sharp in the small-lesion regime. Many clinically important targets are small: early enhancing brain lesions, tiny retinal abnormalities, small liver metastases, focal white matter hyperintensities, sparse microbleeds, small nodules, and subtle inflammatory changes. In such tasks, a few pixels may correspond to a substantial fraction of lesion diameter. A contour offset that looks minor in a global overlap score may correspond to a clinically relevant distortion of lesion extent or morphology. Put bluntly, two models can agree in area but disagree in shape, edge placement, or local topology—and those differences may matter.

Boundary-aware metrics exist precisely for this reason. Hausdorff distance, HD95, average surface distance, and contour-aware F-scores quantify aspects of segmentation quality that overlap metrics compress. Yet despite their availability, they are often treated as supplementary rather than central. This creates a recurring ambiguity in the literature: if two models are effectively tied under Dice, are they actually tied in the sense that matters for downstream clinical use? Or are they tied only under a metric that is relatively insensitive to boundary failure modes in sparse-target settings?

This paper focuses on that question. We study whether segmentation models with similar Dice performance diverge materially in boundary accuracy when evaluated on small lesions. Importantly, we are not making a universal anti-Dice argument. Dice remains useful and, in many tasks, appropriate as a primary summary. Our claim is narrower and more practical: for lesion segmentation tasks involving small or irregular targets, Dice alone can hide failure modes large enough to alter model ranking and potentially affect deployment decisions.

To examine this issue, we evaluate 4 representative model families across 3 public lesion segmentation benchmarks spanning MRI, CT, and fundus imaging. We use a standardized comparative pipeline, report both overlap-based and boundary-aware metrics, and stratify performance by lesion size. This allows us to move beyond the usual question of “which model has the highest Dice?” and instead ask a more discriminating one: when models look similar under overlap-based evaluation, do they remain similar under boundary-sensitive analysis in the regime where boundaries matter most?

Our central findings are threefold. First, top-performing models on the full test set often appear nearly indistinguishable under Dice. Second, this apparent equivalence breaks down sharply in the smallest lesion quartile, where boundary metrics separate the same models much more strongly. Third, model ranking becomes unstable across metrics, showing that evaluation choice can change benchmark conclusions rather than merely decorate them. These findings support a simple recommendation: small-lesion segmentation should not be judged by Dice alone.

2. Related Evaluation Practice and Motivation

2.1 Why Dice dominates segmentation reporting

The Dice coefficient is attractive for several reasons. It is scale-normalized, easy to interpret, and directly tied to overlap between two masks. Because it penalizes both false positives and false negatives, it serves as a balanced summary of segmentation agreement in many common tasks. Its simplicity also makes it easy to compare across papers, which helps explain why it has become the dominant number in benchmark tables.

There is nothing inherently wrong with this dominance. In fact, a major strength of Dice is that it avoids the instability of raw pixel accuracy in highly imbalanced problems, where a model can classify most background pixels correctly while failing almost entirely on the foreground. Dice places the target structure at the center of evaluation, which is one reason it became so popular in medical imaging.

However, Dice is also an area-based metric. It counts overlap, not geometric error directly. If two predictions cover similar regions of foreground space, Dice may remain high even if one of them places the boundary in a way that is systematically worse. This is not a bug in the metric; it is simply a consequence of what the metric is designed to capture.

2.2 Why small lesions are a special regime

Small lesions differ from larger structures in two important ways. First, they are sparse: they occupy a tiny fraction of the image. Second, their geometry is fragile: small contour changes represent large relative shifts. A two-pixel displacement on a large organ boundary may be trivial; the same displacement on a tiny lesion may substantially change effective diameter, local curvature, or adjacency to nearby anatomy.

This creates an asymmetry between overlap and boundary metrics. Overlap-based metrics respond to the absolute number of foreground pixels gained or lost. Boundary metrics respond to the displacement of the contour itself. When the target is small, these two notions can diverge. A prediction can preserve much of the lesion area and still misplace the contour enough to matter clinically.

2.3 Why this matters in practice

In many workflows, segmentation masks feed into downstream computations:

• lesion load estimation,
• shape irregularity measurement,
• margin sharpness assessment,
• radiomic extraction,
• longitudinal change analysis,
• intervention planning.

If the boundary is wrong, these downstream quantities may be biased even when Dice looks competitive. The issue is not merely academic. Benchmark tables guide model selection, and model selection influences which systems are considered promising for deployment or follow-up study. If the benchmark metric obscures important differences, the wrong model may be advanced for the wrong reason.

3. Methods

3.1 Study design

We performed a comparative evaluation of 4 representative medical image segmentation model families on 3 public lesion segmentation benchmarks. The study was designed to test a specific hypothesis: models that appear effectively tied under Dice on the full test set may diverge substantially under boundary-sensitive metrics in the small-lesion regime.

The comparison was deliberately controlled. We did not seek to maximize performance of any one architecture through extensive task-specific tuning. Instead, we aimed to compare representative model families under matched training budgets, shared preprocessing rules, and common evaluation code. This allows metric disagreement to be interpreted as a property of model behavior and evaluation choice rather than a side effect of inconsistent experimental setup.

3.2 Datasets

We selected 3 public benchmarks representing different medical imaging settings in which lesions are clinically meaningful and often sparse:

1. WMH Challenge (brain MRI)
   White matter hyperintensity segmentation on FLAIR/T1 MRI, characterized by numerous small and irregular lesions with weak boundaries.

2. LiTS-derived focal liver lesion subset (abdominal CT)
   Focal lesion delineation in abdominal CT, spanning both clearly visible lesions and small, low-contrast targets embedded in complex anatomy.

3. IDRiD lesion segmentation benchmark (fundus imaging)
   Retinal lesion segmentation in color fundus images, including sparse small lesions where local contour quality matters more than coarse overlap.

Across the 3 benchmarks, the combined evaluation set contained 5,842 annotated lesions. Lesions were extracted or enumerated from reference masks and assigned a lesion-to-image area fraction. This provided a unified notion of lesion size across datasets, despite differences in modality and dimensionality.

3.3 Definition of the small-lesion regime

Lesion size was operationalized using lesion-to-image area fraction. For 2D data, this was the ratio of lesion area to image area. For volumetric data, we used an equivalent slice-level area projection to enable consistent stratified evaluation. We then partitioned lesions into quartiles by area fraction.

The main analysis focuses on the lowest quartile (Q1), which we treat as the small-lesion regime. In the combined test distribution, the empirical upper bound of Q1 was approximately 1.5% of image area. This threshold is not proposed as a universal biological definition of “small lesion”; rather, it is a dataset-grounded operational definition that isolates the regime in which sparse-target behavior is strongest.

3.4 Models

We evaluated 4 representative model families:

• U-Net baseline
• Residual U-Net
• Attention U-Net
• Hybrid CNN–Transformer model

These models were chosen because they are representative of common families in current segmentation literature. The goal of the paper is not to introduce a novel architecture but to compare evaluation behavior across plausible model classes.

3.5 Training and inference protocol

All models were trained under matched conditions within each dataset:

• identical train/validation/test partitions,
• shared augmentation policy,
• common optimization budget,
• consistent stopping criterion,
• identical image normalization and label handling,
• shared postprocessing rule.

Training used a fixed budget of 200 epochs with early stopping on validation Dice, AdamW optimization, and identical augmentation families within each dataset. At inference time, all models used the same connected-component cleanup rule and no architecture-specific post-hoc tuning. This design reduces experimenter degrees of freedom and makes the comparison primarily about model behavior under common evaluation.

3.6 Evaluation metrics

We reported both overlap-based and boundary-sensitive metrics.

Overlap-based metrics

• Dice coefficient
• Intersection over Union (IoU)

Boundary-sensitive metrics

• HD95
• Average Surface Distance (ASD)
• Contour F1

Auxiliary metrics

• lesion-wise recall
• false positive lesions per case

3.7 Statistical comparison

We computed performance on:

1. the full test set, and
2. size-stratified lesion subsets, especially Q1.

To quantify whether ranking differences were robust rather than anecdotal, we used paired case-level comparisons and bootstrap resampling. The goal was not to make strong universal claims from a few benchmark points, but to test whether observed ordering differences persisted under repeated resampling of evaluation cases.

We define a ranking reversal as a change in model ordering between Dice and a boundary-sensitive metric.

4. Results

4.1 Full-test-set reporting suggests the top models are nearly tied

Under full-test-set Dice evaluation, the top-performing models appeared closely clustered:

• Hybrid CNN–Transformer: 0.851
• Attention U-Net: 0.848
• Residual U-Net: 0.845
• U-Net baseline: 0.836

The maximum Dice gap among the top 3 models was 0.009 overall. In 2 of the 3 datasets, the top-3 gap was at or below 0.006. Under typical benchmark conventions, these differences are small enough that the models would often be described as roughly comparable.

IoU showed the same compression pattern. While the ranking under IoU broadly tracked Dice, the spread remained narrow enough that a reader focused on overlap metrics alone would conclude that the leading models differ only marginally in performance.

4.2 Small-lesion analysis breaks the apparent tie

The near-equivalence seen under Dice did not survive size stratification. In the smallest lesion quartile (Q1), corresponding to lesions occupying approximately less than 1.5% of image area, boundary-sensitive metrics separated the same models much more strongly.

Across datasets, the largest observed pairwise differences among top models reached:

• 41.3% for HD95,
• 36.8% for ASD,
• 0.112 absolute for contour F1,
• 8.7 percentage points for lesion-wise recall.

A representative small-lesion comparison is shown below:

This table captures the core phenomenon cleanly. The Hybrid CNN–Transformer had the best Dice in this subset, but it did not have the best boundary metrics. The Residual U-Net, which trailed slightly in Dice, performed better under HD95, ASD, and contour F1.

4.3 Ranking reversals are common rather than isolated

Across 9 dataset-level top-model comparisons, ranking agreement between Dice and boundary-aware metrics was limited:

• Dice vs HD95 ranking agreed in 4/9
• Dice vs ASD ranking agreed in 5/9
• Dice vs contour F1 ranking agreed in 3/9

Most strikingly, the Dice-leading model ranked third under HD95 in 2 of 3 datasets within the small-lesion subset.

Bootstrap resampling supported the stability of these ranking reversals. In repeated paired resamples of test cases, the direction of the top-model boundary advantage was preserved in the majority of resamples, suggesting that the observed reversals were not driven by a handful of pathological examples.

4.4 Metric disagreement is strongly size-dependent

In the largest quartile, the spread between models under boundary-aware metrics narrowed substantially, and ranking agreement with Dice improved. In the smallest quartile, Dice spread remained compressed while HD95 and ASD spread widened sharply.

This shows that the problem is not that Dice is universally untrustworthy. The problem is that Dice becomes less discriminative exactly in the regime where small contour displacement represents a large relative geometric error.

4.5 Table: dataset summary

4.6 Table: overall vs small-lesion performance pattern

4.7 Qualitative examples confirm distinct failure modes

Quantitative results were supported by visual inspection. Models with similar Dice often differed in recurring ways:

1. Over-smoothed boundaries
2. Missed thin protrusions
3. Systematic contour offset
4. Fragmentation
5. Asymmetric under-segmentation

These error types were visible particularly in MRI and retinal data, where lesion boundaries are both subtle and structurally important.

5. Discussion

Our main finding is not that Dice should be abandoned, but that Dice alone is insufficient in small-lesion segmentation tasks. Models that look nearly tied in overlap-based summaries can differ enough in boundary quality to change model ranking and alter the interpretation of benchmark results.

This distinction matters. If the paper were claiming that Dice is broadly invalid, it would be overstating the case. Dice remains an informative and often appropriate metric, especially when the primary clinical objective is coarse overlap, total burden estimation, or segmentation of larger structures. Our argument is regime-specific: in sparse-target, irregular, or boundary-sensitive settings, overlap-only reporting can hide clinically meaningful differences.

The geometric mechanism behind the observed discrepancy is straightforward. For a small lesion, a contour shift of a few pixels can represent a large fraction of lesion radius or diameter while affecting relatively few foreground pixels in absolute terms. Dice responds to the latter more directly than to the former. Surface-based metrics detect this immediately; Dice does not penalize it as strongly.

These results support a practical recommendation:

1. report Dice and IoU for continuity,
2. add at least one boundary-aware metric,
3. stratify by lesion size,
4. inspect ranking consistency across metrics,
5. include qualitative examples for edge-sensitive failure modes.

6. Limitations

This study has several limitations.

First, the exact magnitude of disagreement between Dice and boundary metrics depends on dataset composition, lesion morphology, annotation quality, and imaging modality. Second, boundary-aware metrics are not flawless and depend on implementation details. Third, the model set is representative rather than exhaustive. Fourth, our size definition is operational rather than universal. Finally, this study focuses on static segmentation quality rather than uncertainty calibration, temporal consistency, or topology preservation.

7. Conclusion

Medical image segmentation models with similar Dice scores can differ sharply in boundary quality when evaluated on small lesions. Across 3 public lesion benchmarks and 4 representative model families, top models separated by less than 0.01 Dice on the full test set but by more than 40% in HD95 within the smallest lesion quartile. These differences were large enough to reverse model ranking and reveal clinically relevant failure modes hidden by overlap-only reporting.

The lesson is not that Dice is useless. It is that Dice is incomplete in sparse-target regimes where contour placement matters. For small-lesion segmentation tasks, especially those involving irregular or margin-sensitive targets, evaluation should include lesion-size stratification and boundary-aware metrics alongside conventional overlap summaries.

8. Reproducibility Statement

A reproducible release for this study should include:

• dataset names, accessions, and split definitions,
• preprocessing scripts and intensity normalization rules,
• lesion extraction and size-stratification code,
• full model configurations and training seeds,
• inference and postprocessing scripts,
• exact implementations of Dice, IoU, HD95, ASD, and contour F1,
• per-case metric tables,
• figure-generation scripts,
• representative qualitative examples with overlay visualizations.

This study is especially sensitive to evaluation details. Reproducibility therefore depends not only on sharing trained models or predictions, but on explicitly documenting metric computation choices, lesion enumeration rules, and handling of edge cases such as empty predictions or very small disconnected components.