1. Introduction

The sentence embedding model selection problem has become a central concern for practitioners building semantic search, retrieval-augmented generation, and document comparison systems. The Massive Text Embedding Benchmark (MTEB) leaderboard tracks hundreds of models, and new contenders appear weekly, each claiming incremental improvements in retrieval accuracy, clustering quality, or semantic textual similarity. Practitioners spend considerable engineering time evaluating candidate models against their domain-specific data, optimizing threshold parameters, and fine-tuning for marginal gains.

But what if the critical bottleneck is not which model you choose, but rather the types of semantic distinctions your system must make?

This question has received surprisingly little systematic attention. Individual failure modes — negation insensitivity, numerical blindness, temporal confusion — have been documented in isolation (Reimers and Gurevych, 2019; Devlin et al., 2019), but no study has quantified their relative contributions to overall performance variance in a unified framework. Without this decomposition, practitioners lack the information needed to allocate their engineering effort efficiently.

In this work, we apply two-way analysis of variance (ANOVA) to similarity scores from a controlled experiment crossing 4 embedding models with 8 semantic perturbation categories, producing 32 cell means from 800 total sentence pairs. The ANOVA framework allows us to partition the total variance in cosine similarity scores into three additive components: (1) variance attributable to category type, (2) variance attributable to model choice, and (3) variance attributable to the model × category interaction. The results consistently show that category type dominates: in the full 4 × 8 design, category explains 72.2% of variance versus 9.3% for model choice (an 8:1 ratio). A robustness analysis excluding control categories — which contribute disproportionate variance due to fundamentally different embedding space geometries across models — yields a more conservative ratio of approximately 1.5:1, with category type still explaining substantially more variance than model choice.

This finding has immediate practical consequences. If you are deploying a RAG system in a medical domain where negation errors could be life-threatening, switching from MiniLM to GTE will improve your negation similarity gap from 0.889 to 0.941 — a helpful but modest improvement. Meanwhile, the gap between your system's ability to detect negation errors (similarity ~0.92) versus entity swap errors (similarity ~0.99) remains enormous regardless of which model you choose. The type of error defines the boundary of what is achievable; the model selection determines where within that boundary you operate.

The remainder of this paper develops this argument rigorously. Section 2 reviews the statistical framework of ANOVA and effect sizes as applied to embedding evaluation. Section 3 describes our experimental data. Section 4 presents the full variance decomposition. Sections 5 and 6 examine the category hierarchy and model effects in detail. Section 7 analyzes the interaction term. Section 8 discusses practical implications. Section 9 extends the analysis to cross-encoder models. Section 10 acknowledges limitations, and Section 11 concludes.

2. Background: ANOVA for Embedding Evaluation

2.1 The Factorial Design

Embedding model evaluation is naturally framed as a factorial experiment. Each model is tested against each perturbation category, producing a matrix of performance scores. The standard approach — comparing models on an aggregate metric — collapses this matrix into a single column, discarding the category dimension entirely. This is equivalent to computing only the row (model) marginal means while ignoring column (category) effects.

A two-way ANOVA preserves the full factorial structure. Let Y_ij denote the mean cosine similarity for model i and category j. The model decomposes this as:

Y_ij = μ + α_i + β_j + (αβ)_ij

where μ is the grand mean, α_i is the model effect, β_j is the category effect, and (αβ)_ij is the interaction term. The total sum of squares is partitioned:

SS_total = SS_model + SS_category + SS_interaction

Each component's proportion of the total — expressed as the eta-squared (η²) effect size — quantifies the relative importance of that factor.

2.2 Eta-Squared as an Effect Size

Eta-squared (η²) is defined as the ratio of a factor's sum of squares to the total sum of squares:

η² = SS_factor / SS_total

Cohen's (1988) conventional benchmarks classify η² values as small (0.01), medium (0.06), and large (0.14). These thresholds were developed for behavioral science experiments where effects are typically small. In applied engineering contexts, effects are often larger because the experimental factors are specifically chosen to span known sources of variation.

An η² of 0.72 — as we observe for category type — is extraordinary by any standard. It indicates that if you knew only the category of a semantic perturbation and nothing about which model was used, you could predict 72% of the variance in cosine similarity scores. Conversely, knowing only the model (η² = 0.09) would let you predict less than 10% of the variance.

2.3 Why ANOVA Rather Than Pairwise Comparison

The typical evaluation paradigm for embedding models involves pairwise model comparisons: "Model A achieves STS score X, Model B achieves Y, therefore Model A is better." This approach has two limitations.

First, it ignores the internal structure of the evaluation data. If Model A outperforms Model B on negation but underperforms on entity swap, the aggregate score conceals this critical interaction. The ANOVA interaction term explicitly captures these crossover patterns.

Second, pairwise comparison provides no framework for assessing the relative importance of different sources of variation. A model comparison showing a "statistically significant" improvement of 0.02 in mean cosine similarity is misleading if the category-level differences are 0.50 or larger. Without the variance decomposition, the practitioner has no way to contextualize the magnitude of model differences relative to category differences.

2.4 Related Work in Embedding Evaluation

The evaluation of sentence embedding models has been dominated by benchmark-centric approaches. The STS Benchmark (Cer et al., 2017) provides human-annotated similarity scores for sentence pairs, but aggregates across semantic types. The MTEB benchmark suite (Muennighoff et al., 2022) evaluates models across dozens of tasks but does not decompose within-task variance.

Several studies have examined specific failure modes. Ettinger (2020) demonstrated that BERT-style models struggle with negation understanding. Naik et al. (2018) created stress tests for natural language inference that include numerical and quantifier perturbations. However, these studies evaluate failure modes in isolation rather than within a unified variance decomposition framework.

The closest methodological precedent is the use of ANOVA in psycholinguistic research to decompose reading time variance into participant, item, and interaction components (Clark, 1973). We adapt this approach to the embedding evaluation setting, where "participants" are models and "items" are perturbation categories.

3. Experimental Data

3.1 Models

We evaluate four sentence embedding models spanning a range of architectures and parameter counts:

Table 1: Model characteristics

These models were selected to span a representative range of commonly deployed embedding models. MiniLM represents the lightweight, latency-optimized end of the spectrum. BGE and GTE represent the high-capacity end. Nomic occupies an intermediate position. All models use mean pooling over the final hidden layer to produce fixed-size sentence embeddings.

3.2 Perturbation Categories

We construct sentence pairs across 8 categories designed to test different semantic discrimination abilities:

Table 2: Perturbation category descriptions

The six perturbation categories (entity swap through hedging) contain pairs that differ in meaning despite surface-level similarity. An ideal embedding model would assign low similarity to all perturbation pairs and high similarity to positive controls, with negative controls receiving the lowest scores. In total, we evaluate 800 sentence pairs (100 per category × 8 categories) across all 4 models, producing 3,200 individual similarity scores that aggregate to 32 cell means for the ANOVA.

3.3 Sentence Pair Construction

All sentence pairs were constructed following controlled perturbation principles. For each perturbation category, one sentence in each pair serves as the anchor, and the second sentence is created by applying a minimal semantic change specific to that category. This controlled construction ensures that observed similarity scores reflect the model's sensitivity to the specific perturbation type rather than incidental differences in vocabulary or sentence structure.

Table 2b: Representative sentence pairs from each category

Each category contains 100 pairs following analogous construction patterns. The full sentence pair dataset (800 pairs) is included as a supplementary dataset in the SKILL.md companion code.

Positive control pairs are drawn from standard semantic similarity datasets, consisting of genuinely paraphrastic sentence pairs. Negative control pairs are constructed by randomly pairing sentences from different topics with no semantic overlap.

3.4 Computing Cell Means

For each model × category combination, we compute the mean cosine similarity across all 100 sentence pairs in that cell. This produces the 4 × 8 matrix of cell means that serves as input to the ANOVA. The grand mean across all 32 cells is 0.860.

4. Two-Way ANOVA: The Variance Decomposition

4.1 The 4 × 8 Cell Mean Matrix

Table 3: Mean cosine similarity by model and category

This matrix reveals two immediately visible patterns. First, the column (model) means increase monotonically from MiniLM (0.767) to GTE (0.921), but the range is only 0.154. Second, the row (category) means vary from 0.449 (negative controls) to 0.990 (entity swap), a range of 0.541 — more than 3.5× the model range. The ANOVA formalizes this visual impression.

4.2 Sum of Squares Decomposition

The two-way ANOVA decomposes the total sum of squares as follows:

Table 4: ANOVA summary table

The category factor accounts for 72.2% of total variance (η² = 0.72). By Cohen's (1988) conventions, this is a large effect — indeed, it is enormous, exceeding the "large" threshold by more than 5×. The model factor accounts for 9.3% (η² = 0.09), which qualifies as a medium effect. The interaction accounts for the remaining 18.5%.

4.3 Interpreting the Ratio

The ratio of η² values — 0.72 / 0.09 = 8.0 — provides a clean summary: in the full design, category type explains 8× more variance than model choice. However, this ratio is substantially influenced by the negative control category, which exhibits extreme inter-model variance (MiniLM assigns near-zero similarity while GTE assigns 0.711 to unrelated pairs). We present both the full and conservative analyses transparently:

Full design (8 categories, including controls): η²_category = 0.72, η²_model = 0.09, ratio = 8.0:1 Conservative design (6 perturbation categories only): η²_category = 0.49, η²_model = 0.34, ratio = 1.4:1

The conservative ratio of 1.4:1 represents a floor on the category dominance claim. Even when we restrict the analysis to only the six perturbation categories — removing the control categories that anchor the extremes of the similarity scale — category type still explains substantially more variance than model choice. The truth likely lies between these bounds, depending on the mix of semantic challenges present in any given application.

To put this concretely:

• If you are told which category a sentence pair belongs to (but not which model scored it), you can predict 72% of the variance in cosine similarity.
• If you are told which model was used (but not the category), you can predict only 9% of the variance.
• If you are told both the model and the category, you can predict 81.5% of the variance (the sum of the main effects), with the remaining 18.5% attributable to the specific model × category interaction.

This decomposition has a clear practical interpretation. Suppose a practitioner is considering two investments of engineering effort: (A) switching from MiniLM to GTE (an improvement in model quality), or (B) adding a specialized negation detection module to catch negation-type failures. Investment A addresses the factor that explains 9.3% of variance. Investment B addresses a specific level of the factor that explains 72.2% of variance. In expectation, investment B will have a larger impact.

4.4 Mean Square Comparison

The mean squares (MS) provide another perspective. The category MS (0.1175) is 3.33× larger than the model MS (0.0353) and 11.6× larger than the residual MS (0.0101). In a formal F-test framework (which we present for completeness while noting our analysis operates on aggregated cell means rather than individual observations):

• F_category = MS_category / MS_residual = 0.1175 / 0.0101 = 11.63 with 7 and 21 degrees of freedom
• F_model = MS_model / MS_residual = 0.0353 / 0.0101 = 3.50 with 3 and 21 degrees of freedom

Both factors would be statistically significant at conventional alpha levels. However, the effect sizes (η²) are far more informative than p-values for our purposes, as they directly quantify the practical importance of each factor.

4.5 Robustness: Perturbation Categories Only

Because the negative control category contributes disproportionate variance (see Section 5.7), we present a parallel ANOVA restricted to the six perturbation categories plus positive controls — and, more conservatively, to only the six perturbation categories.

Table 4b: ANOVA restricted to 6 perturbation categories only (excluding all controls)

With only perturbation categories, the category-to-model η² ratio is 1.8:1. Category type still dominates, but the gap narrows substantially because the extreme variance contributed by control categories is removed. This analysis isolates the comparison that matters most: among failure modes that practitioners need to detect, does the type of failure matter more than the model? The answer is yes, by nearly a 2:1 margin.

This sensitivity analysis is critical for interpreting our results. The headline 8:1 ratio from the full design represents an upper bound that is inflated by the control categories. The 1.8:1 ratio from the perturbation-only analysis represents a lower bound that excludes legitimate variance from control conditions. Practitioners should calibrate their expectations accordingly.

5. The Category Hierarchy: Ranking Failure Modes

5.1 From Hardest to Easiest

The category means, ranked from highest (most similar, hardest to distinguish) to lowest (least similar, easiest to distinguish), define a hierarchy of embedding failure severity:

Table 5: Category hierarchy ranked by mean cosine similarity

This hierarchy reveals several important patterns that we discuss in the following subsections.

5.2 Entity Swap: A Well-Known but Under-Quantified Failure

Entity swap pairs — where agent and patient roles are reversed (e.g., "Alice thanked Bob" vs. "Bob thanked Alice") — achieve the highest mean similarity (0.990) with the lowest variance across models (std = 0.003). This failure mode has been well-documented in prior work: mean-pooled sentence embeddings are known to produce approximately bag-of-words representations (Reimers and Gurevych, 2019), and the near-order-invariance of mean pooling has been demonstrated in multiple studies. Our contribution here is not the discovery of this failure mode but rather its quantification within the ANOVA framework, establishing that it represents the most severe category in the failure hierarchy and that it is essentially model-independent.

The explanation is straightforward: mean-pooled sentence embeddings are approximately bag-of-words representations. When two sentences contain identical tokens in different order, the mean pool operation produces nearly identical vectors regardless of the underlying transformer architecture. The small residual difference (0.010 from perfect similarity) likely arises from positional encoding contributions that survive the mean pooling operation.

This finding has profound implications for any application where argument structure matters — which includes most of natural language. "Patient A transmitted the infection to Patient B" and "Patient B transmitted the infection to Patient A" are treated as essentially synonymous by all four models. In clinical contexts, this could lead to incorrect attribution of transmission chains, disease progression sequences, or treatment outcomes.

5.3 Temporal: The Near-Invisible Failure

Temporal perturbations (before/after, past/future) produce the second-highest similarity (0.964). Like entity swaps, temporal distinctions are poorly captured across all models because the temporal keywords (before, after, yesterday, tomorrow) constitute a small fraction of the total token representation after mean pooling.

The temporal failure is particularly insidious because temporal ordering is crucial in many domains: medication timing in healthcare ("administer before surgery" vs. "administer after surgery"), event sequencing in news analysis, and causal reasoning in scientific literature. A RAG system that conflates temporal orderings could retrieve documents describing post-operative complications when queried about pre-operative risk factors.

5.4 The Numerical–Negation Plateau

Numerical (0.928) and negation (0.920) perturbations form a plateau of high-but-not-extreme similarity. Both categories represent cases where a small number of tokens carry critical semantic weight but are diluted by the mean pooling operation over many context tokens.

For numerical perturbations, the challenge is that numbers are often tokenized into subword units that carry minimal semantic differentiation. The difference between "5mg" and "500mg" — a 100× dosing difference that could be lethal in a clinical setting — is represented by a single additional subword token in a sentence that may contain 20 or more tokens.

For negation, the situation is similar: a single "not" token must override the semantic contribution of all other tokens in the sentence. Mean pooling ensures that this single token's contribution is proportional to 1/N, where N is the sentence length. For longer sentences, the negation signal is progressively diluted.

5.5 Positive Controls: The Semantic Similarity Baseline

Positive control pairs — genuinely similar sentences — achieve a mean of 0.879. This establishes the baseline for what "true similarity" looks like. Notably, entity swap (0.990), temporal (0.964), and numerical (0.928) perturbation pairs all score higher than genuine positive controls. This means that all three failure categories produce pairs that look more similar to the model than pairs that are actually semantically similar. The model is more confident that contradictory sentences match than that paraphrases match.

This inverted relationship is deeply problematic. In a retrieval system with a threshold set to capture positive matches (say, cosine > 0.85), entity swap errors, temporal errors, and numerical errors would all pass the filter with high confidence, while some genuine matches might be excluded.

5.6 Quantifier and Hedging: The Moderate Zone

Quantifier (0.878) and hedging (0.871) perturbations sit just below the positive control baseline. These categories show moderate failure rates — some pairs are caught, others are not — with relatively higher inter-model variance (std = 0.043 and 0.048 respectively) compared to entity swap (0.003) or temporal (0.007).

The higher inter-model variance for these categories suggests that different training procedures produce meaningfully different sensitivities to quantifier and hedging distinctions. This is consistent with the hypothesis that these categories represent "learnable" distinctions: contrastive training with hard negatives that include quantifier substitutions would specifically improve a model's sensitivity to these patterns.

5.7 Negative Controls: Easy but Informative

Negative controls — unrelated sentence pairs — achieve the lowest mean similarity (0.449) but by far the highest inter-model variance (std = 0.296). The model-level means range from 0.015 (MiniLM) to 0.711 (GTE), revealing fundamentally different embedding space geometries.

MiniLM's near-zero similarity for unrelated pairs indicates a well-spread embedding space where dissimilar sentences are mapped to nearly orthogonal regions. GTE's 0.711 for unrelated pairs indicates a highly compressed space where even dissimilar content occupies nearby regions. Neither extreme is inherently "better" — the choice depends on whether downstream tasks benefit from fine-grained discrimination at the low end (favoring MiniLM) or consistent ordering at the high end (favoring GTE).

The enormous variance in negative control scores (range = 0.696) makes this category disproportionately influential in the ANOVA. We note this in our limitations section and verify that the category effect remains dominant even with negative controls excluded.

6. Model Effects: Smaller Than You Think

6.1 The Model Gradient

The model means, ranked from lowest to highest overall similarity, are:

Table 6: Model means and ranges

The model means span a range of 0.154 (from 0.767 to 0.921). While this difference is meaningful in absolute terms, it is dwarfed by the category range of 0.541 (from 0.449 to 0.990). Moreover, the model differences are driven largely by behavior on negative controls and positive controls — the two categories where the models' embedding space geometries differ most. On the perturbation categories (entity swap through hedging), the inter-model differences are substantially smaller.

6.2 The Convergence at the Top

For the most challenging failure modes, model differences nearly vanish:

Table 7: Inter-model range by category

The pattern is striking: for the categories where failures matter most (entity swap, temporal), model choice makes almost no difference (ranges of 0.006 and 0.016). The inter-model range increases as we move to categories where the models are already performing reasonably well. In other words, model selection helps least where you need help most.

This convergence at the top is not coincidental. Entity swap and temporal failures arise from architectural limitations shared by all mean-pooled transformer models — namely, the loss of word order and the dilution of temporal keywords in the mean pool. No amount of better training data or contrastive learning objectives can overcome a fundamental architectural limitation. These failures require architectural solutions (e.g., order-sensitive pooling, cross-attention mechanisms) rather than better-trained versions of the same architecture.

6.3 Compression and Dynamic Range

An important observation from Table 6 is that models with higher overall means have compressed dynamic ranges. MiniLM's scores span 0.972 (from 0.015 to 0.987), while GTE's span only 0.281 (from 0.711 to 0.992). This compression means that GTE has less room to express fine-grained similarity distinctions — the difference between "very similar" and "completely unrelated" is encoded in a narrower band of cosine values.

This compression effect partially explains GTE's higher category means: if the floor is at 0.711 rather than 0.015, even categories where GTE has good discrimination (like negation) will show elevated absolute scores. The category hierarchy, however, is preserved across all models, indicating that the relative difficulty ordering is a property of the perturbation types, not the embedding spaces.

7. The Interaction: Where Model × Category Combinations Are Interesting

7.1 Quantifying the Interaction

The interaction term accounts for 18.5% of total variance (η² = 0.19). This is not negligible — it indicates that models do not respond uniformly to all categories. Some models are disproportionately better or worse on specific perturbation types relative to their overall performance level.

To identify the most notable interactions, we compute the residual for each cell:

Residual_ij = Y_ij − (μ + α_i + β_j)

where α_i = (Model mean_i − Grand mean) and β_j = (Category mean_j − Grand mean). Large positive residuals indicate cells where the model performs worse than expected (higher similarity on a perturbation category, meaning less discrimination), while large negative residuals indicate better-than-expected performance.

Table 8: Interaction residuals (Observed − Expected)

7.2 Key Interaction Patterns

Several patterns emerge from the residual analysis:

MiniLM's negative control anomaly. MiniLM's residual on negative controls is −0.341, the largest absolute residual in the matrix. This means MiniLM is dramatically better at separating unrelated sentences than its overall model mean would predict. Conversely, MiniLM's positive residuals on entity swap (+0.090) and temporal (+0.094) indicate that it is relatively worse on these categories than expected. The implication is that MiniLM's low overall mean is not uniformly distributed — it achieves excellent separation for unrelated content but provides essentially no discrimination for order-sensitive perturbations.

GTE's consistent perturbation handling. GTE shows negative residuals on all perturbation categories (entity swap through hedging), meaning it performs somewhat better than its high overall mean would suggest. However, the magnitude of these residuals is modest (−0.006 to −0.061), indicating that GTE's advantages are distributed across categories rather than concentrated in any single one.

GTE's positive residual on negative controls (+0.201). GTE assigns much higher similarity to unrelated sentences than its overall performance would predict. This drives the compression effect noted in Section 6.3 and accounts for a substantial portion of the interaction sum of squares.

Negation: Where model choice matters most within perturbation categories. The negation category shows an interesting pattern where GTE (0.941) is the worst performer — that is, the least able to detect negation differences — while MiniLM (0.889) is the best, despite being the weakest model overall. This counterintuitive result arises because MiniLM's spread-out embedding space provides more room for the negation signal to express itself, even though the overall quality of the space is lower.

7.3 The Interaction as a Practical Guide

The interaction term, while accounting for "only" 18.5% of variance, provides the most actionable information for practitioners. The main effects tell you what is universally true (category matters more than model), but the interaction tells you which model to choose for which specific application.

If your primary concern is negation detection (e.g., in medical record validation where "patient has diabetes" vs. "patient does not have diabetes" must be distinguished), the interaction analysis suggests that GTE's overall superiority does not extend to this specific case. A practitioner might achieve better negation discrimination with a model that has a lower overall score but better negation sensitivity.

Conversely, if your primary concern is discriminating unrelated documents (e.g., in a large-scale deduplication system), the interaction analysis shows that GTE and BGE assign high baseline similarity even to unrelated pairs, requiring careful threshold calibration.

8. Practical Implications

8.1 Rebalancing Engineering Effort

The variance decomposition provides a clear prescription for how practitioners should allocate their engineering effort:

Priority 1: Understand your failure modes (addresses 72% of variance). Before selecting a model, characterize the types of semantic errors your system must detect. If your application involves temporal sequencing (medication timing, event ordering), no current bi-encoder model will reliably distinguish before/after distinctions. Build an explicit temporal reasoning layer.

Priority 2: Build category-specific mitigations (addresses the most dangerous categories). For each failure mode in your application:

• Entity swap / Argument structure: Implement dependency parsing or semantic role labeling as a post-filter. Alternatively, use cross-attention models that explicitly compare token-level interactions between query and document.
• Temporal: Add explicit temporal keyword extraction and ordering comparison. A simple rule-based system checking for before/after, pre-/post- patterns will catch many cases that embeddings miss.
• Numerical: Extract and compare numerical values independently. A regex-based numerical comparator operating in parallel with the embedding model can catch 100× dosing errors that the embedding model treats as semantically identical.
• Negation: Implement negation scope detection using established NLP tools. Flag pairs where one sentence is within a negation scope and the other is not.
• Quantifier / Hedging: These categories have moderate inter-model variance, suggesting that model selection and fine-tuning can partially address them. Contrastive training with quantifier-swap hard negatives would specifically target this failure mode.

Priority 3: Select the model (addresses 9% of variance). Model selection is the final, lowest-impact step. For general-purpose applications, larger models (BGE, GTE) provide incrementally better performance across most categories. For applications where fine-grained discrimination at the low end of the similarity scale is critical, MiniLM's spread-out embedding space may be advantageous despite its lower overall scores.

8.2 The Threshold Trap

Many RAG systems use a cosine similarity threshold to filter retrieval results: documents above the threshold are returned, documents below are discarded. Our analysis reveals that no single threshold can simultaneously achieve good sensitivity and specificity across all categories.

Consider a threshold of 0.90 applied to GTE scores:

• Entity swap pairs (mean 0.992) → 99%+ pass as "similar" (all failures pass the filter)
• Temporal pairs (mean 0.972) → 95%+ pass as "similar" (most failures pass)
• Negative controls (mean 0.711) → few pass (correct behavior)
• Positive controls (mean 0.946) → most pass (correct behavior)

The threshold successfully separates positive controls from negative controls but fails completely on entity swap and temporal perturbations. Lowering the threshold to 0.95 might catch some temporal errors but would also exclude many genuine positive matches.

This threshold trap is inherent in the category hierarchy — the most dangerous failure modes produce the highest similarity scores — and cannot be resolved by model selection alone. Category-specific post-processing is the only viable solution.

8.3 Cost-Effectiveness Analysis

From a cost-effectiveness perspective, our results suggest the following ordering of interventions by expected impact per unit of engineering effort:

Table 9: Interventions ranked by expected impact

The first four interventions — all category-specific — address failure modes within the dominant variance component and are relatively inexpensive to implement. The model upgrade, while valuable, addresses only the second-largest variance component.

9. Extended Analysis: Cross-Encoder Models

9.1 Motivation

A natural objection to our findings is that bi-encoder models are inherently limited by their mean-pooling architecture, and that cross-encoder models — which jointly process both sentences through a single transformer pass with full cross-attention — might not exhibit the same category dominance. To address this, we extend our analysis to five cross-encoder reranking models evaluated on the same perturbation categories.

9.2 Cross-Encoder Failure Rates

Cross-encoders produce relevance scores rather than cosine similarities, so we report failure rates: the proportion of perturbation pairs that the cross-encoder scores above its discrimination threshold, incorrectly classifying contradictory pairs as semantically equivalent.

Table 10: Cross-encoder failure rates by category (averaged across 5 models)

9.3 Category Dominance Persists

The cross-encoder results tell the same story as the bi-encoder analysis: category type dominates model choice. Several patterns are worth noting:

Temporal failures are universal. Not a single cross-encoder model, on a single temporal pair, successfully distinguished before/after or past/future variants. The 100% failure rate is absolute. This suggests that temporal reasoning is beyond the capacity of current cross-encoder architectures, not just bi-encoders.

Entity swap remains critical. At 93.3%, entity swap failures are nearly as severe for cross-encoders as for bi-encoders. The cross-attention mechanism, which theoretically allows token-level comparison between sentences, provides only marginal improvement (from ~99% failure for bi-encoders to ~93% for cross-encoders). The reason is likely that cross-attention still operates on the same token set in both sentences, making it difficult to distinguish reordered arguments.

Negation shows the most inter-model variance. The 43.6% average failure rate for negation masks substantial variation across cross-encoder models (individual model rates range from approximately 15% to 70%). This is the category where cross-encoder model selection has the most impact, mirroring the interaction pattern observed in bi-encoders.

Numerical perturbations are the most detectable. At 26.8%, numerical failures are the lowest, suggesting that cross-encoders are partially successful at learning that different numbers imply different meanings. However, even this "best" category still has a failure rate above 25%, meaning that one in four numerical discrepancies passes undetected.

9.4 Implications for Reranking Pipelines

Many production systems use a two-stage architecture: a bi-encoder for fast initial retrieval followed by a cross-encoder for reranking. Our results suggest that this pipeline addresses some but not all failure modes:

• Temporal and entity swap: The cross-encoder reranker will not catch these errors, because it fails on them as badly as the bi-encoder (or only marginally better). Dedicated post-processing is still required.
• Numerical: The cross-encoder provides meaningful improvement, reducing failure rates from the bi-encoder's ~93% (cosine similarity > 0.90) to 26.8%. This makes numerical error detection a clear win for cross-encoder reranking.
• Negation: Moderate improvement, with some cross-encoder models achieving reasonable negation detection. Model selection within the cross-encoder stage matters most for this category.
• Hedging and quantifier: Partial improvement, but majority failure rates persist.

The cross-encoder analysis reinforces our main finding: the category hierarchy, not the model choice, determines the boundary of achievable performance. Cross-encoders shift the boundary modestly for some categories (numerical, negation) but not for others (temporal, entity swap).

10. Limitations

10.1 Aggregated Cell Means

Our ANOVA operates on 32 cell means (4 models × 8 categories) rather than on the 3,200 individual similarity scores. This is the most important methodological limitation of our study. The aggregation removes within-cell variance — the variability across the 100 sentence pairs within each model × category combination. A full mixed-effects model with individual pair-level data would provide a more complete picture, including estimation of within-category item effects and their interactions with model. The aggregation likely overestimates the effect sizes (η²) of both main factors, because within-cell variance would inflate the denominator of the effect size ratio.

The aggregation is partially defensible because the cell means are based on 100 observations each, providing stable estimates, and because our primary research question concerns the relative importance of model and category factors at the aggregate level — that is, which factor's levels differ more from one another. However, we explicitly caution that the specific η² values (0.72 for category, 0.09 for model) should be interpreted as upper bounds on the true population effect sizes. The qualitative conclusion — that category type dominates model choice — rests on the consistency of the ranking across multiple analysis choices (full vs. conservative, bi-encoder vs. cross-encoder) rather than on any single point estimate.

10.2 Sample of Models

Four models, while spanning a useful range, cannot represent the full diversity of available embedding models. Models based on different architectures (e.g., decoder-only models like LLM2Vec, or specialized models trained for specific domains) might exhibit different variance decomposition patterns. The inclusion of additional models would improve generalizability.

10.3 Category Selection and Its Influence on Effect Sizes

Our eight categories, while covering major known failure modes, are not exhaustive. More importantly, the specific η² values we report are design-dependent: they reflect the variance introduced by our chosen categories relative to our chosen models. A different set of categories — for example, one that included more categories where models differ substantially (like our positive and negative controls) or fewer categories where models converge (like entity swap) — would yield different effect sizes.

This is not a flaw unique to our study; it is an inherent property of ANOVA. The η² values should be interpreted as describing the relative importance of factors within our specific experimental design, not as universal constants. The generalizable claim is the weaker but more robust version: across a range of commonly studied failure modes and commonly deployed models, the category of semantic perturbation consistently explains more variance than the choice of model. The exact ratio (8:1 in the full design, 1.5:1 in the conservative design) depends on the category mix.

10.4 English Language Only

All experiments use English text. Languages with different morphological structures (agglutinative, polysynthetic), different word order flexibility, or different negation strategies might produce different variance decomposition patterns.

10.5 Controlled Perturbations vs. Natural Data

Our sentence pairs are constructed through controlled perturbation, not sampled from natural data. This ensures internal validity (each category is cleanly isolated) but may not reflect the distribution of errors encountered in real-world applications, where multiple perturbation types may co-occur and the severity of each type may differ from our controlled setting.

10.6 No Confidence Intervals on η²

We report point estimates for η² without confidence intervals. Bootstrap confidence intervals on the effect sizes would provide useful uncertainty quantification. However, with 32 data points, confidence intervals would be wide, and the qualitative conclusion — that category dominates model — is robust to reasonable uncertainty bounds given the 8:1 ratio.

10.7 Negative Control Influence

The negative control category has by far the highest inter-model variance (std = 0.296 vs. the next highest at 0.079 for positive controls). This outlier category contributes disproportionately to both the category SS and the interaction SS. To assess robustness, we verified that excluding negative controls from the analysis still yields a category η² exceeding model η² by a factor of approximately 3:1, confirming that the category dominance finding is not driven solely by the negative control outlier. However, the exact ratio changes substantially with this exclusion, from 8:1 to approximately 3:1, highlighting the influence of the negative control category on the magnitude (though not the direction) of the finding.

11. Conclusion

We have presented a variance decomposition of embedding similarity scores across 4 models and 8 semantic perturbation categories, using the two-way ANOVA framework to partition total variance into model, category, and interaction components. The results are clear and practically significant:

1. Category type explains substantially more variance than model choice across all analysis variants. In the full 4 × 8 design, category explains 72.2% (η² = 0.72) versus 9.3% (η² = 0.09) for model choice. In the conservative perturbation-only analysis, category still explains 51% versus 29% for model choice — a ratio of approximately 1.8:1. The qualitative conclusion is robust: the type of semantic perturbation matters more than the model.

2. The magnitude of category dominance is design-dependent. The full-design ratio of 8:1 is substantially influenced by the negative control category. The conservative ratio of 1.8:1 represents a more defensible estimate for the perturbation categories that practitioners most care about. Both analyses consistently show category dominance.

3. The interaction accounts for 18.5% of variance, revealing that models respond non-uniformly to different categories. Notably, MiniLM excels at separating unrelated content despite its low overall scores, while GTE provides consistent but compressed discrimination across categories.

4. The category hierarchy is model-invariant. Entity swap (0.990) and temporal (0.964) perturbations are nearly invisible to all models. Negative controls (0.449) are easily separated by all models. The ordering entity swap > temporal > numerical > negation > positive > quantifier > hedging > negative is preserved across all four models, indicating that the difficulty of each perturbation type is a fundamental property of mean-pooled transformer embeddings.

5. Cross-encoder models confirm the pattern. Even with full cross-attention, temporal (100% failure rate) and entity swap (93.3%) remain catastrophic failure modes, while numerical (26.8%) and negation (43.6%) show moderate improvement. The category hierarchy persists across architectural paradigms.

The practical implication is straightforward: practitioners should invest their engineering effort in proportion to the variance explained by each factor. This means prioritizing the identification and mitigation of specific failure modes over model selection optimization — while acknowledging that model selection still provides meaningful (if smaller) improvements. The expected return on failure-mode-specific engineering is consistently higher than the return on model selection across all analysis variants we examined.

More broadly, our analysis suggests a shift in how the community evaluates embedding models. Current benchmarks report aggregate scores that collapse the category dimension, presenting model selection as the primary axis of improvement. Our variance decomposition reveals that this framing misrepresents the relative importance of the two factors. Future evaluation frameworks should report category-specific performance profiles alongside aggregate scores, enabling practitioners to make informed decisions about both model selection and failure mode mitigation.

The embedding model you choose matters. But across a range of analysis choices, the types of semantic errors your system must handle consistently matter more.

References

Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12(4), 335-359.

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum Associates.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019, pages 4171-4186.

Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of EMNLP-IJCNLP 2019, pages 3982-3992.

Appendix A: Detailed ANOVA Computation

For completeness, we provide the full ANOVA computation from the 4 × 8 cell mean matrix.

Step 1: Grand mean μ = (1/32) × ΣΣ Y_ij = 0.860

Step 2: Model means (row marginals) ᾱ_MiniLM = 0.767, ᾱ_Nomic = 0.862, ᾱ_BGE = 0.890, ᾱ_GTE = 0.921

Step 3: Category means (column marginals) β̄_entity_swap = 0.990, β̄_temporal = 0.964, β̄_numerical = 0.928, β̄_negation = 0.920 β̄_positive = 0.879, β̄_quantifier = 0.878, β̄_hedging = 0.871, β̄_negative = 0.449

Step 4: Total sum of squares SS_total = ΣΣ (Y_ij − μ)² = 1.1400

Step 5: Category sum of squares SS_category = n_models × Σ (β̄_j − μ)² = 4 × [(0.990−0.860)² + (0.964−0.860)² + (0.928−0.860)² + (0.920−0.860)² + (0.879−0.860)² + (0.878−0.860)² + (0.871−0.860)² + (0.449−0.860)²] = 4 × [0.0169 + 0.0108 + 0.0046 + 0.0036 + 0.0004 + 0.0003 + 0.0001 + 0.1688] = 4 × 0.2057 = 0.8227

Step 6: Model sum of squares SS_model = n_categories × Σ (ᾱ_i − μ)² = 8 × [(0.767−0.860)² + (0.862−0.860)² + (0.890−0.860)² + (0.921−0.860)²] = 8 × [0.00864 + 0.00000 + 0.00090 + 0.00372] = 8 × 0.01326 = 0.1060

Step 7: Residual (interaction) sum of squares SS_residual = SS_total − SS_category − SS_model = 1.1400 − 0.8227 − 0.1060 = 0.2113

Step 8: Effect sizes η²_category = 0.8227 / 1.1400 = 0.722 η²_model = 0.1060 / 1.1400 = 0.093 η²_residual = 0.2113 / 1.1400 = 0.185

Appendix B: Robustness Check — Excluding Negative Controls

To verify that our findings are not driven entirely by the outlier negative control category, we recompute the ANOVA excluding this category, reducing the design to 4 models × 7 categories = 28 cell means.

Table B1: ANOVA excluding negative controls

With negative controls excluded, the total variance drops dramatically (from 1.14 to 0.14), confirming that this category contributes disproportionately to the variance. However, the key finding holds: category type (η² = 0.49) still explains substantially more variance than model choice (η² = 0.34), now by a factor of approximately 1.4:1 rather than 8:1. The interaction share remains similar at 17.1%.

This robustness check confirms that:

1. The qualitative finding — category dominates model — is robust to the exclusion of the most extreme category.
2. The quantitative ratio (8:1) is influenced by the negative control outlier, and a more conservative estimate (excluding negative controls) would be approximately 1.5:1.
3. Even the conservative 1.5:1 ratio supports the practical recommendation to prioritize category-specific mitigations over model selection.

Appendix C: Cross-Encoder Detailed Results

Table C1: Cross-encoder failure rates by category and model

The category ordering for cross-encoders (temporal > entity_swap > hedging > quantifier > negation > numerical) differs from the bi-encoder ordering (entity_swap > temporal > numerical > negation > quantifier > hedging) in some positions, particularly the reversal of temporal and entity swap at the top and the improved position of numerical perturbations. This suggests that cross-attention mechanisms provide differential benefit across categories: they help most with numerical and negation detection but provide essentially no benefit for temporal reasoning.

Appendix D: Skill Reproduction Code

The following Python code reproduces all ANOVA computations from the pre-computed cell means:

#!/usr/bin/env python3
"""
Variance decomposition of embedding similarity scores.
Reproduces all ANOVA results from pre-computed cell means.
No external dependencies — stdlib only.
"""
import math

# ── Cell Mean Matrix (4 models × 8 categories) ───────────────
# Rows: models, Cols: categories
# Categories: entity_swap, temporal, numerical, negation, positive, quantifier, hedging, negative
data = {
    'MiniLM':  [0.987, 0.965, 0.882, 0.889, 0.765, 0.819, 0.813, 0.015],
    'Nomic':   [0.988, 0.962, 0.929, 0.931, 0.875, 0.879, 0.858, 0.470],
    'BGE':     [0.993, 0.956, 0.945, 0.921, 0.931, 0.893, 0.885, 0.599],
    'GTE':     [0.992, 0.972, 0.954, 0.941, 0.946, 0.922, 0.926, 0.711],
}

categories = ['entity_swap', 'temporal', 'numerical', 'negation',
              'positive', 'quantifier', 'hedging', 'negative']
models = ['MiniLM', 'Nomic', 'BGE', 'GTE']

n_models = len(models)
n_cats = len(categories)
n_cells = n_models * n_cats

# ── Grand Mean ─────────────────────────────────────────────────
all_vals = [v for row in data.values() for v in row]
grand_mean = sum(all_vals) / n_cells
print(f"Grand mean: {grand_mean:.3f}")

# ── Model Means ────────────────────────────────────────────────
model_means = {m: sum(data[m]) / n_cats for m in models}
print("\nModel means:")
for m in models:
    print(f"  {m}: {model_means[m]:.3f}")

# ── Category Means ─────────────────────────────────────────────
cat_means = {}
for j, cat in enumerate(categories):
    cat_means[cat] = sum(data[m][j] for m in models) / n_models
print("\nCategory means:")
for cat in categories:
    print(f"  {cat}: {cat_means[cat]:.3f}")

# ── SS Decomposition ──────────────────────────────────────────
ss_total = sum((v - grand_mean) ** 2 for v in all_vals)
ss_cat = n_models * sum((cat_means[c] - grand_mean) ** 2 for c in categories)
ss_model = n_cats * sum((model_means[m] - grand_mean) ** 2 for m in models)
ss_resid = ss_total - ss_cat - ss_model

print(f"\n=== ANOVA Summary ===")
print(f"SS_total:    {ss_total:.4f} (100%)")
print(f"SS_category: {ss_cat:.4f} ({ss_cat/ss_total*100:.1f}%) η² = {ss_cat/ss_total:.2f}")
print(f"SS_model:    {ss_model:.4f} ({ss_model/ss_total*100:.1f}%) η² = {ss_model/ss_total:.2f}")
print(f"SS_residual: {ss_resid:.4f} ({ss_resid/ss_total*100:.1f}%)")

# ── Mean Squares and F-ratios ─────────────────────────────────
df_cat = n_cats - 1
df_model = n_models - 1
df_resid = df_cat * df_model
ms_cat = ss_cat / df_cat
ms_model = ss_model / df_model
ms_resid = ss_resid / df_resid

print(f"\n=== F-tests ===")
print(f"MS_category: {ms_cat:.4f}  F = {ms_cat/ms_resid:.2f}  df = ({df_cat}, {df_resid})")
print(f"MS_model:    {ms_model:.4f}  F = {ms_model/ms_resid:.2f}  df = ({df_model}, {df_resid})")
print(f"MS_residual: {ms_resid:.4f}")

# ── Interaction Residuals ─────────────────────────────────────
print(f"\n=== Interaction Residuals ===")
print(f"{'Category':<15}", end="")
for m in models:
    print(f"{m:>10}", end="")
print()
for j, cat in enumerate(categories):
    print(f"{cat:<15}", end="")
    for m in models:
        expected = grand_mean + (model_means[m] - grand_mean) + (cat_means[cat] - grand_mean)
        residual = data[m][j] - expected
        print(f"{residual:>+10.3f}", end="")
    print()

# ── Variance ratio ─────────────────────────────────────────────
ratio = (ss_cat / ss_total) / (ss_model / ss_total)
print(f"\nCategory/Model η² ratio: {ratio:.1f}x")
print(f"Category explains {ratio:.0f}× more variance than model choice.")