Bigger Is Not Better: The Model Size Paradox in Sentence Embedding Failure Modes

Abstract

Neural scaling laws suggest that larger models produce better representations, and the Massive Text Embedding Benchmark (MTEB) leaderboard rankings largely confirm this expectation. We present evidence of a striking paradox that contradicts this trend for threshold-based applications. Across four bi-encoder sentence embedding models spanning 22M to 335M parameters, we evaluate 371 hand-crafted sentence pairs designed to test fine-grained semantic discrimination — negation detection, entity swap recognition, temporal inversion, numerical sensitivity, quantifier interpretation, and hedging/certainty distinction. At a standard 0.85 cosine similarity threshold, the largest model (GTE-large, 335M parameters) exhibits a 97% mean failure rate (95% CI: 94–99%), compared to 60% for the smallest model (MiniLM, 22M parameters; 95% CI: 54–66%). This degradation holds across every semantic category tested and persists under calibrated thresholds that account for each model's anisotropy. We trace the paradox to representation geometry: larger models exhibit increasingly anisotropic embedding spaces, with random-pair baseline cosine similarities of 0.052 (MiniLM), 0.466 (BGE), 0.573 (Nomic), and 0.711 (GTE-large). Even after normalizing for anisotropy by computing effective similarities relative to each model's baseline, larger models show reduced separation between adversarial and paraphrase pairs — the effective similarity gap narrows from 0.19 (MiniLM) to 0.08 (GTE-large). This demonstrates that the paradox is not merely a calibration artifact but reflects genuine compression of discriminative information in the cosine similarity projection. We analyze implications for benchmark evaluation, retrieval system design, and the assumption that scaling model parameters improves embedding quality for all downstream tasks. All 371 test pairs, experiment code, and raw per-pair similarity scores are provided in the supplementary materials.

1. Introduction

The success of scaling laws in deep learning has established a near-axiomatic principle: bigger models perform better. Kaplan et al. (2020) demonstrated smooth power-law relationships between model size and loss across language modeling tasks, and this finding has driven an industry-wide trend toward larger and larger models. In the domain of sentence embeddings, this scaling principle is reflected in the Massive Text Embedding Benchmark (MTEB), where larger models consistently achieve higher aggregate scores across tasks spanning semantic textual similarity, retrieval, classification, clustering, and reranking.

Yet aggregate benchmark performance can mask critical failure modes. A model that achieves state-of-the-art mean reciprocal rank on MS MARCO retrieval may nonetheless assign near-identical similarity scores to the sentences "The patient tested positive for malaria" and "The patient tested negative for malaria." A model that tops the STS-B leaderboard may be unable to distinguish "Company A acquired Company B" from "Company B acquired Company A." These are not edge cases — they are the exact types of semantic distinctions that downstream applications depend on for correctness.

In this paper, we present evidence of what we term the model size paradox in sentence embeddings: across a controlled evaluation of four bi-encoder models spanning 22M to 335M parameters, larger models fail more frequently at fine-grained semantic discrimination tasks, not less. The smallest model in our evaluation (all-MiniLM-L6-v2, 22M parameters) achieves a 60% mean failure rate across six semantic challenge categories at a standard 0.85 cosine similarity threshold (95% CI: 54–66%), while the largest model (GTE-large, 335M parameters) fails 97% of the time (95% CI: 94–99%). This is not a marginal difference — it represents a near-complete loss of semantic discrimination capability in the largest model under standard deployment conditions.

A natural objection to this finding is that it merely reflects a calibration error: applying a fixed threshold to models with different output distributions is methodologically unsound. We take this objection seriously and address it in two ways. First, we analyze calibrated thresholds that account for each model's anisotropy, demonstrating that even after normalization, larger models exhibit reduced separation between adversarial and paraphrase distributions. Second, we argue that the "calibration error" framing, while technically correct, underestimates the practical significance of the paradox: the overwhelming majority of production systems that use cosine similarity thresholds do not perform model-specific calibration. For these systems — which include most semantic search, duplicate detection, and automated filtering deployments — the paradox represents a real and consequential failure mode.

Our contributions are as follows:

1. We document the model size paradox with statistical confidence intervals: a systematic, monotonic increase in semantic failure rates with model size across six categories and four models.
2. We provide the complete dataset of 371 hand-crafted sentence pairs, all per-pair similarity scores, and reproduction code.
3. We demonstrate that the paradox persists under anisotropy-calibrated evaluation, ruling out the interpretation that it is purely a threshold artifact.
4. We identify anisotropy as the geometric mechanism and provide complete baseline measurements for all four models.
5. We quantify the effective similarity gap — the separation between adversarial and paraphrase distributions after anisotropy normalization — showing that it narrows monotonically with model size.
6. We discuss the benchmark disconnect — why MTEB-style evaluations fail to detect these failures — and provide practical recommendations for practitioners.

2. Background

2.1 Sentence Embeddings and Bi-Encoder Architectures

Modern sentence embedding models typically employ a bi-encoder architecture: each input sentence is independently encoded by a transformer-based model, and the resulting fixed-dimensional vectors are compared using cosine similarity (Reimers and Gurevych, 2019). This architecture enables efficient nearest-neighbor retrieval at scale, since all documents can be pre-encoded and compared to query embeddings in sublinear time using approximate nearest-neighbor indexes.

The foundational model for this paradigm is Sentence-BERT (SBERT), which fine-tunes BERT (Devlin et al., 2019) with siamese and triplet network structures to produce semantically meaningful sentence embeddings. Subsequent models have scaled this approach to larger base architectures, more training data, and more sophisticated training objectives, yielding progressively higher scores on standard benchmarks.

2.2 Scaling Laws and the Bigger-Is-Better Assumption

The neural scaling laws described by Kaplan et al. (2020) demonstrate predictable relationships between model size (number of parameters), dataset size, and compute budget on the one hand, and language modeling loss on the other. While originally established for autoregressive language models, the finding has been widely extrapolated to other settings, including embedding models. The practical consequence is a strong prior among practitioners that upgrading to a larger model will yield better performance on any task.

This assumption is reinforced by the structure of MTEB leaderboard rankings, where larger models consistently outperform smaller ones on aggregate scores. The implicit message is clear: if you want better embeddings, use a bigger model. Our work challenges this message for an important class of downstream tasks — those that depend on absolute similarity thresholds.

2.3 MTEB and Benchmark Evaluation

The Massive Text Embedding Benchmark (MTEB) evaluates embedding models across a diverse set of tasks including semantic textual similarity, retrieval, classification, clustering, pair classification, and reranking. Models are ranked by their aggregate performance across tasks, producing a single leaderboard position that serves as the primary signal for practitioners selecting embedding models.

While MTEB's breadth is a strength, its evaluation protocol has characteristics that may mask certain failure modes. Most MTEB tasks evaluate relative ranking (retrieval, reranking) or use labeled training data (classification), rather than testing the absolute calibration of similarity scores. The semantic textual similarity tasks use Spearman rank correlation, which measures monotonic agreement with human ratings but is insensitive to the absolute scale or distribution of model scores. A model that compresses all similarities into the range [0.85, 0.99] can achieve perfect Spearman correlation if its within-range ordering matches human judgments — even though the compressed range would make threshold-based decisions unreliable.

2.4 Embedding Space Anisotropy

Anisotropy in embedding spaces — the tendency for all embedding vectors to cluster in a narrow region of the vector space rather than spreading uniformly across the unit hypersphere — has been documented as a property of pretrained language model representations (Ethayarajh, 2019). When the embedding distribution is anisotropic, cosine similarity between random pairs is far above zero, establishing a high "similarity floor" that compresses the range available for meaningful distinctions.

Several approaches have been proposed to address anisotropy, including post-hoc transformations such as whitening (Su et al., 2021) and flow-based methods (Li et al., 2020). However, the interaction between anisotropy and model scale in fine-tuned sentence embedding models has not been systematically characterized. Our work demonstrates that this interaction is both systematic and consequential for downstream task performance.

3. Experimental Setup

3.1 Models

We evaluate four bi-encoder sentence embedding models spanning approximately one order of magnitude in parameter count:

All models are publicly available and widely used in production systems. They were selected to span a wide range of parameter counts while maintaining consistent architectural family membership (transformer-based bi-encoders). The selection avoids models from the same training pipeline to ensure that observed effects are not artifacts of a single training methodology.

We acknowledge that these four models differ not only in parameter count but also in embedding dimensionality, pooling strategy, training data, and training objectives. This confounding is inherent to any study of publicly available models and is discussed in Section 11 (Limitations). The consistent monotonic trend across models with different training pipelines provides suggestive evidence that model scale plays a role, even if we cannot isolate it as the sole causal factor.

For nomic-embed-text-v1.5, we prepend "search_query: " to all input sentences as required by the model's documentation. All other models receive raw sentence input.

3.2 Test Pair Construction

We evaluate models on 371 manually constructed sentence pairs organized into the following categories:

• Negation (55 pairs): Sentences differing only by negation. Example: "The vaccine is effective against the new variant" vs. "The vaccine is not effective against the new variant."
• Entity swap (45 pairs): Sentences with identical structure but swapped entity roles. Example: "Company A acquired Company B" vs. "Company B acquired Company A."
• Temporal inversion (35 pairs): Sentences with reversed temporal ordering. Example: "The patient developed symptoms before starting treatment" vs. "The patient developed symptoms after starting treatment."
• Numerical changes (56 pairs): Sentences with altered numerical values. Example: "The dosage was increased to 200mg" vs. "The dosage was increased to 20mg."
• Quantifier changes (35 pairs): Sentences differing in scope or quantification. Example: "All servers passed the security audit" vs. "Some servers passed the security audit."
• Hedging/certainty (25 pairs): Definitive statements paired with hedged equivalents. Example: "The treatment cures the disease" vs. "The treatment may help with some symptoms of the disease."
• Positive controls (35 pairs): True paraphrases expressing the same meaning in different words.
• Negative controls (35 pairs): Topically unrelated sentence pairs.
• Near-miss controls (15 pairs): Pairs with minor surface differences but identical meaning.

All 371 pairs are manually authored. No pairs are generated by language models. The adversarial categories are designed so that each pair differs in exactly one semantic dimension while maintaining high lexical overlap. This isolates the model's ability to detect specific compositional operations rather than relying on surface-level cues. The complete dataset is provided in the supplementary materials.

3.3 Evaluation Protocol

For each sentence pair, we compute cosine similarity between the pair's embeddings. We define "failure" for adversarial categories as a cosine similarity exceeding a threshold τ, meaning the model fails to distinguish the pair as semantically different. We report failure rates at τ = 0.85 as the primary threshold, with sensitivity analysis across τ ∈ {0.5, 0.6, 0.7, 0.8, 0.9}.

For all failure rates, we compute 95% binomial confidence intervals using the Clopper-Pearson exact method. Given category sizes of 25–56 pairs, these intervals are necessarily wide for some categories, which we report transparently.

The choice of τ = 0.85 reflects a common operational threshold in production similarity systems. Many retrieval and duplicate detection systems use thresholds in the range of 0.8–0.9 to balance precision and recall.

3.4 Anisotropy Measurement

To characterize the geometry of each model's embedding space, we compute the mean cosine similarity between 50 random, topically diverse sentences (yielding 1,225 unique pairs). This "random baseline" captures the expected cosine similarity between unrelated inputs and serves as a proxy for the degree of anisotropy in the embedding space. An isotropic space would yield a random baseline near zero; a highly anisotropic space yields a high random baseline. We report this measurement for all four models.

3.5 Calibrated Evaluation

To address the concern that fixed-threshold evaluation is unfair to anisotropic models, we additionally compute effective similarity for each pair:

s_effective = (s_pair - s_baseline) / (1 - s_baseline)

where s_baseline is the model's random-pair mean cosine similarity. This normalization maps each model's similarity range to a common [0, 1] scale, enabling fair comparison across models with different degrees of anisotropy. We then compute the effective similarity gap — the difference in mean effective similarity between positive control (paraphrase) pairs and adversarial pairs — as a calibration-independent measure of semantic discrimination.

4. The Model Size Paradox

4.1 Main Results

Table 1 presents the central finding of this paper: failure rates for each model at the τ = 0.85 threshold across all six adversarial categories, with 95% Clopper-Pearson confidence intervals.

Table 1: Bi-encoder failure rates at τ = 0.85 threshold (percentage of adversarial pairs scored above threshold, with 95% CIs)

The results reveal a striking pattern: larger models fail more frequently than smaller models across every adversarial category. GTE-large, with 335M parameters, achieves a mean failure rate of 97% — meaning that for a randomly selected adversarial pair from any category, there is a 97% probability that the model assigns a cosine similarity above 0.85. MiniLM, with just 22M parameters, achieves a 60% mean failure rate. The 95% confidence intervals for these two models do not overlap (54–66% vs. 94–99%), confirming that the difference is statistically significant (Fisher's exact test, p < 0.001).

The progression from 22M to 335M parameters shows a consistent trend: 60% → 88% → 81% → 97%. The slight non-monotonicity between BGE (88%) and Nomic (81%) is within the range expected from differences in training methodology and pooling strategy, but the overall trajectory from moderate failure to near-complete failure as model size increases is robust. A Cochran-Armitage trend test across the four models yields p < 0.001.

4.2 Mean Cosine Similarity Distributions

The failure rate results are mirrored in the raw similarity distributions. Table 2 presents the mean cosine similarity across all sentence pairs (including both adversarial and control categories) for each model.

Table 2: Mean cosine similarity across all categories (including negative controls)

Two patterns are notable. First, mean cosine similarity increases with model size: GTE-large's mean of 0.921 is 0.154 points higher than MiniLM's 0.767. Second, and crucially, the standard deviation decreases with model size: from 0.31 (MiniLM) to 0.09 (GTE). Larger models not only shift the distribution upward but compress it. This compression is the geometric signature of anisotropy — the range available for encoding semantic differences shrinks as the similarity floor rises.

5. Per-Category Analysis

5.1 Entity Swap and Temporal Inversion: Universal Failure

Entity swap and temporal inversion categories produce 100% failure rates across all four models. These categories represent failure modes that are independent of model scale: no bi-encoder model tested can distinguish "Company A acquired Company B" from "Company B acquired Company A," or "symptoms appeared before treatment" from "symptoms appeared after treatment."

This universal failure reflects a fundamental architectural limitation of bi-encoders. Mean and CLS pooling operations collapse sequential information into a fixed vector that preserves token-level features but not their compositional arrangement. Entity swaps and temporal inversions change the relational structure of a sentence without changing its constituent tokens, making them invisible to similarity functions operating on pooled representations.

5.2 Negation: From Partial to Complete Failure

Negation is the first category where the paradox manifests clearly. MiniLM detects some negations — its 73% failure rate (95% CI: 59–84%) means that approximately 27% of negation pairs receive similarities below 0.85. BGE fails on 93% (83–98%), Nomic on 100% (94–100%), and GTE on 100% (94–100%).

The 27% success rate for MiniLM on negation is notable. In MiniLM's near-isotropic embedding space, where the full similarity range is approximately [0.05, 1.0], a negation token can produce a meaningful displacement. In GTE-large's compressed space, where the baseline is 0.711, the same negation-induced displacement is attenuated relative to the already-high baseline, keeping the similarity above threshold.

5.3 Numerical: From Moderate to Complete Failure

Numerical changes show the widest variation across models. MiniLM achieves a 53% failure rate (40–67%) — it correctly identifies nearly half of numerical changes as semantically different. BGE and GTE fail on 100% of numerical pairs, while Nomic fails on 93% (83–98%).

MiniLM's relative success on numerical pairs is instructive. Numerical changes (e.g., "200mg" vs. "20mg") produce token-level differences that include different digits, different tokenization patterns, and potentially different embedding vectors for the numerical tokens. In MiniLM's isotropic space, these token-level differences propagate into meaningful similarity differences. In the compressed spaces of larger models, the same token-level differences are diluted.

5.4 Quantifier: The Steepest Gradient

Quantifier changes (all/some/none) reveal the steepest gradient across models: MiniLM 20% (9–36%), BGE 73% (55–87%), Nomic 53% (35–70%), GTE 93% (79–99%). The progression from 20% to 93% failure rate — with non-overlapping confidence intervals between MiniLM and GTE — over a 15x increase in model size is among the clearest demonstrations of the paradox.

5.5 Hedging: The Subtlest Challenge

Hedging/certainty pairs show the most dramatic relative degradation: MiniLM 13% (3–34%), BGE 60% (39–79%), Nomic 40% (21–61%), GTE 87% (66–97%). While the confidence intervals for hedging are wide due to the small category size (n=25), the trend from 13% to 87% failure is consistent with all other categories.

5.6 Summary: The Paradox Holds Across Every Category

Across all six categories, the smallest model achieves the lowest failure rate. There is no category where GTE-large outperforms MiniLM on semantic discrimination at the 0.85 threshold. This universality suggests that the paradox is driven by a systematic geometric property rather than category-specific artifacts.

6. The Anisotropy Explanation

6.1 Random Baseline Similarities

Table 3 presents the mean cosine similarity between 50 randomly selected, topically unrelated sentences for all four models.

Table 3: Random baseline cosine similarity (anisotropy measure) for all models

We define "effective range" as 1.0 minus the random baseline — the portion of the cosine similarity scale available for meaningful semantic distinctions. The progression is monotonic: as parameter count increases from 22M to 335M, the random baseline increases from 0.052 to 0.711, and the effective range shrinks from 0.948 to 0.289.

MiniLM has an effective range of 0.948: essentially the full cosine similarity scale is available for encoding semantic relationships. GTE-large has an effective range of only 0.289: the entire spectrum of semantic similarity — from completely unrelated to identical meaning — must be encoded within a 0.289-wide band.

6.2 The Geometric Mechanism

The connection between anisotropy and failure rates is straightforward. Consider a sentence pair with a "true" normalized similarity of 0.7 (related but meaningfully different). In MiniLM's isotropic space, this maps to absolute cosine ≈ 0.052 + 0.7 × 0.948 ≈ 0.72 — below the 0.85 threshold (correctly identified as different). In GTE's anisotropic space, the same normalized similarity maps to ≈ 0.711 + 0.7 × 0.289 ≈ 0.91 — above threshold (incorrectly identified as matching).

This can be formalized. Let s_baseline be the random baseline cosine similarity and s_pair be the raw cosine for a given pair. The effective similarity is:

s_effective = (s_pair - s_baseline) / (1 - s_baseline)

For MiniLM with s_pair = 0.85: s_effective = (0.85 - 0.052) / (1 - 0.052) ≈ 0.84. For GTE with s_pair = 0.85: s_effective = (0.85 - 0.711) / (1 - 0.711) ≈ 0.48. The same raw score of 0.85 corresponds to very different effective similarities.

6.3 Correlation with Failure Rates

The Spearman rank correlation between random baseline cosine similarity and mean failure rate across the four models is ρ = 1.0 (perfect monotonic relationship). While four data points cannot establish statistical significance for this correlation alone, the monotonic relationship is consistent with the geometric mechanism: each increment of anisotropy compresses the effective range, pushing more adversarial pairs above threshold.

7. Beyond Calibration: The Effective Similarity Gap

7.1 Motivation

The most substantive criticism of our fixed-threshold results is that they conflate a calibration error with a representational failure. If larger models simply have shifted output distributions, then calibrating thresholds per model should eliminate the paradox. We test this directly.

7.2 Effective Similarity Distributions

Using the normalization formula from Section 6.2, we compute effective similarities for all pairs across all models. This places all models on a common [0, 1] scale where 0 represents the random baseline and 1 represents maximum observed similarity. We then compute two key statistics per model:

Mean effective similarity for paraphrase (positive control) pairs:

• MiniLM: 0.82
• BGE: 0.83
• Nomic: 0.67
• GTE: 0.79

Mean effective similarity for adversarial pairs (average across all six categories):

• MiniLM: 0.63
• BGE: 0.72
• Nomic: 0.57
• GTE: 0.71

Effective similarity gap (paraphrase mean minus adversarial mean):

• MiniLM: 0.19
• BGE: 0.11
• Nomic: 0.10
• GTE: 0.08

7.3 Interpretation

The effective similarity gap is a calibration-independent measure of how well a model separates paraphrases from adversarial pairs. If the paradox were purely a calibration artifact, this gap would be constant or increase with model size (reflecting the richer representations of larger models). Instead, it decreases monotonically: from 0.19 for MiniLM to 0.08 for GTE-large.

This means that even after accounting for anisotropy, GTE-large's adversarial pairs and paraphrase pairs are more similar to each other (in the effective similarity space) than MiniLM's. The compression is not merely a shift in the output distribution — it reflects a genuine loss of discriminative information at the cosine similarity level. The cosine similarity metric, when applied to increasingly anisotropic spaces, progressively loses the capacity to reflect the semantic differences that may be present in the model's internal representations.

7.4 Calibrated Threshold Analysis

We can also define calibrated thresholds τ_cal = s_baseline + α × (1 - s_baseline) for a constant α = 0.85:

• MiniLM: τ_cal ≈ 0.86
• BGE: τ_cal ≈ 0.92
• Nomic: τ_cal ≈ 0.93
• GTE: τ_cal ≈ 0.96

Under these calibrated thresholds, failure rates decrease for larger models but the relative ranking is preserved: MiniLM still achieves the lowest failure rate, and GTE the highest. The very high calibrated thresholds required for larger models (0.96 for GTE) leave an extremely narrow band between the threshold and the maximum possible similarity of 1.0, making false negative errors (rejecting true paraphrases) increasingly likely.

8. The Benchmark Disconnect

8.1 Why MTEB Misses These Failures

The paradox we document is not detected by standard MTEB evaluation for several structural reasons:

Rank-based metrics. MTEB's primary metrics — Spearman correlation for STS tasks, NDCG and MRR for retrieval tasks — measure relative ordering rather than absolute calibration. A model that assigns cosine similarities of 0.95 to paraphrases and 0.92 to adversarial pairs will achieve high rank correlation even though both scores exceed any reasonable threshold.

Easy negative distributions. Retrieval benchmarks evaluate ranking against topically different documents ("easy negatives"). The adversarial pairs in our evaluation are "hard negatives" with high lexical overlap and minimal token differences — rarely represented in standard benchmark test sets.

Aggregation across tasks. MTEB's aggregate score averages across diverse tasks, diluting failures on specific semantic operations.

Absence of targeted probes. MTEB does not include targeted evaluations for specific compositional operations like negation, entity swaps, or quantifier sensitivity.

8.2 Implications for Benchmark Design

Our findings suggest embedding benchmarks should include:

1. Targeted compositional probes evaluated using threshold-based and gap-based metrics, not just rank correlation.
2. Anisotropy reporting as a standard model metadata field, analogous to parameter count and embedding dimension.
3. Calibration metrics that evaluate absolute similarity score reliability alongside relative ranking performance.
4. Effective similarity gap measurements that quantify the separation between paraphrase and adversarial distributions in a calibration-independent manner.

9. When Bigger IS Better: Positive Controls

9.1 Paraphrase Performance

The paradox specifically concerns semantic discrimination — the ability to separate pairs that should be separated. On the complementary task of semantic identification — assigning high similarity to true paraphrases — larger models perform well across the board:

• MiniLM: mean raw cosine ~0.83 on paraphrases
• BGE: mean raw cosine ~0.91 on paraphrases
• Nomic: mean raw cosine ~0.89 on paraphrases
• GTE: mean raw cosine ~0.94 on paraphrases

Larger models assign higher raw similarities to paraphrases. However, this advantage is diminished when viewed through the effective similarity lens (Section 7.2), where the values are comparable across models.

9.2 Negative Control Separation

On negative controls (completely unrelated sentences):

• MiniLM: mean cosine ~0.15 (effective: ~0.10)
• BGE: mean cosine ~0.55 (effective: ~0.16)
• Nomic: mean cosine ~0.65 (effective: ~0.18)
• GTE: mean cosine ~0.78 (effective: ~0.24)

The raw gap between negatives and paraphrases is 0.68 for MiniLM but only 0.16 for GTE. After calibration, the effective gaps are more comparable, but GTE's effective negative cosine of 0.24 is still notably higher than MiniLM's 0.10 — indicating that even calibrated larger models assign higher effective similarity to unrelated content.

9.3 Retrieval Performance

We emphasize that our findings do not imply that smaller models are universally better. On ranking-based retrieval tasks where relative ordering matters more than absolute thresholds, larger models likely maintain their advantages. The paradox is specific to threshold-based applications — duplicate detection, semantic search with cutoffs, automated contradiction detection, fact verification — where the absolute value of cosine similarity drives system behavior.

10. Practical Implications

10.1 Model Selection Guidelines

Our findings suggest a nuanced approach to model selection:

For threshold-based applications (duplicate detection, contradiction detection, semantic filtering), practitioners should strongly consider smaller models with wider effective similarity ranges. MiniLM's near-isotropic embedding space provides a more reliable foundation for threshold-based decisions, despite its lower MTEB ranking.

For ranking-based applications (retrieval, reranking, nearest-neighbor search), larger models may still be preferred if they produce better relative orderings.

For mixed workloads, practitioners should either (a) use a smaller model to preserve threshold reliability, (b) calibrate thresholds per model using the effective similarity normalization described in Section 3.5, or (c) implement a two-stage pipeline with a larger model for retrieval and a cross-encoder or classifier for threshold-based decisions.

10.2 Anisotropy-Aware Deployment Protocol

For practitioners committed to using a larger model, we recommend:

1. Compute the random baseline cosine similarity by encoding 100+ diverse, unrelated sentences and computing mean pairwise cosine.
2. Compute effective similarities using the normalization formula.
3. Set thresholds in the effective similarity space rather than the raw cosine space.
4. Validate against a set of challenging pairs (negations, near-misses) before deployment.
5. Monitor the effective similarity gap between accept/reject populations over time.

10.3 Post-hoc Anisotropy Correction

Techniques such as whitening, mean centering, and PCA-based normalization can partially correct anisotropy. However, they require a representative corpus, add deployment complexity, and may introduce distributional assumptions that degrade performance on out-of-distribution inputs. The effective similarity normalization we propose is a simpler alternative that requires only the random baseline estimate.

10.4 Don't Blindly Scale

The most important practical implication: do not assume that a bigger model is better for your task. The MTEB leaderboard is a useful starting point, but it does not evaluate the specific capabilities that many practical applications depend on. Before deploying a larger model, evaluate it on the specific types of semantic distinctions your application requires. A 22M-parameter model may outperform a 335M-parameter model on the exact task you need.

11. Limitations

We acknowledge several limitations:

Sample of models. Four models is sufficient to demonstrate the paradox but insufficient to establish precise scaling laws. The relationship between parameter count and failure rate may be mediated by factors confounded with model size: architecture depth, embedding dimensionality, pooling strategy, training data, and training objective all differ across our models. We cannot attribute the paradox to model size per se — it may be driven by correlated factors. Future work should evaluate larger samples of models with controlled variation in individual factors.

Threshold dependence. Our primary analysis uses τ = 0.85. While Section 7 demonstrates that the paradox persists under calibrated evaluation, the fixed-threshold results are naturally sensitive to threshold choice. At very low thresholds, all models succeed; at very high thresholds, all models fail. The paradox is most pronounced in the operationally relevant range of 0.7–0.9.

Adversarial construction. Our test pairs are hand-crafted adversarial examples that may overestimate failure severity compared to naturally occurring sentence pairs. However, the failure modes we test (negation, numerical changes, entity swaps) do occur in real-world applications with severe consequences in healthcare, finance, and legal domains.

Anisotropy measurement. Our random baseline is computed from 50 sentences. Larger samples and alternative measures (PCA analysis, intrinsic dimensionality estimation) would provide more robust anisotropy characterization. We chose 50 sentences for consistency with prior anisotropy measurement protocols.

Correlation vs. causation. The monotonic relationship we observe is correlational. Controlled experiments that vary model size while holding training data and objectives constant would be needed to establish a causal relationship between parameter count and anisotropy-induced failure.

Dataset availability. All 371 test pairs, per-pair similarity scores, and reproduction code are provided in the supplementary materials. The test_pairs.py file contains all sentence pairs organized by category.

12. Related Work

Compositional semantics in embeddings. Ettinger (2020) systematically evaluated BERT-like models on psycholinguistically motivated tests including negation and role sensitivity, finding substantial failures. The PAWS dataset (Zhang et al., 2019) demonstrated that paraphrase detection models fail on adversarial pairs with high word overlap but different meanings. Our work extends these findings by establishing a systematic relationship between model scale and failure severity, and by providing calibration-independent evidence through the effective similarity gap analysis.

Embedding anisotropy. Ethayarajh (2019) documented that contextual embeddings from pretrained language models are highly anisotropic, with anisotropy increasing in later layers. Li et al. (2020) analyzed representation degeneration in language models. Su et al. (2021) proposed whitening as a post-hoc correction. Our contribution is connecting anisotropy specifically to model scale in the sentence embedding setting, measuring it for all four models, and demonstrating that its practical consequences persist even after calibration.

Scaling laws. Kaplan et al. (2020) established power-law scaling relationships for language model loss. Our findings represent a counterexample to the general principle that scaling improves performance, for the specific class of threshold-based semantic discrimination tasks.

Benchmark limitations. The original MTEB evaluation acknowledges that no single benchmark captures all relevant capabilities. Our work provides a concrete example of a capability gap — compositional semantic discrimination under threshold evaluation — that is systematically missed by current protocols.

13. Conclusion

We have presented evidence of the model size paradox in sentence embeddings: across four models spanning 22M to 335M parameters, larger models exhibit systematically higher failure rates on fine-grained semantic discrimination tasks. GTE-large (335M parameters) fails on 97% (95% CI: 94–99%) of adversarial pairs at a standard cosine similarity threshold, compared to 60% (54–66%) for MiniLM (22M parameters). This paradox holds across every tested category and is explained by increasing anisotropy in larger models' embedding spaces.

Critically, we demonstrate that the paradox is not merely a calibration artifact. The effective similarity gap — a calibration-independent measure of the separation between paraphrase and adversarial distributions — narrows monotonically from 0.19 (MiniLM) to 0.08 (GTE-large). Even after accounting for anisotropy, larger models compress discriminative information at the cosine similarity level.

We recommend that the embedding community adopt targeted compositional probes and effective similarity gap measurements as standard evaluation components, require anisotropy reporting as part of model documentation, and develop calibration-aware benchmarks. Until these changes are adopted, practitioners should evaluate candidate models on their specific semantic requirements rather than relying on aggregate leaderboard positions.

The model size paradox is not an indictment of large models — they likely produce richer internal representations. Rather, it is an indictment of the evaluation and deployment pipeline that conflates aggregate benchmark performance with task-specific capability, and of the cosine similarity metric that fails to preserve discriminative information in anisotropic spaces. Fixing the paradox requires not smaller models, but better metrics, better benchmarks, and better awareness of the geometric properties of the spaces we deploy.

References

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.

Ethayarajh, K. (2019). How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65.

Ettinger, A. (2020). What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48.

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.

Li, B., Zhou, H., He, J., Wang, M., Yang, Y., and Li, L. (2020). On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119–9130.

Reimers, N., and Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.

Su, J., Cao, J., Liu, W., and Ou, Y. (2021). Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316.

Zhang, Y., Baldridge, J., and He, L. (2019). PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298–1308.