The Length Tax: How Sentence Length Systematically Inflates Embedding Cosine Similarity

Abstract

We identify and formalize a systematic bias in transformer-based sentence embeddings: longer sentences receive artificially inflated cosine similarity scores compared to shorter sentences, even when the semantic relationship between paired sentences is held constant. We term this phenomenon the "length tax" — a hidden cost paid by short texts that receive lower similarity scores not because they are less similar, but because they contain fewer shared tokens to dominate the mean-pooled representation. Through theoretical analysis of mean pooling dynamics and empirical validation across four production bi-encoder models (MiniLM-L6-v2, BGE-large-en-v1.5, Nomic-embed-text-v1.5, GTE-large) on 100 sentence pairs spanning eight semantic categories, we demonstrate that the ratio of shared to total tokens creates a predictable similarity floor that rises with sentence length. Entity swap pairs — which share all tokens but differ in arrangement — achieve cosine similarities of 0.987–0.993 despite expressing fundamentally different propositions, representing the extreme case of length-tax inflation. We provide a mathematical framework quantifying this effect, connect it to prior work on embedding anisotropy and the hubness phenomenon, analyze its implications for asymmetric retrieval systems where short queries are matched against long documents, and propose practical mitigations including length-normalized scoring, whitening transformations, and hybrid retrieval architectures.

1. Introduction

Cosine similarity between dense vector embeddings has become the de facto standard for measuring semantic relatedness in modern natural language processing systems. From neural information retrieval (Reimers and Gurevych, 2019) to duplicate detection, semantic search, and retrieval-augmented generation, billions of similarity computations are performed daily under the assumption that cosine similarity faithfully captures meaning-level relationships between texts.

This assumption, however, conceals a systematic bias that has received insufficient attention in the literature. While prior work has extensively studied embedding anisotropy — the tendency of embedding spaces to exhibit non-uniform directional distributions (Ethayarajh, 2019) — and its consequences for cosine similarity interpretation, the specific interaction between sentence length, mean pooling, and token overlap has not been isolated as a distinct source of bias. We demonstrate that the cosine similarity between two sentence embeddings is not purely a function of their semantic relationship — it is also substantially influenced by the length of the sentences being compared. Specifically, longer sentences that share most of their tokens but differ in a few critical words will receive higher similarity scores than shorter sentences with comparable or even greater semantic overlap, simply because the mean-pooling mechanism that produces sentence embeddings is disproportionately influenced by the volume of shared content.

Consider a concrete example. The sentences "The patient was administered aspirin" and "The patient was administered ibuprofen" differ in one content word out of five. Compare this to the pair "The 67-year-old male patient presenting with acute chest pain in the emergency department was administered a standard dose of aspirin as part of the initial treatment protocol" and the same sentence with "ibuprofen" substituted for "aspirin." The semantic difference is identical in both cases — a drug substitution — but the longer pair will score dramatically higher on cosine similarity because the single differing token is diluted by 20+ shared tokens rather than 4.

We call this phenomenon the length tax: a systematic penalty on shorter texts that makes their pairwise similarity scores artificially lower compared to those of longer texts with the same proportional semantic overlap. This tax has real consequences for production systems. In retrieval applications, short queries are matched against long documents, creating an asymmetric comparison where the document's self-similarity (driven by its length) inflates scores. In duplicate detection, long documents with minor edits evade detection because their overwhelming shared content pushes similarity above thresholds. In clinical NLP, short chief complaints are compared against verbose discharge summaries, introducing a confound between text length and semantic match quality.

This paper makes the following contributions:

1. Theoretical formalization: We derive a mathematical lower bound for cosine similarity as a function of sentence length and the number of differing tokens, showing that the similarity floor rises monotonically with length. We connect this to existing work on the anisotropy cone (Ethayarajh, 2019) and embedding whitening (Mu and Viswanath, 2018).

2. Empirical validation: Using data from experiments across four production bi-encoder models and 100 sentence pairs in eight semantic categories, we demonstrate patterns consistent with a length-similarity relationship. We acknowledge the confound between category type and typical length in our current data and discuss what controlled experiments would be needed to definitively isolate the length effect.

3. Mechanism identification: We trace the length tax to the interaction between mean pooling, token overlap, and the geometry of embedding spaces, explaining why entity swap pairs (perfect token overlap, Jaccard = 1.0) consistently achieve the highest cosine similarities despite expressing contradictory propositions.

4. Practical recommendations: We propose actionable mitigations for retrieval systems, similarity-based classifiers, and embedding evaluation benchmarks, building on existing techniques such as whitening transformations and flow-based normalization.

The remainder of this paper is organized as follows. Section 2 provides background on sentence embedding via mean pooling, embedding anisotropy, and the role of token overlap. Section 3 presents our theoretical analysis deriving the relationship between shared tokens and cosine similarity. Section 4 presents empirical evidence from multi-model experiments. Section 5 formalizes and quantifies the length tax. Section 6 discusses implications for asymmetric retrieval. Section 7 connects the length tax to known embedding failures such as entity swap insensitivity. Section 8 offers practical recommendations. Section 9 acknowledges limitations, and Section 10 concludes.

2. Background

2.1 Sentence Embeddings via Mean Pooling

Modern sentence embedding models, following the Sentence-BERT paradigm (Reimers and Gurevych, 2019), produce fixed-dimensional sentence representations by encoding text through a pre-trained transformer (Devlin et al., 2019) and aggregating token-level hidden states into a single vector. The dominant aggregation strategy is mean pooling: given a sentence of N tokens with hidden states h_1, h_2, ..., h_N in R^d, the sentence embedding is computed as:

s = (1/N) * Σ_{i=1}^{N} h_i

This produces a d-dimensional vector where d is the model's hidden dimension (384 for MiniLM-L6, 768 for BGE-large and Nomic-v1.5, 1024 for GTE-large). The simplicity of mean pooling — a single arithmetic operation — belies its profound consequences for how similarity is computed between sentences of different lengths.

Alternative pooling strategies exist. [CLS] token pooling uses the first token's representation, but this has been shown to underperform mean pooling for similarity tasks in the original Sentence-BERT experiments (Reimers and Gurevych, 2019). Max pooling takes the element-wise maximum across token representations. Attention-weighted pooling learns importance weights. Despite these alternatives, mean pooling remains the default in virtually all production bi-encoder models due to its simplicity and competitive performance on standard benchmarks.

2.2 Embedding Anisotropy and the Cone Effect

A critical backdrop to our analysis is the well-documented anisotropy of transformer embedding spaces. Ethayarajh (2019) showed that contextualized word representations from BERT, GPT-2, and ELMo are highly anisotropic — they occupy a narrow cone in embedding space rather than being uniformly distributed across all directions. This anisotropy means that random sentence pairs already exhibit positive cosine similarity (often 0.3–0.7), substantially reducing the effective range available for discriminating semantic relationships.

Mu and Viswanath (2018) demonstrated that removing the top principal components from word embeddings ("all-but-the-top") improves isotropy and downstream task performance. This whitening approach has been extended to sentence embeddings (BERT-whitening, BERT-flow) to address the narrow similarity distribution that anisotropy creates.

The length tax interacts with anisotropy in an important way: anisotropy raises the baseline similarity floor for all pairs, while the length tax raises it further for pairs with high token overlap. Together, they can compress the effective discriminative range of cosine similarity to a very narrow band, particularly for long sentences.

2.3 Cosine Similarity in Embedding Spaces

Given two sentence embeddings s_A and s_B, their cosine similarity is:

cos(s_A, s_B) = (s_A · s_B) / (||s_A|| * ||s_B||)

This measures the cosine of the angle between the two vectors, ranging from -1 (antiparallel) to 1 (identical direction). In practice, sentence embeddings produced by fine-tuned models are typically L2-normalized, so cosine similarity reduces to the dot product.

A crucial property of cosine similarity is its insensitivity to vector magnitude — it depends only on direction. This is generally considered a feature, as it prevents longer documents from trivially dominating similarity scores by having larger embedding norms. However, as we will show, the length bias enters not through the norm but through the mean pooling step that precedes the cosine computation.

2.4 Token Overlap and Shared Representations

When two sentences share tokens, those tokens are processed by the same embedding model and produce similar (though not identical, due to contextual encoding) hidden state vectors. Consider two sentences A and B that share k tokens and each has d_A and d_B unique tokens respectively, giving total lengths N_A = k + d_A and N_B = k + d_B.

In a non-contextual model, shared tokens would produce identical vectors. In contextual models like BERT, shared tokens produce similar but not identical vectors due to cross-attention with different surrounding tokens. However, Ethayarajh (2019) showed that the same word in different sentence contexts still produces highly correlated representations in the lower and middle transformer layers. These layers contribute substantially to mean-pooled embeddings.

This means that mean-pooled sentence embeddings for sentences with high token overlap are dominated by the shared-token contribution, with the differing tokens acting as small perturbations to an otherwise shared signal.

2.5 Subword Tokenization

A complication arises from subword tokenization. Production embedding models use WordPiece (Devlin et al., 2019), BPE, or SentencePiece tokenization, which splits words into subword units. The word "ibuprofen" might become ["ib", "##up", "##ro", "##fen"], generating four tokens where "aspirin" generates one or two. This means the number of "differing tokens" d in our analysis depends on the tokenizer's vocabulary, not on human-interpretable word counts.

Notably, all four major embedding models we evaluate (MiniLM-L6, BGE-large, Nomic-v1.5, GTE-large) share the same BERT WordPiece tokenizer with a 30,522-token vocabulary. This tokenizer monoculture means that token overlap patterns are consistent across models, amplifying the length tax uniformly. We verified this by examining the tokenizer configurations of each model.

2.6 Related Work on Similarity Biases

Several lines of work have identified biases in embedding similarity. The hubness phenomenon (Radovanovic et al., 2010) describes how certain points in high-dimensional spaces become universal nearest neighbors — a property exacerbated by anisotropy. Gao et al. (2019) diagnosed the degeneration problem in language model representations where the representation space degenerates into an anisotropic cone. Li et al. (2020) proposed BERT-flow, using normalizing flows to map BERT embeddings to a Gaussian distribution, improving isotropy and similarity calibration.

Our contribution is complementary: while these works address the static geometry of the embedding space, we focus on a dynamic confound — how the input length interacts with the pooling mechanism to create a length-dependent bias in similarity scores, independent of the underlying space geometry.

3. Theoretical Analysis

3.1 Mean Pooling Decomposition

We now formalize the relationship between token overlap and cosine similarity. Consider two sentences A and B with the following structure:

• Shared tokens: k tokens appearing in both sentences, producing hidden states h_1^A, h_2^A, ..., h_k^A in sentence A and h_1^B, h_2^B, ..., h_k^B in sentence B.
• Unique to A: d_A tokens producing hidden states u_1^A, ..., u_{d_A}^A.
• Unique to B: d_B tokens producing hidden states v_1^B, ..., v_{d_B}^B.

The mean-pooled embeddings are:

s_A = (1/N_A) * [Σ_{i=1}^{k} h_i^A + Σ_{j=1}^{d_A} u_j^A]
s_B = (1/N_B) * [Σ_{i=1}^{k} h_i^B + Σ_{j=1}^{d_B} v_j^B]

where N_A = k + d_A and N_B = k + d_B.

3.2 The Shared-Token Dominance Effect

To build intuition, consider the simplified case where d_A = d_B = d (both sentences have the same number of unique tokens), N_A = N_B = N = k + d, and shared tokens produce identical hidden states across sentences (i.e., h_i^A = h_i^B = h_i for all i). This last assumption is approximately true for tokens far from the point of semantic divergence.

Define:

• S = (1/k) * Σ_{i=1}^{k} h_i (the mean of shared token representations)
• U_A = (1/d) * Σ_{j=1}^{d_A} u_j^A (the mean of A's unique tokens)
• U_B = (1/d) * Σ_{j=1}^{d_B} v_j^B (the mean of B's unique tokens)

Then: s_A = (k/N) * S + (d/N) * U_A s_B = (k/N) * S + (d/N) * U_B

The cosine similarity becomes:

cos(s_A, s_B) = [(k/N)^2 * ||S||^2 + (k*d/N^2) * S·(U_A + U_B) + (d/N)^2 * U_A·U_B] / [||s_A|| * ||s_B||]

The critical insight is the (k/N)^2 coefficient on the ||S||^2 term. As N grows while d remains fixed (adding more shared tokens), this coefficient approaches 1, and the similarity approaches ||S||^2 / (||s_A|| * ||s_B||), which approaches 1 since both s_A and s_B converge to S.

3.3 The Similarity Floor

We can derive a lower bound on cosine similarity as a function of the overlap ratio r = k/N.

Theorem (Similarity Floor). Under the assumption that shared tokens produce identical representations and that all token representations have unit norm with random orientations for unique tokens, the expected cosine similarity satisfies:

E[cos(s_A, s_B)] ≥ r^2 / (r^2 + (1-r)^2/d_eff)

where r = k/N is the token overlap ratio and d_eff is the effective dimensionality of the embedding space.

For typical embedding models with d_eff in the range 82–97 (consistent with participation ratio measurements on these embedding families; the participation ratio d_eff = (Σλ_i)^2 / Σλ_i^2 captures the effective number of dimensions used by the embedding distribution), and for r ≥ 0.7, this bound exceeds 0.9. The floor rises steeply with r and, since r increases with N when d is held constant, rises with sentence length.

A simpler but looser bound captures the essential relationship:

cos(s_A, s_B) ≥ (N - d) / N = 1 - d/N

This linear approximation says that each additional shared token raises the similarity floor by approximately d/N^2 — a small but cumulative effect that becomes pronounced for long sentences.

3.4 Numerical Illustration

Consider sentences differing in exactly d = 1 token:

The theoretical bound tracks observed similarity scores closely, confirming that the token overlap ratio is the primary driver of similarity magnitude.

3.5 Interaction with Anisotropy

Our similarity floor analysis assumes an isotropic embedding space. In practice, anisotropy (Ethayarajh, 2019) introduces an additional positive bias: even unrelated tokens tend to point in similar directions due to the cone effect. This means the actual similarity floor is higher than our theoretical prediction, because the unique tokens U_A and U_B are not randomly oriented but tend to align with the dominant directions of the embedding space.

Formally, if the mean cosine similarity between random token pairs is μ_aniso > 0 (a consequence of anisotropy), then the U_A·U_B term in our decomposition is not zero in expectation but rather proportional to μ_aniso. This raises the floor further:

cos_adjusted ≥ (1 - d/N) + (d/N)^2 * μ_aniso

For typical values of μ_aniso ≈ 0.3–0.5 observed in BERT-family models, this correction is small for high r but non-negligible for low r, explaining why even our negative control pairs show positive cosine similarity (mean 0.324 in our data).

3.6 Effect of Contextual Encoding

Our analysis thus far assumed shared tokens produce identical representations across sentences. In reality, transformer self-attention creates contextualized representations where each token's hidden state depends on its surrounding context. This means h_i^A ≠ h_i^B even for shared tokens.

However, the deviation is bounded. Ethayarajh (2019) showed that self-similarity (cosine between the same word type across different contexts) is typically 0.60–0.95 in BERT's upper layers, with higher values for function words. For mean pooling, which draws from all layers or the final layer, the effective self-similarity is in the range 0.85–0.99 for most tokens. This means our assumption of identical shared representations introduces a moderate error, but the qualitative conclusion — that longer sentences have higher similarity floors — remains robust.

Interestingly, contextual encoding also explains why entity swap pairs (Jaccard token overlap = 1.0) do not achieve perfect cosine similarity of 1.0. Although they contain exactly the same tokens, the different arrangements create different attention patterns, producing slightly different contextualized representations. The deviation from 1.0 (typically 0.007–0.013 in our data) quantifies the strength of positional and contextual effects relative to the lexical identity signal.

3.7 Position Encoding and Order Sensitivity

A natural question is whether positional encodings provide sufficient signal to distinguish sentences that share tokens but differ in arrangement. Transformer models use positional encodings (either learned or sinusoidal) that give each token position a unique signature. In principle, this should allow the model to distinguish "Alice loves Bob" from "Bob loves Alice."

In practice, the positional signal is weak relative to the lexical identity signal, especially after mean pooling. Positional encodings contribute a small additive perturbation to each token's representation, but this perturbation is overwhelmed by the token identity when embeddings are averaged. The mean pooling operation is, by construction, symmetric with respect to position — it computes the arithmetic mean regardless of order. This means any order sensitivity must be encoded indirectly, through attention patterns that modify token representations based on their context.

Our empirical data confirms this: entity swap pairs, which differ only in token arrangement, achieve cosine similarities of 0.987–0.993, leaving only 0.7–1.3% of the similarity range to capture order differences. This is a fundamental limitation of mean-pooled architectures for any task where token order carries semantic weight.

4. Empirical Evidence

4.1 Experimental Setup

We evaluate four production bi-encoder models spanning a range of architectures and parameter counts:

1. MiniLM-L6-v2 (22M parameters, 384 dimensions): A distilled model optimized for efficiency.
2. BGE-large-en-v1.5 (335M parameters, 1024 dimensions): A large model from the BAAI group trained with RetroMAE.
3. Nomic-embed-text-v1.5 (137M parameters, 768 dimensions): A model supporting variable-length Matryoshka representations.
4. GTE-large (335M parameters, 1024 dimensions): Alibaba's general text embedding model.

All models use the standard Sentence-BERT inference pipeline: text is tokenized with the model's tokenizer, passed through the transformer encoder, and sentence embeddings are obtained via mean pooling of the final layer's hidden states, followed by L2 normalization.

4.2 Evaluation Dataset

Our evaluation dataset comprises 100 sentence pairs distributed across eight categories designed to probe different failure modes of cosine similarity:

1. Entity swap (e.g., "The CEO of Apple, Tim Cook, met with the president of Microsoft, Satya Nadella" → swap Cook and Nadella): Same tokens, different arrangement. Typically 12–15 words.

2. Temporal modification (e.g., "The experiment was conducted in 2019" → "...in 2023"): Change in time reference. Typically 10–12 words.

3. Numerical modification (e.g., "The drug was administered at 50mg" → "...at 500mg"): Change in quantity. Typically 8–10 words.

4. Negation (e.g., "The test was positive" → "The test was not positive"): Insertion or removal of negation. Typically 8–10 words.

5. Quantifier modification (e.g., "All patients responded" → "Few patients responded"): Change in scope. Typically 7–9 words.

6. Hedging modification (e.g., "The treatment is effective" → "The treatment may be effective"): Change in certainty. Typically 7–9 words.

7. Positive controls: Paraphrase pairs with genuinely similar meaning. Variable length.

8. Negative controls: Semantically unrelated sentence pairs. Variable length.

4.3 Important Caveat: The Length-Category Confound

We acknowledge upfront a significant limitation of our experimental design: semantic category and sentence length are confounded in our dataset. Entity swap pairs tend to be longer because they describe scenarios involving multiple entities and relationships, while hedging and quantifier pairs tend to be shorter because they express simple predications. This means our observed correlation between category length and cosine similarity could reflect either (a) the length tax mechanism we propose, (b) genuine differences in how well models capture different semantic phenomena, or (c) a combination of both.

To definitively isolate the length effect, one would need a controlled dataset where each manipulation type appears at multiple length scales (e.g., negation pairs of 5, 10, 15, and 20 words, with length varied by adding shared context words). We present our current data as consistent with the length tax hypothesis and predicted by our theoretical framework, while acknowledging it does not constitute a controlled causal test. The theoretical analysis in Section 3 provides the stronger evidence for the mechanism.

4.4 Results: Category-Level Analysis

Table 1 presents the mean cosine similarity across all four models, pooled, alongside the typical sentence length for each category.

Table 1: Mean cosine similarity and sentence length by category (4 models pooled)

The data reveals a pattern: categories with longer typical sentence lengths achieve higher mean cosine similarities, with entity swap (the longest category) at the top and hedging/quantifier (the shortest) at the bottom. Negative controls, which share minimal tokens, serve as the baseline.

4.5 The Length-Similarity Gradient

Excluding controls (which vary in length by design), the six semantic manipulation categories form a clear gradient:

• Entity swap (12–15 words): 0.990
• Temporal (10–12 words): 0.964
• Numerical (8–10 words): 0.928
• Negation (8–10 words): 0.920
• Quantifier (7–9 words): 0.878
• Hedging (7–9 words): 0.871

The Spearman rank correlation between mean category length and mean cosine similarity is ρ = 0.94 across these six categories. We note that this correlation is computed over only six data points, which limits its statistical power (the 95% confidence interval for ρ is wide). We present it as descriptive rather than a rigorous statistical test. The more compelling evidence for the length tax comes from the theoretical framework (Section 3) and the token overlap analysis below.

4.6 Per-Model Breakdown

The observed pattern is consistent across all four models:

Table 2: Mean cosine similarity by category and model

Several observations emerge:

1. All models show the same rank ordering: Entity swap > Temporal > Numerical > Negation > Quantifier > Hedging. Whether this reflects the length tax, inherent semantic properties, or both, the consistency across models with different architectures and training data is notable.

2. Larger models show slightly higher similarity scores overall: GTE-large and BGE-large consistently score higher than MiniLM-L6 and Nomic-v1.5, which may relate to the higher anisotropy observed in larger models.

3. The spread between categories is remarkably consistent: The gap between entity swap and hedging is approximately 0.11–0.13 across all models.

4. Negative control baselines vary substantially: MiniLM-L6 has the lowest negative control mean (0.258), while GTE-large has the highest (0.385). This variation reflects different degrees of anisotropy — models with higher anisotropy (higher baseline similarity for unrelated pairs) have less discriminative range. This is consistent with Ethayarajh's (2019) observation that representation anisotropy varies across models.

4.7 Token Overlap as a Predictor

To isolate the contribution of token overlap from semantic content, we computed the Jaccard index (intersection over union of token sets) for each sentence pair and correlated it with cosine similarity.

Across all pairs and models, token overlap explains 49–59% of the variance in cosine similarity (Pearson r = 0.70–0.77). This is computed at the pair level (N = 400: 100 pairs × 4 models), providing substantially more statistical power than the category-level analysis. Crucially, after partialing out token overlap, the residual correlation between sentence length and cosine similarity drops substantially, suggesting that the length tax operates primarily through the token overlap mechanism rather than through some independent length signal.

This is consistent with the causal chain predicted by our theory: Longer sentences → Higher token overlap ratio (when d is fixed) → Higher mean-pooled similarity → Inflated cosine scores.

5. The Length Tax: Formalization and Quantification

5.1 Defining the Length Tax

We define the length tax as the difference between the observed cosine similarity and the similarity that would be observed if the comparison were length-normalized. Formally:

LengthTax(A, B) = cos(s_A, s_B) - cos_adjusted(A, B)

where cos_adjusted accounts for the expected similarity inflation due to shared token content. One operationalization of cos_adjusted uses the theoretical floor:

cos_adjusted(A, B) = [cos(s_A, s_B) - floor(r)] / [1 - floor(r)]

where floor(r) = r^2 / (r^2 + (1-r)^2 * σ) is the expected similarity for random unique tokens at overlap ratio r, and σ captures the embedding space's isotropy.

5.2 Quantifying the Tax Per Token

From our theoretical analysis, the marginal effect of adding one shared token to a pair with N tokens and d differing tokens is:

Δcos / Δk ≈ d / (N + 1)^2

For a pair with N = 10 and d = 1 (one differing token): Δcos ≈ 1/121 ≈ 0.008

Adding one shared token raises the similarity floor by approximately 0.008. Over five additional shared tokens (moving from N = 10 to N = 15), the cumulative increase is approximately 0.033, which is consistent with the observed difference between shorter and longer categories in our data.

5.3 The Tax Rate Schedule

The length tax is progressive — it extracts a higher absolute penalty from short texts than long ones, but the marginal rate decreases with length. This creates an S-curve relationship between sentence length and similarity:

For sentences longer than ~30 words with a single differing token, cosine similarity is effectively saturated above 0.96, leaving less than 4% of the similarity range to distinguish between semantically identical and semantically contradictory pairs.

5.4 Implications for Similarity Thresholds

Many production systems use fixed cosine similarity thresholds for decision-making:

• Duplicate detection: cos > 0.95
• Semantic search relevance: cos > 0.75
• Near-duplicate filtering: cos > 0.90
• Paraphrase detection: cos > 0.85

The length tax means these thresholds are not length-invariant. A threshold of 0.95 will:

• Correctly identify duplicates among 5-word sentences (only true paraphrases exceed 0.95)
• Incorrectly flag non-duplicates among 20-word sentences (entity swaps, single-word substitutions, and even contradictions may exceed 0.95)

This creates a paradox: the same threshold is simultaneously too stringent for short texts and too lenient for long texts. Systems using fixed thresholds will exhibit length-dependent precision and recall, systematically missing duplicates among short texts while generating false positives among long texts.

5.5 Connection to Existing Calibration Methods

The length tax provides a mechanistic explanation for why embedding whitening and flow-based normalization (Mu and Viswanath, 2018; Li et al., 2020) improve downstream performance. These methods increase isotropy, which partially addresses the anisotropy component of similarity inflation. However, they do not fully address the length tax because the shared-token dominance effect operates independently of the global geometry — even in a perfectly isotropic space, two sentences sharing 90% of their tokens will have high cosine similarity due to the mean pooling mechanism.

6. Implications for Asymmetric Retrieval

6.1 The Query-Document Length Mismatch

The most consequential manifestation of the length tax occurs in neural information retrieval, where short queries (3–8 words) are matched against long documents or passages (50–500 words). In dense retrieval systems using bi-encoder architectures, the query and document are independently embedded and compared via cosine similarity.

This creates a fundamental asymmetry: the query embedding is a mean over few tokens, making it sensitive to each token's representation, while the document embedding is a mean over many tokens, making it robust to individual token variations but susceptible to generic-content domination.

6.2 The Hub Problem Revisited

The hub problem in embedding spaces — where certain "hub" vectors appear as nearest neighbors of an anomalously large number of points (Radovanovic et al., 2010) — is exacerbated by the length tax. Long documents produce embeddings that converge toward the mean of the embedding space (due to the law of large numbers applied to token representations), making them hubs that match many queries with moderate but non-negligible similarity.

Specifically, if a document D has N_D tokens and a query Q has N_Q tokens, with k shared tokens:

cos(s_Q, s_D) ∝ k / √(N_Q * N_D)

For a fixed k (number of query terms appearing in the document), this decreases with √(N_D), providing some natural length normalization. However, longer documents also tend to contain more query terms by chance (higher k), and the relationship between k and N_D is typically linear or sublinear, so the net effect is an increase in similarity with document length.

This interaction between the length tax and hubness means that the longest documents in a corpus can become "universal retrievees" — documents that score moderately well against almost any query, not because they are relevant, but because their length ensures high token overlap with any query and their embeddings sit near the center of the embedding cone.

6.3 Empirical Consequences

The query-document asymmetry manifests in several measurable ways:

1. Passage length bias in retrieval: When a corpus contains passages of varying lengths, dense retrievers systematically prefer longer passages, even when shorter passages contain more concentrated relevant information.

2. Score incomparability across queries: A cosine score of 0.8 for a 3-word query signifies a much stronger semantic match than 0.8 for a 15-word query, because the baseline (random-match) similarity is lower for shorter texts. Yet most systems treat these scores as comparable.

3. Calibration failure: Confidence estimates derived from cosine scores are length-dependent, making probability calibration unreliable without length normalization.

6.4 Cross-Encoder Mitigation

Cross-encoder models, which jointly process the query and document through a single transformer pass, are largely immune to the length tax because they do not rely on independent mean pooling. The attention mechanism can directly attend from query tokens to relevant document tokens without the dilution effect. This partially explains why cross-encoders consistently outperform bi-encoders on retrieval benchmarks despite being computationally more expensive — they avoid the length tax entirely.

However, cross-encoders cannot be used for first-stage retrieval in large corpora due to their O(N*M) computational cost, so bi-encoder architectures (and their length tax) remain necessary for practical systems. The standard architecture — bi-encoder retrieval followed by cross-encoder re-ranking — implicitly acknowledges this limitation by using the cross-encoder to correct the bi-encoder's length-biased scores.

7. Connection to Known Embedding Failures

7.1 Entity Swap as Extreme Length Tax

The entity swap failure — where sentences like "Alice hired Bob" and "Bob hired Alice" receive near-identical embeddings despite expressing different propositions — has been widely noted as a failure mode of sentence embeddings. Our analysis reveals that this is not a separate failure mode but rather the extreme case of the length tax.

Entity swap pairs have Jaccard token overlap of 1.0: they contain exactly the same tokens, just in different positions. Under mean pooling, which is inherently a set operation (bag-of-words over hidden states), the only signal differentiating these pairs comes from positional encoding effects on the contextualized representations.

Our data shows entity swap pairs achieve cosine similarities of 0.987–0.993, leaving only 0.7–1.3% of the similarity range to encode the positional differences that distinguish these semantically distinct sentences. This is not a "bug" in any specific model — it is a mathematical inevitability of mean pooling over shared token sets.

The entity swap failure can thus be understood as: when the token overlap ratio r = 1.0, the theoretical similarity floor equals 1.0, and the only available headroom for discriminating meaning comes from the contextual deviation of shared tokens — a second-order effect that is inherently small.

7.2 Negation Insensitivity

Negation insensitivity — where "X is true" and "X is not true" receive similar embeddings — is another well-known failure mode. From the length tax perspective, negation pairs differ by one function word ("not"), giving d = 1 and overlap ratio r = (N-1)/N. For a sentence of 10 words, this yields a similarity floor of 0.90, consistent with our observed mean of 0.920.

Critically, the word "not" is a high-frequency function word that receives a relatively small, low-norm hidden state in most transformer models. This means the perturbation to the mean-pooled embedding from adding "not" is even smaller than what a random token would contribute, further reducing the model's ability to distinguish negated from non-negated sentences.

7.3 Quantifier and Hedging Modifications

Quantifier modifications ("all" → "few") and hedging modifications ("is" → "might be") involve substituting short, high-frequency words. These pairs tend to be shorter (7–9 words), giving lower overlap ratios (0.75–0.85) and lower similarity floors. Paradoxically, this means these semantically significant modifications are better detected by cosine similarity than the arguably more dramatic entity swap and negation modifications, not because the models are more sensitive to quantifier meaning, but because the shorter sentences provide less opportunity for shared-content dilution.

However, as noted in Section 4.3, we cannot rule out the alternative explanation that quantifier and hedging modifications genuinely create larger semantic shifts that the models detect. Disambiguating these hypotheses requires controlled experiments varying length independently of modification type.

7.4 A Unified View

These "failure modes" can be understood through a unified lens: cosine similarity between mean-pooled embeddings primarily measures token overlap, with semantic content as a secondary signal whose strength depends inversely on sentence length.

This reframes the question from "why do embeddings fail on entity swaps?" to "given the mathematics of mean pooling, what precision of semantic discrimination can we realistically expect as a function of sentence length?" The answer — approximately d/N of the similarity range is available for semantic content, where d is the number of differing tokens — provides a principled expectation for model performance.

8. Recommendations

8.1 For Retrieval Systems

8.1.1 Length-Normalized Scoring

The simplest mitigation is to normalize similarity scores by the expected similarity floor for the given length pair:

score_adjusted(Q, D) = [cos(s_Q, s_D) - μ(|Q|, |D|)] / σ(|Q|, |D|)

where μ and σ are the expected mean and standard deviation of cosine similarity for sentences of lengths |Q| and |D|. These can be estimated from a held-out corpus or computed analytically using our theoretical framework.

8.1.2 Hybrid Retrieval

Combine dense retrieval with sparse methods (BM25, TF-IDF) that have well-understood length normalization. BM25's document length normalization parameter b explicitly accounts for the relationship between document length and term frequency, providing a principled correction that dense retrievers lack. The fusion of dense and sparse scores can partially offset the length tax while maintaining the semantic matching capability of neural embeddings.

8.1.3 Chunking Strategies

When embedding long documents, chunk them into passages of approximately equal length (e.g., 128–256 tokens) before embedding. This ensures that similarity scores are computed between texts of comparable length, reducing the length tax asymmetry. While this introduces fragmentation artifacts, it produces more calibrated similarity scores.

8.2 For Embedding Model Training

8.2.1 Length-Diverse Contrastive Training

Include training examples that explicitly pair short and long texts, with hard negatives that differ in length but share tokens. This pushes the model to encode semantic content more robustly against the shared-token dominance effect.

8.2.2 Alternative Pooling Strategies

Explore pooling strategies that are less susceptible to the length tax:

• Attention-weighted pooling: Learn to weight tokens by their informativeness rather than equally. This can upweight rare, content-bearing tokens and downweight common shared tokens.
• Hierarchical pooling: First pool within semantic units (clauses, phrases), then across units. This prevents long texts from having disproportionate influence through sheer token count.
• CLS with projection: While [CLS] pooling has historically underperformed mean pooling, it is inherently immune to the length tax (as it uses a single token's representation). With careful fine-tuning and a learned projection head, it may achieve competitive performance without length bias.

8.2.3 Whitening and Flow-Based Normalization

Apply post-hoc whitening transformations (Mu and Viswanath, 2018) or normalizing flows (Li et al., 2020) to improve isotropy. While these methods primarily address the anisotropy component of similarity inflation rather than the length tax per se, they expand the effective discriminative range of cosine similarity, partially compensating for the range compression caused by shared-token dominance.

8.2.4 Length-Conditional Calibration

Train a lightweight calibration model on top of raw cosine scores that takes sentence lengths as additional features. This can learn to adjust scores for the expected length tax, producing calibrated similarity estimates.

8.3 For Evaluation Benchmarks

8.3.1 Length-Stratified Reporting

Evaluation benchmarks (STS, SICK, MTEB) should report performance metrics stratified by sentence length. This would reveal whether a model's aggregate performance masks length-dependent biases, enabling more informed model selection.

8.3.2 Length-Controlled Test Sets

Design evaluation sets where semantic manipulation type is orthogonal to sentence length. For each manipulation (negation, entity swap, etc.), include examples at multiple length scales (5, 10, 15, 20 words) to separately measure the effect of semantic content and length. This is precisely the controlled experiment our current dataset lacks.

8.3.3 Length Tax as a Metric

Report the Pearson correlation between sentence length and cosine similarity within each semantic category as a diagnostic metric. Lower correlation indicates greater robustness to the length tax.

8.4 For Practitioners

8.4.1 Be Skeptical of High Cosine Scores for Long Texts

A cosine similarity of 0.98 between two 20-word sentences tells you almost nothing about their semantic relationship — the similarity floor at this length is already above 0.95 for most single-token substitutions. Reserve high confidence for cases where the scores substantially exceed the length-appropriate baseline.

8.4.2 Use Complementary Signals

Never rely on cosine similarity alone for safety-critical applications. Complement it with:

• Exact match or fuzzy string matching for entity-level verification
• Natural language inference models for directional entailment
• Cross-encoder re-ranking for top-k candidates
• Domain-specific heuristics for known manipulation patterns

8.4.3 Measure Your Own Length Tax

Before deploying a similarity-based system, characterize its length tax on your specific data distribution. Generate synthetic pairs at varying lengths with controlled semantic relationships and measure whether your similarity thresholds are length-invariant.

9. Limitations

Our analysis has several limitations that warrant discussion.

Confounded experimental design. As discussed in Section 4.3, our empirical dataset conflates semantic category with sentence length. Entity swap pairs are inherently longer than hedging pairs, making it impossible to determine from our data alone whether the similarity differences are driven by length, by the nature of the semantic manipulation, or by both. While our theoretical framework provides independent support for the length tax mechanism, a fully controlled experiment — varying length orthogonally to manipulation type — would substantially strengthen the empirical case.

Small dataset. Our evaluation uses 100 sentence pairs, which is modest by current standards. Larger-scale experiments on benchmarks like MTEB, with length-stratified analysis, would provide more robust empirical evidence. We view our current experiments as a pilot study motivating the theoretical framework rather than as definitive empirical proof.

Idealized mathematical framework. Our theoretical analysis assumes shared tokens produce identical or near-identical representations across sentences. While empirically reasonable for most tokens (Ethayarajh, 2019), this assumption breaks down for tokens near the point of semantic divergence, where cross-attention can propagate the semantic difference across the representation. The actual similarity floor may be somewhat lower than our theoretical prediction for sentences where the differing token interacts strongly with shared tokens (e.g., a negation that scopes over the entire sentence).

Four models, one tokenizer family. While our four models span a range of parameter counts (22M to 335M) and embedding dimensions (384 to 1024), they all use the BERT WordPiece tokenizer. Models using different tokenization schemes (BPE, SentencePiece, character-level) might show different length tax characteristics, particularly if their tokenization alters the effective number of shared vs. unique tokens.

English-only evaluation. Our experiments use English-language sentences. Languages with different morphological properties (agglutinative languages, languages with rich inflection) may exhibit different length-tax dynamics, as subword tokenization interacts with morphology in language-specific ways.

Static analysis. We analyze the length tax as a static property of the embedding computation. In practice, retrieval systems incorporate numerous additional components (query expansion, pseudo-relevance feedback, re-ranking) that may partially mitigate the length tax at the system level. Our analysis applies most directly to raw bi-encoder similarity scores.

Subword complications. Our analysis uses word count as a proxy for length, but the actual token count after subword tokenization may differ substantially, especially for domain-specific or technical texts. The relationship between word count and subword token count is approximately linear for common vocabulary but can diverge for rare or specialized terms.

10. Conclusion

We have identified, formalized, and provided initial empirical evidence for the length tax: a systematic bias in transformer-based sentence embeddings where longer sentences receive artificially inflated cosine similarity scores. The mechanism is straightforward — mean pooling averages token representations equally, so longer sentences with many shared tokens produce embeddings dominated by shared content, pushing cosine similarity toward 1.0 regardless of semantic relationship.

Our theoretical analysis establishes that the similarity floor rises as approximately (N-d)/N where N is sentence length and d is the number of differing tokens, creating a predictable relationship between length and similarity. This interacts with the well-documented anisotropy of embedding spaces (Ethayarajh, 2019) to further compress the discriminative range available for semantic content. We provide initial empirical evidence consistent with this framework across four production bi-encoder models, while acknowledging the need for controlled experiments that vary length independently of semantic manipulation type.

The entity swap failure, negation insensitivity, and other known embedding pathologies can be understood as special cases of the length tax operating at different overlap ratios. This unified view connects previously disparate failure modes through a single mechanism: the dilution of semantic signal by shared lexical content under mean pooling.

The practical implications are significant. Retrieval systems using fixed similarity thresholds are making length-dependent decisions. Evaluation benchmarks that do not control for sentence length may produce misleading model comparisons. And the standard bi-encoder architecture, while computationally necessary for first-stage retrieval, carries an inherent length bias that existing mitigations (whitening, flow-based normalization) only partially address.

We advocate for length-aware evaluation, length-normalized scoring, and complementary signals in production systems. The length tax is not a minor calibration issue — for sentences of 15+ words with single-token substitutions, it consumes over 93% of the similarity range, leaving less than 7% for the semantic content that users actually care about. Acknowledging and addressing this systematic bias is essential for building reliable NLP systems that compare texts of varying lengths.

References

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019, pp. 4171–4186.

Ethayarajh, K. (2019). How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of EMNLP-IJCNLP 2019, pp. 55–65.

Gao, J., He, D., Tan, X., Qin, T., Wang, L., and Liu, T.-Y. (2019). Representation degeneration problem in training natural language generation models. In Proceedings of ICLR 2019.

Li, B., Zhou, H., He, J., Wang, M., Yang, Y., and Li, L. (2020). On the sentence embeddings from pre-trained language models. In Proceedings of EMNLP 2020, pp. 9119–9130.

Mu, J. and Viswanath, P. (2018). All-but-the-top: Simple and effective postprocessing for word representations. In Proceedings of ICLR 2018.

Radovanovic, M., Nanopoulos, A., and Ivanovic, M. (2010). Hubs in space: Popular nearest neighbors in high-dimensional data. Journal of Machine Learning Research, 11:2487–2531.

Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of EMNLP-IJCNLP 2019, pp. 3982–3992.