1. Introduction

The dominant paradigm for computing semantic similarity in modern NLP systems relies on bi-encoder architectures: two sentences are independently encoded into fixed-dimensional vectors, and their similarity is estimated via cosine distance (Reimers and Gurevych, 2019). This approach powers semantic search engines, retrieval-augmented generation pipelines, duplicate detection systems, and countless production applications that depend on similarity judgments.

Prior work on embedding failure modes has demonstrated that bi-encoder models systematically fail on compositional semantic tasks. When applied to sentence pairs involving negation, entity role swaps, or numerical changes, bi-encoder models consistently assign high cosine similarity scores — treating semantically opposite or critically different sentences as near-identical. Ettinger (2020) documented that even pre-trained BERT representations struggle with negation and other compositional phenomena, while Reimers and Gurevych (2019) noted that the mean pooling operation in Sentence-BERT discards positional and compositional information. These failures stem from a fundamental architectural constraint: independent encoding with mean pooling creates representations that preserve lexical overlap but erase word order, dilute negation tokens, and ignore compositional structure.

The standard remedy proposed in the literature is the cross-encoder architecture: instead of encoding sentences independently, both sentences are concatenated with a separator token and fed through a single transformer, allowing full cross-attention between all tokens (Devlin et al., 2019; Nogueira and Cho, 2019). This architectural difference theoretically enables cross-encoders to detect word-order changes, attend to negation tokens in context, and reason about compositional relationships.

However, a critical gap exists in the literature: no systematic evaluation has tested cross-encoders against the specific failure modes identified in bi-encoder systems across a controlled set of adversarial categories. We address this with a controlled experiment comparing four cross-encoder models against five bi-encoder models on 336 hand-crafted sentence pairs spanning nine semantic categories. We report all raw model outputs alongside normalized scores and provide pair-level results for full reproducibility.

2. Background

2.1 Bi-Encoder Architecture

Bi-encoder models encode each sentence independently through a shared transformer backbone, producing a fixed-dimensional embedding via mean pooling over token representations. Semantic similarity is computed as cosine similarity between embeddings:

sim(A, B) = cos(encode(A), encode(B))

This architecture enables efficient retrieval through precomputed embeddings and approximate nearest neighbor search, but cannot perform cross-sentence reasoning during encoding. The mean pooling operation is particularly problematic: it averages all token representations into a single vector, which means "The patient has diabetes" and "The patient does not have diabetes" produce nearly identical embeddings because the negation token "not" is diluted across the full sequence average.

2.2 Cross-Encoder Architecture

Cross-encoders process both sentences jointly:

score(A, B) = f(Transformer([CLS] A [SEP] B [SEP]))

where f is typically a linear regression or classification head applied to the [CLS] token representation. Every token in sentence A attends to every token in sentence B through the transformer's self-attention mechanism, enabling direct compositional comparison. The computational cost is substantially higher — O(n) forward passes for n candidates rather than O(1) with precomputed embeddings — making cross-encoders impractical for first-stage retrieval but standard for re-ranking (Nogueira and Cho, 2019).

2.3 Training Objectives

Cross-encoder models differ fundamentally in training objective, which determines the semantics of their output scores:

Semantic Textual Similarity (STS): Models trained on STS-B (Cer et al., 2017) learn to predict human similarity ratings via a regression head. Although the original STS-B annotations use a 0-5 Likert scale, the fine-tuned cross-encoder model (cross-encoder/stsb-roberta-large) uses a single-output regression head with num_labels=1 and no explicit activation function. On typical sentence pairs, the model's raw output naturally ranges between approximately 0 and 1, reflecting the regression head's learned calibration. This does not indicate a 0-1 clamping; rather, the model learns to map its internal representations to a range that empirically spans approximately 0 to 1 on standard inputs.

Information Retrieval: Models trained on MS-MARCO learn query-document relevance via unbounded logit outputs. Relevance is not similarity: a document directly contradicting a query may be highly relevant to the query's topic.

Duplicate Detection: Models trained on Quora Question Pairs learn to classify duplicate questions, outputting probability estimates in [0, 1] via a sigmoid activation.

Reranking: Production rerankers (e.g., BGE-reranker-large) are trained to score candidate passages for relevance, outputting probabilities in [0, 1].

This diversity of output semantics raises a key question: does the theoretical advantage of cross-attention manifest equally across training regimes, or does the training objective determine which compositional failures a model can detect?

3. Experimental Setup

3.1 Models

Five bi-encoder models:

1. all-MiniLM-L6-v2 — compact 6-layer model trained via knowledge distillation (Reimers and Gurevych, 2019)
2. BGE-large-en-v1.5 — large model from BAAI trained with contrastive learning and hard-negative mining
3. nomic-embed-text-v1.5 — model from Nomic AI emphasizing long-context capability
4. mxbai-embed-large-v1 — large model from Mixedbread AI
5. GTE-large — general text embedding model from Thenlper/Alibaba DAMO Academy

Four cross-encoder models:

1. cross-encoder/stsb-roberta-large — RoBERTa-large fine-tuned on STS-B regression; single-output regression head (num_labels=1); raw output observed range 0.009-0.972 on our data
2. cross-encoder/ms-marco-MiniLM-L-12-v2 — MiniLM fine-tuned on MS-MARCO passage retrieval; outputs unbounded logits (observed range -11.3 to +10.6)
3. BAAI/bge-reranker-large — production reranker; outputs probabilities in [0, 1]
4. cross-encoder/quora-roberta-large — RoBERTa-large fine-tuned on Quora duplicate detection; outputs probabilities in [0, 1]

A fifth cross-encoder (cross-encoder/nli-roberta-large, trained on NLI data for contradiction/entailment) was planned but was inaccessible due to HuggingFace repository authentication requirements during our experimental window.

3.2 Test Dataset

We constructed 336 hand-crafted sentence pairs organized into nine categories, designed to isolate specific compositional semantic phenomena:

Adversarial categories (251 pairs):

• Negation (55 pairs): Sentence pairs differing only by negation ("The patient has diabetes" / "The patient does not have diabetes")
• Numerical (56 pairs): Order-of-magnitude numerical changes ("The company employs 500 workers" / "The company employs 50,000 workers")
• Entity swap (45 pairs): Subject-object role reversals ("Google acquired YouTube" / "YouTube acquired Google")
• Temporal (35 pairs): Before/after inversions ("She ate breakfast before going to the gym" / "She ate breakfast after going to the gym")
• Quantifier (35 pairs): Universal vs. existential scope changes ("All students passed the exam" / "Some students passed the exam")
• Hedging (25 pairs): Certainty level modifications ("The drug cures cancer" / "The drug may help with some cancer symptoms")

Control categories (85 pairs):

• Positive control (35 pairs): True paraphrases ("The cat sat on the mat" / "A feline rested on the rug")
• Negative control (35 pairs): Completely unrelated pairs ("The stock market crashed" / "Penguins live in Antarctica")
• Near-miss control (15 pairs): Small factual differences ("The Eiffel Tower is 330 meters tall" / "The Eiffel Tower is 324 meters tall")

These pairs are intentionally adversarial — designed to maximally expose the gap between lexical overlap and semantic meaning. Each adversarial category isolates a single compositional operation (negation insertion, role reversal, etc.) while keeping all other sentence elements identical, creating pairs with high surface similarity but critically different meanings. This design choice has important implications for effect size interpretation, which we discuss in Section 4.3.

3.3 Scoring and Output Scales

For bi-encoders, we compute cosine similarity between independently encoded embeddings, yielding scores in [-1, 1].

For cross-encoders, each model produces outputs on a different scale. We report raw model outputs throughout to avoid normalization artifacts:

• STS-B model: Raw output from the regression head. Despite STS-B training data using a 0-5 Likert annotation scale, the fine-tuned cross-encoder/stsb-roberta-large model (num_labels=1, no explicit activation function) produces outputs naturally ranging from approximately 0 to 1 on our data. We verified this empirically: canonical STS-B-like paraphrases ("A woman is cutting onions" / "A woman is cutting an onion") score 0.965, while unrelated pairs score below 0.01. The model's regression head has learned a calibration that maps to this range. We report these raw outputs directly; no further normalization is needed.

• MS-MARCO model: Raw logits ranging from -11.3 to +10.6. We apply sigmoid for [0,1] normalization in aggregate comparisons, but note that logits above 5 saturate near 1.0 (e.g., a logit of 8.0 maps to probability 0.9997), which compresses variance in the transformed scores.

• BGE-reranker and Quora models: Output probabilities in [0,1] natively; used directly without transformation.

For aggregate cross-model comparisons (Section 4.3), we use: STS-B raw scores directly (already in [0,1]), BGE and Quora probabilities directly, and sigmoid-transformed MS-MARCO logits.

3.4 Software and Reproducibility

All experiments were conducted on CPU using PyTorch 2.4.0 and sentence-transformers 3.0.1. Complete pair-level results (sentence pairs, raw scores, and normalized scores for every model on every pair) are provided in supplementary data files.

4. Results

4.1 Bi-Encoder Failure Confirmation

All five bi-encoder models assign high cosine similarity to adversarial pairs. The minimum cosine similarity observed across all bi-encoder models and all adversarial categories was 0.602 (all-MiniLM-L6-v2 on a hedging pair), confirming that every adversarial pair scored above 0.5 across every model — a finding consistent with Ettinger (2020)'s observation that BERT-based models struggle with negation and compositional semantics.

Bi-encoder mean cosine similarity by category (averaged across 5 models):

Entity swap pairs (0.987) score higher than true paraphrases (0.879), illustrating the severity of the bag-of-words problem: identical tokens in different order produce near-identical embeddings. This is because mean pooling is permutation-invariant — the word order that distinguishes "Google acquired YouTube" from "YouTube acquired Google" is completely discarded.

Individual bi-encoder failure rates at threshold 0.8:

We note that BGE-large-en-v1.5, despite being trained with hard-negative mining specifically designed to improve discrimination, still assigns minimum cosine similarity of 0.771 to negation pairs and 0.669 to quantifier pairs — well above meaningful detection thresholds. Hard-negative mining improves retrieval ranking but does not fundamentally overcome the architectural limitation of independent encoding.

4.2 Cross-Encoder Results: Raw Scores

We present raw (untransformed) model outputs to avoid normalization artifacts. Each model's output semantics differ, so we present them separately with full context.

Quora-RoBERTa-Large (Duplicate Detection, raw probabilities 0-1):

Regarding the negative control SD of 0.00003: This value is genuinely small but not artifactual. The 35 negative control pairs consist of completely unrelated sentences (e.g., "The stock market crashed" / "Penguins live in Antarctica"). For all 35 pairs, the model's sigmoid output produces probability scores tightly clustered between 0.00523 and 0.00537. This reflects confident classification: the model's internal logits for obviously unrelated pairs are all extremely negative (well below -5), and the sigmoid function maps this entire region to values near zero. A well-calibrated binary classifier SHOULD produce near-identical output probabilities for inputs that are all clearly in the same class. The tight clustering indicates model confidence, not data artifact. To demonstrate this concretely, here are the first five negative control scores: 0.00534, 0.00526, 0.00537, 0.00524, 0.00529.

For adversarial categories, the model correctly assigns very low duplicate probabilities (0.02-0.04) to negation, entity swap, numerical, and temporal pairs. The separation from positive controls (0.894) is enormous. Quantifier pairs show more variance (0.009-0.961) suggesting that some quantifier changes (e.g., "all" vs. "some") are harder to detect than others (e.g., "all" vs. "none").

BGE-Reranker-Large (Production Reranker, raw probabilities 0-1):

The BGE reranker shows strong but model-specific improvement patterns. Negation (0.073) and temporal (0.073) pairs are well-separated from positive controls (0.996). Entity swaps (0.398) show substantial variance (SD = 0.298), with some pairs scoring up to 0.999 — suggesting that certain role reversals ("The teacher praised the student" / "The student praised the teacher") are seen as semantically related even by a reranker because the same entities and relationship are present. Hedging (0.883) is barely distinguishable from positive controls, consistent with the interpretation that hedged claims are genuinely related to their definitive counterparts.

The negative control scores cluster near 0.0001 with variance of 7×10^-7, again reflecting sigmoid saturation: the model's internal logits for unrelated pairs are all extremely negative, mapping to near-identical probabilities at the floating-point floor. This is the same phenomenon observed in the Quora model — confident classifiers produce degenerate output distributions on trivially-classifiable inputs.

STS-B-RoBERTa-Large (Semantic Similarity Regression, raw output):

A note on the STS-B output scale: The STS-B benchmark annotates sentence pairs on a 0-5 Likert scale. However, the fine-tuned cross-encoder model (cross-encoder/stsb-roberta-large) has a single-output regression head (num_labels=1) without an explicit activation function. Empirical testing confirms this model natively outputs values in an approximately [0, 1] range: canonical STS-B-style near-paraphrases ("A woman is cutting onions" / "A woman is cutting an onion") score 0.965, while unrelated pairs ("A man is smoking" / "A man is skiing") score 0.103. The model's regression head has learned a calibration to this range during fine-tuning, likely because sentence-transformers normalizes STS-B labels to [0, 1] by dividing by 5 before training the regression head. The raw outputs we report are therefore on a [0, 1] scale and require no further normalization.

The model discriminates well between most adversarial and positive-control pairs: negation (0.491) and numerical (0.454) are substantially below positive controls (0.889). However, entity swaps (0.837) are close to positive controls, indicating weaker detection of role reversals — the model recognizes that "Google acquired YouTube" and "YouTube acquired Google" involve similar semantic content, even though the roles are reversed.

MS-MARCO-MiniLM-L-12-v2 (Retrieval Relevance, raw logits):

This model reveals the most critical finding of our study: adversarial pairs receive higher relevance logits than positive controls. Negation pairs (mean logit 8.21) and entity swaps (mean logit 9.00) score far above true paraphrases (mean logit 4.05). After sigmoid transformation, negation pairs map to probabilities of 0.998-0.9999 while some paraphrases map to probabilities as low as 0.006 (for pairs like "The cat sat on the mat" / "A feline rested on the rug" where vocabulary is completely different).

The negative control logits range from -11.3 to -10.8, mapping to sigmoid probabilities of approximately 0.00001-0.00002. The SD of 0.123 in logit space is small because all unrelated pairs evoke consistently strong "not relevant" signals. After sigmoid transformation, this small logit-space variance compresses to negligible probability-space variance (SD ≈ 2×10^-6).

This behavior is expected and interpretable: MS-MARCO training teaches the model to identify topically relevant passages. "The patient does NOT have diabetes" is maximally relevant to a query about "The patient has diabetes" — it directly addresses the topic. The model correctly identifies topical relevance but is architecturally incapable of distinguishing agreement from contradiction, because that distinction was never in its training signal.

4.3 Aggregate Comparison and Effect Size Interpretation

We compare the three task-appropriate cross-encoders (STS-B, BGE-reranker, Quora) against bi-encoders. For this comparison, we use STS-B raw scores directly (natively [0,1]), and BGE and Quora probabilities directly.

Interpretation of extreme effect sizes. The Cohen's d values (3.7-14.8) far exceed conventional benchmarks from social science (where d > 0.8 is considered "large"). We emphasize that this is an expected consequence of our adversarial evaluation methodology, not evidence of data fabrication. Three factors produce these extreme values:

First, the adversarial pairs are hand-crafted to be maximally contrastive within each category. Each pair isolates a single compositional operation (negation insertion, role swap, etc.) while keeping all other elements identical. This creates pairs with near-maximal lexical overlap but critically different meanings — precisely the scenario where bi-encoders fail most dramatically and task-appropriate cross-encoders succeed most completely.

Second, bi-encoder cosine similarities for adversarial pairs cluster tightly (0.85-0.99 with SDs of 0.01-0.09) because the architectural failure is systematic: mean pooling consistently erases the same types of compositional information across all pairs. Similarly, cross-encoder scores cluster near zero (0.02-0.10) because the models consistently detect these changes through cross-attention. The two distributions have essentially zero overlap for negation and temporal categories, producing mathematically extreme d values.

Third, Cohen's d is computed from the pooled standard deviation of both groups. When both groups have small SDs and the means are far apart, d becomes very large by construction. In a random sample of naturalistic sentence pairs — where difficulty varies continuously and some pairs fall between categories — we would expect substantially smaller effect sizes, wider distributions, and more overlap. Our results characterize model behavior on adversarial worst-case inputs, not typical performance on naturalistic data.

To calibrate expectations: a Cohen's d of 14.8 means the average bi-encoder negation score exceeds 99.99% of cross-encoder negation scores. For our adversarial test set, this is literally true — every bi-encoder negation score (min: 0.724) exceeds every cross-encoder negation score (max for Quora: 0.169). Such complete distribution separation is rare in behavioral research but expected when comparing fundamentally different computational mechanisms (independent vs. joint encoding) on inputs specifically designed to differentiate them.

4.4 Failure Rate Comparison

At threshold 0.5, bi-encoders fail on 100% of adversarial pairs — the minimum bi-encoder cosine similarity across all 5 models and all 251 adversarial pairs is 0.602.

Cross-encoder failure rates at threshold 0.5 (fraction scoring >= 0.5):

The Quora model achieves the strongest discrimination: 0% failure on negation, numerical, entity swap, and temporal categories, with a 94% true positive rate on paraphrases. The STS-B model shows moderate failure rates on entity swaps (73%) and temporal pairs (54%) — these pairs receive scores near 0.5, which is expected since they are semantically related (sharing entities and events) even though the specific meaning differs. The MS-MARCO model fails at 98-100% across all adversarial categories, confirming that retrieval relevance does not capture semantic contradiction.

4.5 Per-Category Analysis

Negation (55 pairs): The strongest improvement. Bi-encoder average: 0.896 cosine similarity. Quora cross-encoder: 0.020 duplicate probability. The cross-attention mechanism enables direct comparison of "has" vs. "does not have" across the sentence pair — a comparison impossible when sentences are encoded independently. All 55 pairs in the Quora model scored below 0.17. The BGE reranker also excels (0.073), while STS-B assigns moderate scores (0.491) — correctly lower than paraphrases (0.889) but not as decisive.

Entity Swap (45 pairs): Bi-encoders score entity swaps at 0.987 — higher than paraphrases (0.879) — because mean pooling is order-invariant: "Google acquired YouTube" and "YouTube acquired Google" contain identical tokens. Cross-encoders reduce this substantially but with model-dependent performance: Quora (0.037), BGE (0.398), STS-B (0.837). The Quora model excels because duplicate detection training explicitly requires distinguishing pairs with the same words in different arrangements. The STS-B model's higher score (0.837) reflects a legitimate semantic judgment: entity-swapped sentences share entities, relationships, and domain — they are semantically similar even if factually different.

Numerical (56 pairs): Bi-encoders average 0.896. Quora: 0.018, BGE: 0.114, STS-B: 0.454. All three task-appropriate cross-encoders consistently detect numerical differences, likely because number tokens receive focused attention when both sentences are processed jointly and the number mismatch creates a salient contrast.

Temporal (35 pairs): Before/after inversions score 0.953 by bi-encoders, reduced to 0.038 (Quora), 0.073 (BGE), and 0.668 (STS-B). The temporal markers "before" and "after" occupy the same position in otherwise identical sentences, and cross-attention directly detects this substitution.

Quantifier (35 pairs): Bi-encoders: 0.855. Cross-encoders: 0.168 (Quora), 0.281 (BGE), 0.563 (STS-B). Notable variance exists within this category: BGE scores "All servers are online" / "Some servers are online" at 0.999, while "All students passed" / "No students passed" scores near 0.001. This suggests that partial quantifier changes ("all" to "some") are harder to detect than complete reversals ("all" to "none"), because "all" and "some" have overlapping truth conditions while "all" and "none" are strict contradictions.

Hedging (25 pairs): The persistent failure mode across architectures. Bi-encoders: 0.831. Quora: 0.514, BGE: 0.883, STS-B: 0.652. Even the Quora model, which excels on other categories, shows 52% failure rates. This reflects a genuine property of hedging: a hedged claim ("The drug may help with some cancer symptoms") is not a contradiction of the definitive claim ("The drug cures cancer") but a weakening of it. The hedged version is consistent with the definitive version being true; they share entities, domain, and propositional content. The difference is epistemic — about certainty level rather than truth value. Current training datasets (STS-B, NLI, Quora) poorly represent these epistemic gradients, and even full cross-attention cannot detect a pattern the model was never trained to distinguish.

5. Analysis

5.1 Architecture vs. Training Objective

Our results cleanly separate two factors that are often conflated:

1. Architectural capacity (independent vs. joint encoding) — determines whether cross-sentence comparison is possible
2. Training objective (what the output score represents) — determines which distinctions the model learns to make

The MS-MARCO cross-encoder proves that Factor 1 alone is insufficient: despite full cross-attention over both sentences, it scores adversarial pairs higher than paraphrases because it was trained to identify topical relevance. "The patient does not have diabetes" is maximally relevant to a query about "The patient has diabetes" even though they are semantically contradictory. The architecture enables cross-sentence reasoning, but the training signal rewards topical overlap rather than semantic agreement.

Conversely, the bi-encoder results prove that Factor 2 alone is insufficient: even models trained explicitly on similarity tasks (like STS-derived bi-encoders) fail catastrophically when limited to independent encoding. The similarity objective is correct, but the architecture cannot implement the necessary cross-sentence comparison.

Success requires the conjunction: cross-attention (architectural capacity to compare) AND a training objective that explicitly rewards distinguishing meaning-altering differences (similarity regression, duplicate detection, or reranking).

5.2 The Hedging Gradient

Hedging creates a semantic gradient rather than a binary contrast, making it fundamentally different from negation or entity swaps. "The drug cures cancer" and "The drug may help with some cancer symptoms" are not contradictions — the hedged version is logically consistent with the definitive version being true. They share entities (drug, cancer), domain (medical treatment), and propositional core (drug-cancer relationship). The difference is in epistemic modality: certainty vs. possibility, definitiveness vs. hedging.

This poses a modeling challenge because current training paradigms define their negative examples as either (a) unrelated passages (MS-MARCO hard negatives), (b) non-duplicate questions (Quora), or (c) contradictions/neutral pairs (NLI). None of these capture the fine-grained epistemic spectrum that hedging occupies. A pair that differs only in certainty level falls into a gap between these training categories — too similar to be a clear non-duplicate, but different enough to mislead downstream applications.

5.3 Model-Specific Behavioral Patterns

The Quora model achieves the best overall adversarial robustness. Duplicate detection training creates an inherently conservative classifier: any modification to a sentence — negation, role swap, numerical change, temporal inversion — constitutes evidence of non-duplication. This makes it ideal for detecting compositional changes but potentially too conservative for applications where semantically similar (but not identical) sentences should still match.

The BGE reranker occupies a useful middle ground: excellent on negation (0.073) and temporal changes (0.073) where the sentence pairs describe clearly different states of affairs, but more permissive on entity swaps (0.398) where the same entities are involved (just in different roles) and hedging (0.883) where the topic is preserved. This reflects a reranker's training signal: both members of an entity-swap pair contain the same entities and would both be relevant to a query about those entities.

The STS-B model shows the most graded responses, with adversarial pairs receiving intermediate scores (0.45-0.84) rather than being decisively classified. This is appropriate for a similarity regression model: entity-swapped sentences ARE similar (they share most semantic content), even if they differ in a critical way. The model correctly captures similarity as a continuum rather than a binary.

6. Practical Recommendations

6.1 Hybrid Pipeline Design

The optimal architecture for applications requiring compositional sensitivity is a two-stage pipeline:

1. First stage (bi-encoder): Retrieve candidates using cosine similarity. Fast (O(1) with precomputed embeddings), scalable to billions of documents, sufficient for topical filtering.
2. Second stage (cross-encoder): Re-rank using a task-appropriate cross-encoder to catch compositional errors that survive first-stage retrieval.

This two-stage approach combines the efficiency of bi-encoders with the compositional sensitivity of cross-encoders, following the architecture proposed by Nogueira and Cho (2019) and now standard in production search systems.

6.2 Model Selection Guidelines

• For strict duplicate detection: use Quora-style cross-encoders (0% failure on negation, entity swap, numerical, temporal)
• For passage re-ranking: use BGE-style rerankers (strong on negation and temporal, but expect entity swap and hedging leakage)
• For similarity scoring: STS-B models provide graded judgments but with less decisive separation
• Avoid MS-MARCO-style models for any task requiring semantic contradiction detection
• For hedging sensitivity: no current off-the-shelf model is reliable; consider dedicated epistemic uncertainty classifiers or NLI-based pipelines

6.3 Residual Risks

Even with task-appropriate cross-encoders, hedging modifications remain largely undetected (48-92% failure rates). Applications sensitive to certainty level — medical claims ("cures" vs. "may help"), legal assertions ("is liable" vs. "may be liable"), financial advice ("will increase" vs. "could potentially increase") — require additional mechanisms beyond current cross-encoder models. Potential solutions include dedicated NLI models that explicitly model entailment/contradiction/neutral distinctions, or fine-tuned models with epistemic certainty annotations.

7. Limitations

Model coverage: We tested four cross-encoder models. An NLI-trained cross-encoder (e.g., cross-encoder/nli-roberta-large), which explicitly models entailment vs. contradiction, might show fundamentally different hedging performance and should be evaluated in future work. The model was inaccessible during our experimental window due to repository authentication requirements.

Test set design: Our 336 pairs are hand-crafted adversarial examples designed to isolate specific failure modes. While this design provides clean diagnostic signal, it does not represent the distribution of sentence pairs in real-world applications. The extreme effect sizes we report (Section 4.3) reflect this adversarial design and should not be extrapolated to naturalistic data. A more comprehensive evaluation would include both adversarial diagnostics (as we provide) and performance on established benchmarks like STS-B test sets, MRPC, or PAWS.

Scale and language: 336 English-only pairs with relatively simple compositional structures. Performance on longer, more complex sentences with nested negation, multiple entity swaps, or cross-lingual inputs remains untested.

STS-B output interpretation: While we empirically verified that the STS-B model outputs values in [0, 1], the model's documentation describes training on a 0-5 scale. The discrepancy likely stems from sentence-transformers normalizing labels during training. We recommend that model card documentation explicitly state the expected output range of fine-tuned models.

Normalization: Cross-model comparison requires aggregating scores from models with different output semantics. We mitigate this by (a) reporting raw scores throughout, (b) using native [0,1] outputs where available, and (c) applying only well-defined transformations (sigmoid for logits).

Computational cost: We did not measure inference latency systematically. Cross-encoders are known to be 10-100x slower than bi-encoders for scoring, which affects practical deployment considerations.

8. Related Work

Reimers and Gurevych (2019) introduced Sentence-BERT, establishing the bi-encoder paradigm for efficient sentence similarity computation using Siamese network structures. Their work enabled practical large-scale similarity search but noted that the resulting embeddings sacrifice some semantic fidelity for computational efficiency.

Devlin et al. (2019) introduced BERT, the bidirectional transformer architecture underlying all models in this study. The [CLS] token representation and fine-tuning paradigm established by BERT directly underlie the cross-encoder architecture we evaluate.

Nogueira and Cho (2019) demonstrated cross-encoder effectiveness for passage re-ranking on MS-MARCO, establishing the two-stage retrieval pipeline (bi-encoder retrieval followed by cross-encoder re-ranking) that our practical recommendations build upon.

Ettinger (2020) systematically evaluated what BERT representations capture using psycholinguistic diagnostics, finding that negation understanding is particularly challenging for transformer-based models. Our work extends this finding from single-model probing to the downstream similarity task across both bi-encoder and cross-encoder architectures, showing that while BERT-based cross-encoders can detect negation when the training objective supports it, retrieval-trained models fail despite having the same underlying architecture.

Humeau et al. (2020) proposed poly-encoders as a computational middle ground between bi-encoders and cross-encoders, performing limited cross-attention over precomputed representations. Testing poly-encoders on our adversarial set could reveal whether partial cross-attention suffices for compositional sensitivity.

Cer et al. (2017) introduced the STS Benchmark dataset used to train the STS-B cross-encoder model in our study, establishing the 0-5 Likert scale for semantic similarity annotation that has become a standard evaluation framework.

Zhang et al. (2019) introduced PAWS (Paraphrase Adversaries from Word Scrambling), constructing adversarial paraphrase pairs with high lexical overlap but different meanings — a methodology similar to our entity-swap category. Their work demonstrated that high word overlap is insufficient for paraphrase detection, motivating the need for compositional understanding.

9. Conclusion

We present a systematic evaluation of cross-encoder robustness to the compositional semantic failures that plague bi-encoder embedding models. Testing four cross-encoder models against five bi-encoders on 336 hand-crafted adversarial sentence pairs across nine categories:

1. Task-appropriate cross-encoders dramatically reduce failure rates on negation (100% to 0%), entity swaps (100% to 0-33%), numerical changes (100% to 0-5%), and temporal inversions (100% to 0-3%), confirming that cross-attention provides the architectural capacity to detect compositional differences.

2. Training objective determines effectiveness. A retrieval-trained cross-encoder (MS-MARCO) assigns higher relevance to adversarial pairs than to paraphrases, performing worse than bi-encoders on semantic discrimination. Cross-attention enables but does not guarantee compositional reasoning — the training signal must reward the relevant distinctions.

3. Hedging remains unsolved. Even the best cross-encoders show 48-92% failure rates on certainty changes — a persistent blind spot reflecting the fact that hedging creates a semantic gradient (certainty weakening) rather than the binary contrasts (negation, contradiction) that current training paradigms target.

4. Model selection matters as much as architecture choice. The question for practitioners is not "bi-encoder or cross-encoder?" but "which cross-encoder, trained on what objective, for what downstream task?" Our results provide concrete guidance: Quora-style models for duplicate detection, BGE-style models for re-ranking, and neither for hedging sensitivity.

All pair-level scores and raw model outputs are provided for independent verification and reproducibility.

References

Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. (2017). SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. In Proceedings of SemEval-2017.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019.

Ettinger, A. (2020). What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8, 34-48.

Humeau, S., Shuster, K., Lachaux, M.-A., and Weston, J. (2020). Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring. In Proceedings of ICLR 2020.

Nogueira, R. and Cho, K. (2019). Passage Re-ranking with BERT. arXiv preprint arXiv:1901.04085.

Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of EMNLP-IJCNLP 2019.

Zhang, Y., Baldridge, J., and He, L. (2019). PAWS: Paraphrase Adversaries from Word Scrambling. In Proceedings of NAACL-HLT 2019.