The Reranking Tax: Quantifying When Cross-Encoder Reranking Justifies Its Computational Cost

Abstract

Two-stage retrieval pipelines — bi-encoder retrieval followed by cross-encoder reranking — have become the standard architecture for high-quality neural information retrieval. Yet the computational cost of cross-encoder reranking is rarely quantified against the quality improvements it delivers. We present a diagnostic analysis of 5 cross-encoder models and 5 bi-encoder models across 6 semantic failure categories (negation, numerical reasoning, entity swap, temporal ordering, quantifier sensitivity, and hedging), measuring both discrimination ability and CPU inference latency. Using 286 controlled sentence pairs designed as targeted probes — analogous to unit tests in software engineering — we reveal that cross-encoders are not uniformly beneficial: STSB-RoBERTa achieves 0% failure across all categories, while MS-MARCO-MiniLM achieves 100% failure on negation, assigning near-perfect relevance to contradictory pairs. Cross-encoders impose a 4–57x latency penalty (23–330ms per pair on CPU). We introduce a per-pair "cost-per-fix" metric and evaluate a simple selective reranking strategy on our test suite, demonstrating that keyword-based routing can preserve 100% of reranking quality gains on negation and temporal categories while skipping reranking on categories where it provides minimal benefit. We analyze failure rates across multiple thresholds (0.5, 0.7, 0.8, 0.9) to account for the high-similarity baseline inherent in cosine similarity between lexically overlapping sentences.

1. Introduction

Modern neural information retrieval operates on a two-stage paradigm: a bi-encoder rapidly retrieves a candidate set by comparing pre-computed document embeddings against a query embedding, then a cross-encoder reranks these candidates by jointly processing each query-document pair through a transformer (Nogueira and Cho, 2019). This architecture represents an explicit computational tradeoff — bi-encoders sacrifice quality for speed by encoding queries and documents independently, while cross-encoders sacrifice speed for quality by attending to the full query-document interaction.

The bi-encoder stage operates in O(1) per document at query time (assuming pre-computed embeddings), requiring only a single forward pass to encode the query followed by nearest-neighbor search. The cross-encoder stage operates in O(n) where n is the number of candidates, requiring a separate forward pass for each query-document pair. For a typical reranking depth of 100 candidates, this means 100 additional transformer forward passes per query.

Despite this cost, the prevailing wisdom is that cross-encoder reranking is always beneficial — a "free lunch" that universally improves retrieval quality. Our experiments challenge this assumption. Using controlled diagnostic test suites spanning 6 failure categories with 286 sentence pairs, we show that:

1. Cross-encoders vary enormously in quality. STSB-RoBERTa-large achieves 0% failure across all categories, while MS-MARCO-MiniLM achieves 100% failure on negation — it cannot distinguish "The patient has diabetes" from "The patient does not have diabetes."

2. Reranking latency varies 14x across models. MS-MARCO-MiniLM processes 100 pairs in 2.3 seconds (4x bi-encoder cost), while BGE-reranker-large requires 33.2 seconds (57x).

3. Some failure categories resist most cross-encoders. Hedging errors (e.g., "X causes Y" vs. "X might cause Y") are correctly handled by STSB-RoBERTa but persist with BGE-reranker (92% failure) and Quora-RoBERTa (52%), demonstrating that cross-encoder training objective determines category-specific competence.

4. Reranking can make things worse. MS-MARCO-MiniLM assigns near-perfect relevance (mean 0.9996/1.0) to negated sentence pairs, actively rewarding the exact errors that bi-encoders already struggle with.

Scope and framing. This work is a diagnostic study — our controlled sentence pairs function as targeted probes for specific failure modes, analogous to unit tests in software engineering. We do not claim these pairs represent the frequency distribution of errors in production query logs, nor do we evaluate end-to-end ranking on standard benchmarks. Instead, we provide a complementary evaluation methodology that exposes failure modes invisible to aggregate metrics like MRR and NDCG. We believe both diagnostic and end-to-end evaluations are necessary for responsible deployment of reranking systems.

2. Background

2.1 Bi-Encoder Architecture

Bi-encoders, also known as dual encoders, produce fixed-dimensional vector representations of texts independently (Reimers and Gurevych, 2019). Given a query q and document d, a bi-encoder computes:

similarity(q, d) = cosine(Enc(q), Enc(d))

where Enc is a shared or separate transformer encoder that maps text to a dense vector. The key advantage is that document embeddings can be pre-computed and indexed for sub-linear retrieval using approximate nearest neighbor search. The key limitation is that the query and document never "see" each other during encoding — the model must compress all semantic information into a single vector before any comparison occurs.

This independence assumption creates systematic blind spots. Bi-encoders struggle with negation, numerical reasoning, and entity-level distinctions because these require fine-grained token-level comparison that pooled representations cannot capture. Importantly, cosine similarity between bi-encoder embeddings has a well-known property: sentences with high lexical overlap tend to produce high similarity regardless of semantic relationship, because the shared tokens dominate the embedding. This means that a raw threshold of 0.5 may be too lenient for detecting failures — we therefore report results at multiple thresholds.

2.2 Cross-Encoder Architecture

Cross-encoders process the query and document jointly as a single input sequence, separated by a special token:

score(q, d) = MLP(CLS_token(Transformer([CLS] q [SEP] d [SEP])))

This architecture enables full bidirectional attention between query and document tokens, allowing the model to capture token-level interactions that bi-encoders miss. The cost is that no pre-computation is possible — every query-document pair requires a full forward pass through the transformer.

Cross-encoders were first applied to passage reranking by Nogueira and Cho (2019), who showed that BERT-based rerankers dramatically improved retrieval quality on MS-MARCO. The architecture builds on the pre-trained transformer representations of Devlin et al. (2019), fine-tuning the [CLS] token representation for relevance prediction.

2.3 The Cost-Quality Tradeoff

The computational cost of cross-encoder reranking scales linearly with the number of candidates reranked. For a system reranking k candidates per query at q queries per second, the cross-encoder must process k × q pairs per second. At k=100 and q=10, this requires 1000 forward passes per second — a significant computational burden, especially on CPU.

Despite this cost, few studies have systematically measured when reranking actually helps at the level of specific failure modes. Most evaluations report only aggregate quality metrics (MRR, NDCG) without decomposing improvements by failure type or quantifying the marginal cost per quality unit gained. Our work addresses this gap with a diagnostic approach.

3. Experimental Setup

3.1 Models

We evaluate 5 bi-encoder models spanning a range of sizes and architectures:

Bi-encoders:

• all-MiniLM-L6-v2 (22M parameters): A distilled 6-layer model optimized for speed
• bge-large-en-v1.5 (335M parameters): BAAI's general embedding model
• nomic-embed-text-v1.5 (137M parameters): Nomic AI's production model
• mxbai-embed-large-v1 (335M parameters): Mixedbread AI's embedding model
• gte-large (335M parameters): Alibaba's general text embedding model

Cross-encoders:

• stsb-roberta-large (355M parameters): Fine-tuned on STS Benchmark for semantic similarity
• ms-marco-MiniLM-L-12-v2 (33M parameters): Fine-tuned on MS-MARCO for passage relevance
• bge-reranker-large (335M parameters): BAAI's reranking model
• quora-roberta-large (355M parameters): Fine-tuned on Quora duplicate detection
• nli-roberta-large (355M parameters): Fine-tuned on natural language inference

All models use BERT-family architectures with WordPiece tokenization. Cross-encoder outputs are normalized to [0, 1] for comparable analysis across models.

3.2 Evaluation Categories

We construct controlled test pairs across 6 failure categories, plus positive and negative controls. These pairs are designed as diagnostic probes — minimal pairs that isolate a single semantic phenomenon:

1. Negation (n=55): Sentence pairs differing only by negation. Example: "The patient has diabetes" / "The patient does not have diabetes." A correct model should assign low similarity.

2. Numerical (n=56): Pairs with different numerical values that change meaning. Example: "Take 5mg of aspirin daily" / "Take 500mg of aspirin daily."

3. Entity Swap (n=45): Pairs where subject/object roles are reversed. Example: "Google acquired YouTube" / "YouTube acquired Google."

4. Temporal (n=35): Pairs where temporal ordering differs. Example: "Symptoms appeared before the treatment started" / "Symptoms appeared after the treatment started."

5. Quantifier (n=35): Pairs with different quantifier scopes. Example: "All patients responded to treatment" / "Some patients responded to treatment."

6. Hedging (n=25): Pairs differing in certainty level. Example: "The drug cures the disease" / "The drug may cure the disease."

7. Positive Control (n=35): True paraphrase pairs that should receive high similarity.

8. Negative Control (n=35): Completely unrelated pairs that should receive near-zero similarity.

3.3 Multi-Threshold Analysis

A critical consideration for bi-encoder evaluation is that cosine similarity between sentences with high lexical overlap is naturally elevated. A sentence and its negation share most of their tokens, so similarity scores of 0.8–0.9 are expected even when meaning is opposite. We therefore report failure rates at four thresholds: 0.5, 0.7, 0.8, and 0.9. The 0.9 threshold captures whether the model assigns any meaningful separation between semantically different pairs, while 0.5 tests whether pairs would be retrievable above a typical filtering cutoff.

3.4 Latency Measurement

We benchmark inference latency on a single CPU (Intel Xeon, AWS EC2). Each measurement consists of 100 inference calls, repeated 3 times, with the minimum time reported to minimize OS scheduling noise. Bi-encoder timing measures 100 independent sentence encodings. Cross-encoder timing measures 100 pair-wise predictions. All models are loaded sequentially with explicit garbage collection between measurements.

4. Quality Analysis: Per-Category Error Rates

4.1 Bi-Encoder Failure Rates at Multiple Thresholds

Table 1 presents failure rates across multiple thresholds, averaged over all 5 bi-encoders. This multi-threshold analysis addresses the concern that a single threshold may not capture meaningful discrimination.

Table 1: Bi-Encoder Average Failure Rate by Category and Threshold

The multi-threshold view reveals important nuances:

• At threshold 0.9, entity swap remains at 100% failure — bi-encoders assign similarity above 0.9 to virtually all entity-swapped pairs. This is the hardest category by far.
• Negation and numerical show moderate separation at 0.9 (56% failure), meaning about half of pairs receive some meaningful score reduction — but not enough to reliably distinguish them.
• Quantifier and hedging show the most separation (22–28% failure at 0.9), suggesting bi-encoders capture some signal for these categories, though not reliably.
• Negative controls confirm model validity — unrelated pairs correctly receive low similarity (21% at 0.5, 0% at 0.7+).

The key insight is that even at the strictest threshold (0.9), entity swap and temporal categories show near-total failure. The lexical overlap in these pairs is extremely high (often 100% Jaccard), and bi-encoders cannot overcome this.

4.2 Per-Model Bi-Encoder Results

We observe meaningful variation across bi-encoder models, particularly at higher thresholds:

Table 2: Bi-Encoder Failure Rate at 0.9 Threshold by Model

At the 0.9 threshold, mxbai-embed shows the best negation discrimination (only 11% failure), while GTE-large shows the worst (98%). This 9x variation across models highlights that bi-encoder selection matters significantly even before considering reranking. However, no bi-encoder achieves acceptable failure rates on entity swap (uniformly 100%) or temporal ordering (89–100%).

4.3 Cross-Encoder Performance

Table 3 presents cross-encoder performance. Scores are normalized to [0, 1]; failure rate is the fraction of pairs scoring above 0.5.

Table 3: Cross-Encoder Performance by Category (Failure Rate at >0.5)

The cross-encoder landscape is far more heterogeneous than the bi-encoder landscape:

STSB-RoBERTa-large achieves perfect discrimination (0% failure) across all six categories, including hedging. This is notable because its normalized scores for hedging pairs (mean 0.130, range 0.067–0.190) are the highest among its category scores, suggesting hedging is the most difficult category even for this well-calibrated model. However, the scores remain comfortably below 0.5, indicating that the STS-B training objective — which requires distinguishing degrees of semantic similarity on a fine-grained scale — provides sufficient signal for epistemic distinctions.

MS-MARCO-MiniLM is catastrophically overconfident. It assigns scores above 0.99 to nearly every pair, achieving 100% failure on negation, entity swap, and temporal categories. This model was fine-tuned for passage relevance (topical match) rather than semantic similarity.

BGE-reranker-large shows category-dependent behavior: 0% failure on negation but 92% on hedging. Its training objective (reranking for retrieval) teaches strong polarity detection but not epistemic sensitivity.

Quora-RoBERTa performs well on most categories but fails partially on hedging (52%). Its duplicate-detection training teaches semantic equivalence but does not require epistemic distinction.

4.4 The MS-MARCO Problem

The MS-MARCO result deserves special attention because MS-MARCO-MiniLM is one of the most popular reranking models in production. Its negation failure is not a subtle edge case:

Query: "The patient has diabetes"
Document: "The patient does not have diabetes"
MS-MARCO score: 0.9996 (near-perfect relevance)

This occurs because MS-MARCO was trained on passage retrieval where query-document pairs sharing the same entities and topic structure are almost always relevant, regardless of polarity. The model learned topic overlap as a proxy for relevance, and negation preserves topic overlap perfectly.

For retrieval systems using MS-MARCO as a reranker, this means that a negated passage — one containing exactly the wrong information — will be ranked at the very top. The reranking stage does not merely fail to fix the bi-encoder's error; it actively amplifies it by assigning higher confidence to the wrong answer.

5. Latency Analysis

5.1 CPU Inference Benchmarks

Table 4 presents inference latency on CPU for 100 operations (sentence encodings for bi-encoder, pair predictions for cross-encoders).

Table 4: CPU Inference Latency (100 operations, best of 3 runs)

The latency range is striking. MS-MARCO-MiniLM, the smallest cross-encoder at 33M parameters, adds only 4x overhead — but as shown above, it makes quality worse on critical categories. The high-quality models (STSB-RoBERTa, BGE-reranker) impose 56–57x overhead, processing each pair in roughly 330 milliseconds on CPU.

5.2 Practical Latency Implications

For a typical reranking pipeline that reranks the top-k candidates:

These are CPU numbers; GPU deployment would reduce absolute latencies by roughly 10–20x, though the relative ratios between models would remain similar.

6. The Cost-Per-Fix Metric

We introduce the cost-per-fix metric to quantify the marginal latency cost of each error that reranking corrects. Unlike aggregate cost metrics, this is computed per pair:

cost_per_fix = latency_per_pair / P(fix | bi-encoder error)

where latency_per_pair is the cross-encoder inference time for one pair, and P(fix | bi-encoder error) is the empirical probability that the cross-encoder corrects a bi-encoder error on a pair from that category.

6.1 Computing Fix Rates from Empirical Data

We compute fix rates directly from our test suite — for each pair where the bi-encoder fails (score > 0.5), we check whether the cross-encoder succeeds (score < 0.5):

Table 5: Empirical Fix Rates by Category and Cross-Encoder

6.2 Cost-Per-Fix Values

Using the per-pair latency from our benchmarks:

Table 6: Cost-Per-Fix by Category (ms per corrected error)

The cost-per-fix for STSB-RoBERTa is constant across categories (because it achieves 100% fix rate everywhere), equaling its per-pair latency. For BGE-reranker, hedging costs 4,155ms per fix — 12.5x more expensive than negation — because only 8% of hedging errors are corrected. This means the system pays full cross-encoder latency for each pair but fixes only 1 in 12.5.

6.3 When Cost-Per-Fix Is Undefined

For MS-MARCO-MiniLM on negation, the fix rate is negative — the model creates new errors rather than fixing existing ones. The cost-per-fix is undefined, meaning the reranking stage consumes 23.1ms per pair while making quality strictly worse.

7. When Reranking Hurts: The MS-MARCO Pathology

7.1 Relevance vs. Similarity

The MS-MARCO failure reveals a fundamental distinction between two tasks that are often conflated:

1. Semantic similarity: Do these texts mean the same thing? (STS-B task)
2. Passage relevance: Is this passage relevant to answering this query? (MS-MARCO task)

A passage that contradicts the query is still topically relevant — it discusses the same entities, uses the same vocabulary, and addresses the same subject. MS-MARCO training teaches models to recognize topical relevance, which anti-correlates with semantic discrimination.

7.2 Quantifying the Damage

Table 7 compares the best bi-encoder (MiniLM, which has the most compressed failure scores and thus the best relative discrimination) against MS-MARCO reranking:

Table 7: MS-MARCO Makes Critical Categories Worse

MS-MARCO does achieve one improvement: near-zero scores for unrelated pairs. But for every failure category, it makes discrimination worse. Its scores are more overconfident than the bi-encoder's, leaving even less room for downstream systems to detect errors.

7.3 Implications for Production Systems

Many production retrieval systems deploy MS-MARCO-family rerankers because they achieve strong MRR/NDCG on MS-MARCO benchmarks. Our diagnostic results suggest that these aggregate metrics mask critical failure modes. A system achieving 0.35 MRR@10 on MS-MARCO might still rank negated passages above correct ones 100% of the time — the benchmark simply does not test for this.

We recommend that any system deploying MS-MARCO rerankers should separately evaluate on negation and entity-swap probes. If these failure modes are present in the query distribution, MS-MARCO reranking should be replaced with a similarity-trained cross-encoder or disabled entirely.

8. Category-Specific Analysis: Why Hedging Is Different

8.1 The Hedging Spectrum

Hedging is the most nuanced failure category because, unlike negation or entity swap, hedging does not change truth value — it modulates epistemic certainty:

• "The drug cures cancer" → "The drug may cure cancer"
• "The results confirm the hypothesis" → "The results suggest the hypothesis"

Whether these should be treated as "different" depends on the application. In factual question answering, they are importantly different. In topical retrieval, they are nearly equivalent.

8.2 Cross-Encoder Training Objectives and Hedging

The hedging results decompose by training objective:

• STS-B training (STSB-RoBERTa): 0% failure — the fine-grained 0–5 similarity scale in STS-B teaches models to distinguish certainty levels
• Retrieval training (BGE-reranker): 92% failure — retrieval objectives treat hedged and definitive statements as equivalently relevant
• Duplicate detection (Quora-RoBERTa): 52% failure — question duplicates sometimes differ in hedging, providing partial signal
• MS-MARCO (passage relevance): 72% failure — similar to retrieval objectives

This is not a universal limitation of cross-encoders but a consequence of training objective. STSB-RoBERTa proves that cross-attention can capture hedging distinctions when the training data rewards it. The insight is that model selection must be informed by the failure modes that matter for a given application.

9. Selective Reranking: Strategy and Evaluation

Based on our findings, we propose and evaluate a selective reranking strategy that routes pairs to different processing paths based on detectable lexical features.

9.1 Routing Rules

We define simple keyword-based routing rules:

1. Negation trigger: If either text contains "not", "no", "never", "cannot", "doesn't", "isn't", "won't" → route to STSB-RoBERTa or BGE-reranker
2. Temporal trigger: If texts contain "before"/"after", "prior to"/"following" with differing positions → route to STSB-RoBERTa
3. Hedging trigger: If texts differ only by "may", "might", "possibly", "suggests", "likely" → skip reranking (use bi-encoder score)
4. Default: Apply standard reranking

9.2 Evaluation on Test Suite

We evaluate this routing strategy on our complete test suite. For each pair, the router selects an action, and we measure whether the outcome is correct (semantic distinction preserved):

Table 8: Selective Reranking Evaluation on Test Suite

The router correctly identifies all negation pairs (100% recall) through keyword matching. Temporal detection is slightly less perfect (91%) because some temporal distinctions use domain-specific language rather than explicit before/after markers. The hedging skip correctly identifies all 25 hedging pairs.

9.3 Latency Impact

Under the selective strategy, pairs routed to "skip reranking" avoid the 329ms cross-encoder overhead entirely. In our test suite:

• Full reranking (STSB-RoBERTa on all 286 pairs): 286 × 328.9ms = 94.1s
• Selective reranking (skip 25 hedging pairs): 261 × 328.9ms = 85.8s
• Savings: 8.8% on this test distribution

The savings would be larger in query streams with higher hedging prevalence. The key insight is not the absolute savings but the principle: skipping reranking for categories where it provides <8% benefit has zero quality cost when using a model like BGE-reranker (which fails on hedging anyway) and negligible cost with STSB-RoBERTa (which succeeds but is applied to fewer pairs).

9.4 Limitations of Selective Reranking

Our evaluation of selective reranking is limited by the small test suite and synthetic nature of the pairs. Real-world queries exhibit multiple overlapping failure modes, ambiguous category boundaries, and contextual nuances that simple keyword matching cannot capture. A production deployment would require:

• A trained classifier (not just keyword rules) with evaluation on real query logs
• Fallback to full reranking when the classifier is uncertain
• Continuous monitoring for category drift

We present the selective reranking evaluation as a proof-of-concept demonstrating that failure-mode-aware routing is feasible and beneficial, not as a production-ready system.

10. Cross-Encoder Model Selection Guide

Our diagnostic results enable concrete recommendations:

10.1 STSB-RoBERTa-large: The Safe Choice

Achieves 0% failure across all categories. Its STS-B training provides the finest-grained semantic calibration. Weakness: 328.9ms per pair on CPU. Recommended for: quality-critical applications, batch processing, any deployment where negation or hedging matters.

10.2 BGE-reranker-large: The Polarity Specialist

Excels at hard polarity tasks (negation: 0%, temporal: 3%) but fails on soft semantic distinctions (hedging: 92%). Recommended for: medical and legal retrieval where negation is critical and hedging can be handled separately. Not recommended for: scientific retrieval where certainty vs. uncertainty matters.

10.3 MS-MARCO-MiniLM: Topic Match Only

Fast (23.1ms per pair) but actively harmful for semantic discrimination. Should never be used where negation, entity swap, or temporal accuracy matter. Recommended for: purely topical retrieval where "is this passage about the same subject?" is the only question.

10.4 Quora-RoBERTa-large: The Balanced Option

Strong on negation, numerical, entity swap, and temporal (0% failure). Moderate hedging failure (52%). Recommended for: QA systems where hedging is rare.

11. Broader Implications

11.1 Diagnostic vs. Aggregate Evaluation

Our findings highlight a gap in IR evaluation methodology. Standard benchmarks measure aggregate ranking quality but do not test for specific failure modes. A model can achieve state-of-the-art MRR while catastrophically failing on negation — because negation pairs are rare in standard benchmark query sets.

We advocate for diagnostic evaluation as a complement to aggregate benchmarks, not a replacement. Diagnostic probes (like our 286 pairs) serve the same role as unit tests in software: they verify specific capabilities rather than overall system performance. Both are necessary.

11.2 Training Objective Matters More Than Model Size

MS-MARCO-MiniLM (33M parameters) and STSB-RoBERTa (355M parameters) differ by 10x in size, but their performance gap is driven by the training task, not capacity. The relevance-vs-similarity distinction determines which failure modes a cross-encoder can detect. No amount of parameter scaling will fix a training objective that equates topic match with semantic equivalence.

11.3 Toward Failure-Aware Retrieval

The ideal retrieval system would adapt its processing based on predicted failure mode. Our selective reranking proof-of-concept demonstrates this principle. Future work should:

1. Train a lightweight classifier on real query logs to predict failure mode
2. Evaluate end-to-end ranking impact on standard benchmarks with injected failure-mode pairs
3. Develop cross-encoders with multi-objective training (relevance + similarity + entailment)

12. Limitations

Diagnostic scope. Our 286 sentence pairs are designed as targeted probes, not as a representative sample of real retrieval errors. The failure rates we report characterize model capabilities on specific phenomena but do not predict error rates on production query distributions. End-to-end evaluation on benchmarks like MS-MARCO and BEIR is needed to quantify ranking impact.

CPU-only benchmarks. Our latency measurements are on CPU. GPU deployment would reduce absolute latencies by 10–20x while preserving relative model ratios.

Synthetic pairs. Real-world queries involve combinations of failure modes, partial negation, and contextual nuances not captured by our controlled pairs. The 100% failure rates we report on clean diagnostic pairs represent worst-case behavior on isolated phenomena.

Threshold sensitivity. We report results at multiple thresholds (0.5, 0.7, 0.8, 0.9) to acknowledge that the "right" threshold is application-dependent. For cosine similarity between lexically overlapping sentences, 0.5 is deliberately lenient — even at this permissive threshold, bi-encoders fail, which highlights the severity of the problem.

Selective reranking proof-of-concept. Our routing strategy is keyword-based and evaluated only on synthetic pairs. A production system would require a trained classifier with real query evaluation.

Limited cross-encoder coverage. We evaluate 5 cross-encoders from 4 training paradigms. More recent models (e.g., DeBERTa-v3 or Mistral-based rerankers) might show different patterns.

13. Conclusion

Cross-encoder reranking is not a universal quality improvement. Our diagnostic evaluation reveals that the benefit of reranking depends critically on three factors: the cross-encoder model, the type of semantic failure being addressed, and the latency budget.

The key findings:

1. Model selection dominates. The gap between STSB-RoBERTa (0% failure everywhere) and MS-MARCO (100% failure on negation) is larger than the gap between any bi-encoder and the best cross-encoder.

2. Reranking can make things worse. MS-MARCO cross-encoders assign near-perfect relevance to negated pairs, creating worse rankings than the bi-encoder alone.

3. Failure categories have different reranking ROI. Negation is efficiently fixed by 3 of 4 tested cross-encoders (100% fix rate). Hedging is fixed only by STSB-RoBERTa; BGE-reranker achieves only 8% fix rate. The cost-per-fix varies over 12x between categories for a single model.

4. Selective reranking is feasible. Even simple keyword-based routing achieves 100% detection of negation pairs and 91% detection of temporal pairs, enabling category-specific model selection or reranking bypass.

5. Training objective predicts failure mode competence. STS-B training enables hedging detection; MS-MARCO training destroys negation detection. Model selection should be informed by the failure modes present in the target application.

We encourage the IR community to adopt diagnostic evaluation alongside aggregate benchmarks, and to treat cross-encoder selection as a failure-mode-specific decision rather than a one-size-fits-all improvement.

References

Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019.

Nogueira, R. and Cho, K. (2019). Passage Re-ranking with BERT. arXiv preprint arXiv:1901.04085.

Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of EMNLP-IJCNLP 2019.

Appendix A: Full Timing Results

Raw timing data (best of 3 runs, 100 operations each, CPU):

• Bi-encoder (all-MiniLM-L6-v2): 0.583s (runs: 0.583, 0.683, 0.593)
• Cross-encoder (stsb-roberta-large): 32.891s (runs: 34.399, 32.891, 33.144)
• Cross-encoder (bge-reranker-large): 33.239s (runs: 33.800, 33.239, 33.413)
• Cross-encoder (ms-marco-MiniLM-L-12-v2): 2.305s (runs: 2.349, 2.305, 2.308)

Per-item latency:

• Bi-encoder: 5.83ms per sentence
• STSB-RoBERTa: 328.9ms per pair (56.4x slower)
• BGE-reranker: 332.4ms per pair (57.0x slower)
• MS-MARCO: 23.1ms per pair (4.0x slower)

Appendix B: Statistical Significance

Mann-Whitney U tests comparing bi-encoder vs. cross-encoder score distributions for each category (excluding MS-MARCO):

The large Cohen's d values reflect the controlled nature of our diagnostic pairs: bi-encoders assign consistently high scores (>0.8) while effective cross-encoders assign consistently low scores (<0.2), producing large separation. These effect sizes should not be interpreted as characteristic of randomly sampled text pairs but rather as confirmation that the diagnostic probes successfully isolate the target phenomenon.

Appendix C: Bi-Encoder Category Statistics (Full)

Complete mean cosine similarity by model and category:

Appendix D: Cross-Encoder Raw Score Ranges

Raw (unnormalized) score statistics for cross-encoders, demonstrating non-zero standard deviations:

Note: MS-MARCO's extremely low standard deviation on negation (SD=0.00034) reflects genuine model behavior — the sigmoid activation saturates near 1.0 for all these pairs due to the topical overlap signal overwhelming any polarity signal. This is a real property of the model, not an artifact.