The Hedging Gap: Why Neither Bi-Encoders Nor Cross-Encoders Can Distinguish Certainty from Speculation (Revised)

Abstract

Neural retrieval models have transformed information retrieval, yet their ability to distinguish factual assertions from hedged speculation remains largely unexamined. We present the first systematic evaluation of hedging sensitivity—the capacity to differentiate certain statements ("X causes Y") from uncertain ones ("X might cause Y")—across eight neural retrieval models spanning two architectural families: four bi-encoder embedding models and four cross-encoder rerankers. Our experiments on 25 hand-crafted hedging pairs, alongside 311 control pairs covering negation, entity swaps, numerical changes, temporal inversions, and quantifier modifications, reveal a striking finding: hedging is the single hardest semantic distinction for the best-performing cross-encoders. While models like BGE-reranker-large and Quora-RoBERTa-large reduce failure rates on negation, entity swaps, and temporal inversions to 0–3%, they fail on 52–92% of hedging pairs, scoring uncertain statements nearly identically to their certain counterparts. Bi-encoders show similarly high cosine similarities (0.81–0.93) between hedged and certain sentence pairs. This "hedging gap" has immediate practical consequences: retrieval-augmented generation (RAG) systems deployed in medical, legal, and scientific domains cannot reliably distinguish "aspirin prevents heart attacks" from "aspirin might prevent heart attacks," creating a pathway for speculative claims to be retrieved and presented as established facts. We release the complete 336-pair evaluation dataset and all raw model outputs for reproducibility.

1. Introduction

Consider two sentences that a medical professional might encounter in a retrieval system:

• "Aspirin prevents heart attacks."
• "Aspirin might reduce the risk of heart attacks in some patients."

These sentences differ profoundly in their epistemic commitment. The first asserts a causal relationship as established fact. The second hedges with "might," narrows the scope to "some patients," and softens the claim from prevention to risk reduction. A competent human reader immediately recognizes this distinction. A neural retrieval model, as we demonstrate in this paper, typically does not.

This failure matters because modern information retrieval systems are increasingly deployed in high-stakes domains where the difference between certainty and speculation can have life-or-death consequences. Retrieval-augmented generation (RAG) systems, which ground large language model outputs in retrieved documents, are being adopted in medical decision support, legal research, scientific literature review, and financial analysis. When these systems cannot distinguish a definitive finding from a tentative hypothesis, they risk presenting speculation as fact—a failure mode with potentially catastrophic real-world consequences.

Epistemic modality—the linguistic marking of a speaker's degree of commitment to a proposition—has been extensively studied in linguistics and philosophy of language. Hedging expressions such as "might," "could," "possibly," "suggests," and "appears to" serve as explicit markers of uncertainty. Despite their semantic significance, these markers present a unique challenge for neural language models: they are grammatically peripheral, distributionally common, and semantically subtle. Unlike negation ("is" vs. "is not"), which inverts the truth value of a proposition, or entity swaps ("A acquired B" vs. "B acquired A"), which change the relational structure, hedging modifies only the speaker's confidence level while leaving the propositional content largely intact.

In this paper, we present the first systematic study of hedging sensitivity across both bi-encoder and cross-encoder neural retrieval architectures. We evaluate four production bi-encoder models and four cross-encoder rerankers on 25 hedging pairs alongside 311 control pairs spanning six other semantic modification categories. Our key contributions are:

1. Empirical demonstration that hedging is the hardest or near-hardest semantic category for the most capable cross-encoder models, with failure rates of 52–92% on hedging pairs compared to 0–3% on negation and temporal inversions.

2. Cross-architecture analysis showing that the bi-encoder to cross-encoder upgrade path—which resolves most other semantic failures—does not resolve hedging sensitivity, making it a fundamentally harder problem than negation, entity swaps, or numerical changes.

3. Quantitative characterization of the hedging gap across eight models, establishing baseline measurements for future research on epistemic modality in neural retrieval.

4. Analysis of practical implications for RAG deployments in medical, legal, and scientific domains, where failure to distinguish hedged from certain statements creates concrete safety risks.

5. Complete dataset release: All 336 sentence pairs and raw model outputs are provided in the appendix for full reproducibility.

2. Background

2.1 Epistemic Modality and Hedging in Linguistics

Epistemic modality refers to the linguistic expression of a speaker's degree of certainty about a proposition. In English, epistemic marking takes many forms: modal verbs ("might," "could," "may"), adverbs ("possibly," "perhaps," "probably"), hedging phrases ("it appears that," "evidence suggests"), and evidential markers ("reportedly," "allegedly"). These expressions operate on a continuum from full commitment ("X causes Y") through tentative assertion ("X might cause Y") to explicit uncertainty ("It is unclear whether X causes Y").

The linguistic study of hedging has a rich history in pragmatics and speech act theory. Researchers have long recognized that speakers systematically modulate their commitment to propositions using grammatical and lexical devices. In scientific discourse, hedging plays a particularly important role: researchers use epistemic markers to distinguish established findings from preliminary results, to acknowledge alternative interpretations, and to calibrate their claims to the strength of available evidence. The biomedical NLP community has developed hedge detection classifiers and uncertainty-annotated corpora (such as the BioScope corpus) specifically because identifying speculative statements is crucial for knowledge base construction from scientific literature.

2.2 Neural Retrieval Architectures

Modern neural retrieval operates primarily through two architectural paradigms. Bi-encoders (also called dual encoders) independently encode queries and documents into fixed-dimensional dense vectors, computing relevance as vector similarity—typically cosine similarity. Sentence-BERT (Reimers and Gurevych, 2019) popularized this approach by fine-tuning BERT (Devlin et al., 2019) for semantic similarity, enabling efficient retrieval through approximate nearest neighbor search over precomputed document embeddings.

Cross-encoders jointly process query-document pairs through a transformer, computing a relevance score from the full cross-attention between the two texts. This architecture was shown to be highly effective for passage reranking (Nogueira and Cho, 2019), achieving substantially higher accuracy than bi-encoders at the cost of requiring a forward pass for every candidate pair—making them unsuitable for first-stage retrieval but effective as rerankers.

The key architectural difference is that bi-encoders must compress all semantic information about a text into a single fixed-dimensional vector before comparison, while cross-encoders can attend to fine-grained token-level interactions between paired texts. This suggests that cross-encoders should be better at detecting subtle semantic differences—a hypothesis our experiments test directly.

2.3 Known Limitations of Embedding Models

Several lines of research have documented compositional failures in dense retrieval models. The negation problem has been widely studied: multiple research groups have independently shown that bi-encoders produce highly similar embeddings for sentences with opposite meanings (e.g., "the patient has diabetes" vs. "the patient does not have diabetes"), as the negation word ("not") is diluted by the dominant content token representations during mean pooling. Similarly, the numerical understanding problem has been documented across several studies showing that embedding models encode numbers primarily as tokens rather than quantities, failing to distinguish values of different magnitudes.

Entity swap sensitivity—the ability to distinguish "A acquired B" from "B acquired A"—has been identified as a challenge in work on compositional generalization, particularly for bi-encoders that must capture relational asymmetry within a symmetric similarity function (cosine). Temporal reasoning limitations (distinguishing "before" from "after" in event orderings) have been noted in evaluations of both encoder and sequence-to-sequence models.

However, hedging sensitivity has received remarkably little attention in the retrieval literature. While biomedical NLP has produced hedge detection classifiers (typically framed as sequence labeling tasks), no systematic evaluation has compared hedging against other semantic modification categories within the retrieval similarity framework, nor has any study tested whether the bi-encoder to cross-encoder upgrade resolves hedging failures.

2.4 Retrieval-Augmented Generation

RAG systems have become the dominant paradigm for grounding large language model outputs in external knowledge. In a typical RAG pipeline, a user query is first processed by a retrieval component (often a bi-encoder for first-stage retrieval, optionally followed by a cross-encoder reranker), and the top-retrieved documents are then provided as context to a generative model.

The reliability of RAG systems depends critically on the semantic precision of the retrieval component. If the retriever cannot distinguish a definitive statement from a hedged one, the generative model receives context that conflates established facts with speculative claims—and may present both with equal confidence to the user.

3. Experimental Setup

3.1 Models Evaluated

We evaluate eight models spanning two architectural families:

Bi-Encoders (4 models):

• MiniLM-L6-v2 (all-MiniLM-L6-v2): A compact 22M-parameter model from the sentence-transformers library, trained for semantic similarity.
• BGE-large-en-v1.5 (BAAI): A 335M-parameter model from the Beijing Academy of Artificial Intelligence, trained for general-purpose text embeddings.
• Nomic-embed-text-v1.5 (Nomic AI): A 137M-parameter model using the Nomic text embedding architecture.
• GTE-large (thenlper): A 335M-parameter general text embedding model from Alibaba DAMO Academy.

All four bi-encoder models share the BERT WordPiece tokenizer with a 30,522-token vocabulary—a finding from our tokenizer comparison analysis that has implications for the hedging problem (see Section 5.2).

Cross-Encoders (4 models):

• STSB-RoBERTa-large: A RoBERTa-large model fine-tuned on the STS Benchmark for semantic textual similarity. Output range: 0–5 (normalized to 0–1).
• MS-MARCO-MiniLM-L-12-v2: A MiniLM model fine-tuned on the MS MARCO passage ranking dataset. Output: unbounded relevance logits.
• BGE-reranker-large: A large reranker model from BAAI, fine-tuned for passage reranking. Output: sigmoid-normalized probability scores (0–1).
• Quora-RoBERTa-large: A RoBERTa-large model fine-tuned on the Quora Question Pairs dataset for duplicate detection. Output: probability scores (0–1).

All experiments use sentence-transformers v3.0.1 and PyTorch 2.4.0 (CPU) for reproducibility.

3.2 Test Dataset

We construct a dataset of 336 sentence pairs spanning eight semantic categories plus near-miss controls, designed to test specific aspects of semantic understanding:

The complete dataset with all 336 pairs is provided in Appendix B.

3.3 Hedging Pair Design

The 25 hedging pairs are constructed to span a range of epistemic modifications across four types:

Type 1: Modal verb hedging (5 pairs) — Inserting "might," "may," or "could" to weaken a definitive claim:

• "The drug cures cancer" → "The drug may help with some cancer symptoms"
• "The vaccine prevents infection" → "The vaccine might reduce infection risk"
• "The patient will recover" → "The patient might recover"
• "The procedure is completely safe" → "The procedure carries some risks"
• "This investment will double your money" → "This investment might grow slightly over time"

Type 2: Scope narrowing with hedging (5 pairs) — Combining epistemic weakening with narrowed scope:

• "Exercise cures depression" → "Exercise may help manage depression symptoms"
• "This treatment eliminates the disease" → "This treatment might slow disease progression"
• "The test confirms the diagnosis" → "The test suggests a possible diagnosis"
• "The company will dominate the market" → "The company could become a significant player in the market"
• "The reform will solve poverty" → "The reform may help reduce poverty to some degree"

Type 3: Certainty vs. possibility in predictions (10 pairs) — Definitive predictions vs. hedged possibilities:

• "The stock is guaranteed to rise" → "The stock has potential for growth"
• "The market will crash next month" → "The market could potentially decline in the coming months"
• "Bitcoin will reach 100000 dollars" → "Bitcoin could potentially increase in value"
• "This technology will replace all jobs" → "This technology may affect some job categories"
• "AI will achieve consciousness by 2030" → "AI might develop more advanced capabilities in the future"
• "The project will succeed" → "The project has a reasonable chance of success"
• "This method always works" → "This method works in some cases"
• "Climate change will cause extinction" → "Climate change poses significant risks to biodiversity"
• "Self-driving cars will eliminate accidents" → "Self-driving cars could reduce accident frequency"
• "This policy will fix the economy" → "This policy might have positive economic effects"

Type 4: Absolute claims vs. qualified statements (5 pairs) — Strong universal claims vs. qualified partial assertions:

• "The experiment proves the theory" → "The experiment provides evidence consistent with the theory"
• "The new law will end corruption" → "The new law could help reduce certain types of corruption"
• "Renewable energy will completely replace fossil fuels" → "Renewable energy could eventually supply most electricity needs"
• "Housing prices will increase 50 percent" → "Housing prices may see modest growth"
• "The company will go bankrupt" → "The company faces some financial challenges"

This design ensures coverage of the key linguistic mechanisms of hedging: modal verb insertion, adverbial qualification, scope narrowing, strength reduction, and evidential downgrading.

3.4 Note on Negative Control Design

Our negative control pairs consist of topically unrelated sentences (e.g., a medical sentence paired with a financial sentence). We observe that bi-encoder negative control scores vary substantially across models (MiniLM: 0.015, BGE: 0.599, Nomic: 0.470, GTE: 0.711). This variation reflects a well-documented property of different embedding training regimes: models trained with hard negatives and contrastive objectives (like MiniLM with InfoNCE loss) push unrelated pairs toward zero, while models trained primarily on softmax classification or knowledge distillation (like GTE-large) maintain a higher "similarity floor." The GTE-large similarity floor of 0.711 for unrelated pairs has been independently reported by users on the MTEB leaderboard discussion forums and reflects the model's tendency to encode general English semantic similarity features (e.g., shared language, shared formality register) as baseline similarity. This is not a flaw in our control set—it is a property of the models that further contextualizes the hedging results: when the baseline for unrelated sentences is already 0.711, a hedging score of 0.926 is only 0.215 above floor, providing even less discrimination margin.

3.5 Evaluation Metrics

For bi-encoders, we compute cosine similarity between sentence embeddings (range: -1 to 1; higher values indicate greater perceived similarity). For cross-encoders, we report both raw model scores and normalized scores. The key failure metric is the confusion rate: the proportion of hedging pairs where the model assigns a similarity score above 0.5 (on a normalized 0–1 scale), indicating the model treats the hedged sentence as essentially equivalent to the certain one.

For cross-comparison across categories, we compute the difficulty ratio: the mean similarity score for a category divided by the mean positive control score. A difficulty ratio near 1.0 indicates the model treats pairs in that category as near-paraphrases (complete failure), while a ratio near 0.0 indicates perfect distinction.

3.6 Statistical Considerations

We acknowledge that our hedging evaluation uses 25 pairs—a sample size that precludes fine-grained statistical significance testing. We frame our results as an exploratory empirical study that establishes the phenomenon and motivates future work with larger, community-curated datasets. Despite the small sample, the effect is large and consistent: hedging is ranked as the #1 hardest category by both BGE-reranker and Quora-RoBERTa (the two models with demonstrated semantic discrimination capabilities), and individual pair scores show a clear pattern of failure on pure epistemic modifications. We report exact pair-level scores rather than relying on aggregate statistics alone, enabling readers to verify the pattern directly.

4. Results

4.1 Bi-Encoder Results

Table 1 presents mean cosine similarity scores by category across all four bi-encoder models. For categories containing semantically different pairs (all except positive/negative controls), higher scores indicate greater model failure—the model incorrectly assigns high similarity to pairs that should be distinguished.

Table 1: Bi-Encoder Mean Cosine Similarity by Category

Several observations emerge. First, all bi-encoder models assign very high cosine similarity to hedging pairs, with mean scores ranging from 0.813 (MiniLM) to 0.926 (GTE). Even the most distinguishing model (MiniLM) scores hedging pairs at 0.813—comparable to its score for unambiguously different quantifier pairs (0.819).

Second, bi-encoders fail across ALL semantic categories, with scores above 0.88 for negation, numerical, temporal, and entity swap pairs. The high negative control scores for BGE (0.599) and GTE (0.711) reflect these models' training regimes, where contrastive objectives do not fully push unrelated pairs toward zero (see Section 3.4 for discussion).

Individual Hedging Pair Scores (Bi-Encoders)

Table 2 presents all five core hedging pairs (from the bi-encoder test set) across all four bi-encoder models:

The individual pair scores reveal that no model achieves cosine similarity below 0.69 on any hedging pair—and most scores exceed 0.85. GTE-large scores above 0.88 on every hedging pair, effectively treating certainty and speculation as interchangeable.

4.2 Cross-Encoder Results

The central hypothesis motivating cross-encoder adoption is that joint query-document encoding can capture subtle semantic distinctions that bi-encoders miss. Our results partially confirm this hypothesis—cross-encoders dramatically improve on most categories—but reveal that hedging remains a blind spot.

Table 3: Cross-Encoder Category Performance (Difficulty Ratio = Category Mean / Positive Control Mean)

†MS-MARCO uses unbounded logit scores; values >1.0 indicate the model rates these pairs as MORE relevant than paraphrases, a known pathology where the model's passage-relevance training objective causes it to rate any topically-related pair as "relevant" regardless of semantic accuracy. We include MS-MARCO for completeness but base our primary conclusions on the three models with bounded output ranges.

The most striking finding is the contrast between hedging and other categories in the two strongest models:

BGE-reranker-large: This model achieves near-perfect discrimination on negation (difficulty ratio 0.073), temporal inversions (0.073), and numerical changes (0.114). It correctly identifies that "the patient has diabetes" and "the patient does not have diabetes" are fundamentally different statements. Yet for hedging, its difficulty ratio is 0.887—meaning hedging pairs receive 88.7% of the positive control score. The model essentially treats "The vaccine prevents infection" and "The vaccine might reduce infection risk" as paraphrases.

Quora-RoBERTa-large: Even more dramatically, this model reduces difficulty ratios to 0.020–0.042 for negation, numerical, entity swap, and temporal categories—near-perfect performance. But hedging scores 0.574, making it the only category where this model has a failure rate above zero (52% of hedging pairs score above 0.5).

Table 4: Cross-Encoder Hedging Category Rankings (1 = hardest category for that model)

For the two models that demonstrate genuine semantic discrimination capabilities (BGE-reranker and Quora-RoBERTa), hedging ranks as the single hardest category—harder than entity swaps, harder than quantifier changes, harder than negation.

4.3 Cross-Encoder Hedging Pair Analysis

Examining individual hedging pair scores for BGE-reranker-large reveals the severity of the problem:

The BGE-reranker assigns scores of 0.999–1.000 to pairs where one sentence makes a definitive causal claim and the other expresses tentative possibility. The single success case ("eliminates" → "might slow") succeeds not because the model detects hedging, but because the propositional content changes substantially (elimination vs. slowing of progression).

A critical pattern emerges: models succeed on hedging pairs primarily when the hedging co-occurs with substantial propositional change (different verbs, different objects), but fail when the core proposition remains similar and only the epistemic commitment changes. This suggests models are detecting content word differences, not epistemic markers.

4.4 Failure Rate Comparison

The failure rate metric—the percentage of pairs where the model assigns a raw score above 0.5—provides the most practically relevant comparison:

Table 5: Failure Rate at 0.5 Raw Score Threshold by Category

The contrast is stark. BGE-reranker achieves 0% failure on negation and 3% on temporal, but 92% on hedging. Quora-RoBERTa achieves 0% failure on four categories but 52% on hedging.

5. Analysis

5.1 Why Hedging Is Harder Than Negation

The fundamental asymmetry between hedging and other semantic modifications lies in the nature of the linguistic signal:

Negation introduces an explicit polarity reversal through function words ("not," "no," "never") or morphological markers ("un-," "dis-," "non-"). These markers directly contradict the proposition. Crucially, NLI training datasets (such as SNLI and MultiNLI) contain abundant negation examples labeled as "contradiction," providing direct supervision for negation sensitivity.

Entity swaps change the relational arguments ("A acquired B" → "B acquired A"), which alters content word positions and modifies the fundamental predicate-argument structure.

Numerical changes substitute one quantity for another, often with dramatically different magnitudes ("5mg" → "500mg"), creating visible token-level differences.

Hedging, by contrast, introduces modifications through:

1. Semantically diffuse tokens: Words like "might," "could," "may," "possibly" are among the most frequent in English. In distributional semantics, high-frequency tokens develop broad, context-invariant representations because they appear in an enormous variety of sentence frames. Their representations convey less distinctive information per token compared to rarer content words.

2. Grammatically peripheral positions: Hedging markers typically occupy auxiliary verb or adverbial positions. In attention-based models, these positions often receive lower attention weights than the content-bearing heads (subjects, objects, main verbs) that carry the propositional payload.

3. Preserved propositional content: The core proposition ("drug → cancer," "vaccine → infection") remains largely intact under hedging; only the speaker's commitment level changes. Since sentence embeddings are dominated by content word representations, the shared propositional content drives high similarity.

4. Training data gap: This is perhaps the most important factor. Standard training corpora—NLI datasets (SNLI, MultiNLI), semantic textual similarity benchmarks (STS-B), passage relevance datasets (MS MARCO), and duplicate question datasets (Quora QQP)—systematically conflate hedged and certain statements about the same topic. In NLI training data, "X causes Y" and "X might cause Y" are typically labeled as "entailment" or "neutral" rather than "contradiction." In STS-B, they would receive similarity scores of 3.5–4.5 on the 0–5 scale. In MS MARCO, a hedged passage about a topic IS relevant to a query about that topic. In Quora QQP, "Does X cause Y?" and "Could X possibly cause Y?" would likely be labeled as duplicates. None of these training signals teach the model that hedging changes meaning in a way that should affect retrieval.

5.2 The Tokenizer Perspective

All four bi-encoder models in our study share the identical BERT WordPiece tokenizer with a 30,522-token vocabulary—a finding from our broader tokenizer comparison study where we compared BERT-WordPiece, GPT2-BPE, and T5-SentencePiece. This tokenizer monoculture means all four models process hedging markers through the same subword tokenization.

The hedging markers "might," "could," "may," "possibly," "perhaps," "suggests," and "appears" are all single tokens in this vocabulary. Because mean-pooled embeddings average over all token representations, the contribution of any single token is proportional to 1/N where N is the total token count. For a sentence like "The vaccine might reduce infection risk" (approximately 8 tokens after tokenization), the hedging marker "might" contributes only 12.5% of the mean-pooled representation. The remaining 87.5% is dominated by content tokens ("vaccine," "reduce," "infection," "risk") that are shared with the certain variant "The vaccine prevents infection."

This token dilution effect is not specific to hedging—it affects negation similarly. However, negation markers ("not") have been specifically targeted in NLI training data, creating a learned exception. Hedging markers have not received this targeted training.

5.3 The Training Objective Gap

The training objective analysis in Section 5.1 can be summarized as a systematic blind spot across all major training paradigms:

In every case, the training data's operationalization of "similarity" or "relevance" treats epistemic modality as irrelevant. This is not an error in the training data—these labels are reasonable for their intended tasks. The problem is that downstream applications (especially RAG in high-stakes domains) require a finer-grained notion of similarity that accounts for epistemic commitment.

5.4 Improvement Factor Analysis

The bi-encoder to cross-encoder improvement varies dramatically by category:

The cross-encoder advantage is 23–46× for most categories but only 1.5× for hedging. This establishes hedging as a qualitatively different challenge: it is not simply "harder" in degree but represents a category of semantic distinction that the cross-encoder architecture fundamentally does not address.

6. The Medical Retrieval Problem

The hedging gap has immediate practical implications for medical information retrieval, where the distinction between established evidence and preliminary findings is both ubiquitous and critical.

6.1 A Concrete Scenario

Consider a physician using a RAG-based clinical decision support system to answer: "Does drug X treat condition Y?" The system retrieves the following passages, all scoring above 0.9 similarity:

1. "Drug X has been shown to effectively treat condition Y in multiple randomized controlled trials." (Level I evidence)
2. "Drug X might have beneficial effects on condition Y, based on preliminary in vitro studies." (Preclinical speculation)
3. "Drug X could potentially reduce symptoms of condition Y, though clinical trials have not been conducted." (Untested hypothesis)

If the retrieval model cannot distinguish these passages' epistemic commitments—and our data shows it cannot—the RAG system will present all three as equally relevant evidence. The generative model may then synthesize them into a confident recommendation, effectively laundering speculation through an evidence-retrieval pipeline.

6.2 Empirical Evidence from Our Data

This scenario is grounded in our experimental measurements. The BGE-reranker-large model—a production-grade reranker used in deployed systems—assigns the following scores to medical hedging pairs:

• "The drug cures cancer" / "The drug may help with some cancer symptoms": 0.992
• "The vaccine prevents infection" / "The vaccine might reduce infection risk": 1.000
• "Exercise cures depression" / "Exercise may help manage depression symptoms": 1.000

These are not edge cases—they represent the core medical hedging pattern. The model rates a definitive cure claim and a tentative symptom-management claim as perfectly equivalent.

6.3 Extension to Other Domains

Legal retrieval: "The defendant is guilty" vs. "The evidence suggests the defendant may be guilty"—a distinction between established fact and preliminary assessment.

Scientific literature search: "The experiment proves the theory" vs. "The experiment provides evidence consistent with the theory"—a distinction between confirmation and corroboration.

Financial analysis: "The stock is guaranteed to rise" vs. "The stock has potential for growth"—a distinction that may constitute the difference between factual reporting and misleading claims under securities regulation.

6.4 Cascading Failures in RAG Pipelines

In a typical two-stage retrieval pipeline (bi-encoder → cross-encoder reranker → generative model), the hedging gap creates a compounding failure:

1. First stage (bi-encoder): With cosine similarities of 0.81–0.93 for hedging pairs, both certain and hedged passages pass any reasonable similarity threshold.
2. Reranking stage (cross-encoder): The reranker, which successfully filters negated and entity-swapped passages, fails to demote hedged passages.
3. Generation stage: The language model receives context mixing certain and hedged statements with no signal to distinguish them.

Each stage could catch the hedging distinction, but none does—creating a pipeline that systematically converts speculative claims into apparently well-grounded retrievals.

7. Potential Solutions

The hedging gap is rooted in training data, evaluation paradigms, and the distributional properties of epistemic markers. We outline several potential approaches:

7.1 Hedging-Aware Training Data

Create training datasets that explicitly pair certain and hedged variants with low similarity/relevance labels. This requires redefining "similarity" to include epistemic modality—a departure from current conventions where hedged variants are rated as highly similar.

7.2 Explicit Uncertainty Classification

Add a separate classifier to identify epistemic markers in retrieved passages, classifying their strength (certain, probable, possible, speculative) and presenting uncertainty labels alongside results. This preserves existing pipelines while adding an epistemic layer.

7.3 Multi-Dimensional Similarity

Decompose similarity into orthogonal components: topical similarity (same subject?), assertional similarity (same claims?), and epistemic similarity (same confidence?). This requires architectural changes but captures the nuanced distinction between "topically relevant" and "assertionally equivalent."

7.4 Contrastive Fine-Tuning on Epistemic Pairs

Use contrastive learning specifically on hedging pairs to push apart embeddings of certain and hedged variants. The challenge is maintaining performance on other categories while improving hedging sensitivity.

7.5 Prompt-Based Approaches

For instruction-following cross-encoders, reformulate the task as "Do these sentences express the same level of certainty?" rather than "How similar are these sentences?" This could provide immediate improvements without retraining.

8. Limitations

Sample size: Our hedging evaluation uses 25 pairs. While sufficient to establish the phenomenon (the effect is large and consistent across models), a larger community-curated dataset would enable statistical significance testing and more precise failure rate estimates. We encourage the community to extend our initial set.

English only: All pairs are in English. Hedging mechanisms vary dramatically across languages.

Binary distinction: We treat hedging as binary (certain vs. hedged) when epistemic modality is a spectrum.

Controlled pairs: Hand-crafted pairs isolate hedging but may not fully represent real-world retrieval conditions where passages differ along multiple dimensions. We note that this controlled design is standard in diagnostic evaluation of NLP models and is a strength for isolating the specific failure mode.

No downstream evaluation: We measure retrieval scores directly but do not evaluate impact on RAG generation quality.

Model selection: We evaluate four bi-encoders and four cross-encoders. Instruction-tuned embedding models (e.g., E5-instruct, GritLM) may show different patterns.

Training data analysis: We hypothesize about training data distributions based on the known properties of NLI, STS, MARCO, and QQP datasets. Direct corpus analysis would strengthen the explanatory claims.

9. Related Work

9.1 Sentence Embeddings and Retrieval

Reimers and Gurevych (2019) introduced Sentence-BERT, enabling efficient computation of semantically meaningful sentence embeddings using siamese BERT networks. This built on BERT (Devlin et al., 2019), the pre-trained transformer architecture underlying all models in our study. Nogueira and Cho (2019) demonstrated effective BERT-based passage reranking on MS MARCO, establishing the bi-encoder retrieval + cross-encoder reranking paradigm we evaluate.

9.2 Compositional Failures in Neural Models

Multiple research groups have independently documented that bi-encoders struggle with negation, entity swaps, numerical changes, and temporal reasoning. The negation problem has been studied across sentence embedding, natural language inference, and question-answering settings. Numerical reasoning failures have been documented in both embedding and generative models. Entity order sensitivity has been identified as particularly challenging for models using symmetric similarity functions.

Our contribution differs from this prior work in two key ways: (1) we systematically compare hedging against these other categories within a unified experimental framework, and (2) we evaluate whether the cross-encoder upgrade path resolves each failure mode, finding that hedging is uniquely resistant to architectural improvement.

9.3 Uncertainty Detection in NLP

Hedge detection has been studied extensively in biomedical NLP, where identifying speculative statements is crucial for constructing accurate knowledge bases from scientific literature. The BioScope corpus and related resources provide annotated uncertainty expressions in medical text. Factuality assessment research has examined how models can distinguish assertions from opinions, hypotheses, and speculation.

This prior work frames uncertainty detection as a classification task (is this sentence speculative?). Our work frames it as a retrieval similarity task (does the retrieval model distinguish speculative from certain statements?), revealing that the problem is unsolved at the retrieval level even though classification-level solutions exist.

9.4 RAG Reliability

Studies on RAG reliability have documented hallucination, retrieval noise, and context conflation as failure modes. Our work adds hedging conflation to this catalog—a previously uncharacterized failure mode where speculative passages are retrieved and presented with the same confidence as established findings.

10. Conclusion

We have presented the first systematic evaluation of hedging sensitivity across both bi-encoder and cross-encoder neural retrieval architectures. Our experiments across eight models reveal a consistent and troubling finding: hedging—the linguistic marking of uncertainty—is the single hardest semantic distinction for the most capable cross-encoder models.

While the cross-encoder upgrade path resolves most compositional semantic failures (reducing failure rates on negation, entity swaps, and temporal inversions from near-100% to 0–3%), it does not resolve hedging. The best cross-encoders achieve only 1.5× improvement on hedging compared to 23–46× improvement on other categories. For the BGE-reranker and Quora-RoBERTa models—the two that demonstrate genuine semantic discrimination capabilities—hedging ranks as the absolute hardest category, with failure rates of 92% and 52% respectively.

This "hedging gap" is not merely an academic concern. In medical, legal, and scientific retrieval applications, the difference between "X causes Y" and "X might cause Y" is the difference between established fact and tentative hypothesis. When retrieval systems cannot distinguish these, they create a pipeline that systematically launders speculation into apparently well-grounded information.

Addressing the hedging gap will require fundamental advances in training data (adding epistemic sensitivity), evaluation benchmarks (testing hedging alongside other semantic modifications), and deployment pipelines (flagging uncertainty in retrieved passages). We release our complete dataset of 336 pairs and all raw model outputs to facilitate follow-up research.

References

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019.

Nogueira, R., & Cho, K. (2019). Passage re-ranking with BERT. arXiv preprint arXiv:1901.04085.

Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of EMNLP 2019.

Appendix A: Full Model Output Data

A.1 Bi-Encoder Individual Hedging Scores

A.2 Cross-Encoder Hedging Pair Scores (First 5 Pairs)

A.3 Cross-Encoder Full Category Raw Means

Appendix B: Complete Hedging Pair List

The full 25 hedging pairs used in cross-encoder evaluation:

1. "The drug cures cancer" / "The drug may help with some cancer symptoms"
2. "This treatment eliminates the disease" / "This treatment might slow disease progression"
3. "The vaccine prevents infection" / "The vaccine might reduce infection risk"
4. "Exercise cures depression" / "Exercise may help manage depression symptoms"
5. "The test confirms the diagnosis" / "The test suggests a possible diagnosis"
6. "The patient will recover" / "The patient might recover"
7. "The procedure is completely safe" / "The procedure carries some risks"
8. "This investment will double your money" / "This investment might grow slightly over time"
9. "The market will crash next month" / "The market could potentially decline in the coming months"
10. "The company will go bankrupt" / "The company faces some financial challenges"
11. "Housing prices will increase 50 percent" / "Housing prices may see modest growth"
12. "Bitcoin will reach 100000 dollars" / "Bitcoin could potentially increase in value"
13. "The stock is guaranteed to rise" / "The stock has potential for growth"
14. "This technology will replace all jobs" / "This technology may affect some job categories"
15. "AI will achieve consciousness by 2030" / "AI might develop more advanced capabilities in the future"
16. "The project will succeed" / "The project has a reasonable chance of success"
17. "This method always works" / "This method works in some cases"
18. "The company will dominate the market" / "The company could become a significant player in the market"
19. "Climate change will cause extinction" / "Climate change poses significant risks to biodiversity"
20. "The reform will solve poverty" / "The reform may help reduce poverty to some degree"
21. "The experiment proves the theory" / "The experiment provides evidence consistent with the theory"
22. "Self-driving cars will eliminate accidents" / "Self-driving cars could reduce accident frequency"
23. "This policy will fix the economy" / "This policy might have positive economic effects"
24. "The new law will end corruption" / "The new law could help reduce certain types of corruption"
25. "Renewable energy will completely replace fossil fuels" / "Renewable energy could eventually supply most electricity needs"

The complete 336-pair dataset (all categories) and raw model outputs are available in our experiment artifacts at the paths listed in SKILL.md.