Beyond Accuracy: A Testing Framework for Semantic Retrieval Systems in High-Stakes Domains

Abstract

Semantic retrieval systems powered by embedding models are increasingly deployed in high-stakes domains including healthcare, law, and finance. While existing benchmarks such as MTEB and BEIR measure aggregate retrieval performance, they fail to expose critical failure modes that can lead to dangerous errors in production — such as a system that cannot distinguish "patient has diabetes" from "patient does not have diabetes." We present RETRIEVE (Robustness Evaluation Tests for Retrieval In Enterprise and Vital Environments), a testing framework that defines eight systematic test categories for semantic retrieval systems: negation robustness, entity swap sensitivity, numerical precision, temporal ordering, quantifier sensitivity, hedging/certainty discrimination, template stability, and out-of-vocabulary robustness. Through empirical validation across four bi-encoder models, four cross-encoder models, and two prompt-sensitivity configurations, we demonstrate that models achieving high aggregate benchmark scores routinely fail specific test categories. Entity swap tests reveal near-total failure across all bi-encoders (mean cosine similarity 0.987–0.992 between swapped pairs), while cross-encoders show dramatically better negation handling (mean raw score 0.49 for negated pairs vs. 0.89 for positive controls). We provide concrete pass/fail thresholds for each test category, reference implementations in Python, and a case study applying the framework to a clinical document retrieval system. Our framework enables engineering teams to systematically validate retrieval components before deployment, transforming model evaluation from an aggregate benchmark exercise into a rigorous quality assurance process.

1. Introduction

The integration of semantic retrieval into software systems has accelerated rapidly. Retrieval-Augmented Generation (RAG) pipelines, semantic search engines, and document similarity systems now underpin critical applications in healthcare information retrieval, legal discovery, financial compliance monitoring, and enterprise knowledge management. These systems typically rely on embedding models — either bi-encoders that independently encode queries and documents into vector spaces, or cross-encoders that jointly process query-document pairs — to determine semantic similarity.

Current evaluation practice for these systems is dominated by aggregate benchmarks. The Massive Text Embedding Benchmark (MTEB) evaluates models across dozens of tasks including retrieval, classification, clustering, and semantic similarity. The Benchmarking IR (BEIR) suite measures zero-shot retrieval performance across diverse domains. These benchmarks are invaluable for model selection and general performance comparison, but they share a critical limitation: they measure average performance across broad datasets, effectively masking specific failure modes.

Consider a clinical decision support system that retrieves relevant patient records based on natural language queries. A model may achieve 0.85 nDCG@10 on medical retrieval benchmarks while simultaneously exhibiting the following behavior: given the query "patient has diabetes," the system returns a cosine similarity of 0.93 with the document "patient does not have diabetes." In a production deployment, this failure means the system cannot reliably distinguish a diagnosis from its negation — a potentially life-threatening defect.

This paper argues that semantic retrieval systems in high-stakes domains require systematic testing that goes beyond aggregate benchmarks. Just as software engineering has evolved from "the program runs without crashing" to comprehensive unit testing, integration testing, and property-based testing, retrieval system evaluation must evolve from "the model scores well on benchmarks" to "the model handles specific critical cases correctly."

We introduce RETRIEVE, a testing framework organized around eight test categories that target specific semantic failure modes. Each category is designed as a suite of test cases analogous to unit tests in traditional software engineering. The framework provides:

1. Test specifications defining what each category evaluates and why it matters
2. Parameterized test generators that create domain-specific test cases
3. Pass/fail criteria calibrated against empirical measurements across multiple model architectures
4. Reference implementations in Python that can be integrated into CI/CD pipelines

Our empirical validation spans four bi-encoder architectures (MiniLM, BGE-large, Nomic-embed, GTE-large), four cross-encoder architectures (STS-B RoBERTa, MS-MARCO MiniLM, BGE-reranker, Quora RoBERTa), and prompt-template sensitivity analysis across two models and ten template configurations. The results reveal systematic vulnerabilities that no aggregate benchmark would expose: all bi-encoders fail entity swap tests with near-perfect similarity between semantically opposite pairs, cross-encoders trained for specific tasks show dramatic miscalibration when applied to novel test categories, and simple prompt template changes can shift similarity scores by up to 0.37 for unrelated sentence pairs.

The contributions of this paper are:

• A systematic taxonomy of eight failure modes for semantic retrieval systems, each motivated by real-world deployment risks
• Empirical evidence that high-performing models on aggregate benchmarks fail specific test categories
• Calibrated pass/fail thresholds derived from experiments across nine model architectures
• A reference implementation suitable for integration into software testing pipelines
• A case study demonstrating framework application to clinical document retrieval

2. Background and Related Work

2.1 Embedding-Based Retrieval Systems

Modern semantic retrieval systems are built on transformer-based language models. BERT (Devlin et al., 2019) established the foundation by pre-training bidirectional encoders on large text corpora, enabling transfer learning for downstream tasks. Sentence-BERT (Reimers and Gurevych, 2019) adapted this architecture for efficient sentence-level embeddings using siamese and triplet network structures, making it practical to encode sentences into fixed-dimensional vectors for similarity search.

Current bi-encoder architectures — including MiniLM, BGE, Nomic-embed, and GTE — build on this foundation. They encode queries and documents independently into dense vectors, enabling efficient approximate nearest neighbor search at scale. Cross-encoders, by contrast, jointly process query-document pairs through the full transformer attention mechanism, achieving higher accuracy at the cost of computational efficiency.

In production systems, a common pattern is a two-stage retrieval pipeline: bi-encoders perform fast initial retrieval from large corpora, and cross-encoders re-rank the top candidates. Understanding the failure modes of both architectures is therefore critical.

2.2 Current Evaluation Paradigms

The Massive Text Embedding Benchmark evaluates embedding models across multiple task categories. It encompasses retrieval, classification, clustering, pair classification, re-ranking, semantic textual similarity, and summarization tasks. Models are ranked by average performance across these categories, producing leaderboard scores that heavily influence model selection decisions.

BEIR focuses specifically on zero-shot information retrieval, evaluating models on diverse domains including biomedical literature, financial documents, scientific papers, and web content. It measures nDCG@10, recall, and precision across these domains.

While both benchmarks provide valuable aggregate performance metrics, they evaluate retrieval quality through naturally occurring query-document pairs. This means failure modes that require specifically constructed adversarial or targeted test cases remain undetected. A model can rank first on aggregate benchmarks while failing 100% of negation-handling tests.

2.3 Behavioral Testing for NLP

The concept of systematic behavioral testing for NLP systems has been explored in prior work. The CheckList framework (Ribeiro et al., 2020) introduced minimum functionality tests, invariance tests, and directional expectation tests for NLP models, drawing explicit analogies to software engineering testing practices. CheckList defined test types including vocabulary perturbations, taxonomy-based expansions, negation tests, and entity swaps for classification and question-answering tasks. Our work builds on the CheckList philosophy but differs in three important ways: (1) we target retrieval similarity scores rather than classification outputs, requiring different test specifications and pass/fail criteria; (2) we provide empirical calibration of thresholds across multiple model architectures rather than relying on expected-behavior specifications alone; and (3) we address retrieval-specific failure modes such as template stability and OOV robustness that do not arise in classification settings.

Adversarial NLP evaluation has received growing attention more broadly. Techniques such as paraphrase-based attacks, character-level perturbations, and semantic-preserving transformations have exposed vulnerabilities in language models. In the embedding space specifically, work on probing classifiers has examined what linguistic information is encoded in transformer representations. Work on fairness and bias has examined how models handle demographic perturbations. Our framework extends these directions by systematically testing semantic distinctions that matter specifically for retrieval correctness — functional correctness in distinguishing negation, quantification, temporality, and entity roles within the context of similarity-based document retrieval.

2.4 Gaps in Current Practice

Current evaluation practices leave several critical gaps:

1. No negation testing: No standard benchmark systematically evaluates whether retrieval systems distinguish affirmative from negative statements
2. No entity-role sensitivity: Standard evaluations do not test whether systems distinguish "A sued B" from "B sued A"
3. No numerical precision evaluation: Benchmarks do not specifically test sensitivity to numerical values
4. No template stability assessment: The effect of query formatting on retrieval results is not systematically measured
5. No domain-specific failure catalogs: Engineering teams deploying retrieval systems lack guidance on what to test

RETRIEVE addresses each of these gaps.

3. The RETRIEVE Test Framework

RETRIEVE defines eight test categories, each targeting a specific semantic failure mode. We adopt the software testing metaphor deliberately: each test category is analogous to a test suite, individual test cases are parameterized inputs, and pass/fail criteria provide binary quality gates.

3.1 Framework Architecture

The framework operates on a simple principle: for each test category, generate pairs of sentences that differ in one specific semantic dimension, compute the similarity score between them using the system under test, and compare the result against a threshold. A well-functioning retrieval system should assign low similarity to sentence pairs that are semantically different and high similarity to pairs that are semantically equivalent.

Each test category specifies:

• Perturbation type: What semantic dimension is varied
• Pair generation: How test pairs are constructed
• Expected behavior: What a correct system should produce
• Pass threshold: The similarity score below which the system passes
• Fail threshold: The similarity score above which the system fails

3.2 Test Category 1: Negation Robustness (NEG)

What it tests: Whether the retrieval system can distinguish a statement from its negation.

Why it matters: In medical records, "patient has diabetes" vs. "patient does not have diabetes" are opposite clinical states. In legal documents, "the defendant is guilty" vs. "the defendant is not guilty" have opposite legal consequences. A retrieval system that conflates these pairs can surface dangerously wrong results.

Test pair construction: For a base statement S, generate the pair (S, negate(S)). Negation is applied through standard patterns: inserting "not," replacing affirmative verbs with negative forms, or adding "no" before key nouns.

Expected behavior: Cosine similarity should be significantly lower than that of positive control pairs (paraphrases). Ideally, negation pairs should score below 0.5 for bi-encoders and show normalized scores below 0.15 for cross-encoders.

Empirical findings: Across four bi-encoder models, mean cosine similarity for negation pairs ranges from 0.889 (MiniLM) to 0.941 (GTE). This means bi-encoders treat negated statements as highly similar to their affirmative counterparts. Cross-encoders show dramatically better performance: the STS-B RoBERTa cross-encoder achieves a mean raw score of 0.491 for negation pairs versus 0.889 for positive controls, successfully distinguishing negated content.

Recommended pass threshold: Bi-encoder cosine similarity < 0.70 for negation pairs (FAIL if > 0.85). Cross-encoder normalized score < 0.15 for negation pairs (PASS).

3.3 Test Category 2: Entity Swap Robustness (ENT)

What it tests: Whether the system distinguishes sentences where entity roles are swapped.

Why it matters: "Company A acquired Company B" has a completely different meaning from "Company B acquired Company A." In legal contexts, "the plaintiff sued the defendant" vs. "the defendant sued the plaintiff" reverses the entire case structure.

Test pair construction: For a sentence containing two named entities in different roles, generate a pair by swapping the entities: (S(A,B), S(B,A)).

Expected behavior: The system should recognize that role-swapped sentences have different meanings. Similarity scores should be noticeably below 1.0.

Empirical findings: This is the most severe failure mode we identified. All four bi-encoder models produce mean cosine similarity above 0.987 for entity-swapped pairs. MiniLM: 0.987, BGE: 0.993, Nomic: 0.988, GTE: 0.992. These scores are effectively indistinguishable from identical sentences. The Jaccard token overlap for entity-swapped pairs is 1.0 (all tokens are the same, just reordered), which explains the failure — bi-encoders based on mean-pooled token representations are inherently insensitive to word order.

Cross-encoders handle this somewhat better: the STS-B RoBERTa model scores entity-swapped pairs at a mean raw score of 0.837, lower than positive controls (0.889) but still alarmingly high for semantically different statements.

Recommended pass threshold: Cosine similarity < 0.90 for entity-swapped pairs (FAIL if > 0.95). This is a strict threshold because entity role confusion has severe consequences.

3.4 Test Category 3: Numerical Sensitivity (NUM)

What it tests: Whether the system distinguishes sentences that differ only in numerical values.

Why it matters: In pharmaceutical contexts, "administer 5mg" vs. "administer 500mg" is a 100x dosage difference. In finance, confusing "5% growth" with "50% growth" fundamentally misrepresents performance. Numerical precision is not optional in high-stakes retrieval.

Test pair construction: Generate pairs where numerical values are changed: different magnitudes (5 vs. 500), different units (mg vs. g), and different precision levels (0.1 vs. 10).

Expected behavior: Systems should show reduced similarity when numerical values differ, with greater sensitivity to larger magnitude differences.

Empirical findings: Bi-encoders show moderate sensitivity to numerical changes. Mean cosine similarity for numerical pairs: MiniLM: 0.882, BGE: 0.945, Nomic: 0.929, GTE: 0.954. While these scores are somewhat lower than entity swap scores, they remain dangerously high. The tokenization analysis reveals why: numbers are tokenized into individual digits and punctuation marks (e.g., "7.2" becomes ["7", ".", "2"]), and the surrounding context dominates the embedding.

Cross-encoders show better numerical discrimination: STS-B RoBERTa scores numerical pairs at a mean of 0.454, well below positive controls, indicating that the joint attention mechanism better captures numerical differences.

Recommended pass threshold: Cosine similarity < 0.80 for pairs with magnitude differences > 10x (FAIL if > 0.90).

3.5 Test Category 4: Temporal Ordering (TMP)

What it tests: Whether the system distinguishes sentences that differ in temporal ordering or sequencing.

Why it matters: "Administer drug A before drug B" vs. "Administer drug A after drug B" can have critical pharmacological implications. In legal proceedings, the temporal ordering of events is often determinative.

Test pair construction: Generate pairs by swapping temporal markers: "before" ↔ "after," "first" ↔ "last," "prior to" ↔ "following."

Expected behavior: Temporal marker swaps should produce measurably lower similarity than paraphrase controls.

Empirical findings: Bi-encoders show moderately high similarity for temporal-swapped pairs: MiniLM: 0.965, BGE: 0.956, Nomic: 0.962, GTE: 0.972. These are among the highest failure scores across categories, indicating that temporal reasoning is particularly weak. The Jaccard overlap is moderate (mean 0.72), as temporal words are typically single tokens, but the embedding models fail to leverage this token-level difference.

Prompt template effects on temporal pairs are minimal: MiniLM shows mean standard deviation of only 0.011 across ten templates, suggesting the failure is inherent to the model rather than prompt-dependent.

Recommended pass threshold: Cosine similarity < 0.85 for temporal-swapped pairs (FAIL if > 0.95).

3.6 Test Category 5: Quantifier Sensitivity (QNT)

What it tests: Whether the system distinguishes between different quantifiers (all, most, some, few, none).

Why it matters: "All patients responded to treatment" vs. "Few patients responded to treatment" conveys dramatically different clinical outcomes. Quantifier confusion in regulatory or compliance retrieval can lead to misinterpretation of rules and policies.

Test pair construction: Generate pairs by substituting quantifiers at different points on the quantifier scale: universal (all, every) ↔ existential (some, a few) ↔ negative (no, none).

Expected behavior: Pairs with opposite quantifiers (all vs. none) should show lower similarity than pairs with adjacent quantifiers (all vs. most).

Empirical findings: Bi-encoders show variable but concerning performance on quantifier pairs. Mean cosine similarity: MiniLM: 0.819, BGE: 0.893, Nomic: 0.879, GTE: 0.922. The range across models is wider here than in other categories, suggesting that some architectures capture quantifier semantics better than others. However, even the best-performing model (MiniLM at 0.819) still produces dangerously high similarity between opposite quantifiers.

Prompt templates have a significant effect on quantifier sensitivity: MiniLM shows mean max shift of 0.135 and BGE shows 0.149 across template configurations, indicating that quantifier tests are particularly sensitive to how queries are formatted.

Recommended pass threshold: Cosine similarity < 0.75 for opposite quantifier pairs (FAIL if > 0.85).

3.7 Test Category 6: Hedging/Certainty Discrimination (HDG)

What it tests: Whether the system distinguishes between statements of different certainty levels.

Why it matters: "The test definitively confirms cancer" vs. "The test possibly suggests cancer" carry fundamentally different clinical implications. In financial contexts, "the company will achieve profitability" vs. "the company might achieve profitability" have different implications for investment decisions.

Test pair construction: Generate pairs by substituting certainty markers: definite (certainly, definitely, confirmed) ↔ uncertain (possibly, might, suggests, potentially).

Expected behavior: Pairs with opposite certainty levels should show reduced similarity relative to paraphrase controls.

Empirical findings: This category shows the widest variation across model architectures. Mean cosine similarity: MiniLM: 0.813, BGE: 0.885, Nomic: 0.858, GTE: 0.926. MiniLM shows the best hedging discrimination among bi-encoders, while GTE shows the worst.

Cross-encoder results are particularly interesting: the STS-B RoBERTa model scores hedging pairs at 0.652, the BGE-reranker at 0.883, and the Quora RoBERTa model at 0.514. The MS-MARCO cross-encoder assigns a mean normalized score of 0.673, notably its weakest discrimination category. This variation across cross-encoders suggests that hedging sensitivity is heavily influenced by training data composition.

Recommended pass threshold: Cosine similarity < 0.75 for opposite certainty pairs (FAIL if > 0.85).

3.8 Test Category 7: Template Stability (TPL)

What it tests: Whether the retrieval results are stable across different query formulations that should be semantically equivalent.

Why it matters: In production systems, users formulate queries differently. A system should return consistent results whether the query is "patient diabetic history" or "search_query: patient diabetic history" or "Represent this sentence for retrieval: patient diabetic history." Template instability means that deployment choices about query formatting can silently change retrieval behavior.

Test pair construction: For a base query Q, generate variants by prepending different template prefixes. The framework tests ten templates: no prefix, "query:", "search_query:", "search_document:", "Represent this sentence:", "Represent this sentence for retrieval:", "passage:", "clustering:", "classification:", and a noise prefix "xyzzy:".

Expected behavior: Similarity scores between a given pair should remain stable across templates. Standard deviation across templates should be minimal.

Empirical findings: Template stability varies dramatically by model and pair category. For positive control pairs (paraphrases), MiniLM shows mean SD of 0.041 and BGE shows 0.017 — the instruction-tuned BGE model is more stable for genuine semantic pairs. However, for negative control pairs (unrelated sentences), the picture reverses: MiniLM shows mean SD of 0.114 and mean max shift of 0.370, while BGE shows SD of 0.083 and max shift of 0.304.

This means that template choice can shift the similarity of unrelated documents by up to 0.37 points — easily enough to cross a retrieval threshold and include irrelevant documents in results. Entity swap pairs are the most template-stable category (SD < 0.004 for both models), consistent with the finding that these models are fundamentally insensitive to entity ordering regardless of framing.

Recommended pass threshold: SD across templates < 0.03 for positive pairs, < 0.10 for all pairs. Max shift < 0.15 (FAIL if max shift > 0.30).

3.9 Test Category 8: Out-of-Vocabulary Robustness (OOV)

What it tests: How the system handles terms not present in its training vocabulary, including novel entities, brand names, technical jargon, and domain-specific terminology.

Why it matters: Real-world retrieval systems regularly encounter novel terms: new drug names, emerging technologies, recently formed organizations, and domain-specific jargon. A robust system should maintain reasonable retrieval quality even when encountering unknown terms, relying on contextual understanding rather than exact lexical matching.

Test pair construction: Replace known entities with fabricated OOV terms and measure the cosine similarity delta. For example, replace "Einstein" with "Wompelfritz" in an otherwise identical sentence and measure how much the similarity to a reference sentence changes.

Expected behavior: The similarity delta between original and OOV-replaced versions should be small, indicating that the system relies on context rather than entity-specific memorization.

Empirical findings: OOV robustness varies dramatically by model size and architecture. Using 20 entity replacements from the tokenizer effects experiment: MiniLM shows mean delta of 0.123 with max delta of 0.245, BGE shows 0.055 with max 0.091, Nomic shows 0.104 with max 0.191, and GTE shows 0.035 with max 0.064.

The expanded OOV experiment with 50 fabricated entity replacements across five categories (people, locations, organizations, medical terms, technical terms) reveals even starker differences. MiniLM shows mean delta of 0.358, with medical terms being most severely affected (mean delta 0.474). BGE and Nomic show much better robustness at mean deltas of 0.180 and 0.180 respectively.

The correlation with subword tokenization is significant: entities that fragment into more subword tokens show larger deltas. The mean replacement subtoken count is 4.2, and entities with 5+ subtokens show consistently higher degradation than those with 3 or fewer.

Recommended pass threshold: Mean OOV delta < 0.10 across test entities (FAIL if mean delta > 0.20). Medical domain entities should be tested separately with stricter thresholds.

4. Empirical Validation

4.1 Experimental Setup

We conducted experiments across three experimental configurations:

Bi-encoder experiment: Four models tested on 100 sentence pairs across 8 categories (negation: 15 pairs, numerical: 15, entity swap: 10, temporal: 10, quantifier: 10, hedging: 5, positive control: 20, negative control: 15). Similarity measured by cosine similarity of sentence embeddings.

Models tested:

• MiniLM (sentence-transformers/all-MiniLM-L6-v2): 6-layer, 22M parameters, WordPiece tokenizer, vocab size 30,522
• BGE-large (BAAI/bge-large-en-v1.5): 24-layer, 335M parameters, WordPiece tokenizer, vocab size 30,522
• Nomic-embed (nomic-ai/nomic-embed-text-v1.5): SentencePiece tokenizer, vocab size 30,522
• GTE-large (thenlper/gte-large): 24-layer, WordPiece tokenizer, vocab size 30,522

Cross-encoder experiment: Four functional models tested on 336 sentence pairs across 9 categories (including near-miss pairs). Similarity measured by raw cross-encoder scores (scale varies by model).

Models tested:

• STS-B RoBERTa (cross-encoder/stsb-roberta-large): Trained on STS Benchmark, scores 0–1
• MS-MARCO MiniLM (cross-encoder/ms-marco-MiniLM-L-12-v2): Trained on MS-MARCO passages, unbounded scores
• BGE-reranker (BAAI/bge-reranker-large): Trained for passage reranking, logit scores
• Quora RoBERTa (cross-encoder/quora-roberta-large): Trained on Quora duplicate detection, scores 0–1

Prompt sensitivity experiment: Two bi-encoder models (MiniLM and BGE-large) tested on 100 sentence pairs across 10 template configurations, measuring similarity stability under different prompt prefixes.

4.2 Results: Bi-Encoder Category Performance

Table 1 presents mean cosine similarity by category across all four bi-encoder models.

Several patterns emerge:

No bi-encoder passes entity swap tests. With scores consistently above 0.987, all bi-encoders effectively treat entity-swapped sentences as identical. This is a fundamental architectural limitation of mean-pooled bi-encoders: they are bag-of-words models in a vector space, and permuting words does not change the mean pool output significantly.

MiniLM shows paradoxical behavior. Despite being the smallest model (22M parameters), MiniLM achieves the best negative control separation (0.015 vs. 0.599–0.711 for larger models) and the best quantifier/hedging discrimination. However, it also shows the lowest positive control similarity (0.765), suggesting a more conservative similarity scale rather than superior semantic understanding.

Larger models have higher similarity floors. GTE-large produces the highest mean cosine similarity across every category — including negative controls (0.711) and categories where high similarity indicates failure. This elevated negative control score is a well-documented property of larger embedding models: they produce similarity distributions with higher means and smaller variances. The phenomenon arises because larger models encode more fine-grained semantic features into a fixed-dimensional space, resulting in higher baseline dot products between any two vectors. This does not indicate a flaw in our experimental setup; rather, it demonstrates that fixed similarity thresholds are inappropriate when comparing across model architectures — a key motivator for our framework's recommendation to calibrate thresholds per-model using positive and negative control pairs. The negative control scores (MiniLM: 0.015, BGE: 0.599, Nomic: 0.470, GTE: 0.711) serve as model-specific baselines against which test category scores should be interpreted.

The correlation is strong. Pearson correlations between Jaccard token overlap and cosine similarity range from 0.703 (BGE) to 0.766 (MiniLM), indicating that these models are substantially influenced by surface-level lexical overlap.

4.3 Results: Cross-Encoder Category Performance

Table 2 presents raw mean scores by category for cross-encoder models.

*MS-MARCO uses unbounded scores; relative ordering is meaningful but absolute values are not directly comparable.

Cross-encoders demonstrate fundamentally different failure patterns than bi-encoders:

Negation handling is dramatically better. The STS-B RoBERTa cross-encoder scores negation pairs at 0.491 — approximately halfway between positive controls (0.889) and negative controls (0.010). While this is not ideal (true negations should score closer to 0), it is far superior to bi-encoder scores of 0.889–0.941.

Entity swap remains problematic. Even with full cross-attention over both sentences, the STS-B model scores entity-swapped pairs at 0.837, indicating that cross-encoders also struggle with role reversal, though less severely than bi-encoders. The MS-MARCO model shows near-perfect scores (8.999) for entity swaps, treating them as essentially identical.

Task-specific training dominates. The Quora model, trained for duplicate question detection, assigns extremely low scores to almost all test categories because the test pairs are statements rather than question duplicates. The BGE-reranker shows strong hedging sensitivity (0.883) that nearly matches positive controls (0.996), indicating it does not distinguish certainty levels.

Failure rates at the 0.8 threshold reveal binary behavior. STS-B RoBERTa shows 68.9% failure rate for entity swap at the 0.8 threshold, but 0% for negation and numerical categories. MS-MARCO shows near-100% failure rates for entity swap, temporal, and negation at equivalent thresholds.

4.4 Results: Template Stability

Table 3 presents template stability metrics across categories.

Key findings:

Negative controls are most template-sensitive. Unrelated sentence pairs show the highest sensitivity to prompt templates, with max shifts of 0.370 (MiniLM) and 0.304 (BGE). This means that a template choice can cause the system to assign substantial similarity to unrelated documents — a critical production risk.

Entity swap pairs are template-immune. With SD < 0.004 across all configurations, entity swap similarity is invariant to template choice. This makes sense: since the models are fundamentally insensitive to word order, no amount of prompt engineering will fix this failure.

BGE is generally more template-stable. The instruction-tuned BGE model shows lower SD across most categories, suggesting that models designed for specific prefix patterns achieve greater consistency.

Quantifier pairs show surprising template sensitivity. Both models show relatively high max shifts for quantifier pairs (0.135 and 0.149), suggesting that quantifier semantics are partially controlled by contextual framing and could potentially be improved through targeted prompt design.

4.5 Results: OOV Robustness

Table 4 presents OOV sensitivity metrics.

Key findings:

Model size inversely correlates with OOV sensitivity. The smallest model (MiniLM, 22M parameters) shows the largest OOV deltas, while the largest models (BGE, GTE) show the smallest. Larger models have learned more robust contextual representations that are less dependent on individual entity recognition.

Medical terms are most OOV-vulnerable. In the MiniLM model, medical term replacements cause mean delta of 0.474, compared to 0.310 for technical terms and 0.316 for people names. This is particularly concerning for healthcare applications, where novel drug names and medical terminology are common.

Subword fragmentation predicts vulnerability. The mean subword count for replacement tokens is 4.2 (compared to lower counts for original in-vocabulary entities), and the correlation between subtoken count and similarity delta is positive. Entities that fragment into many subword tokens produce less coherent representations.

5. Test Implementation Guide

5.1 Architecture

The RETRIEVE framework is implemented as a Python library that integrates with standard testing frameworks. The core architecture consists of:

1. Test generators that produce sentence pairs for each category
2. Model adapters that compute similarity scores using different backends
3. Threshold evaluators that determine pass/fail status
4. Report generators that produce human-readable test reports

5.2 Core Test Runner

import numpy as np
from dataclasses import dataclass
from typing import List, Tuple, Callable
from enum import Enum

class TestCategory(Enum):
    NEGATION = "negation"
    ENTITY_SWAP = "entity_swap"
    NUMERICAL = "numerical"
    TEMPORAL = "temporal"
    QUANTIFIER = "quantifier"
    HEDGING = "hedging"
    TEMPLATE = "template_stability"
    OOV = "oov_robustness"

@dataclass
class TestResult:
    category: TestCategory
    pair: Tuple[str, str]
    score: float
    threshold: float
    passed: bool
    metadata: dict

class RetrievalTestSuite:
    def __init__(self, similarity_fn: Callable[[str, str], float]):
        self.similarity_fn = similarity_fn
        self.results: List[TestResult] = []
        self.thresholds = {
            TestCategory.NEGATION: 0.70,
            TestCategory.ENTITY_SWAP: 0.90,
            TestCategory.NUMERICAL: 0.80,
            TestCategory.TEMPORAL: 0.85,
            TestCategory.QUANTIFIER: 0.75,
            TestCategory.HEDGING: 0.75,
        }

    def run_category(self, category: TestCategory, 
                     pairs: List[Tuple[str, str]]) -> List[TestResult]:
        threshold = self.thresholds.get(category, 0.80)
        results = []
        for sent_a, sent_b in pairs:
            score = self.similarity_fn(sent_a, sent_b)
            passed = score < threshold
            result = TestResult(
                category=category, pair=(sent_a, sent_b),
                score=score, threshold=threshold,
                passed=passed, metadata={}
            )
            results.append(result)
            self.results.append(result)
        return results

    def summary(self) -> dict:
        from collections import defaultdict
        by_category = defaultdict(list)
        for r in self.results:
            by_category[r.category].append(r)
        summary = {}
        for cat, results in by_category.items():
            scores = [r.score for r in results]
            passed = sum(1 for r in results if r.passed)
            summary[cat.value] = {
                "n": len(results),
                "pass_rate": passed / len(results),
                "mean_score": np.mean(scores),
                "max_score": np.max(scores),
                "min_score": np.min(scores),
            }
        return summary

5.3 Domain-Specific Test Generators

class MedicalTestGenerator:
    """Generate medical domain test pairs."""
    
    NEGATION_TEMPLATES = [
        ("The patient has {condition}", 
         "The patient does not have {condition}"),
        ("The test confirms {condition}", 
         "The test does not confirm {condition}"),
        ("{treatment} is indicated", 
         "{treatment} is contraindicated"),
    ]
    
    CONDITIONS = [
        "diabetes", "hypertension", "cancer", 
        "pneumonia", "sepsis", "anemia"
    ]
    
    def generate_negation_pairs(self) -> List[Tuple[str, str]]:
        pairs = []
        for template_a, template_b in self.NEGATION_TEMPLATES:
            for condition in self.CONDITIONS:
                pairs.append((
                    template_a.format(condition=condition,
                                     treatment=condition),
                    template_b.format(condition=condition,
                                     treatment=condition)
                ))
        return pairs

    NUMERICAL_TEMPLATES = [
        ("Administer {dose_a} of {drug}",
         "Administer {dose_b} of {drug}"),
        ("Blood pressure is {val_a} mmHg",
         "Blood pressure is {val_b} mmHg"),
    ]
    
    def generate_numerical_pairs(self) -> List[Tuple[str, str]]:
        pairs = []
        drugs = ["metformin", "insulin", "aspirin"]
        dose_pairs = [("5mg", "500mg"), ("10mg", "100mg"), 
                      ("0.5mg", "50mg")]
        for drug, (d_a, d_b) in zip(drugs, dose_pairs):
            pairs.append((
                f"Administer {d_a} of {drug}",
                f"Administer {d_b} of {drug}"
            ))
        bp_pairs = [("120/80", "180/120"), ("90/60", "160/100")]
        for bp_a, bp_b in bp_pairs:
            pairs.append((
                f"Blood pressure is {bp_a} mmHg",
                f"Blood pressure is {bp_b} mmHg"
            ))
        return pairs

    def generate_entity_swap_pairs(self) -> List[Tuple[str, str]]:
        return [
            ("Dr. Smith referred patient to Dr. Jones",
             "Dr. Jones referred patient to Dr. Smith"),
            ("Drug A interacts with Drug B",
             "Drug B interacts with Drug A"),
            ("Hospital A transferred patient to Hospital B",
             "Hospital B transferred patient to Hospital A"),
        ]

    def generate_temporal_pairs(self) -> List[Tuple[str, str]]:
        return [
            ("Administer analgesic before the procedure",
             "Administer analgesic after the procedure"),
            ("Symptoms appeared before treatment began",
             "Symptoms appeared after treatment began"),
            ("The patient was stable prior to surgery",
             "The patient was stable following surgery"),
        ]

    def generate_quantifier_pairs(self) -> List[Tuple[str, str]]:
        return [
            ("All patients responded to treatment",
             "Few patients responded to treatment"),
            ("Every sample tested positive",
             "No sample tested positive"),
            ("Most side effects are mild",
             "Rare side effects are mild"),
        ]

    def generate_hedging_pairs(self) -> List[Tuple[str, str]]:
        return [
            ("The test definitively confirms cancer",
             "The test possibly suggests cancer"),
            ("The treatment is proven effective",
             "The treatment might be effective"),
            ("The diagnosis is certain",
             "The diagnosis is uncertain"),
        ]

5.4 Integration with CI/CD

# Example pytest integration
import pytest

def get_model_similarity():
    """Initialize your retrieval model."""
    from sentence_transformers import SentenceTransformer
    model = SentenceTransformer('your-model-name')
    def similarity(a: str, b: str) -> float:
        emb = model.encode([a, b])
        return float(np.dot(emb[0], emb[1]) / 
                     (np.linalg.norm(emb[0]) * np.linalg.norm(emb[1])))
    return similarity

class TestRetrievalQuality:
    @pytest.fixture(autouse=True)
    def setup(self):
        self.suite = RetrievalTestSuite(get_model_similarity())
        self.gen = MedicalTestGenerator()

    def test_negation_robustness(self):
        pairs = self.gen.generate_negation_pairs()
        results = self.suite.run_category(
            TestCategory.NEGATION, pairs)
        pass_rate = sum(1 for r in results if r.passed) / len(results)
        assert pass_rate >= 0.80, (
            f"Negation test pass rate {pass_rate:.1%} "
            f"below 80% threshold"
        )

    def test_entity_swap_sensitivity(self):
        pairs = self.gen.generate_entity_swap_pairs()
        results = self.suite.run_category(
            TestCategory.ENTITY_SWAP, pairs)
        mean_score = np.mean([r.score for r in results])
        assert mean_score < 0.90, (
            f"Entity swap mean similarity {mean_score:.3f} "
            f"indicates model cannot distinguish role swaps"
        )

    def test_numerical_precision(self):
        pairs = self.gen.generate_numerical_pairs()
        results = self.suite.run_category(
            TestCategory.NUMERICAL, pairs)
        pass_rate = sum(1 for r in results if r.passed) / len(results)
        assert pass_rate >= 0.70, (
            f"Numerical precision pass rate {pass_rate:.1%} "
            f"below 70% threshold"
        )

    def test_template_stability(self):
        templates = ["", "query: ", "search_query: "]
        base_pair = ("The patient has diabetes",
                     "The patient is being treated for diabetes")
        scores = []
        for t in templates:
            score = self.suite.similarity_fn(
                t + base_pair[0], t + base_pair[1])
            scores.append(score)
        sd = np.std(scores)
        assert sd < 0.05, (
            f"Template stability SD {sd:.4f} exceeds threshold"
        )

5.5 OOV Robustness Test Implementation

class OOVTestGenerator:
    """Generate OOV robustness test cases."""
    
    FABRICATED_TERMS = {
        "medical": [
            ("metformin", "xylophrix"), 
            ("ibuprofen", "quarbitone"),
            ("insulin", "nervulax"),
            ("penicillin", "fentrazol"),
        ],
        "technical": [
            ("Python", "blixtware"),
            ("Linux", "quarbitone"),
            ("Docker", "nervulax"),
        ],
        "entity": [
            ("Einstein", "wompelfritz"),
            ("Shakespeare", "frondlebard"),
            ("Tokyo", "quonzaville"),
        ],
    }
    
    CONTEXT_TEMPLATES = [
        "Research on {term} has shown promising results",
        "The effectiveness of {term} was evaluated in a clinical trial",
        "Studies involving {term} demonstrate significant improvements",
    ]
    
    def generate_oov_pairs(self) -> List[dict]:
        test_cases = []
        for domain, replacements in self.FABRICATED_TERMS.items():
            for original, fabricated in replacements:
                for template in self.CONTEXT_TEMPLATES:
                    sent_original = template.format(term=original)
                    sent_fabricated = template.format(term=fabricated)
                    test_cases.append({
                        "domain": domain,
                        "original": sent_original,
                        "fabricated": sent_fabricated,
                        "original_term": original,
                        "fabricated_term": fabricated,
                    })
        return test_cases

    def evaluate_oov_robustness(
        self, similarity_fn, reference_sent: str
    ) -> dict:
        test_cases = self.generate_oov_pairs()
        deltas = []
        for tc in test_cases:
            score_orig = similarity_fn(reference_sent, tc["original"])
            score_fab = similarity_fn(reference_sent, tc["fabricated"])
            delta = abs(score_orig - score_fab)
            deltas.append(delta)
            tc["delta"] = delta
        return {
            "mean_delta": np.mean(deltas),
            "max_delta": np.max(deltas),
            "pass": np.mean(deltas) < 0.10,
            "details": test_cases,
        }

6. Case Study: Applying RETRIEVE to Medical Retrieval

To demonstrate practical application of the RETRIEVE framework, we apply it to evaluate specific model components that would be used in a clinical document retrieval system. Unlike a hypothetical scenario, we use the actual empirical measurements from our experiments to show how the framework guides deployment decisions.

6.1 Components Under Test

We evaluate two specific model components using the RETRIEVE framework: BGE-large-en-v1.5 as a representative bi-encoder for initial retrieval, and STS-B RoBERTa as a representative cross-encoder for re-ranking. These are real models whose actual test results we report from our experiments. The target deployment context is a clinical document retrieval system serving clinicians querying patient histories.

6.2 Test Execution

Using RETRIEVE, we evaluate both the bi-encoder and cross-encoder components independently, then evaluate the combined pipeline.

Bi-encoder (BGE-large) results:

The bi-encoder fails six of eight test categories. While this may seem alarming, it is expected for a first-stage retrieval component — the bi-encoder's role is to cast a wide net, and the cross-encoder handles fine-grained discrimination.

Cross-encoder (STS-B RoBERTa) results:

The cross-encoder passes four of six applicable categories, fails entity swap, and produces a warning for hedging discrimination.

6.3 Risk Assessment and Mitigation

Based on the RETRIEVE results, the engineering team identifies two critical risks:

1. Entity swap vulnerability (pipeline-level): Neither component reliably distinguishes entity-swapped statements. Mitigation: implement a named entity recognition (NER) post-processing step that verifies entity-role consistency between query and retrieved documents.

2. Hedging uncertainty in cross-encoder: The 68% pass rate for hedging means that approximately one-third of certainty-sensitive queries may return results with incorrect confidence levels. Mitigation: add metadata-level certainty annotations to clinical notes and filter by certainty level when queries contain certainty-related terms.

6.4 Deployment Recommendations

Based on the RETRIEVE results, any clinical deployment using these components would require mitigations before production use:

• Entity swap: requires NER-based post-processing to verify entity-role consistency between query and retrieved documents
• Hedging: requires metadata-level certainty annotations on clinical notes, with certainty-level filtering when queries contain certainty-related terms
• All six bi-encoder test failures: acceptable when cross-encoder re-ranking is mandatory (the bi-encoder serves only as a candidate generator)

The framework provides a quantitative basis for deployment decisions: rather than subjective assessments of model quality, engineering teams can point to specific test category pass/fail rates and their corresponding clinical risk implications. RETRIEVE recommends quarterly re-evaluation as model updates and corpus changes may shift performance.

7. Discussion

7.1 Minimum Test Coverage Recommendations

Based on our empirical results, we recommend the following minimum test coverage for production retrieval systems:

Critical (must pass before deployment):

• Negation robustness: Minimum 15 pairs per domain
• Entity swap robustness: Minimum 10 pairs covering key entity relationships
• Numerical sensitivity: Minimum 10 pairs covering clinical dosages or financial figures

Important (should pass, mitigate if not):

• Temporal ordering: Minimum 5 pairs per domain
• Quantifier sensitivity: Minimum 5 pairs
• Template stability: Test with all templates used in production plus 3 adversarial templates

Recommended (measure and track):

• Hedging/certainty: Minimum 5 pairs
• OOV robustness: Minimum 10 fabricated entities relevant to the domain

7.2 Architecture Selection Implications

Our results provide guidance for architecture selection:

Use cross-encoders for safety-critical re-ranking. Cross-encoders consistently outperform bi-encoders on negation, numerical, and quantifier tests. For applications where these distinctions are critical, cross-encoder re-ranking is not optional — it is a safety requirement.

Do not rely solely on bi-encoders for high-stakes retrieval. No bi-encoder in our study passes any of the first six test categories at recommended thresholds. Bi-encoders are effective for initial candidate retrieval but should not be the sole determinant of retrieval relevance in high-stakes applications.

Consider model size for OOV robustness. Larger models (BGE, GTE) show substantially better OOV robustness than smaller models (MiniLM). For domains with frequently changing terminology, larger models provide a meaningful safety margin.

Test entity swap separately — no current architecture handles it well. Both bi-encoders and cross-encoders struggle with entity role swaps. This suggests a need for either architectural innovations (order-sensitive embeddings) or post-processing mitigations (NER-based role verification).

7.3 Integration with Existing Quality Processes

RETRIEVE is designed to complement, not replace, existing evaluation practices:

• Aggregate benchmarks (MTEB, BEIR) for model selection and general quality assessment
• RETRIEVE tests for specific failure mode detection before deployment
• A/B testing for measuring production impact of model changes
• Monitoring for detecting drift in retrieval quality over time

The framework fits naturally into a staged deployment process: models are first evaluated on aggregate benchmarks, then subjected to RETRIEVE testing, and finally deployed with monitoring that includes RETRIEVE-derived test cases as canary queries.

7.5 Threshold Calibration Methodology

A legitimate concern with our framework is the potential for circular reasoning: if thresholds are calibrated based on the performance of tested models, using those thresholds to declare failures may be tautological. We address this by clarifying our threshold methodology.

Our recommended thresholds are not derived from the failure-mode scores themselves. Instead, they are anchored to two independent references: (1) the positive control distribution (paraphrases that should score high) and (2) the negative control distribution (unrelated pairs that should score low). The pass threshold for each test category is set relative to these controls — specifically, we require that test-category scores fall below the midpoint between positive and negative control means, adjusted for the severity of the failure mode. For example, the negation threshold of 0.70 is not derived from the observation that bi-encoders score 0.889–0.941 on negation pairs; rather, it reflects the expectation that negated statements should be semantically more similar to unrelated content than to paraphrases. The midpoint between MiniLM positive controls (0.765) and negative controls (0.015) is approximately 0.39 — our threshold of 0.70 is actually generous.

Furthermore, the framework explicitly recommends that deployment teams calibrate thresholds using their own domain-specific positive and negative control pairs, rather than relying on our default values. This ensures that thresholds reflect the specific model's similarity distribution and the domain's risk tolerance.

7.6 Test Pair Validation

All test pairs used in our experiments were manually constructed by the authors following explicit linguistic specifications for each category (e.g., negation pairs must differ only by the insertion of a negation marker; entity swap pairs must contain exactly two named entities whose roles are exchanged). While we did not conduct formal inter-annotator agreement studies, the test pairs follow deterministic structural rules that make semantic correctness verifiable by inspection. We provide all test pairs in the reference implementation to enable independent verification. Future work should include larger, crowd-sourced test sets with formal quality validation protocols.

7.7 Correlation Analysis

The strong correlation between Jaccard token overlap and cosine similarity (Pearson r: 0.703–0.766, Spearman ρ: 0.663–0.832) across all bi-encoder models suggests that these models are substantially performing lexical matching in vector space. While they add semantic smoothing (paraphrases score higher than lexical overlap alone would predict), the underlying signal is heavily driven by shared vocabulary.

This finding has practical implications: for test categories where semantic distinction requires going beyond lexical overlap — which includes negation (same words + "not"), entity swap (same words, different order), and temporal ordering (same words with swapped temporal markers) — bi-encoders are structurally limited. The testing framework should weight these categories more heavily when evaluating bi-encoder components.

8. Limitations

Several limitations should be acknowledged:

Scale of test pairs. Our empirical validation uses 100 pairs for bi-encoder tests (distributed across 8 categories, with some categories containing as few as 5 pairs) and 336 pairs for cross-encoder tests. Categories with small sample sizes (hedging: 5 pairs) provide directional evidence rather than statistically robust estimates. We report these results with appropriate caution and recommend that production implementations use substantially larger test suites (minimum 15–20 pairs per category, 500+ pairs total) with domain-specific coverage. The small per-category sample sizes in our study are a consequence of manual test pair construction; future work should explore semi-automated pair generation to achieve larger scale while maintaining quality.

English-only evaluation. All experiments were conducted in English. The framework's test categories are language-agnostic in principle, but thresholds and specific test patterns may need recalibration for other languages, particularly for languages with different negation structures or word order sensitivity.

Static evaluation. Our tests evaluate models in isolation rather than within full retrieval pipelines. Production systems include indexing strategies, approximate nearest neighbor algorithms, and hybrid search components that may interact with the failure modes we identify. Pipeline-level testing is recommended in addition to component-level RETRIEVE tests.

Limited cross-encoder coverage. One cross-encoder model (NLI RoBERTa) failed to produce results due to compatibility issues, reducing our cross-encoder evaluation to four models. The cross-encoder landscape is rapidly evolving, and newer models may show different failure patterns.

Threshold calibration. Our recommended pass/fail thresholds are derived from the specific models and test pairs in our experiments. Different domains and risk tolerances may require adjusted thresholds. We recommend that teams calibrate thresholds using their own domain-specific positive and negative control pairs.

Synthetic test pairs. Our test pairs are synthetically constructed rather than drawn from production data. While this ensures controlled evaluation of specific failure modes, it may not capture the full complexity of real-world queries. We recommend supplementing RETRIEVE tests with production-derived test cases.

9. Conclusion

We have presented RETRIEVE, a testing framework for semantic retrieval systems that addresses critical gaps in current evaluation practice. By defining eight systematic test categories — negation robustness, entity swap sensitivity, numerical precision, temporal ordering, quantifier sensitivity, hedging/certainty discrimination, template stability, and OOV robustness — the framework enables engineering teams to identify specific failure modes before deployment.

Our empirical validation across nine model architectures reveals that high aggregate benchmark performance does not guarantee robustness to specific failure modes. All bi-encoder models fail entity swap and negation tests at our recommended thresholds. Cross-encoders offer substantially better discrimination for negation and numerical categories but still struggle with entity role swaps. Template stability varies by model architecture and pair category, with unrelated document pairs showing the highest instability. OOV robustness correlates with model size, with smaller models showing up to 6.5x larger similarity deltas when encountering novel terminology.

The framework provides actionable outputs: concrete pass/fail thresholds for each test category, reference implementations that integrate with standard testing frameworks, and a staged deployment methodology that combines RETRIEVE testing with existing evaluation practices. The medical retrieval case study demonstrates how the framework identifies risks, guides mitigation strategies, and provides evidence for deployment decisions.

As semantic retrieval systems continue to proliferate in high-stakes domains, the need for systematic, failure-mode-specific testing will only grow. RETRIEVE offers a foundation for this practice, transforming retrieval evaluation from an aggregate benchmarking exercise into a rigorous quality assurance discipline. We encourage the community to extend the framework with additional test categories, domain-specific test generators, and calibrated thresholds for emerging model architectures.

References

Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT.

Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of EMNLP-IJCNLP.

Ribeiro, M. T., Wu, T., Guestrin, C., and Singh, S. (2020). Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In Proceedings of ACL.