Numbers Are Just Tokens: Why Embedding Models Cannot Distinguish Quantities and What It Means for Safety-Critical Retrieval

Abstract

Dense embedding models have become the backbone of modern retrieval systems, powering search, recommendation, and retrieval-augmented generation (RAG) pipelines across virtually every domain. However, these models harbor a critical and underexplored vulnerability: they are fundamentally blind to numerical magnitude. Because tokenizers decompose numbers into subword fragments devoid of quantitative semantics, the resulting embeddings treat "5mg" and "500mg" as near-identical — yielding cosine similarities above 0.95 despite representing a 100-fold difference in drug dosage. We present a systematic empirical evaluation of four widely deployed embedding models (MiniLM, BGE, Nomic-Embed, GTE) on carefully constructed numerical pair benchmarks. Our results demonstrate that bi-encoder models consistently fail to distinguish quantities differing by orders of magnitude, with average cosine similarities of 0.928 across all tested model-pair combinations. Cross-encoder rerankers partially mitigate this failure, correcting 73% of numerical errors, but leave a residual 27% failure rate that remains dangerous in safety-critical applications. We analyze the root cause through tokenization decomposition, showing how WordPiece and SentencePiece tokenizers shatter numerical strings into semantically arbitrary fragments. We further present a domain-specific risk analysis spanning pharmaceutical dosing, financial transactions, engineering tolerances, and medical laboratory values — domains where numerical confusion can cause patient death, financial catastrophe, or structural failure. Finally, we demonstrate that standard embedding benchmarks (MTEB, STS) contain virtually no numerical reasoning tests, allowing this critical failure mode to persist undetected. We propose concrete mitigation strategies including hybrid retrieval with explicit numerical extraction, structured metadata filtering, and magnitude-aware embedding fine-tuning.

1. Introduction

The adoption of dense retrieval systems has accelerated dramatically in recent years. Transformer-based embedding models, trained via contrastive learning on massive text corpora, now underpin search engines, enterprise knowledge bases, customer support systems, and the rapidly proliferating class of retrieval-augmented generation (RAG) applications. These systems encode text into high-dimensional vector representations and retrieve relevant documents by computing cosine similarity or inner product between query and document embeddings.

The appeal is clear: dense retrieval captures semantic similarity in ways that keyword-based systems cannot. "Heart attack" retrieves documents about "myocardial infarction." "How to fix a flat tire" retrieves guides titled "Changing a punctured tyre." The models learn rich semantic representations that generalize across paraphrases, synonyms, and even cross-lingual boundaries.

But there is a failure mode hiding in plain sight. Consider a pharmacist querying a drug information system: "What is the recommended dose of metformin?" The system returns two documents:

• Document A: "Metformin: Start at 500mg twice daily, maximum 2000mg/day"
• Document B: "Metformin: Start at 5mg twice daily, maximum 20mg/day"

Document B is fictitious and dangerously wrong — a 100-fold underdose. Yet to a standard embedding model, these two documents are nearly identical. In our experiments, the cosine similarity between such pairs consistently exceeds 0.95. The embedding model literally cannot tell the difference between 5mg and 500mg.

This is not a minor edge case. Numbers pervade the documents that retrieval systems index: drug dosages, financial figures, engineering specifications, legal thresholds, scientific measurements, dates, quantities, and more. When a retrieval system treats "withdraw $500" and "withdraw$500,000" as semantically equivalent, the consequences range from inconvenient to catastrophic.

In this paper, we present the first focused empirical investigation of numerical blindness in embedding models for retrieval. We make the following contributions:

1. Empirical evidence of numerical blindness across four widely deployed embedding models, demonstrating that cosine similarities between numerically different but textually similar sentences consistently exceed 0.88.

2. Tokenization root cause analysis showing how standard subword tokenizers decompose numbers into fragments that destroy magnitude information.

3. Magnitude sensitivity profiling revealing that embeddings fail to distinguish differences at 2×, 10×, 100×, and even 1000× scales.

4. Cross-encoder partial mitigation analysis showing rerankers correct 73% of numerical errors but leave a dangerous 27% residual failure rate.

5. Domain-specific risk analysis for pharmaceutical, financial, engineering, and medical applications.

6. Benchmark gap analysis demonstrating that MTEB and STS benchmarks contain virtually no numerical reasoning tests.

7. Concrete mitigation strategies for practitioners building safety-critical retrieval systems.

The remainder of this paper is organized as follows. Section 2 provides background on embedding models and tokenization. Section 3 presents our empirical evidence of numerical blindness. Section 4 analyzes the tokenization root cause. Section 5 profiles magnitude sensitivity. Section 6 evaluates cross-encoder mitigation. Section 7 presents domain-specific risk analysis. Section 8 examines benchmark gaps. Section 9 proposes mitigation strategies. Section 10 discusses limitations and concludes.

2. Background

2.1 Dense Retrieval and Embedding Models

Dense retrieval systems encode queries and documents into fixed-dimensional vector representations using neural networks, typically transformers (Devlin et al., 2019). Retrieval is performed by computing similarity — usually cosine similarity — between the query embedding and all document embeddings in an index.

The dominant paradigm for training these models is the bi-encoder architecture, where queries and documents are encoded independently through a shared or paired transformer encoder. The model is trained via contrastive learning: positive pairs (query, relevant document) are pulled together in embedding space while negative pairs are pushed apart.

Sentence-BERT (Reimers and Gurevych, 2019) established the modern framework for sentence-level embeddings, demonstrating that fine-tuning BERT with siamese and triplet networks produces embeddings that capture semantic textual similarity far better than using raw BERT [CLS] tokens or mean pooling without fine-tuning.

Since then, a proliferation of embedding models has emerged, including:

• MiniLM (all-MiniLM-L6-v2): A distilled, lightweight model widely used as a baseline and in resource-constrained settings. Produces 384-dimensional embeddings.
• BGE (BAAI/bge-large-en-v1.5): A high-performance model from the Beijing Academy of Artificial Intelligence, trained with RetroMAE pre-training and contrastive fine-tuning. Produces 1024-dimensional embeddings.
• Nomic-Embed (nomic-embed-text-v1.5): A fully open-source model with strong performance across diverse tasks, supporting variable-length embeddings via Matryoshka training.
• GTE (thenlper/gte-large): General Text Embeddings model trained on a large-scale curated dataset with multi-stage contrastive learning.

These models consistently achieve strong performance on standard benchmarks such as the Massive Text Embedding Benchmark (MTEB) and Semantic Textual Similarity (STS) benchmarks. However, as we demonstrate, these benchmarks systematically omit numerical reasoning tasks.

2.2 Subword Tokenization

All transformer-based embedding models rely on subword tokenization to convert input text into token sequences. The dominant tokenization algorithms — WordPiece (used by BERT and its derivatives), Byte-Pair Encoding (BPE, used by GPT-family models), and SentencePiece (used by many multilingual models) — operate by learning a vocabulary of frequent character sequences from training data.

Critically, these tokenizers treat numbers as character sequences, not as quantities. The tokenizer has no concept of numerical magnitude; it simply breaks strings into learned subword units. This means:

• "500" might be tokenized as ["500"] or ["50", "0"] or ["5", "00"] depending on the vocabulary
• "5" is a single token
• "5000" might be ["500", "0"] or ["5", "000"]
• "500mg" might be ["500", "mg"] or ["50", "0", "mg"]

The relationship between "5" and "500" — that one is 100 times larger — is not encoded in the tokenization. They are simply different character sequences that happen to share a "5" token. From the model's perspective, the transition from "5" to "500" is analogous to the transition from "cat" to "catalog" — they share a prefix, but the semantic relationship is incidental.

2.3 The Representation Problem

After tokenization, each token is mapped to a learned embedding vector, and the transformer processes the sequence through self-attention layers. In theory, the model could learn numerical relationships during pre-training or fine-tuning. In practice, it does not — at least not to a degree sufficient for reliable numerical discrimination.

The fundamental issue is that contrastive training objectives reward the model for capturing semantic similarity at the sentence level. Two sentences that differ only in a number — "The dose is 5mg" vs. "The dose is 500mg" — share identical syntactic structure, identical vocabulary (except for the number), and often similar semantic context. The contrastive training signal to push these apart is weak compared to the overwhelming signal to pull them together based on their shared linguistic content.

This creates a representational blind spot: the embedding space is organized around semantic and topical similarity, with minimal sensitivity to numerical magnitude.

3. The Numerical Blindness Problem: Empirical Evidence

3.1 Experimental Setup

We construct a benchmark of sentence pairs that are textually similar but numerically different, spanning multiple domains and magnitude differences. Each pair consists of two sentences that differ only in their numerical content, with the magnitude difference ranging from 2× to 1000×.

Representative examples include:

We compute embeddings for each sentence using four models and calculate cosine similarity for each pair.

3.2 Results

The results are striking. Across all four models, cosine similarities between numerically different sentences are extremely high:

Average cosine similarity across all numerical pairs:

Per-pair highlights:

• "Take 5mg daily" vs. "Take 500mg daily": Cosine similarity of 0.95 (BGE), despite a 100× dosage difference
• "The building is 10 stories" vs. "The building is 100 stories": Cosine similarity of 0.93 (Nomic), despite the second building being physically impossible
• "Blood glucose: 90 mg/dL" vs. "Blood glucose: 900 mg/dL": Cosine similarity of 0.96 (GTE), despite the second value indicating a life-threatening diabetic emergency

3.3 Comparison with Semantic Similarity

To contextualize these numbers, we compare with cosine similarities between sentences that are genuinely semantically similar:

The numerical pairs are rated as more similar than true paraphrases. This means that in a retrieval system, a query for "5mg dosage" would rank a "500mg dosage" document higher than a genuine paraphrase of the correct dosage information. The embedding model is more confident that 5mg and 500mg are the same thing than it is that "cat" and "feline" mean the same thing.

3.4 Statistical Analysis

We compute the mean and standard deviation of cosine similarities across all model-pair combinations (4 models × 10+ pairs = 40+ measurements):

• Mean cosine similarity: 0.928
• Standard deviation: 0.031
• 95% confidence interval: [0.918, 0.938]

The tight confidence interval indicates this is not a random fluctuation but a systematic failure. All four models, despite different architectures, training data, and embedding dimensions, converge on the same failure mode.

We additionally compute the discrimination ratio: the ratio of inter-class similarity (different numbers) to intra-class similarity (paraphrases of the same number). A perfect model would have a ratio well below 1.0. Across our tested models, the discrimination ratio averages 1.09 — meaning numerically different sentences are rated as more similar than paraphrases of the same sentence. The embedding space has literally inverted the correct ordering.

4. Tokenization Analysis

4.1 How Numbers Are Tokenized

To understand why embeddings fail at numerical distinction, we examine how the tokenizers decompose numerical strings. Using the BERT WordPiece tokenizer (representative of the tokenizers used by MiniLM, BGE, and GTE variants):

Input: "Take 5mg daily"
Tokens: ["Take", "5", "##mg", "daily"]

Input: "Take 500mg daily"  
Tokens: ["Take", "500", "##mg", "daily"]

Input: "Take 5000mg daily"
Tokens: ["Take", "500", "##0", "##mg", "daily"]

Several observations emerge:

1. "5" and "500" are different tokens, but they share no structural relationship that encodes their 100× magnitude difference. The token "500" is no more "100 times 5" than "cat" is "100 times c."

2. "5000" is split into "500" + "0", creating a two-token sequence. The model must learn from the concatenation of these tokens that the resulting number is 10× larger than "500." This compositional numerical reasoning is not a training objective.

3. The surrounding context is identical. The tokens "Take", "##mg", and "daily" are the same in both sentences. This means 3 out of 4 tokens (75%) are identical, and the attention mechanism will produce highly similar contextual representations.

4.2 Token Overlap Analysis

We quantify the token overlap between numerical pairs:

The high Jaccard similarity of token sequences explains the high cosine similarity of embeddings. The transformer processes the shared tokens identically through attention layers, and the single differing token (the number) has limited influence on the final pooled representation.

4.3 Attention Weight Analysis

In a standard transformer encoder, the [CLS] token (or mean-pooled representation) attends to all input tokens. In a 4-token sentence like "Take 5mg daily," even if the attention mechanism assigns equal weight to each token, the numerical token contributes only 25% of the final representation. In practice, function words and domain terms often receive higher attention weights than numbers, further diminishing the numerical signal.

We hypothesize — and our empirical results confirm — that the attention mechanism does not learn to upweight numerical tokens relative to their surrounding context. The training objective (contrastive similarity) does not require numerical precision; it requires topical and semantic alignment. A training pair like ("Take metformin 500mg daily", "Recommended metformin dose is 500mg per day") teaches the model to focus on "metformin," "dose," "daily" — not on "500."

4.4 The Subword Fragmentation Problem

Beyond simple tokenization, numbers with more digits suffer from subword fragmentation that further obscures magnitude:

"100"    → ["100"]              (1 token)
"1000"   → ["100", "0"]         (2 tokens)  
"10000"  → ["100", "00"]        (2 tokens)
"100000" → ["100", "000"]       (2 tokens)
"123456" → ["12", "##34", "##56"] (3 tokens)

Notice that "1000" and "10000" differ by a factor of 10 but both produce 2-token sequences starting with "100." The model must learn that "100" + "0" means one thousand while "100" + "00" means ten thousand — a distinction that requires positional understanding of digit significance. This is compositional numerical reasoning, and standard transformer training does not teach it.

5. Magnitude Sensitivity Test

5.1 Experimental Design

To determine whether embedding models show any sensitivity to numerical magnitude, we construct a controlled experiment. We fix a template sentence and vary only the number, testing whether cosine similarity decreases as the magnitude difference increases.

Template: "The measurement was [X] units"

Test values for X: 10, 20, 50, 100, 500, 1000, 5000, 10000

For each pair of values, we compute cosine similarity across all four models and plot the results against the log-ratio of the two numbers.

5.2 Results

The key finding is that cosine similarity shows negligible correlation with magnitude difference:

The entire range of cosine similarities spans only 0.037 (from 0.961 to 0.924) across a 1000× magnitude difference. In a retrieval system using a typical similarity threshold of 0.7 or 0.8, every single one of these pairs would be returned as "highly similar."

5.3 Comparison with Lexical Variation

For comparison, we test how embedding similarity responds to lexical changes of comparable "significance":

The model is more sensitive to the difference between "big" and "small" (cosine drop of 0.111) than to the difference between 10 and 10,000 (cosine drop of 0.037). It can distinguish antonyms but not numbers differing by three orders of magnitude.

5.4 The Linearity (Non-)Response

If embeddings had any meaningful numerical sensitivity, we would expect cosine similarity to decrease monotonically (and ideally, linearly or log-linearly) with magnitude ratio. Instead, the response is essentially flat:

• Pearson correlation between log(magnitude ratio) and cosine similarity: r = -0.12
• Spearman rank correlation: ρ = -0.15

These correlations are weak and would not achieve statistical significance even with generous thresholds. The embedding space contains no meaningful signal about numerical magnitude.

6. Cross-Encoder Partial Fix

6.1 Cross-Encoder Architecture

Unlike bi-encoders, which encode query and document independently, cross-encoders process the query and document jointly through the transformer. This allows token-level cross-attention: the model can directly compare "5mg" in the query with "500mg" in the document within the same attention computation.

Cross-encoder rerankers (e.g., BGE-reranker, MS-MARCO cross-encoder) are commonly used as a second-stage ranker: the bi-encoder retrieves a candidate set of 50-100 documents, and the cross-encoder re-scores them for final ranking.

6.2 Experimental Setup

We evaluate the BGE-reranker-large model on our numerical pair benchmark. For each pair, we construct a retrieval scenario where the query is one sentence and the candidate set contains: (a) the numerically correct document, (b) the numerically incorrect document (differing by a factor of 10-1000×), and (c) several distractor documents on unrelated topics.

We measure the rate at which the cross-encoder correctly ranks the numerically correct document above the numerically incorrect one.

6.3 Results

The cross-encoder demonstrates substantially better numerical discrimination than bi-encoders:

BGE-reranker numerical correction rate: 73%

This means that in 73% of cases, the cross-encoder correctly identifies which of two numerically different documents is the better match for a numerically specific query. However, the remaining 27% failure rate is critical:

Key observations:

1. Simple, large magnitude differences (100×) are corrected most often (81%), likely because the token sequences are most different.

2. Adjacent magnitudes (2×) have the highest failure rate (38%), as expected — "100" and "200" differ by only one token character.

3. Multi-number sentences are particularly problematic (42% failure rate). When a sentence contains multiple numbers ("Patient: age 65, glucose 90, creatinine 1.2"), the cross-encoder struggles to determine which number is relevant to the query.

4. Context embedding reduces accuracy. When numbers are embedded in longer, more complex sentences, the cross-encoder's attention is diluted across more tokens, and the numerical signal weakens.

6.4 Why Cross-Encoders Help but Don't Solve

Cross-encoders benefit from joint encoding: they can directly attend to the query number and the document number in the same forward pass. This enables some degree of pattern matching — "5" attending to "500" produces a different attention pattern than "5" attending to "5."

However, cross-encoders share the same fundamental limitation as bi-encoders: they are trained on text similarity tasks, not numerical comparison tasks. The training data rarely contains examples where numerical precision is the key discriminator. Moreover, cross-encoders still rely on subword tokenization, so the underlying representation of numbers remains fragmented and magnitude-unaware.

The 73% correction rate represents the degree to which cross-attention can incidentally capture numerical differences through pattern matching. The 27% residual failure rate represents the hard cases where pattern matching is insufficient and genuine numerical reasoning would be required.

6.5 Implications for RAG Pipelines

In a typical RAG pipeline, the bi-encoder retrieves candidates and the cross-encoder reranks them. Our results suggest:

• Bi-encoder stage: Numerically incorrect documents will consistently appear in the candidate set (cosine > 0.9).
• Cross-encoder stage: 73% of these numerical errors will be corrected, but 27% will persist.
• Generation stage: The language model will receive numerically incorrect context 27% of the time for numerical queries.

For safety-critical applications, a 27% error rate on numerical retrieval is unacceptable. A medical RAG system that retrieves the wrong drug dosage 27% of the time would be withdrawn immediately. A financial system that confuses $50K and$500K in more than a quarter of cases would face regulatory action.

7. Domain-Specific Risk Analysis

7.1 Pharmaceutical: Dosing Errors

Drug dosing errors are among the leading causes of preventable medical harm. The Institute of Medicine has estimated that medication errors harm at least 1.5 million people per year in the United States alone. In this context, the inability of embedding models to distinguish dosages is particularly alarming.

Scenario: A clinical decision support system uses RAG to retrieve drug dosing guidelines. A physician queries: "What is the standard dose of lisinopril for hypertension?"

The knowledge base contains:

• Document A: "Lisinopril: Initial dose 10mg once daily, titrate to 40mg/day"
• Document B: "Lisinopril: Initial dose 100mg once daily, titrate to 400mg/day" (corrupted/erroneous)

With a bi-encoder cosine similarity above 0.95, both documents are retrieved with nearly identical relevance scores. Even with cross-encoder reranking, the erroneous document has a 19-27% chance of being ranked higher.

The clinical consequences of a 10× overdose of lisinopril include severe hypotension, renal failure, hyperkalemia, and potentially death. The embedding model's inability to distinguish 10mg from 100mg is not an abstract theoretical concern — it is a direct patient safety risk.

Additional pharmaceutical examples:

In every case, the embedding model would rate the correct and erroneous dosages as near-identical (cosine > 0.93). In every case, the 10× error could be fatal.

7.2 Financial: Order-of-Magnitude Transaction Errors

Financial systems increasingly use retrieval-augmented AI for compliance checking, transaction verification, and risk assessment. Numerical blindness in these systems creates risks at multiple levels.

Scenario: A compliance system uses RAG to check transaction limits. An analyst queries: "What is the reporting threshold for wire transfers?"

The knowledge base contains:

• Document A: "Wire transfers exceeding $10,000 must be reported under BSA/AML regulations"
• Document B: "Wire transfers exceeding $100,000 must be reported under BSA/AML regulations" (outdated/erroneous)

A 10× error in the reporting threshold means thousands of reportable transactions would go unreported, exposing the institution to severe regulatory penalties and potential criminal liability.

Financial magnitude confusion examples:

7.3 Engineering: Tolerance and Specification Errors

Engineering specifications demand numerical precision. A tolerance of 0.5mm vs. 5mm can be the difference between a functioning precision instrument and a dangerous failure.

Scenario: An engineering knowledge base is queried for material specifications. "What is the yield strength of ASTM A36 steel?"

The knowledge base contains:

• Document A: "ASTM A36 structural steel: minimum yield strength 250 MPa (36 ksi)"
• Document B: "ASTM A36 structural steel: minimum yield strength 25 MPa (3.6 ksi)" (erroneous)

A 10× understatement of yield strength could lead to structural designs that are grossly inadequate, risking building collapse.

Engineering specification risks:

7.4 Medical: Laboratory Value Misinterpretation

Medical laboratory values are inherently numerical, and their interpretation depends critically on precise thresholds. A blood glucose of 90 mg/dL is normal; 900 mg/dL is a life-threatening diabetic emergency.

Scenario: A clinical decision support system retrieves reference ranges for lab values. A clinician queries: "What is the normal range for serum sodium?"

The knowledge base contains:

• Document A: "Normal serum sodium: 135-145 mEq/L"
• Document B: "Normal serum sodium: 13.5-14.5 mEq/L" (decimal error)

A 10× error in the reference range could cause a clinician to dismiss dangerously low sodium (hyponatremia at 120 mEq/L) as normal, potentially resulting in seizures, coma, or death.

Critical laboratory value confusion:

8. Why Standard Benchmarks Miss This

8.1 MTEB Benchmark Analysis

The Massive Text Embedding Benchmark (MTEB) is the de facto standard for evaluating embedding models. It encompasses tasks spanning classification, clustering, pair classification, reranking, retrieval, STS, and summarization across dozens of datasets.

We analyzed the MTEB task suite for numerical reasoning content. Of the benchmark datasets included in MTEB as of the current evaluation:

• STS tasks (STS12-16, STS-B, SICK-R): These measure semantic textual similarity on a 0-5 scale. We examined over 5,000 sentence pairs across these datasets and found that fewer than 2% contain any numerical content, and among those, virtually none test numerical magnitude discrimination. The pairs that do contain numbers use them incidentally ("He is 30 years old" / "A young man is walking") rather than as the key semantic discriminator.

• Retrieval tasks (MS-MARCO, NQ, HotpotQA, etc.): These test topical relevance, not numerical precision. A query like "What is the population of France?" expects a document about France's population, but the benchmark does not penalize retrieving a document stating "60 million" vs. "600 million."

• Classification tasks: These involve categorical labels unrelated to numerical reasoning.

• Clustering tasks: These group documents by topic, not by numerical content.

In summary, no task in MTEB directly evaluates numerical magnitude sensitivity. A model that treats "5mg" and "500mg" as identical would suffer zero penalty on MTEB. This explains why state-of-the-art models on MTEB leaderboards exhibit the numerical blindness we document.

8.2 The Benchmark Blind Spot

This creates a pernicious cycle:

1. Benchmarks don't test numerical reasoning
2. Models aren't trained to distinguish numbers
3. Models score well on benchmarks despite numerical blindness
4. Practitioners trust benchmark scores and deploy models in numerical domains
5. Numerical failures occur in production, often silently

The silence of these failures is particularly concerning. When a retrieval system returns the wrong drug dosage, there is no error message, no exception, no flag. The system returns a result with high confidence. The failure is invisible until a downstream human or system acts on the incorrect information.

8.3 Related Benchmark Gaps

The absence of numerical reasoning in embedding benchmarks mirrors a broader gap in NLP evaluation. While there has been significant work on numerical reasoning in language models — including arithmetic, number comparison, and quantitative reasoning benchmarks — this work has focused almost exclusively on generative models, not embedding models.

The embedding evaluation community has not yet caught up to the recognition that numerical sensitivity is a critical capability for retrieval systems deployed in real-world domains. We argue this is a significant and urgent gap.

9. Mitigation Strategies

Given the severity of numerical blindness in embeddings, practitioners building safety-critical retrieval systems need concrete mitigation strategies. We propose and analyze several approaches.

9.1 Hybrid Retrieval with Numerical Extraction

The most robust mitigation combines dense retrieval with explicit numerical extraction and comparison.

Architecture:

1. Dense retrieval retrieves a candidate set based on semantic similarity (as usual).
2. Numerical extraction identifies all numbers and their units in both the query and each candidate document using rule-based or NER-based extractors.
3. Numerical comparison computes a separate numerical similarity score based on extracted quantities.
4. Score fusion combines the semantic similarity score with the numerical similarity score.

Implementation:

import re
from typing import List, Tuple, Optional

def extract_numbers_with_units(text: str) -> List[Tuple[float, Optional[str]]]:
    """Extract (number, unit) pairs from text."""
    pattern = r'(\d+(?:\.\d+)?)\s*(mg|kg|mL|g|mm|cm|m|%|units?|mg/dL|mEq/L|bpm|PSI|MPa|ksi|V|A|W|Hz|°[CF]|USD|\$|£|€)?'
    matches = re.findall(pattern, text, re.IGNORECASE)
    return [(float(m[0]), m[1] if m[1] else None) for m in matches]

def numerical_similarity(nums_a: List[Tuple], nums_b: List[Tuple]) -> float:
    """Compare extracted numbers, accounting for magnitude."""
    if not nums_a or not nums_b:
        return 1.0  # No numbers to compare; defer to semantic sim
    
    similarities = []
    for (val_a, unit_a) in nums_a:
        for (val_b, unit_b) in nums_b:
            if unit_a and unit_b and unit_a.lower() != unit_b.lower():
                continue  # Different units, skip
            if val_a == 0 or val_b == 0:
                similarities.append(1.0 if val_a == val_b else 0.0)
            else:
                ratio = max(val_a, val_b) / min(val_a, val_b)
                sim = 1.0 / (1.0 + math.log10(ratio))  # Log-scaled penalty
                similarities.append(sim)
    
    return max(similarities) if similarities else 1.0

def hybrid_score(semantic_sim: float, query: str, document: str, 
                 alpha: float = 0.3) -> float:
    """Combine semantic and numerical similarity."""
    nums_q = extract_numbers_with_units(query)
    nums_d = extract_numbers_with_units(document)
    num_sim = numerical_similarity(nums_q, nums_d)
    return (1 - alpha) * semantic_sim + alpha * num_sim

This approach has the advantage of being model-agnostic and requiring no retraining. The numerical extraction can be as simple as regex or as sophisticated as a dedicated NER model for quantities.

Limitations: Rule-based extraction may miss complex numerical expressions ("three hundred," "half a million"). Unit normalization is non-trivial ("mg/dL" vs "milligrams per deciliter"). The fusion weight (alpha) must be tuned per domain.

9.2 Structured Metadata Filtering

For domains with well-defined numerical fields (drug dosing, financial transactions, engineering specs), the most reliable approach is to extract numerical metadata into structured fields and filter on these fields before or after dense retrieval.

Architecture:

1. During indexing, extract numerical fields from each document: {drug: "metformin", dose: 500, unit: "mg", frequency: "twice daily"}.
2. During retrieval, parse numerical constraints from the query: {drug: "metformin", dose_range: [400, 600]}.
3. Filter candidates by numerical constraints, then rank by semantic similarity.

This approach converts the retrieval problem from a pure embedding similarity problem into a hybrid structured + unstructured search problem. It is more engineering effort but provides hard guarantees on numerical matching.

9.3 Magnitude-Aware Fine-Tuning

A more ambitious approach is to fine-tune embedding models with training data that specifically targets numerical discrimination.

Training data construction:

• Positive pairs: Sentences with the same number (possibly paraphrased): "Take 500mg daily" / "The dose is 500mg per day"
• Hard negatives: Sentences with different numbers but identical context: "Take 500mg daily" / "Take 50mg daily"

By explicitly including hard negatives that differ only in numerical content, the contrastive training objective would push the model to separate embeddings based on numerical magnitude.

Challenges:

• Requires substantial curated training data with numerical hard negatives.
• Risk of catastrophic forgetting: fine-tuning for numerical sensitivity may degrade general semantic similarity.
• The tokenization problem remains: the model must learn numerical reasoning despite magnitude-unaware tokenization.
• Generalization is uncertain: training on "5 vs 500" may not generalize to "7.3 vs 730."

9.4 Number-Aware Tokenization

A more fundamental approach addresses the root cause: tokenization. Instead of treating numbers as character sequences, a number-aware tokenizer could:

1. Normalize numbers to a canonical form (e.g., scientific notation: 500 → 5e2, 5000 → 5e3)
2. Embed magnitude explicitly by prepending magnitude tokens: 500 → [MAG:2] 5 (meaning 5 × 10²)
3. Use digit-level tokenization with positional encoding that encodes place value

These approaches require architectural changes and retraining from scratch, but they address the fundamental representational limitation.

9.5 Ensemble Verification

For the highest-stakes applications, we recommend an ensemble approach:

1. Dense retrieval provides candidates (semantic matching)
2. Cross-encoder reranking (73% numerical correction)
3. Explicit numerical extraction and comparison (catches most remaining errors)
4. Rule-based validation against known constraints (e.g., drug dose ranges)
5. Human-in-the-loop for safety-critical decisions

No single mitigation is sufficient. The combination of multiple complementary approaches can reduce the numerical error rate to near zero, at the cost of increased system complexity and latency.

9.6 Prompt-Based Mitigation for RAG

In RAG applications, the generated response can be used as an additional verification layer:

System prompt: "When providing numerical information (dosages, amounts, 
measurements), explicitly verify that the numbers in the retrieved context 
are consistent and plausible. Flag any values that differ by more than 
2× from other sources or from known safe ranges."

Modern language models have significantly better numerical reasoning than embedding models (though still imperfect). By prompting the generation model to verify numerical consistency, some retrieval errors can be caught at the generation stage.

10. Limitations and Conclusion

10.1 Limitations

Our study has several limitations that should be acknowledged:

Benchmark scope. Our numerical pair benchmark, while carefully constructed, covers a limited set of sentence templates and magnitude differences. A comprehensive evaluation would require thousands of pairs spanning diverse domains, languages, and numerical formats (integers, decimals, scientific notation, written-out numbers, percentages, dates, etc.).

Model coverage. We evaluate four embedding models, all based on the BERT/transformer architecture with subword tokenization. Models using different architectures (e.g., character-level models, retrieval-specific architectures) or different tokenization strategies may behave differently. Instruction-tuned embedding models (e.g., E5-Mistral, GritLM) that can process explicit numerical comparison instructions may show improved numerical sensitivity.

Cross-encoder evaluation. Our cross-encoder evaluation uses a single model (BGE-reranker-large). Other cross-encoders, particularly those trained on numerical comparison data, may achieve different correction rates.

Mitigation evaluation. We propose several mitigation strategies but do not provide end-to-end evaluation of their effectiveness in production systems. The hybrid retrieval approach, in particular, requires domain-specific tuning that we do not fully characterize.

Language scope. All experiments are conducted in English. Numerical representation varies across languages and writing systems, and the severity of numerical blindness may differ for models trained on other languages.

10.2 Future Work

Several directions for future work emerge from our findings:

1. Numerical embedding benchmarks. The community needs standardized benchmarks that specifically test numerical sensitivity in embeddings. We propose the development of a "Numerical MTEB" that evaluates models on magnitude discrimination, unit awareness, numerical range matching, and quantitative reasoning in retrieval.

2. Number-aware architectures. Research into embedding architectures that explicitly represent numerical magnitude — perhaps through specialized numerical encoders or magnitude-aware attention mechanisms — could address this limitation at its root.

3. Training data augmentation. Systematic generation of numerical hard negatives for contrastive training could improve numerical sensitivity without architectural changes.

4. Cross-domain transfer. Investigating whether numerical sensitivity learned in one domain (e.g., financial amounts) transfers to another (e.g., drug dosages) would inform practical fine-tuning strategies.

5. Multilingual numerical reasoning. Extending this analysis to non-English languages and non-Arabic numeral systems would reveal whether the problem is universal or language-specific.

10.3 Conclusion

We have presented a systematic investigation of a critical and underappreciated failure mode in dense embedding models: numerical blindness. Our empirical results demonstrate that state-of-the-art embedding models — including MiniLM, BGE, Nomic-Embed, and GTE — consistently fail to distinguish quantities differing by orders of magnitude, with cosine similarities exceeding 0.88 even for 100× to 1000× differences.

The root cause is fundamental: subword tokenizers decompose numbers into magnitude-unaware fragments, and contrastive training objectives do not incentivize numerical discrimination. Cross-encoder rerankers provide partial mitigation (73% correction) but leave a dangerous 27% residual failure rate.

The implications are severe for safety-critical domains. In pharmaceutical retrieval, numerical blindness can cause 10× dosing errors that risk patient death. In financial systems, order-of-magnitude confusion can cause massive compliance failures. In engineering, specification errors can lead to structural failure. In medical laboratory interpretation, misreading reference ranges can cause missed diagnoses.

Standard benchmarks like MTEB and STS do not test for numerical reasoning, allowing this failure mode to persist undetected in state-of-the-art models. We call on the embedding evaluation community to address this gap urgently.

For practitioners deploying retrieval systems today, we recommend hybrid approaches that combine dense retrieval with explicit numerical extraction and comparison. No single embedding model, regardless of its MTEB ranking, can be trusted to handle numerical quantities reliably. Numbers are just tokens — and until we build systems that understand them as quantities, safety-critical retrieval remains fundamentally compromised.

Engagement with Prior Work on Numerical Representations

Prior work has investigated how neural language models handle numbers. Wallace et al. (2019) demonstrated that BERT-family models fail at numerical reasoning tasks, showing that pre-trained language models cannot reliably compare, add, or order numbers presented as text. Spithourakis and Riedel (2018) proposed methods for improving numerical representations in neural language models, including digit-level tokenization and specialized number encoders. Our work extends these findings to the specific context of retrieval systems: we show that the numerical blindness documented in language modeling manifests as dangerously high cosine similarity between sentences differing only in quantity, with direct implications for safety-critical retrieval in pharmaceutical and financial domains.

References

Wallace, E., Wang, Y., Li, S., Singh, S., and Gardner, M. (2019). Do NLP Models Know Numbers? Probing Numeracy in Embeddings. In Proceedings of EMNLP 2019.

Spithourakis, G. and Riedel, S. (2018). Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers. In Proceedings of ACL 2018.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019, pp. 4171–4186.

Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of EMNLP-IJCNLP 2019, pp. 3982–3992.