Out-of-Vocabulary Robustness in Sentence Embeddings: How Embedding Models Differ on Unknown Entities

Abstract

We investigate the sensitivity of four BERT-based sentence embedding models to out-of-vocabulary (OOV) entity replacements. Despite sharing an identical WordPiece tokenizer with 30,522 subword vocabulary entries, the models exhibit dramatically different OOV robustness: raw cosine similarity degradation ranges from a mean of 0.035 (GTE-Large) to 0.123 (MiniLM-L6), a 3.5x gap. Through controlled experiments replacing 20 known entities with fabricated OOV words and analyzing 100 baseline sentence pairs across 8 semantic categories, we demonstrate that OOV robustness is a learned property of the embedding space, not a consequence of tokenizer architecture. Crucially, we introduce a normalized OOV sensitivity metric that accounts for each model's dynamic similarity range, revealing that the raw-delta ranking can be misleading: when normalized against usable similarity range, Nomic-Embed consumes the largest fraction of its dynamic range per OOV replacement (25.7%), while MiniLM (16.5%), BGE (16.7%), and GTE (15.1%) cluster more closely together. We observe a descriptive pattern across the four models tested wherein larger models tend to show lower raw OOV sensitivity, though we emphasize this is an observation from a small model sample rather than a proven relationship. Entity swap pairs (identical token sets, reordered) achieve near-perfect similarity (>0.987) across all models, confirming mean-pooling bag-of-words behavior, yet OOV replacement impacts diverge sharply. With Cohen's d effect sizes exceeding 1.9 between the most and least robust models on raw deltas, the practical significance is clear. These findings have direct implications for retrieval-augmented generation (RAG) systems deployed in specialized domains where novel entities are common.

1. Introduction

Sentence embedding models are foundational components of modern natural language processing (NLP) systems. They underpin semantic search, retrieval-augmented generation (RAG), question answering, and document clustering. In many high-value application domains—biomedical research, legal document analysis, scientific literature retrieval, and cybersecurity threat intelligence—the text frequently contains specialized terminology, proper nouns, and neologisms that were absent from the model's training data.

When such out-of-vocabulary (OOV) terms appear, subword tokenizers decompose them into sequences of known subword units. The quality of the resulting sentence embedding then depends on how well the model can compose meaningful representations from these fragmentary pieces. This compositional capability varies across models, but the factors driving this variation remain underexplored.

In this work, we conduct controlled experiments to isolate the effect of OOV entities on sentence embeddings. Our experimental design exploits a critical observation: several popular embedding models share the exact same WordPiece tokenizer with identical vocabulary size (30,522 entries), yet they produce dramatically different embeddings when encountering novel words. By holding tokenization constant, we can attribute any differences in OOV robustness to the embedding model's learned representations rather than to tokenizer design choices.

Our contributions are as follows:

1. We demonstrate a 3.5x gap in raw OOV sensitivity across four models that share an identical tokenizer, proving that OOV robustness is learned during training rather than determined by subword segmentation.
2. We introduce a normalized OOV sensitivity metric that expresses each model's OOV degradation as a fraction of its usable similarity dynamic range, revealing that the raw delta ranking can be misleading—Nomic-Embed, despite having a moderate raw delta, consumes the largest fraction of its dynamic range per OOV replacement.
3. We observe a descriptive pattern wherein larger models tend to show lower raw OOV sensitivity, while acknowledging that four data points cannot establish a robust statistical trend.
4. We provide per-entity analysis revealing that culturally prominent entities (e.g., "Einstein," "Shakespeare") suffer the largest degradation when replaced, while generic terms show more stability.
5. We offer practical recommendations for practitioners selecting embedding models for domains with high OOV rates.

2. Background and Related Work

2.1 Subword Tokenization

Modern NLP models rely on subword tokenization algorithms to handle open-vocabulary text. WordPiece (Devlin et al., 2019) and Byte Pair Encoding (BPE; Sennrich et al., 2016) are the two dominant approaches. Both construct a fixed-size vocabulary of subword units from a training corpus and decompose unknown words into sequences of known subword tokens at inference time.

WordPiece, used in BERT and its derivatives, employs a greedy longest-match-first algorithm. For example, the fabricated word "Xylophrix" is decomposed into ["x", "##yl", "##op", "##hri", "##x"]—five subword tokens that individually carry minimal semantic content. The critical question is whether the model can compose these fragments into a useful representation.

SentencePiece (Kudo and Richardson, 2018) provides an alternative framework that treats tokenization as an unsupervised text segmentation problem. Some models in our study use SentencePiece-based tokenizers, though with identical vocabulary sizes and—crucially—identical tokenization outputs on our test data.

2.2 Sentence Embeddings

Sentence-BERT (SBERT; Reimers and Gurevych, 2019) established the paradigm of fine-tuning BERT-like models with siamese and triplet network architectures to produce sentence embeddings suitable for cosine similarity comparison. The standard approach uses mean pooling over token embeddings to produce fixed-size sentence vectors.

Mean pooling has an important consequence for our study: it creates bag-of-words-like behavior where token order has minimal impact on the final embedding. We confirm this empirically in our entity swap experiments (Section 4.1).

2.3 OOV Handling in Neural Models

Prior work has examined OOV handling primarily at the word embedding level. Character-level models and subword models were developed partly to address the OOV problem. However, the interaction between subword tokenization and downstream sentence-level representations has received less attention. Our work specifically examines how different training procedures affect the model's ability to compose subword fragments into meaningful entity representations.

3. Experimental Setup

3.1 Models

We evaluate four sentence embedding models that span a range of architectures and sizes while sharing compatible tokenization:

Table 1: Model characteristics

A critical design feature of our experiment is that all four models produce identical tokenization on our test data. Despite the Nomic model using a SentencePiece tokenizer rather than WordPiece, the vocabulary size is identical (30,522), and empirical verification confirms that every word in our test set is tokenized identically across all four models. This allows us to attribute any differences in embedding behavior solely to the learned representations.

All models use mean pooling over the final hidden layer to produce sentence-level embeddings.

3.2 OOV Sensitivity Test

We constructed 20 sentence pairs to measure OOV sensitivity. Each pair consists of:

• Original sentence: Contains a well-known entity (e.g., "Google," "Einstein," "penicillin")
• Modified sentence: The entity is replaced with a fabricated OOV word (e.g., "Xylophrix," "Wompelfritz," "flobnaxitol")

The fabricated words were designed to be phonotactically plausible but completely absent from any training corpus. We note that some fabricated words contain recognizable morphemes as subword fragments (e.g., "Quarbitone" contains "##bit" and "##one"; "Frondlebard" contains "##bard"). Truly random character sequences (e.g., "xyzqwrt") would provide a purer control, but would also be unrealistic—real OOV terms encountered in domain-specific text, such as medical compounds (e.g., "pembrolizumab"), chemical nomenclature (e.g., "dimethylformamide"), and proper nouns from non-English languages, routinely contain recognizable morphemes. Our fabricated words thus approximate the structure of realistic OOV encounters.

The key measurement is the cosine similarity delta: the drop in cosine similarity between the original pair and the modified pair, where both pairs share the same semantic frame but differ in the entity name.

Table 2: OOV replacement pairs with tokenization details

The replacement words consistently produce more subword tokens than the originals (mean 4.2 vs. a mix of 1-4), as expected for truly novel words not in the tokenizer's vocabulary.

3.3 Baseline Semantic Pairs

We also constructed 100 sentence pairs across 8 semantic categories to establish baseline behavior:

• Entity swap (10 pairs): Same words reordered (e.g., "Alice thanked Bob" vs. "Bob thanked Alice")
• Temporal (10 pairs): Time-shifted variants (e.g., "yesterday" vs. "tomorrow")
• Negation (15 pairs): Affirmative vs. negated sentences
• Numerical (15 pairs): Different numerical values in otherwise identical frames
• Quantifier (10 pairs): Different quantity words (e.g., "few" vs. "many")
• Hedging (5 pairs): Certain vs. uncertain phrasing
• Positive (20 pairs): Semantically similar sentence pairs
• Negative (15 pairs): Semantically unrelated sentence pairs

These categories allow us to assess whether OOV sensitivity patterns correlate with other aspects of semantic discrimination. Critically, the positive and negative control pairs serve a dual purpose: they establish each model's dynamic similarity range—the span between meaningful semantic similarity and effective dissimilarity—which we use to normalize OOV deltas (Section 4.3).

3.4 Metrics

For each sentence pair, we compute:

• Cosine similarity between the two sentence embeddings
• Token Jaccard overlap: the Jaccard similarity coefficient between the two sentences' token sets
• OOV delta: for OOV pairs, the absolute difference in cosine similarity between the original-entity pair and the replacement-entity pair

We also compute:

• Pearson and Spearman correlations between token Jaccard overlap and cosine similarity across all 100 baseline pairs
• Cohen's d effect size for pairwise OOV sensitivity comparisons between models
• Normalized OOV delta: the raw OOV delta divided by each model's dynamic range (mean positive cosine − mean negative cosine), expressing the OOV impact as a fraction of the model's usable similarity spectrum

4. Results

4.1 Entity Swap Confirms Bag-of-Words Behavior

Entity swap pairs—where sentences contain identical tokens in different order—achieve near-perfect cosine similarity across all models:

Table 3: Entity swap cosine similarity

All models score above 0.987 on entity swaps, confirming that mean pooling produces effectively order-invariant representations. This is important context for interpreting OOV results: since token order barely matters, the OOV impact must arise from the representations of individual subword tokens and their composition, not from positional encoding effects.

4.2 Raw OOV Sensitivity Varies 3.5x Across Models

The central finding of this study is the dramatic variation in OOV sensitivity despite identical tokenization:

Table 4: OOV sensitivity summary statistics (raw cosine deltas)

The most sensitive model (MiniLM, mean delta = 0.123) is 3.48x more affected by OOV replacements than the most robust model (GTE, mean delta = 0.035) in raw terms. Even the ratio between BGE and GTE—both with 1024 dimensions and 24 layers—is 1.56x, indicating that architectural similarity alone does not determine OOV robustness.

The variance also differs dramatically: MiniLM's OOV deltas range from 0.036 to 0.245 (range = 0.209), while GTE's range is only 0.014 to 0.064 (range = 0.050). This suggests that larger models not only have lower mean sensitivity but also more consistent behavior across different OOV entities.

4.3 Normalized OOV Sensitivity: Accounting for Dynamic Range

A critical methodological consideration raised during peer review is that raw cosine deltas can be misleading when models operate in fundamentally different similarity ranges. MiniLM assigns near-zero cosine similarity (mean 0.015) to unrelated sentence pairs, giving it a wide dynamic range of ~0.97 between "completely unrelated" and "near-identical." GTE, by contrast, assigns mean 0.711 to unrelated pairs, compressing its usable range to ~0.28. A raw delta of 0.035 in GTE's compressed space may be proportionally more damaging than it appears.

To account for this, we compute normalized OOV deltas:

• Dynamic range = mean positive cosine − mean negative cosine
• Normalized delta = raw delta / dynamic range

This expresses the OOV impact as a fraction of each model's usable similarity spectrum—answering the question: what fraction of the meaningful similarity range is consumed by an OOV replacement?

Table 5: Normalized OOV sensitivity

The normalized analysis substantially revises the raw-delta picture. The key findings:

1. Nomic-Embed is the most OOV-sensitive model when normalized. Despite having the second-highest raw delta, Nomic consumes 25.7% of its usable dynamic range per OOV replacement on average—far more than any other model. Its worst-case normalized delta (47.2%, for Shakespeare → Frondlebard) means nearly half the meaningful similarity range is consumed by a single entity replacement.

2. MiniLM, BGE, and GTE cluster together. When normalized, these three models show remarkably similar proportional OOV impact: 16.5%, 16.7%, and 15.1% respectively. MiniLM's apparently dramatic raw sensitivity is substantially explained by its wider dynamic range.

3. GTE remains the most robust even after normalization, but the advantage over MiniLM and BGE is modest (15.1% vs. 16.5-16.7%) rather than dramatic.

4. The raw ranking changes. In raw deltas: MiniLM > Nomic > BGE > GTE (most to least sensitive). After normalization: Nomic > BGE ≈ MiniLM > GTE. The practical implication is that Nomic-Embed users in OOV-heavy domains should be particularly cautious, as OOV replacements erode a disproportionate share of its ability to discriminate between related and unrelated content.

Table 6: Per-entity normalized deltas (selected entities, as % of dynamic range)

The per-entity normalized deltas show no single model dominates across all entities. Nomic tends to show the largest normalized deltas overall, but MiniLM leads on culturally prominent entities (Fibonacci, Einstein), and BGE shows surprising vulnerability on some entities (Google, DNA) once normalized. This heterogeneity underscores that OOV sensitivity is entity-dependent and model-dependent in complex ways.

4.4 Same Tokenizer, Different Robustness

The most striking aspect of these results is that all four models tokenize every word in our test set identically. The fabricated word "Xylophrix" is decomposed into ["x", "##yl", "##op", "##hri", "##x"] by every model. The word "Wompelfritz" becomes ["wo", "##mp", "##el", "##fr", "##itz"] for all four. Yet the impact on the final sentence embedding varies substantially.

Consider the Einstein → Wompelfritz replacement:

• MiniLM raw delta: 0.219 (normalized: 29.2% of dynamic range)
• Nomic raw delta: 0.073 (normalized: 18.0% of dynamic range)
• BGE raw delta: 0.055 (normalized: 16.6% of dynamic range)
• GTE raw delta: 0.046 (normalized: 19.7% of dynamic range)

The same five subword tokens ["wo", "##mp", "##el", "##fr", "##itz"] cause markedly different perturbations across models, both in raw and normalized terms. This conclusively demonstrates that OOV robustness is a property of the learned embedding space, not the tokenizer.

4.5 Effect Sizes Between Models

Cohen's d effect sizes quantify the practical significance of the inter-model differences in raw OOV deltas:

Table 7: Cohen's d for pairwise OOV sensitivity comparisons (raw deltas)

Using conventional thresholds (small = 0.2, medium = 0.5, large = 0.8), most pairwise comparisons show very large effect sizes. The MiniLM vs. Nomic comparison is the smallest (d = 0.384), which is expected given their relatively similar raw deltas. The Nomic vs. GTE comparison yields the largest effect size (d = 2.586), indicating an enormous practical difference in raw OOV impact.

4.6 Descriptive Pattern: Model Size and OOV Sensitivity

We observe a descriptive pattern across our four models:

Table 8: Model characteristics and OOV sensitivity

In raw deltas, there is a broad trend where larger models show lower sensitivity: the two 335M-parameter models (BGE, GTE) show notably lower raw deltas than the two smaller models (MiniLM at 22M, Nomic at 137M). However, we deliberately refrain from computing correlation coefficients or fitting trend lines to four data points, as such statistics would carry misleading precision. The pattern is suggestive but cannot establish a robust quantitative relationship.

Notably, BGE and GTE have nearly identical architectures (1024d, 24 layers, ~335M parameters) but differ in raw OOV sensitivity by 1.56x, indicating that factors beyond model size—including training data composition, contrastive learning objectives, and hard negative mining strategies—play substantial roles. The normalized analysis further complicates any simple size-based narrative: Nomic (137M) is more OOV-sensitive relative to its own discrimination capacity than MiniLM (22M).

4.7 Per-Entity Analysis

Individual entity replacements reveal consistent patterns across models:

Table 9: Raw OOV delta by entity (all models), sorted by MiniLM delta

Several patterns emerge from this per-entity analysis:

High-impact entities tend to be culturally iconic. Fibonacci, Einstein, Hamlet, and Shakespeare—all deeply embedded in Western cultural and educational corpora—show the largest deltas, particularly for MiniLM. This suggests that smaller models develop highly specialized, token-specific representations for well-known entities, which are then severely disrupted when those tokens are replaced.

Medical/scientific terms show moderate impact. DNA, insulin, penicillin, and cancer occupy a middle ground. These terms are frequent in training data but are used in more formulaic contexts, potentially allowing the surrounding sentence structure to partially compensate.

Technology/brand names show lower impact. Google, Windows, Python, and MIT show relatively smaller deltas. One possible explanation is that these terms appear in diverse contexts during training, leading to less specialized representations that are easier to approximate from subword fragments.

Token count change does not predict delta. The Fibonacci → Zragnacci replacement maintains the same token count (4→4) yet causes the largest delta for MiniLM (0.245). Meanwhile, Google → Xylophrix increases from 1 to 5 tokens but causes only a modest 0.052 delta. This further confirms that the compositional quality of subword representations, not the tokenization granularity, drives OOV sensitivity.

4.8 Single-Token vs. Multi-Token Originals

We examined whether entities that are represented as a single token (e.g., "google," "einstein") behave differently from multi-token originals (e.g., "Bitcoin" → ["bit", "##co", "##in"]):

Table 10: OOV delta by original token count

The differences between single-token and multi-token originals are minimal and inconsistent in direction. For MiniLM, multi-token originals show slightly higher deltas, while for GTE, single-token originals show slightly higher deltas. With only 3 multi-token cases, we cannot draw strong conclusions, but the data suggest that the original word's tokenization granularity is not a major factor.

4.9 Token Overlap and Semantic Similarity

Across the 100 baseline pairs, we measured the correlation between token Jaccard overlap and cosine similarity:

Table 11: Token-cosine correlation across models

Interestingly, the models with higher raw OOV sensitivity (MiniLM, Nomic) also show stronger correlation between token overlap and cosine similarity. This is consistent with a picture where models that rely more heavily on surface-level token identity are more susceptible to token-level perturbations. However, we note this is an observation across only four models and should not be over-interpreted.

4.10 Category-Level Analysis

The baseline category analysis reveals consistent performance ordering across semantic tasks:

Table 12: Mean cosine similarity by category

Several observations are relevant to our OOV analysis:

Entity swap uniformity. All models achieve >0.987 on entity swaps, confirming that mean pooling makes token order nearly irrelevant. The small residual differences (0.987 to 0.993) may reflect minor positional encoding contributions.

Negation insensitivity. All models score >0.889 on negation pairs, indicating poor negation handling. This is a known limitation of mean-pooled sentence embeddings and is not the focus of this study but provides useful context.

Negative pair discrimination varies enormously. MiniLM assigns near-zero similarity (0.015) to unrelated sentence pairs, while GTE assigns 0.711. This dramatically different baseline behavior reveals fundamental differences in how these models organize their embedding spaces—and is precisely why normalized OOV metrics (Section 4.3) are essential for fair comparison.

5. Analysis and Discussion

5.1 Why OOV Robustness Varies Across Models

We propose three complementary hypotheses for the observed variation in OOV robustness:

Hypothesis 1: Representational capacity. A 1024-dimensional embedding space has 2.67x more capacity than a 384-dimensional one. When a known entity like "Einstein" is replaced with subword fragments ["wo", "##mp", "##el", "##fr", "##itz"], the model must compose these into a single entity representation via mean pooling. In a higher-dimensional space, the composition of fragmentary subword embeddings has more room to land in a region that preserves the sentence's overall semantic content. In a lower-dimensional space, the perturbation from switching tokens is proportionally larger.

Hypothesis 2: Training exposure breadth. Larger models are typically trained on more data and for more steps. This broader exposure may produce subword representations that are more compositionally regular—meaning that even arbitrary combinations of subwords yield embeddings in sensible regions of the space, rather than pathological outlier regions.

Hypothesis 3: Depth of contextualization. Models with more transformer layers (24 for BGE/GTE vs. 6 for MiniLM) have greater capacity to contextualize subword tokens relative to the surrounding sentence. When an OOV word like "Wompelfritz" appears in a sentence about physics, a deeper model can leverage the context to partially infer the entity's role and properties, even without recognizing the specific name. A shallower model has less opportunity for such contextualization.

The relative contributions of these hypotheses cannot be fully disentangled in our experiment. The observation that BGE and GTE—both with 1024 dimensions and 24 layers—still differ by 1.56x in raw deltas underscores that training procedure specifics play a significant role alongside architectural factors.

5.2 Confounding Variables

An important caveat for interpreting our results is that the four models differ in many dimensions simultaneously, not just parameter count. Key confounding variables include:

Training data volume and diversity. MiniLM was distilled from a larger teacher model using a specific corpus. BGE was trained on large-scale text pairs curated by BAAI. GTE was trained on a diverse mixture of text pairs. Nomic used a distinct training pipeline with its own data composition. The volume, domain coverage, and quality of training data all plausibly affect OOV robustness, and we cannot isolate these effects from model size.

Contrastive learning objectives. The models employ different training objectives: some use standard contrastive loss, others use hard negative mining, and the specifics of negative sampling strategies can dramatically affect how models handle edge cases like OOV tokens. A model trained with more diverse negative examples may develop more robust subword composition as a byproduct of learning finer-grained distinctions.

Knowledge distillation. MiniLM is a distilled model, which means its representations are shaped by a teacher model's behavior. The distillation process may systematically affect how the student handles OOV tokens compared to models trained from scratch.

Architecture details beyond size. Attention head counts, intermediate layer sizes, and other architectural choices differ across models and may influence OOV handling independently of raw parameter count.

We therefore emphasize that our findings should be interpreted as practical observations about specific, widely-used models rather than as evidence for a causal mechanism linking model size to OOV robustness. The practical recommendation—test your chosen model's OOV sensitivity in your target domain—holds regardless of the underlying causal story.

5.3 The Dynamic Range Problem and Why Normalization Matters

Our normalized analysis (Section 4.3) reveals a critical insight that has broader methodological implications for embedding model evaluation. Different models operate in fundamentally different cosine similarity regimes:

• MiniLM uses nearly the full [-0.07, 0.99] range, with unrelated sentences near 0.0
• GTE compresses everything into roughly [0.67, 0.99], with even unrelated sentences above 0.7

When comparing raw cosine deltas across models with such different operating regimes, we are not comparing like with like. A 0.035 drop in GTE's compressed space removes 15.1% of its useful range, while a 0.123 drop in MiniLM's wide space removes 16.5%—a proportionally similar impact despite the 3.5x difference in raw deltas.

This does not invalidate the raw delta analysis—in absolute terms, a practitioner using fixed thresholds cares about raw deltas. But for understanding each model's relative vulnerability to OOV perturbation, normalization is essential.

The most striking outcome of normalization is the re-ranking of Nomic-Embed. Its moderate raw delta (0.104) is revealed to be the most proportionally damaging (25.7% of dynamic range) because Nomic operates in a moderately compressed similarity range (dynamic range = 0.405) but suffers nearly as much raw degradation as MiniLM (which has a 0.750 dynamic range to absorb the impact). This makes Nomic the model most at risk for OOV-induced rank inversions—cases where an OOV replacement causes a relevant document to be scored below an irrelevant one.

5.4 Entity Prominence and OOV Impact

The per-entity analysis reveals a pattern where culturally prominent entities—those likely to have highly specialized, semantically rich single-token embeddings—show the largest OOV deltas. We hypothesize a mechanism:

1. During training, the model encounters "Einstein" in many physics-related contexts and develops a specialized embedding that encodes rich semantic associations (physics, relativity, genius, etc.).
2. When "Einstein" is replaced with "Wompelfritz," these specialized associations are lost entirely. The subword fragments ["wo", "##mp", "##el", "##fr", "##itz"] carry no physics-related information.
3. The resulting embedding is pulled toward a generic, uninformative region of the space.
4. Models with deeper contextualization can partially compensate by leveraging contextual cues from the surrounding sentence, but shallower models cannot.

This mechanism is consistent with the observation that technology terms (Google, Windows, Python) show smaller deltas: these terms appear in more varied contexts during training and may develop less specialized representations.

5.5 Implications for RAG Systems

Our findings have direct practical implications for retrieval-augmented generation systems:

Model selection in specialized domains. In domains with high OOV rates—biomedical NER, patent search, cybersecurity threat intelligence—practitioners should evaluate their chosen model's OOV sensitivity empirically. The normalized analysis suggests that Nomic-Embed users are at greatest proportional risk, while GTE-Large offers the best combination of low raw and normalized sensitivity.

Threshold calibration. Systems that use cosine similarity thresholds for retrieval decisions must account for OOV sensitivity. The appropriate threshold adjustment depends on both the model's raw OOV sensitivity and its dynamic range. A model operating in a compressed range (like GTE) may tolerate less absolute threshold margin than one with a wide range (like MiniLM).

Entity normalization. Pre-processing steps that normalize OOV entities (e.g., replacing specialized terms with generic category markers like "[DRUG]" or "[PERSON]") may be especially important for Nomic-Embed, given its high proportional sensitivity to OOV replacements.

Cost-benefit analysis. The computational cost of larger models must be weighed against the accuracy cost of OOV sensitivity. For a medical document retrieval system where entities like novel drug names are frequent, the additional latency of a 335M-parameter model may be justified by improved OOV robustness—but the normalized analysis suggests the gains are more modest than raw deltas imply.

6. Limitations

Our study has several important limitations that should guide interpretation:

Limited model sample. We evaluate only four models. While these span a useful range of sizes and architectures, any patterns observed across four data points are inherently preliminary. We deliberately avoid fitting trend lines or computing correlation statistics on four points, as such analyses would carry misleading precision. Conclusions about general relationships between model characteristics and OOV robustness should be treated as exploratory hypotheses requiring validation on a much larger model sample.

Exploratory OOV pair count. Twenty OOV replacement pairs, while sufficient to identify that the phenomenon exists and to establish statistically significant effect sizes between models (Cohen's d > 1.0 for most pairs), represent a small sample of the vast space of possible entities and replacements. This sample size is appropriate for an exploratory study identifying the OOV sensitivity phenomenon and demonstrating its variation across models, but it is not sufficient for definitive model rankings or fine-grained conclusions. A larger-scale validation study with hundreds of replacements across multiple domains would provide more reliable and generalizable estimates.

Fabricated OOV word design. Our fabricated words were designed to be phonotactically plausible, but they inevitably contain recognizable morphemic subwords (e.g., "Frondlebard" contains "##bard," "Quarbitone" contains "##bit" and "##one," "dravomycin" contains "##cin"). These incidental morphemic fragments could provide partial semantic cues that would not be present in truly random character sequences (e.g., "xyzqwrt" or "bkfmpl"). However, we argue that our approach reflects realistic OOV encounters better than random strings: real domain-specific OOV terms—medical compounds (e.g., "pembrolizumab"), chemical nomenclature (e.g., "dimethylformamide"), novel proper nouns—routinely contain recognizable morphemes. A study using both phonotactically plausible and purely random OOV words would help disentangle these effects.

Single replacement word per entity. Each entity is replaced with exactly one fabricated OOV word. Different replacement words might produce different deltas due to accidental subword overlaps with meaningful words, varying token counts, or other factors. Multiple replacements per entity would provide more robust per-entity estimates.

English only. All experiments use English text. OOV handling may differ significantly for morphologically rich languages or languages with different writing systems.

Confounded variables. As discussed in Section 5.2, model size, training data, training objective, architecture depth, and other factors all co-vary across our models. We observe patterns but cannot establish causal mechanisms. The practical recommendations stand regardless of the causal story, but mechanistic understanding requires carefully controlled ablation studies.

No [UNK] token analysis. All fabricated words are decomposed into subword sequences; none triggers the [UNK] token. The behavior with true [UNK] tokens (which can occur with certain characters or scripts) may differ.

Static embeddings only. We evaluate only the final sentence embedding via mean pooling. Analysis of individual layer representations or attention patterns could provide deeper mechanistic insights.

Normalization approach. Our dynamic range normalization uses the mean positive and mean negative cosine similarities as anchors. Alternative normalization approaches (e.g., using percentile-based ranges, or computing per-pair normalization) might yield different relative rankings. The approach we use provides a simple, interpretable metric, but it is one of several reasonable choices.

7. Conclusion

We have demonstrated that out-of-vocabulary robustness in sentence embedding models is primarily a learned property of the embedding space rather than a consequence of tokenizer architecture. Through controlled experiments with four models sharing identical tokenization (vocabulary size 30,522), we find:

1. OOV sensitivity varies 3.5x in raw deltas (mean cosine delta 0.035 to 0.123) across models, with GTE-Large being the most robust and MiniLM-L6 the most sensitive in absolute terms.
2. Normalized sensitivity tells a different story. When OOV deltas are expressed as fractions of each model's usable similarity dynamic range, Nomic-Embed emerges as the most proportionally sensitive (25.7%), while MiniLM (16.5%), BGE (16.7%), and GTE (15.1%) cluster together. This reveals that raw deltas can be misleading when models operate in fundamentally different similarity regimes.
3. Identical tokenization, divergent robustness: the same subword decomposition produces dramatically different embedding perturbations across models, conclusively separating the tokenizer's role from the embedding model's role.
4. A descriptive pattern suggests larger models may be more robust in raw OOV deltas, but with only four models tested and multiple confounding variables (training data, objectives, distillation), we present this as an observation to be validated rather than a proven relationship.
5. Culturally prominent entities show the largest OOV deltas, consistent with the hypothesis that specialized single-token representations are lost when replaced with generic subword fragments.
6. Dynamic range normalization is essential for fair cross-model comparison of OOV sensitivity, and we recommend its adoption in future evaluation studies.

These findings have immediate practical relevance: practitioners deploying sentence embeddings in domains with frequent novel entities (biomedicine, law, specialized sciences) should evaluate their chosen model's OOV sensitivity empirically, with particular attention to both raw deltas and normalized metrics. The additional computational cost of larger models may be justified by improved OOV robustness, but the gains are more nuanced than raw cosine deltas alone suggest.

Future work should expand this analysis to a broader set of models (including decoder-based and multilingual embeddings), develop controlled ablation studies to isolate the causal factors behind OOV robustness, investigate targeted fine-tuning strategies for enhancing compositional subword representation, and validate the findings with larger OOV test sets spanning multiple domains.

References

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019, pages 4171-4186.

Kudo, T. and Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of EMNLP 2018: System Demonstrations, pages 66-71.

Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of EMNLP-IJCNLP 2019, pages 3982-3992.

Sennrich, R., Haddow, B., and Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. In Proceedings of ACL 2016, pages 1715-1725.

Appendix A: Complete OOV Delta Data

Table A1: Full OOV delta matrix (all 20 pairs × 4 models)

Appendix B: Tokenization Verification

To verify that all four models produce identical tokenizations, we compared the subword splits for all test words across models. Representative examples:

All 30 subword-split words in our test set produce identical token sequences across all four models, confirming that differences in embedding behavior cannot be attributed to tokenization differences.

Appendix C: Normalized OOV Delta Computation

The normalized OOV delta metric is computed as follows:

For each model m:

1. Compute the mean cosine similarity over 20 semantically similar (positive) pairs: μ_pos(m)
2. Compute the mean cosine similarity over 15 semantically unrelated (negative) pairs: μ_neg(m)
3. Dynamic range: DR(m) = μ_pos(m) − μ_neg(m)
4. For each OOV pair i, normalized delta: δ_norm(m, i) = δ_raw(m, i) / DR(m)

This metric answers: "What fraction of the model's useful similarity range is consumed by this OOV replacement?" A normalized delta of 0.25 means the OOV replacement erodes 25% of the distance between "semantically similar" and "semantically unrelated" in that model's embedding space.

Table C1: Complete normalized OOV deltas (% of dynamic range)