Relational Displacement in Three General-Purpose Text Embedding Models: Tokenizer-Induced Collisions and the Limits of Vector Arithmetic

Emma Leonhart

Abstract

It is well established that embedding spaces encode relational structure as vector arithmetic — from word2vec analogies (Mikolov et al., 2013) through TransE translations (Bordes et al., 2013) to modern knowledge graph embeddings. What remains underexplored is where this encoding breaks down and what the failure modes reveal about the topology of embedding spaces. We apply standard relational displacement analysis to three general-purpose text embedding models (mxbai-embed-large 1024-dim, nomic-embed-text 768-dim, all-minilm 384-dim) using Wikidata triples from two domains: Engishiki (a Japanese historical text with dense romanized non-Latin terminology) and a broad sample of country-level entities via P31 (instance of). Across all three models, 30 relations manifest as consistent displacements universally, with up to 109 per model. The self-diagnostic correlation between geometric consistency and prediction accuracy (r = 0.861, 95% CI [0.773, 0.926]) reproduces across models and domains. The Engishiki-seeded dataset is retained deliberately: its dense romanized Japanese, Arabic, Irish, and indigenous-language terminology exposes a large-scale embedding collision — 147,687 cross-entity embedding pairs at cosine similarity ≥ 0.95, traceable to WordPiece diacritic stripping. We analyze the geometry of this collapse, showing that colliding embeddings occupy the densest regions of the embedding space — 71% fall in the dense-collision quartile — crowding into already-saturated neighborhoods rather than drifting into empty space. This three-regime structure (dense-collision, functional, sparse) sets hard limits on relational displacement methods regardless of algorithmic sophistication. All code and data are publicly available, and the analysis reproduces end-to-end in approximately 30 minutes per model on commodity hardware.

1. Introduction

That embedding spaces encode relational structure as vector arithmetic is well established. The word2vec analogy king - man + woman ≈ queen (Mikolov et al., 2013) demonstrated this for distributional word embeddings. TransE (Bordes et al., 2013) formalized the insight for knowledge graphs, training embeddings such that h + r ≈ t for each triple (head, relation, tail). Subsequent work introduced rotations (RotatE; Sun et al., 2019), complex-valued embeddings (ComplEx; Trouillon et al., 2016), geometric constraints for hierarchical relations (box embeddings; Vilnis et al., 2018), and extensive theoretical analysis of which relation types admit which geometric representations (e.g., Wang et al., 2014; Kazemi & Poole, 2018).

It is also well known that these methods work best on functional (many-to-one) and bijective (one-to-one) relations, and struggle with symmetric, transitive, or many-to-many relations. TransE explicitly cannot model symmetric relations (Bordes et al., 2013); RotatE was designed partly to address this gap. The characterization of which relation types encode as consistent displacements — non-transitive, bijective relations succeed; symmetric and semantically overloaded relations fail — is a consequence of the mathematics, not a new empirical finding.

What we contribute is not the relational displacement itself, but its application as a diagnostic for embedding space topology. We apply standard relational displacement analysis to general-purpose text embedding models — models trained for semantic similarity, not for knowledge graph completion — and use the pattern of success and failure to characterize the structure of the embedding space. Specifically:

1. We use a deliberately domain-specific seed (Engishiki, a Japanese historical text) alongside a broader country-level sample to expose how different regions of the same embedding space behave under identical relational tests.

2. We show that the Engishiki-seeded data, rich in romanized Japanese, Arabic, Irish, and indigenous-language terminology, reveals a large-scale embedding collision — 147,687 cross-entity embedding pairs at cosine ≥ 0.95, driven by WordPiece diacritic stripping.

3. We analyze the geometry of the collapse zone, showing that colliding embeddings are not merely close to each other but are sparse and distant from the well-structured regions where relational displacement succeeds — establishing hard topological limits on vector arithmetic methods.

1.1 Key Findings

1. Relational displacement reproduces in untrained models. Of 159 predicates tested (≥10 triples each), 86 produce consistent displacement vectors in mxbai-embed-large, with 30 universal across all three models tested. This confirms that general-purpose models inherit the same relational structure that trained KGE models exploit.

2. The self-diagnostic correlation holds. The correlation between geometric consistency and prediction accuracy (r = 0.861, 95% CI [0.773, 0.926]) means the displacement consistency metric predicts which relations will function as vector arithmetic — reproducing the known functional/relational split without ground-truth labels.

3. Embedding collapse is large-scale and dense-collision. 147,687 cross-entity pairs collapse to cosine ≥ 0.95. Geometric analysis shows colliding embeddings are 2.4× denser (mean k-NN distance 0.106 vs 0.258) and 71% fall in the dense-collision (densest) quartile. The collapse zone is not distant and sparse — it is crowded and central.

4. Three-regime structure sets hard limits. The embedding space partitions into dense-collision (saturated, where collisions concentrate), functional (vector arithmetic works), and sparse (sparse) regimes. No relational displacement method — learned or discovered — can extract consistent structure from the embedding collision zone. This is a property of the space, not the method.

5. Engishiki highlights the dense-collision regime. The domain-specific seed is not a limitation but a feature: it floods the embedding space with exactly the kind of input (romanized non-Latin scripts) that triggers dense-collision crowding, making the phenomenon measurable at scale.

2. Related Work

2.1 Knowledge Graph Embedding

TransE (Bordes et al., 2013) established that relations can be modeled as translations (h + r ≈ t) in learned embedding spaces. Subsequent work analyzed which relation types each model can represent: TransE handles antisymmetric and compositional relations but cannot model symmetric ones; RotatE (Sun et al., 2019) handles symmetry via rotation; ComplEx (Trouillon et al., 2016) handles symmetry and antisymmetry via complex-valued embeddings. Wang et al. (2014) and Kazemi & Poole (2018) provided systematic analyses of the relation type expressiveness of different KGE architectures. Our work does not introduce a new embedding method but applies the known displacement test to general-purpose (non-KGE) models as a diagnostic for embedding space topology.

2.2 Word Embedding Analogies

Mikolov et al. (2013) showed that king - man + woman ≈ queen holds in word2vec. Subsequent work (Linzen, 2016; Rogers et al., 2017; Schluter, 2018) showed these analogies are less robust than initially claimed, often reflecting frequency biases and dataset artifacts. Ethayarajh et al. (2019) formalized the conditions under which analogy recovery succeeds, showing it requires the relation to be approximately linear and low-rank in the embedding space. Our work is consistent with these findings: the relations we recover are exactly those that satisfy the linearity condition (functional, bijective), and those that fail are those the theory predicts will fail (symmetric, many-to-many).

2.3 Neurosymbolic Integration

Logic Tensor Networks (Serafini & Garcez, 2016), Neural Theorem Provers (Rocktäschel & Riedel, 2017), and DeepProbLog (Manhaeve et al., 2018) integrate logical reasoning into neural architectures. These constructive approaches build systems that reason logically. Our work is analytical rather than constructive, but we make no claim that the analytical approach is novel in itself — probing pre-trained representations for structure is standard practice.

2.4 Probing and Representation Analysis

Probing classifiers (Conneau et al., 2018; Hewitt & Manning, 2019) test what linguistic properties are encoded in learned representations. Our displacement consistency metric is analogous to a probe, but operates at the relational level and uses vector arithmetic rather than learned classifiers. The key methodological difference is that we use the failure pattern of the probe — which relations don't encode — as the primary finding, rather than the successes.

2.5 Embedding Space Topology and Failure Modes

The glitch token phenomenon (Li et al., 2024) documents poorly trained embeddings for low-frequency tokens in LLMs. Our collision finding extends this to sentence-embedding models, showing that entire classes of input (romanized non-Latin scripts, diacritical text) collapse into degenerate regions. Recent work on embedding space topology has identified stratified sub-manifolds within learned representations, independently supporting the multi-regime structure we observe.

2.6 Tokenizer-Induced Information Loss

WordPiece (Schuster & Nakajima, 2012) and BPE (Sennrich et al., 2016) tokenizers are known to struggle with out-of-vocabulary and non-Latin text. Rust et al. (2021) showed that tokenizer quality strongly predicts downstream multilingual model performance. Our collision analysis provides a geometric characterization of this failure: tokenizer-induced information loss creates measurable topological defects in the embedding space — sparse, distant regions where distinct inputs become indistinguishable.

3. Method

3.1 Problem Formulation

Given:

• An embedding function $f: \text{Text} \to \mathbb{R}^d$ (any text embedding model)
• A knowledge base $\mathcal{K} = {(s, p, o)}$ of subject-predicate-object triples

Find: The subset of predicates $P^* \subseteq P$ whose triples manifest as consistent displacement vectors in the embedding space.

Definition (Relational Displacement). For a triple $(s, p, o) \in \mathcal{K}$, the relational displacement is the vector $\mathbf{g}_{s,p,o} = f(o) - f(s)$, connecting the subject's embedding to the object's embedding. This is the standard TransE formulation applied without training.

Definition (Displacement Consistency). For a predicate $p$ with triples ${(s_1, p, o_1), \ldots, (s_n, p, o_n)}$, the mean displacement is $\mathbf{d}$p = \frac{1}{n}\sum{i=1}^{n} \mathbf{g}_{s_i, p, o_i}. The consistency of $p$ is the mean cosine alignment of individual displacements with the mean:

$$\text{consistency}(p) = \frac{1}{n}\sum_{i=1}^{n} \cos(\mathbf{g}_{s_i,p,o_i}, \mathbf{d}_p)$$

A predicate with consistency > 0.5 encodes as a consistent relational displacement: its triples are approximated by a single vector operation. This threshold is not novel — it corresponds to the standard criterion for meaningful directional agreement in high-dimensional spaces.

3.2 Data Pipeline

1. Entity Import. Two seed strategies: (a) Breadth-first search from Engishiki (Q1342448), importing 500 entities with all triples and linked entities (14,796 items total) — deliberately chosen to produce dense romanized non-Latin terminology that stresses the embedding space; (b) Broad P31 (instance of) sampling across country-level entities to provide a domain-general baseline.

2. Embedding. Each entity's English label is embedded using mxbai-embed-large (1024-dim) via Ollama. Aliases receive separate embeddings. Total: 41,725 embeddings from the Engishiki seed.

3. Relational Displacement Computation. For each entity-entity triple, compute the displacement vector between subject and object label embeddings. Total: 16,893 entity-entity triples across 1,472 unique predicates. This is the standard h + r ≈ t test from TransE, applied without training.

3.3 Discovery Procedure

For each predicate $p$ with $\geq 10$ entity-entity triples:

1. Compute all relational displacements ${\mathbf{g}_i}$
2. Compute mean displacement $\mathbf{d}_p$
3. Compute consistency: mean alignment of each $\mathbf{g}_i$ with $\mathbf{d}_p$
4. Compute pairwise consistency: mean cosine similarity between all pairs of displacements
5. Compute magnitude coefficient of variation: stability of displacement magnitudes

Note on unit-norm embeddings. mxbai-embed-large returns L2-normalized embeddings (||v|| = 1.0000). Consequently, displacement magnitudes are a deterministic function of cosine similarity: ||f(o) - f(s)|| = sqrt(2(1 - cos(f(o), f(s)))). The MagCV metric therefore carries no information independent of cosine distance for this model. We retain it for cross-model comparability, as other models (e.g., BioBERT) do not necessarily normalize.

3.4 Prediction Evaluation

For each discovered operation ($\text{consistency} > 0.5$), we evaluate prediction accuracy using leave-one-out:

For each triple $(s, p, o)$:

1. Compute $\mathbf{d}_{p}^{(-i)}$ = mean displacement excluding this triple
2. Predict: $\hat{\mathbf{o}} = f(s) + \mathbf{d}_{p}^{(-i)}$
3. Rank all entities by cosine similarity to $\hat{\mathbf{o}}$
4. Record the rank of the true object $o$

We report Mean Reciprocal Rank (MRR) and Hits@k for k ∈ {1, 5, 10, 50}.

3.5 Composition Test

To test whether operations can be chained, we find all two-hop paths $s \xrightarrow{p_1} m \xrightarrow{p_2} o$ where both $p_1$ and $p_2$ are discovered operations. We predict:

$$\hat{\mathbf{o}} = f(s) + \mathbf{d}$${p_1} + \mathbf{d}{p_2}

and evaluate whether the true $o$ appears in the top-k nearest neighbors. We test 5,000 compositions.

4. Results

4.1 Operation Discovery

Of 159 predicates with ≥10 triples, 86 (54.1%) produce consistent displacement vectors:

Table 1. Distribution of discovered operations by consistency.

The top 15 discovered operations:

Table 2. Top 15 relations by displacement consistency (alignment with mean displacement). N = number of triples. Pairwise = mean cosine similarity between all pairs of displacements. MagCV = coefficient of variation of displacement magnitudes. Cos Dist = mean cosine distance between subject and object.

4.2 Prediction Accuracy

Leave-one-out evaluation of all 86 discovered operations:

Table 3. Prediction results for selected operations (full table in supplementary). MRR = Mean Reciprocal Rank. H@k = Hits at rank k.

Aggregate statistics across all 86 operations:

Table 4. Aggregate prediction statistics with bootstrap confidence intervals (10,000 resamples). All correlations survive Bonferroni correction across 3 tests (adjusted alpha = 0.017).

The correlation between displacement consistency and prediction accuracy (r = 0.861, 95% CI [0.773, 0.926]) is the central methodological finding: the discovery metric is also the quality metric. A predicate's geometric consistency, computable without any held-out evaluation, predicts how well that predicate will function as a vector operation. Note that this correlation is not tautological despite the apparent circularity: consistency is computed over all triples, while MRR is computed via leave-one-out evaluation where each prediction uses a mean displacement that excludes the test triple. A predicate could have high alignment (all displacements point the same direction) but poor prediction (if the mean displacement points to a crowded region where many non-target entities cluster). The correlation is an empirical finding about the geometry of these embedding spaces, not a mathematical necessity. The effect size between strong (>0.7) and moderate (0.5-0.7) operations is Cohen's d = 3.092 — a large effect, indicating the alignment threshold cleanly separates high-performing from marginal operations.

4.3 Two-Hop Composition

Over 5,000 tested two-hop compositions (S + d₁ + d₂):

Table 5. Two-hop composition results.

Selected successful compositions (Rank ≤ 5):

Table 6. Successful two-hop compositions. Note: all examples involve Fujiwara no Tadahira because our dataset is seeded from Engishiki (Q1342448), a Japanese historical text. Tadahira is one of the most densely connected entities in this neighborhood, appearing in many two-hop paths. The composition mechanism itself is general — the examples reflect dataset composition, not a limitation of the method.

4.4 Failure Analysis

Predicates that resist vector encoding:

Table 7. Predicates with lowest consistency. Pattern = our characterization of why the displacement is inconsistent.

Three failure modes emerge:

1. Symmetric predicates (sibling, spouse, shares-border-with, diplomatic-relation): No consistent displacement direction because f(A) - f(B) and f(B) - f(A) are equally valid. Alignment ≈ 0.

2. Sequence predicates (follows, followed-by): The displacement from "Monday" to "Tuesday" has nothing in common with the displacement from "Chapter 1" to "Chapter 2." The relationship type is consistent but the direction in embedding space is domain-dependent.

3. Semantically overloaded predicates (instance-of, subclass-of, part-of): "Tokyo is an instance of city" and "7 is an instance of prime number" produce wildly different displacement vectors because the predicate covers too many semantic domains.

Instance-of (P31) at 0.244 is particularly notable. It is the most important predicate in Wikidata (835 triples in our dataset) and a cornerstone of first-order logic, yet it does not function as a vector operation. This suggests that embedding spaces systematically under-represent relational structure: the space encodes entities well but predicates poorly.

4.5 Cross-Model Generalization

To test whether discovered operations are model-agnostic or artifacts of a single model's training, we ran the full pipeline on two additional embedding models: nomic-embed-text (768-dim) and all-minilm (384-dim). All three models were given identical input: the same Wikidata entities seeded from Engishiki (Q1342448) with --limit 500.

Table 8. Operations discovered per model. All three models discover operations despite different architectures and dimensionalities.

30 operations are universal — discovered by all three models. These include demographics-of-topic (avg alignment 0.925), culture (0.923), economy-of-topic (0.896), flag (0.883), coat of arms (0.777), and central bank (0.793). The universal operations are exclusively functional predicates, confirming the functional-vs-relational split across architectures.

Table 9. Cross-model operation overlap. 30 universal operations constitute the model-agnostic core.

Cross-model consistency correlations (alignment scores on shared predicates): mxbai vs all-minilm r = 0.779, mxbai vs nomic r = 0.554, nomic vs all-minilm r = 0.358. The positive correlations confirm that consistency is not random — predicates that work well in one model tend to work well in others, though the strength varies by model pair.

This result is the core evidence for the model-agnostic claim: the same logical operations emerge across three unrelated embedding models with different architectures, different dimensionalities, and different training data. The operations are properties of the semantic relationships themselves, not artifacts of any particular model.

5. Discussion

5.1 Relation Types and Displacement: Confirming the Known Pattern

The pattern across Tables 2 and 7 confirms what the KGE literature predicts: consistent displacements emerge for functional (many-to-one) and bijective (one-to-one) relations, and fail for symmetric, transitive, or many-to-many relations. Each country has one flag, one coat of arms, one head of state — these produce consistent displacements. Symmetric relations (sibling, spouse, shares-border-with) produce no consistent direction because f(A) - f(B) and f(B) - f(A) are equally valid.

This is not a new finding. It follows directly from the mathematics of translational models (Bordes et al., 2013; Wang et al., 2014). What is notable is that the same pattern holds in general-purpose text embedding models with no relational training signal, confirming that the structure is a property of the semantic relationships themselves, not of the training objective.

5.2 The Self-Diagnostic Correlation

The r = 0.861 correlation between consistency and prediction accuracy means the displacement consistency metric is self-calibrating: it predicts which relations will function as vector arithmetic without needing ground-truth evaluation data. This is practically useful for applying relational displacement as a diagnostic to new embedding spaces.

5.3 Collision Geography

We independently measure two properties of each embedding: (a) its local density (mean k-NN distance) and (b) whether it collides with a semantically distinct entity at cosine ≥ 0.95. The question is whether collisions are uniformly distributed across density levels or concentrated in particular regions. This is an empirical question, not a definitional one — dense regions could in principle have few collisions if the model separates semantically distinct entities effectively even in crowded neighborhoods.

5.4 The Embedding Collapse: Geometry of Oversymbolic Crowding

Empirical evidence: the Jinmyōchō collapse. Our collision analysis finds 147,687 cross-entity embedding pairs with cosine similarity ≥ 0.95 that represent genuine semantic collisions: different text mapped to near-identical vectors. The collisions are dominated by romanized non-Latin-script terms — the embedding vector for "Hokkaidō" has cosine similarity ≥ 0.95 with the vectors for 1,428 other entities. This does not mean all 1,428 entities normalize to the same string; rather, the WordPiece tokenizer maps diacritic-bearing text to similar subword sequences, producing embedding vectors that are geometrically near-identical despite representing distinct concepts. "Jinmyōchō" similarly collides with 504 unique texts spanning romanized Japanese (kugyō, Shōtai), Arabic (Djazaïr, Filasṭīn), Irish (Éire), Brazilian indigenous languages (Aikanã, Amanayé), and IPA characters — all of which tokenize to overlapping subword sequences after diacritic stripping.

The mechanism is tokenizer-induced. mxbai-embed-large's WordPiece tokenizer strips diacritical marks during normalization — "Hokkaidō" tokenizes to ['hokkaido'], "Tōkyō" to ['tokyo'], "România" to ['romania']. Terms whose semantic content is carried primarily by diacritics lose that content at tokenization, collapsing into shared or similar subword sequences. This is consistent with Rust et al. (2021)'s finding that tokenizer quality predicts multilingual performance, but here we characterize the geometric consequence rather than the downstream task impact.

The collapse zone is dense, not sparse. Geometric analysis of 16,067 colliding embeddings (vs. 74,760 non-colliding) reveals a finding opposite to what naive intuition might suggest. Colliding embeddings do not occupy empty space far from the well-structured manifold — they crowd into the densest regions of the embedding space:

1. Colliding embeddings are 2.4× denser than non-colliding ones. Mean k-NN distance for colliding embeddings is 0.106, vs 0.258 for non-colliding (ratio 0.41×). Colliding entities are tightly packed together.

2. 71% of colliding embeddings fall in the dense-collision (densest quartile) regime, vs the expected 25% if uniformly distributed. Only 3.2% fall in the sparse (sparsest quartile) regime. The collision zone is overwhelmingly dense-collision.

3. The collapse zone is not geometrically isolated. The distance from a colliding embedding to its nearest non-colliding neighbor (mean 0.119) is nearly identical to the equivalent non-colliding-to-non-colliding distance (mean 0.121, ratio 0.98×). The centroids of the two populations are close (cosine distance 0.038).

This means tokenizer-induced information loss does not push embeddings into distant, empty regions — it collapses them into already-crowded neighborhoods where distinct inputs cannot be differentiated. The colliding embeddings sit among the well-structured embeddings, not apart from them. This is an dense-collision phenomenon: the model's representational capacity in these regions is saturated, and the tokenizer's diacritic stripping removes the only information that could distinguish these inputs.

This extends the glitch token phenomenon (Li et al., 2024) from individual tokens in LLMs to entire classes of input in sentence-embedding models, but with a geometric twist: the failure mode is not sparse under-representation but dense over-crowding. The scale — 147,687 colliding pairs from a single domain seed — suggests that any application of embedding-based reasoning to multilingual or diacritic-rich text will encounter regions where the space is too crowded to discriminate.

Why the Engishiki seed matters. The domain-specific seed is not a limitation but a deliberate experimental choice. Engishiki (Q1342448) is a 10th-century Japanese text whose entities include romanized shrine names (Jinmyōchō, Shikinaisha), historical Japanese personal names, and linked entities from Arabic, Irish, and indigenous-language Wikipedia articles. This floods the embedding space with exactly the inputs that trigger embedding collision, making the phenomenon measurable at scale. The country-level P31 sample provides the domain-general baseline against which the collapse is measured.

5.5 Hard Limits on Vector Arithmetic

The three-regime structure implies that relational displacement methods — whether learned (TransE, RotatE) or discovered (this work) — are bounded by the topological quality of the underlying embedding space. No amount of algorithmic sophistication can extract consistent displacements from a region where distinct concepts have collapsed into the same coordinates. The embedding collision is particularly insidious because it is invisible to standard evaluation: the model appears to embed these inputs normally, and the resulting vectors sit in well-populated regions of the space, but they carry no discriminative information for the colliding inputs.

This has practical implications for any system that chains embedding-based reasoning with knowledge from non-Latin-script domains: RAG systems retrieving over multilingual corpora, knowledge graph completion over Wikidata's non-English entities, and cross-lingual transfer learning.

5.6 Limitations

1. Three embedding models. We validate across mxbai-embed-large (1024-dim), nomic-embed-text (768-dim), and all-minilm (384-dim), finding 30 universal relations. All three are English-language text embedding models trained on similar corpora. Testing on multilingual models or domain-specific models (e.g., biomedical) would further characterize the generality of the three-regime structure.

2. Collision geometry analysis covers one seed. The distance metrics characterizing the embedding collision zone (Section 5.4) are computed from the Engishiki-seeded dataset. Multi-seed analysis would test whether the same crowding pattern holds across domains.

3. Label embeddings only. We embed entity labels (short text strings), not descriptions or full articles. This deliberately mirrors how these models are used in practice for entity linking and knowledge graph completion (short query strings, not full documents). Richer textual representations might shift some entities out of the sparse zone, but the label-only setting represents a common real-world deployment pattern for these models.

4. Relational displacement, not full FOL. We test which binary relations encode as consistent vector arithmetic. Full first-order logic includes quantifiers, variable binding, negation, and complex formula composition, none of which we test. The title of this paper reflects the scope: relational displacement and its failure modes, not a claim about discovering FOL.

6. Conclusion

We apply standard relational displacement analysis to three general-purpose text embedding models and confirm the known finding: functional and bijective relations encode as consistent vector displacements, while symmetric and many-to-many relations do not. This holds across models not trained for knowledge graph completion, confirming that the relational structure is a property of the semantic relationships, not the training objective.

The primary contribution is the embedding collision finding. A deliberately domain-specific seed (Engishiki) exposes 147,687 cross-entity embedding collapses at cosine ≥ 0.95, traceable to WordPiece diacritic stripping. Geometric analysis reveals that these collapses occupy the densest regions of the embedding space — 71% fall in the dense-collision quartile, with 2.4× smaller k-NN distances than non-colliding embeddings. The failure mode is not sparse under-representation but dense over-crowding: tokenizer-induced information loss pushes diacritic-rich inputs into already-saturated neighborhoods where distinct concepts become indistinguishable.

The practical implication is that embedding-based reasoning over multilingual or diacritic-rich text — RAG systems, knowledge graph completion, cross-lingual transfer — will encounter regions where the embedding space provides no discriminative information, and no amount of relational modeling can compensate for what the tokenizer has already destroyed.

All code and data are publicly available. The full analysis reproduces end-to-end in approximately 30 minutes on commodity hardware with a local Ollama instance.

References

Bordes, A., Usunier, N., Garcia-Durán, A., Weston, J., & Yakhnenko, O. (2013). Translating Embeddings for Modeling Multi-relational Data. NeurIPS, 26.

Conneau, A., Kruszewski, G., Lample, G., Barrault, L., & Baroni, M. (2018). What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. ACL.

Ethayarajh, K., Duvenaud, D., & Hirst, G. (2019). Towards understanding linear word analogies. ACL.

Hewitt, J., & Manning, C. D. (2019). A structural probe for finding syntax in word representations. NAACL.

Kazemi, S. M., & Poole, D. (2018). SimplE embedding for link prediction in knowledge graphs with baseline model comparison. NeurIPS.

Li, Y., Liu, Y., Deng, G., Zhang, Y., & Song, W. (2024). Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection. Proceedings of the ACM on Software Engineering, 1(FSE). https://doi.org/10.1145/3660799

Linzen, T. (2016). Issues in evaluating semantic spaces using word analogies. RepEval Workshop.

Manhaeve, R., Dumančić, S., Kimmig, A., Demeester, T., & De Raedt, L. (2018). DeepProbLog: Neural probabilistic logic programming. NeurIPS.

Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. NeurIPS.

Rocktäschel, T., & Riedel, S. (2017). End-to-end differentiable proving. NeurIPS.

Rogers, A., Drozd, A., & Li, B. (2017). The (too many) problems of analogical reasoning with word vectors. StarSem.

Rust, P., Pfeiffer, J., Vulić, I., Ruder, S., & Gurevych, I. (2021). How good is your tokenizer? On the monolingual performance of multilingual language models. ACL.

Schluter, N. (2018). The word analogy testing caveat. NAACL.

Schuster, M., & Nakajima, K. (2012). Japanese and Korean voice search. ICASSP.

Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. ACL.

Serafini, L., & Garcez, A. d'A. (2016). Logic Tensor Networks: Deep learning and logical reasoning from data and knowledge. NeSy Workshop.

Sun, Z., Deng, Z.-H., Nie, J.-Y., & Tang, J. (2019). RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. ICLR.

Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., & Bouchard, G. (2016). Complex embeddings for simple link prediction. ICML.

Vilnis, L., Li, X., Xiang, S., & McCallum, A. (2018). Probabilistic embedding of knowledge graphs with box lattice measures. ACL.

Wang, Z., Zhang, J., Feng, J., & Chen, Z. (2014). Knowledge graph embedding by translating on hyperplanes. AAAI.