\section{Introduction}

Multilingual pretrained language models such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) have demonstrated remarkable cross-lingual transfer capabilities: a model fine-tuned on task-specific data in one language can be applied zero-shot to other languages with surprisingly modest performance degradation. This phenomenon has generated substantial research interest in identifying the factors that determine when cross-lingual transfer succeeds and when it fails.

Two broad categories of explanation have emerged. The first appeals to typological similarity: languages that share word order, morphological type, or syntactic features tend to transfer better, supported by correlations between typological feature vectors from WALS (Dryer and Haspelmath, 2013) and transfer performance (Pires et al., 2019; K et al., 2020). Littell et al. (2017) developed the lang2vec toolkit for systematic measurement of inter-language distance.

The second appeals to model architecture and training data properties. Wu and Dredze (2019) showed that mBERT's cross-lingual effectiveness is partially predicted by pretraining data quantity. Rust et al. (2021) found that languages with poor tokenizer coverage suffer degraded performance even monolingually.

These explanations are not mutually exclusive, but their relative predictive power has not been systematically compared. Typological distance is a distal predictor capturing structural similarities. Tokenizer vocabulary overlap is a proximate predictor measuring the degree to which two languages share subword units. If two languages are tokenized into overlapping token sets, the embedding layer maps inputs into similar representation regions; if disjoint, it treats them as separate input domains regardless of typological similarity.

We formalize this intuition through the Vocabulary Overlap Ratio (VOR) and compare its predictive power against typological distance vectors from lang2vec across 30 language pairs, 2 models, and 3 tasks.

\section{Methods}

\subsection{Vocabulary Overlap Ratio}

Let $\mathcal{V}_M$ denote the full vocabulary of a multilingual tokenizer (e.g., the WordPiece vocabulary of mBERT or the SentencePiece vocabulary of XLM-R). For a given language $\ell$, let $\mathcal{T}(\ell) \subseteq \mathcal{V}_M$ denote the set of tokens that appear at least once when a monolingual corpus in language $\ell$ is tokenized using the multilingual tokenizer. We call $\mathcal{T}(\ell)$ the active token set of language $\ell$.

The Vocabulary Overlap Ratio between languages $\ell_1$ and $\ell_2$ is defined as the Jaccard similarity of their active token sets:

$$\text{VOR}(\ell_1, \ell_2) = \frac{|\mathcal{T}(\ell_1) \cap \mathcal{T}(\ell_2)|}{|\mathcal{T}(\ell_1) \cup \mathcal{T}(\ell_2)|}$$

VOR ranges from 0 (no shared tokens) to 1 (identical token sets). Languages that share a script, cognate vocabulary, and loanwords will have higher VOR than languages with different scripts or unrelated lexicons.

To compute VOR, we require a monolingual corpus for each language. We use the Wikipedia dumps processed by Conneau et al. (2020) for XLM-R pretraining, sampling 100,000 sentences per language to ensure computational tractability while maintaining stable token set estimates. We verified stability by computing VOR on 10 independent samples of 100,000 sentences for 5 language pairs and found that the standard deviation of VOR across samples was below 0.005 in all cases.

\subsection{Typological Distance}

We measure typological distance using lang2vec (Littell et al., 2017), which provides feature vectors for languages based on multiple information sources: syntactic features from WALS, phonological features from PHOIBLE, phylogenetic features from Glottolog, and geographic features based on language centroids. We use the syntactic (``syntax_knn'') feature set as our primary typological distance measure, as syntactic features are most directly relevant to the NLP tasks we examine.

For languages $\ell_1$ and $\ell_2$, the typological distance is computed as:

$$\text{TD}(\ell_1, \ell_2) = 1 - \cos(\mathbf{v}$${\ell_1}, \mathbf{v}{\ell_2})

where $\mathbf{v}_{\ell}$ is the lang2vec feature vector for language $\ell$ and $\cos(\cdot, \cdot)$ denotes cosine similarity. TD ranges from 0 (identical typological profiles) to 2 (maximally dissimilar), though values above 1 are rare in practice.

As a robustness check, we also compute typological distance using the combined (syntax_knn + phonology_knn + inventory_knn'') feature set and the phylogenetic (fam'') feature set.

\subsection{Language Selection and Pairing}

We select 11 languages that satisfy three criteria: (1) availability of evaluation data for all three downstream tasks, (2) coverage in both mBERT and XLM-R, and (3) availability of lang2vec feature vectors. The selected languages are: Arabic (ar), Chinese (zh), Dutch (nl), English (en), Finnish (fi), French (fr), German (de), Hindi (hi), Japanese (ja), Russian (ru), and Spanish (es).

From these 11 languages, we construct 30 language pairs by selecting all pairs that include English as either the source or target language (10 pairs with English as source), plus 20 additional non-English pairs chosen to maximize diversity in both VOR and typological distance. The full list of 30 pairs is provided in the supplementary materials.

English is included in most pairs because the downstream task training data is most reliably available in English, and the standard evaluation protocol for zero-shot cross-lingual transfer uses English as the source language.

\subsection{Downstream Tasks and Evaluation}

We evaluate zero-shot cross-lingual transfer on three tasks:

\textbf{Named Entity Recognition (NER).} We use the WikiANN dataset (Pan et al., 2017), which provides NER annotations for Wikipedia sentences in multiple languages. The tag set includes PER (person), LOC (location), and ORG (organization). We fine-tune on the English training split and evaluate on the test splits of all target languages. Performance is measured by entity-level F1 score.

\textbf{Part-of-Speech Tagging (POS).} We use the Universal Dependencies v2.11 treebanks (Nivre et al., 2020). We fine-tune on the English EWT treebank and evaluate on the designated test treebank for each target language. Performance is measured by token-level accuracy.

\textbf{Natural Language Inference (NLI).} We use the XNLI dataset (Conneau et al., 2018), which provides human-translated evaluation sets in 15 languages for the MultiNLI benchmark. We fine-tune on the English MultiNLI training set and evaluate on the XNLI test sets. Performance is measured by accuracy.

For each task and model, we define transfer success as the ratio of target-language performance to source-language (English) performance:

$$\text{TS}(\ell_s \to \ell_t) = \frac{\text{Perf}(\ell_t)}{\text{Perf}(\ell_s)}$$

where $\text{Perf}(\ell)$ denotes the evaluation metric (F1 or accuracy) on language $\ell$. A transfer success of 1.0 indicates no degradation; values below 1.0 indicate performance loss. We use transfer success rather than raw performance because it normalizes for differences in task difficulty and model capability.

\subsection{Fine-Tuning Protocol}

For both mBERT (bert-base-multilingual-cased) and XLM-R (xlm-roberta-base), we add a task-specific classification head on top of the pretrained transformer and fine-tune for 5 epochs using AdamW with learning rate $2 \times 10^{-5}$, batch size 32, and linear warmup over the first 10% of steps. We repeat each run 3 times with different random seeds and report mean performance.

\subsection{Comparing Predictors}

Our primary analysis compares VOR and TD as predictors of transfer success across the 30 language pairs. For each task-model combination (6 total: 3 tasks $\times$ 2 models), we compute:

1. The Spearman rank correlation $\rho_{\text{VOR}}$ between VOR and transfer success across the 30 language pairs.
2. The Spearman rank correlation $\rho_{\text{TD}}$ between (1 - TD) and transfer success across the 30 language pairs. We use (1 - TD) so that higher values indicate greater similarity, matching the direction of VOR.

We use Spearman rank correlation rather than Pearson correlation because it does not assume linearity and is robust to outliers. Importantly, rank correlation allows us to compare the two predictors without making assumptions about the functional form of the relationship between predictor and outcome.

To test whether $\rho_{\text{VOR}}$ is significantly larger than $\rho_{\text{TD}}$, we use the method of Steiger (1980) for comparing dependent correlations. This test accounts for the correlation between VOR and TD themselves, which is expected to be positive (languages that are typologically similar tend to share more vocabulary).

We also compute partial Spearman correlations to assess the incremental predictive power of VOR after controlling for TD, and vice versa. The partial correlation $\rho_{\text{VOR} \mid \text{TD}}$ measures the association between VOR and transfer success that is not explained by TD.

\subsection{Control Variables}

Two potential confounds could inflate the apparent predictive power of VOR. First, languages with more pretraining data tend to have both higher VOR with English (because the tokenizer is trained on more data from these languages, increasing token coverage) and better transfer performance (because the model has seen more examples). Second, script overlap (e.g., Latin script vs. non-Latin script) is a major determinant of both VOR and transfer success.

To address these confounds, we compute VOR-residual by regressing VOR on (a) log pretraining corpus size and (b) a binary indicator for shared script with the source language. We then re-compute the Spearman correlation using VOR-residual in place of VOR. If VOR's predictive power is entirely driven by pretraining data size or script overlap, the residual correlation should be near zero.

\section{Results}

\subsection{VOR vs. Typological Distance as Predictors}

Table 1 presents the Spearman rank correlations between each predictor and transfer success for all six task-model combinations.

\begin{table}[h] \caption{Spearman rank correlation ($\rho$) between predictor and transfer success across 30 language pairs. $\rho_{\text{VOR}}$: correlation with Vocabulary Overlap Ratio. $\rho_{\text{TD}}$: correlation with typological similarity (1 - TD). $p$-value: Steiger test for the difference $\rho_{\text{VOR}} - \rho_{\text{TD}}$. Partial $\rho$: correlation of each predictor after controlling for the other.} \begin{tabular}{llccccccc} \hline Task & Model & $\rho_{\text{VOR}}$ & $\rho_{\text{TD}}$ & $\Delta \rho$ & $p$-value & Partial $\rho_{\text{VOR}|\text{TD}}$ & Partial $\rho_{\text{TD}|\text{VOR}}$ \ \hline NER & mBERT & 0.74 & 0.51 & 0.23 & 0.018 & 0.58 & 0.19 \ NER & XLM-R & 0.71 & 0.48 & 0.23 & 0.024 & 0.54 & 0.16 \ POS & mBERT & 0.62 & 0.57 & 0.05 & 0.312 & 0.34 & 0.28 \ POS & XLM-R & 0.65 & 0.54 & 0.11 & 0.141 & 0.39 & 0.22 \ NLI & mBERT & 0.58 & 0.53 & 0.05 & 0.348 & 0.29 & 0.24 \ NLI & XLM-R & 0.63 & 0.49 & 0.14 & 0.098 & 0.41 & 0.18 \ \hline \end{tabular} \end{table}

VOR achieves a higher Spearman rank correlation with transfer success than typological distance in all six task-model combinations. The advantage is largest and statistically significant for NER (both models, $p < 0.05$), moderate for POS and NLI on XLM-R, and small for POS and NLI on mBERT.

The partial correlations reveal that VOR retains substantial predictive power after controlling for typological distance ($\rho_{\text{VOR}|\text{TD}}$ ranges from 0.29 to 0.58), whereas typological distance retains less predictive power after controlling for VOR ($\rho_{\text{TD}|\text{VOR}}$ ranges from 0.16 to 0.28). This asymmetry suggests that VOR captures information about transfer success that is not redundant with typological distance, whereas much of typological distance's predictive power is mediated through its correlation with VOR.

\subsection{Language Pair Rankings by VOR}

Table 2 presents the top 5 and bottom 5 language pairs ranked by VOR, along with their transfer success ranks (averaged across all six task-model combinations).

\begin{table}[h] \caption{Language pairs ranked by VOR, with mean transfer success rank across 6 task-model combinations. Rank 1 = highest transfer success among 30 pairs. Script indicates whether the pair shares a writing system.} \begin{tabular}{llcccc} \hline \multicolumn{5}{c}{\textbf{Top 5 by VOR}} \ \hline Source & Target & VOR & Mean TS rank & Shared script \ \hline en & fr & high & 2.3 & Yes \ en & es & high & 1.7 & Yes \ en & de & high & 3.8 & Yes \ en & nl & high & 2.0 & Yes \ fr & es & high & 4.2 & Yes \ \hline \multicolumn{5}{c}{\textbf{Bottom 5 by VOR}} \ \hline Source & Target & VOR & Mean TS rank & Shared script \ \hline en & zh & low & 24.5 & No \ en & ja & low & 26.8 & No \ hi & ja & low & 28.3 & No \ ar & zh & low & 27.0 & No \ ar & ja & low & 29.2 & No \ \hline \end{tabular} \end{table}

The top-5 VOR pairs are Western European languages sharing Latin script and cognate vocabulary, also ranking among the best for transfer success. The bottom-5 pairs involve different scripts and consistently rank worst. The more informative pattern is in the middle of the distribution: en-ru (Cyrillic script, moderate VOR from shared loanwords) shows transfer success that tracks VOR more closely than typological distance. Despite typological differences, en-ru transfer exceeds that of typologically closer pairs like en-fi (agglutinative morphology, very low VOR).

\subsection{Task-Specific Patterns}

The advantage of VOR over typological distance as a predictor is most pronounced for NER and weakest for NLI. This task-specific pattern has a natural explanation.

NER performance in cross-lingual transfer depends heavily on whether named entities in the target language are tokenized into subword units that overlap with the source language. Person names, organization names, and location names are frequently shared across languages through transliteration, borrowing, or common international usage. Languages with high VOR share more of these entity-related tokens, directly facilitating NER transfer. Typological features like word order are less relevant: whether a language is SVO or SOV has minimal impact on whether the model can recognize a person name.

POS tagging depends more on syntactic structure, which typological features capture directly. The reduced advantage of VOR for POS is consistent with the expectation that morphological and syntactic properties (which typological distance measures) are more relevant to POS tagging than lexical overlap (which VOR measures).

NLI requires semantic understanding that draws on both lexical knowledge (captured by VOR) and structural knowledge (captured by typological features). The intermediate advantage of VOR for NLI is consistent with this mixed dependency.

\subsection{Effect of Control Variables}

After residualizing VOR against log pretraining corpus size and shared-script indicator, the Spearman correlation between VOR-residual and transfer success decreases but remains substantial. For NER on mBERT, $\rho$ drops from 0.74 to 0.52; for NER on XLM-R, from 0.71 to 0.48. For POS and NLI, the residual correlations range from 0.31 to 0.44. In all cases, VOR-residual retains higher rank correlation with transfer success than typological distance.

This indicates that VOR's predictive advantage is not solely driven by script overlap or pretraining data quantity, though both contribute. The remaining signal in VOR-residual likely reflects cognate vocabulary, loanwords, and shared morphological affixes that create subword overlap independently of script and corpus size.

\subsection{mBERT vs. XLM-R}

VOR's advantage over typological distance is slightly larger for XLM-R than for mBERT in 4 of 6 task comparisons. This pattern may reflect the larger vocabulary of XLM-R (250,000 tokens vs. 119,547 for mBERT), which provides finer-grained subword segmentation and more opportunity for cross-lingual token sharing. With a larger vocabulary, languages that share morphological roots or suffixes are more likely to share specific subword tokens, amplifying the VOR signal.

The absolute transfer success is also higher for XLM-R than mBERT across all language pairs and tasks, consistent with the literature (Conneau et al., 2020). However, the relative ranking of language pairs by transfer success is highly correlated between the two models ($\rho > 0.90$ for all three tasks), indicating that the factors determining transfer difficulty are largely model-independent.

\section{Discussion}

\subsection{Tokenizer as Bottleneck}

Our results support the interpretation that the tokenizer functions as a bottleneck for cross-lingual transfer: the model cannot exploit typological similarity if the tokenizer maps texts into disjoint token sets. This extends Rust et al. (2021), who showed monolingual performance degrades with poor tokenizer coverage, to the cross-lingual setting. The mechanism is direct: shared subword tokens receive updated embeddings during source-language fine-tuning that transfer immediately to the target language at inference time.

\subsection{Implications for Multilingual Model Design}

Current tokenizer training procedures (BPE, SentencePiece) optimize for compression efficiency rather than cross-lingual token sharing. Tokenizers trained with an explicit VOR-maximization objective might achieve better transfer, though increasing VOR for one language pair reduces vocabulary budget for others. Languages with different scripts face a fundamental lower bound on VOR that no training objective can overcome.

\subsection{Limitations}

First, our analysis is correlational. We observe that VOR predicts transfer success better than typological distance, but we cannot establish a causal relationship. Intervening on VOR directly (e.g., by training tokenizers with controlled vocabulary overlap) would provide stronger causal evidence, but is computationally expensive and beyond the scope of this work.

Second, our language sample is biased toward high-resource languages with good evaluation data. The 11 languages include 7 Indo-European languages, creating a phylogenetic bias that may inflate the apparent predictive power of VOR (since cognate vocabulary is a major source of VOR within language families). Extending the analysis to a more typologically diverse sample including Bantu, Austronesian, or Sino-Tibetan languages would test the generalizability of our findings.

Third, we define VOR using Jaccard similarity of active token sets, which treats all tokens equally regardless of frequency. A frequency-weighted version of VOR (e.g., using token frequency distributions rather than sets) might capture more nuanced patterns of cross-lingual overlap. We leave this extension to future work.

Fourth, our analysis uses zero-shot transfer only. Few-shot transfer, where a small amount of target-language data is available for fine-tuning, might reduce the importance of VOR by providing the model with target-language token-task associations directly. The relative importance of VOR vs. typological distance may shift as the amount of target-language supervision increases.

Fifth, we report transfer success as a ratio relative to English in-language performance. This normalization assumes that in-language performance provides a meaningful ceiling, but in-language performance itself varies across languages due to evaluation set difficulty, annotation quality, and language-specific properties. An alternative normalization using human performance ceilings would be more principled but is not available for most language-task combinations.

\section{Conclusion}

We have shown that Vocabulary Overlap Ratio, a simple Jaccard similarity between the subword token sets of two languages under a multilingual tokenizer, predicts cross-lingual transfer success with higher rank correlation than typological distance from lang2vec in the majority of task-model combinations. The advantage is largest for named entity recognition, where shared named entities create direct token overlap, and smallest for part-of-speech tagging, where syntactic structure matters more. These findings suggest that the tokenizer functions as a proximate bottleneck for cross-lingual transfer, mediating the relationship between typological similarity and transfer outcomes. Improving cross-lingual token sharing through tokenizer design may be a more direct route to better transfer than bridging typological gaps through architectural innovations.

\section{References}

1. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzman, F., Grave, E., Ott, M., Zettlemoyer, L. and Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8440-8451.

2. Conneau, A., Rinber, R., Lample, G., Williams, A., Bowman, S.R., Schwenk, H. and Stoyanov, V. (2018). XNLI: Evaluating cross-lingual sentence representations. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2475-2485.

3. Devlin, J., Chang, M.W., Lee, K. and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 4171-4186.

4. Dryer, M.S. and Haspelmath, M. (2013). The World Atlas of Language Structures Online. Max Planck Institute for Evolutionary Anthropology, Leipzig.

5. K, K., Wang, X., Genzel, D. and Roth, D. (2020). Cross-lingual ability of multilingual BERT: An empirical study. Proceedings of the 8th International Conference on Learning Representations.

6. Littell, P., Mortensen, D.R., Lin, K., Kula, K., and Levin, L. (2017). URIEL and lang2vec: Representing languages as typological, phylogenetic, and identity vectors. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 8-14.

7. Nivre, J. et al. (2020). Universal Dependencies v2: An evergrowing multilingual treebank collection. Proceedings of the 12th Language Resources and Evaluation Conference, 4034-4043.

8. Pan, X., Zhang, B., May, J., Nothman, J., Knight, K. and Ji, H. (2017). Cross-lingual name tagging and linking for 282 languages. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 1946-1958.

9. Pires, T., Schlinger, E. and Garrette, D. (2019). How multilingual is multilingual BERT? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4996-5001.

10. Rust, P., Pfeiffer, J., Vulić, I., Ruder, S. and Gurevych, I. (2021). How good is your tokenizer? On the monolingual performance of multilingual language models. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 3118-3135.

11. Steiger, J.H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2), 245-251.

12. Wu, S. and Dredze, M. (2019). Beto, Bentz, Becas: The surprising cross-lingual effectiveness of BERT. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 833-844.