Introduction

Zipf's law states that the frequency of a word in natural language is inversely proportional to its rank: $f(r) \propto r^{-\alpha}$ with $\alpha \approx 1$. This power-law regularity underpins assumptions in language modeling, compression, and vocabulary design. However, modern LLM tokenizers use BPE (Byte Pair Encoding) and its variants, which produce token vocabularies that differ fundamentally from word-level distributions.

Recent work has shown that BPE's iterative merging process drives token frequencies toward Zipfian distributions as vocabulary size increases, and that downstream task performance peaks when token distributions most closely follow Zipf's law[zipf2025]. Yet systematic analysis of where and how Zipf's law breaks down across different corpus types remains limited.

This work addresses three questions:

• How do Zipf exponents differ between code and natural language token distributions?
• Where in the rank-frequency curve does the power-law fit break down?
• Does the degree of Zipfian adherence predict tokenizer compression efficiency?

We analyze 36 combinations of 4 tokenizers $\times$ 9 corpora, fitting the Zipf-Mandelbrot model and performing piecewise decomposition into head, body, and tail regions.

Methods

Corpora

We use two corpus types:

• Natural language: Tatoeba parallel sentences for 7 languages (English, German, French, Chinese, Japanese, Arabic, Finnish), 200 sentences per language. Dataset revision pinned for reproducibility.
• Code: Python and Java function bodies from CodeSearchNet, 200 samples per language.

Tokenizers

Four BPE-family tokenizers spanning vocabulary sizes from 32K to 200K: GPT-4o (o200k\_base, 200K vocab), GPT-4 (cl100k\_base, 100K), Qwen2.5-7B (152K), and Mistral-7B (32K). All model revisions are pinned.

Zipf-Mandelbrot Fitting

For each tokenizer-corpus pair, we:

• Tokenize the corpus and compute the rank-frequency distribution.
• Fit the Zipf-Mandelbrot model $f(r) = C/(r+q)^\alpha$ via OLS on log-transformed data, with grid search over $q \in {0, 0.5, 1, 2, 5, 10}$.
• Report the best-fit $\alpha$, $q$, and $R^2$.
• Perform piecewise fitting on head (top 10%), body (10--90%), and tail (bottom 10%) regions.
• Detect breakpoints via sliding window analysis.

Correlation Analysis

We compute Pearson and Spearman correlations between the global Zipf exponent $\alpha$ and the compression ratio (characters per token) across all 36 analyses.

Results

Global Zipf Exponents

Average Zipf exponents (α) and R² by corpus type.

Code produces higher Zipf exponents than natural language (Table). A Mann-Whitney U test confirms this difference is statistically significant ($p < 0.05$). This indicates steeper rank-frequency curves in code: a smaller set of high-frequency tokens (keywords, braces, indentation) dominates, while the long tail of identifiers and string literals drops off more sharply.

Among natural languages, the exponent varies substantially: English has $\alpha \approx 0.84$ across tokenizers, while Arabic with Mistral reaches $\alpha = 2.67$ (reflecting severe tokenization fragmentation). GPT-4o, with its 200K vocabulary, consistently produces the lowest $\alpha$ values, indicating flatter, more uniform token usage.

Piecewise Breakdown

Representative piecewise Zipf exponents (GPT-4o tokenizer).

The piecewise analysis (Table) reveals a universal pattern:

• The head region (most frequent 10% of tokens) has sub-Zipfian exponents ($\alpha < 1$), indicating a flatter distribution than Zipf predicts. High-frequency tokens are more uniformly distributed than expected.
• The body (10--90%) shows the strongest Zipfian behavior, with exponents closest to the global fit. For code, body exponents exceed 1, reflecting the steep drop from keywords to identifiers.
• The tail (least frequent 10%) universally collapses to $\alpha \approx 0$, because most rare tokens appear exactly once (hapax legomena). This frequency plateau represents the primary mode of Zipf breakdown.

In 34 of 36 analyses, the tail $\alpha$ is effectively zero. The two exceptions—Arabic with GPT-4 ($\alpha_\text{tail} = 11.7$) and Mistral ($\alpha_\text{tail} = 8.3$)—exhibit extreme tail collapse where even among rare tokens there is a steep hierarchy, caused by aggressive byte-level fallback tokenization.

Correlation with Compression

The correlation between $\alpha$ and compression ratio is modestly negative in the pooled analysis, but the evidence differs by test:

• Pearson $r = -0.337$ ($p = 0.044$)
• Spearman $\rho = -0.261$ ($p = 0.124$)

Exploratory within-type analyses retain the same sign:

• Natural language only ($n=28$): Pearson $r=-0.568$ ($p=0.0016$), Spearman $\rho=-0.571$ ($p=0.0015$)
• Code only ($n=8$): Pearson $r=-0.701$ ($p=0.0526$), Spearman $\rho=-0.857$ ($p=0.0065$)

This pattern is counterintuitive and should still be interpreted cautiously because the code subset is small and one of two code-side tests is marginal. Higher $\alpha$ (steeper, more Zipfian distributions) is associated with lower compression ratios, both overall and in these subgroup analyses. The explanation lies in what drives high $\alpha$: it often indicates that the tokenizer fragments text into many single-use tokens (e.g., Arabic with Mistral), producing a steep rank-frequency curve but poor compression. Conversely, tokenizers with large, well-allocated vocabularies (e.g., GPT-4o) produce flatter distributions ($\alpha \approx 0.8$) and better compression, because their vocabulary efficiently covers the input space.

Discussion

Our findings challenge the simple narrative that "more Zipfian = better." While prior work shows that vocabulary expansion drives token distributions toward Zipf's law, our cross-corpus analysis reveals that the direction of this relationship is confounded by vocabulary adequacy.

The three-regime structure (flat head, Zipfian body, collapsed tail) appears universal across corpus types and tokenizers. This suggests that BPE tokenization inherently produces distributions that only approximate Zipf's law in the mid-frequency range, with systematic deviations at both extremes.

The code-vs-language difference ($\Delta\alpha \approx 0.34$) is robust across all four tokenizers and likely reflects the lower lexical diversity of programming languages, where a fixed set of keywords and syntactic elements dominates the frequency distribution.

Limitations

• Corpus sizes are small (200 samples per language), limiting statistical power for tail analysis.
• Only BPE-family tokenizers are tested; unigram and WordPiece tokenizers may show different behavior.
• OLS on log-log data introduces known biases compared to MLE; our results are valid for comparative analysis but absolute $\alpha$ values may be slightly biased.
• The negative alpha-compression correlation may partly reflect confounding by language script complexity rather than a causal relationship.

Conclusion

We provide the first systematic cross-corpus analysis of Zipf's law adherence in BPE token distributions. Our key findings are: (1) code has $~$33% higher Zipf exponents than natural language; (2) Zipf's law holds best in the mid-frequency body but universally breaks down in the tail; (3) higher Zipf exponents can coincide with worse compression, contradicting naive expectations. The entire analysis is reproducible via an agent-executable SKILL.md.

References

• [zipf2025] Pre-trained Models Perform the Best When Token Distributions Follow Zipf's Law. arXiv:2507.22543, 2025.

• [piantadosi2014] S. T. Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic Bulletin & Review, 21(5):1112--1130, 2014.

• [zipf1949] G. K. Zipf. Human Behavior and the Principle of Least Effort. Addison-Wesley, 1949.

• [sennrich2016] R. Sennrich, B. Haddow, and A. Birch. Neural Machine Translation of Rare Words with Subword Units. ACL, 2016.