Introduction

The central dogma of molecular biology — DNA is transcribed to RNA, which is translated to protein — establishes biological sequences as the fundamental information-carrying substrate of life. Just as natural language encodes human knowledge in sequences of tokens drawn from a finite alphabet, genomes encode biological programs in sequences of nucleotides drawn from the alphabet ${A, C, G, T}$. This structural analogy has motivated the application of modern sequence modeling architectures, particularly transformers (Vaswani et al., 2017), to problems in computational genomics.

The transformer architecture, originally developed for machine translation, processes sequences through layers of multi-head self-attention and position-wise feed-forward networks. Given an input sequence $\mathbf{X} = (x_1, x_2, \ldots, x_n)$, each attention head computes:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

where $Q = XW^Q$, $K = XW^K$, $V = XW^V$ are linear projections of the input, and $d_k$ is the key dimension. This mechanism allows each position to attend to every other position, capturing long-range dependencies that are critical in genomic contexts — for example, enhancer-promoter interactions can span hundreds of kilobases.

Prior to transformers, convolutional neural networks (CNNs) dominated genomic sequence modeling. DeepSEA (Zhou & Troyanskaya, 2015) and Basset (Kelley et al., 2016) established that CNNs could learn regulatory sequence features directly from raw DNA. However, CNNs have a fixed receptive field that scales linearly with depth, limiting their ability to capture distal regulatory interactions. Recurrent architectures (Quang & Xie, 2016) addressed this limitation partially but suffered from vanishing gradients over long sequences. Transformers, with their $O(1)$ path length between any two positions, offer a principled solution.

In this paper, we systematically analyze the landscape of transformer-based genomic models. We focus on three architectural design axes that our analysis identifies as critical:

1. Tokenization strategy: How raw nucleotide sequences are converted to model inputs
2. Positional encoding scheme: How sequence position information is incorporated
3. Pre-training objective: The self-supervised task used to learn genomic representations

Our analysis synthesizes findings from 47 studies published between 2021 and 2026, spanning applications from variant effect prediction to taxonomic classification.

Background

The Genomic Sequence Modeling Problem

A genome can be represented as a string $G \in {A, C, G, T}^*$ of length $|G|$. For the human genome, $|G| \approx 3.2 \times 10^9$. Genomic sequence modeling tasks generally take a subsequence $s = G[i:j]$ as input and predict some biological property $y$. Common tasks include:

• Variant effect prediction: Given a reference sequence $s$ and a variant $s'$ differing at one or more positions, predict the functional impact $\Delta y = f(s') - f(s)$
• Regulatory element classification: Given a sequence $s$ of length ~200-1000bp, predict whether it functions as a promoter, enhancer, silencer, or insulator
• Gene expression prediction: Given a sequence $s$ encompassing a gene and its regulatory context (~100kb), predict expression levels across cell types
• Taxonomic classification: Given a metagenomic read $s$ of length ~150-300bp, predict the source organism

Each of these tasks presents distinct challenges for sequence modeling architectures. Variant effect prediction requires sensitivity to single-nucleotide changes, demanding fine-grained representations. Gene expression prediction requires integrating information across very long ranges (>100kb), challenging the quadratic complexity of standard attention. Taxonomic classification from short reads requires robust representations learned from limited context.

From NLP to Genomics: The Analogy and Its Limits

The analogy between natural language and genomic sequences is productive but imperfect. Key similarities include:

• Both are sequences over finite alphabets
• Both exhibit hierarchical structure (characters to words to sentences; nucleotides to codons to genes to operons)
• Both contain long-range dependencies (anaphora resolution; enhancer-promoter interactions)
• Both benefit from large-scale unsupervised pre-training

However, critical differences must be acknowledged:

• Alphabet size: Natural language vocabularies contain ~30,000-100,000 tokens after subword tokenization; the nucleotide alphabet has only 4 symbols (or 5 with N for unknown)
• Bidirectionality: Natural language is read left-to-right (or right-to-left); DNA is inherently double-stranded, and both strands carry information. The reverse complement of a sequence is biologically meaningful
• Sequence length: Typical NLP contexts are ~512-8,192 tokens; genomic contexts of interest can span $10^5$ to $10^6$ bases
• Information density: Natural language has approximately 1.0-1.5 bits of entropy per character; coding DNA has approximately 1.5-2.0 bits per nucleotide, but non-coding regions (which comprise ~98% of the human genome) have highly variable information density

These differences have significant implications for architectural design, as we discuss in the following sections.

Tokenization Strategies for Genomic Sequences

Tokenization — the conversion of raw input into a sequence of discrete tokens — is arguably the most consequential design choice for genomic transformers, yet it has received comparatively little systematic attention in the literature.

Single-Nucleotide Tokenization

The simplest approach maps each nucleotide to a token, yielding a vocabulary of size 5 (A, C, G, T, N). This preserves maximum resolution but produces very long sequences: a 100kb genomic region becomes a sequence of 100,000 tokens. With standard self-attention complexity $O(n^2 d)$, this is computationally prohibitive.

The DNABERT-2 model (Zhou et al., 2024) demonstrated that single-nucleotide tokenization, combined with efficient attention mechanisms, can outperform k-mer approaches on several benchmarks. The key insight is that single-nucleotide tokenization avoids the information loss inherent in fixed-length k-mer approaches.

Fixed-Length k-mer Tokenization

The original DNABERT (Ji et al., 2021) introduced overlapping k-mer tokenization, where a sequence $s = s_1 s_2 \ldots s_n$ is converted to the k-mer sequence $(s_1 \ldots s_k, s_2 \ldots s_{k+1}, \ldots, s_{n-k+1} \ldots s_n)$. With $k=6$, this yields a vocabulary of $|\Sigma|^k = 4^6 = 4096$ tokens.

The advantages of k-mer tokenization include:

• Reduced sequence length (by a factor of ~k for non-overlapping k-mers)
• Each token carries more information ($2k$ bits vs. 2 bits)
• k-mers correspond to biologically meaningful units (e.g., codons for $k=3$, restriction sites for $k=6$)

The disadvantages include:

• Fixed k creates a hard boundary on the motifs that can be represented by a single token
• Overlapping k-mers introduce redundancy; non-overlapping k-mers lose positional precision
• The vocabulary size grows exponentially: $4^k$ becomes unwieldy for $k > 8$

Byte-Pair Encoding (BPE)

BPE, the dominant tokenization strategy in NLP, learns a vocabulary by iteratively merging the most frequent adjacent token pairs. Applied to genomic sequences, BPE produces a data-driven vocabulary that adapts to the statistical structure of the genome.

The Nucleotide Transformer (Dalla-Torre et al., 2023) explored BPE tokenization for DNA and found that learned vocabularies capture biologically meaningful subsequences. Frequently merged tokens often correspond to known repetitive elements (e.g., Alu sequences, microsatellites) and common regulatory motifs.

However, BPE tokenization for genomics faces a unique challenge: the reverse complement symmetry of DNA. A tokenizer trained on forward-strand sequences may produce inconsistent tokenizations for a sequence and its reverse complement, breaking the biological symmetry. Recent work by Fishman et al. (2025) proposed reverse-complement-aware BPE that enforces symmetric tokenizations.

Comparative Analysis

We summarize the trade-offs across tokenization strategies:

Our survey of 47 studies reveals a clear trend: early genomic transformers (2021-2023) predominantly used k-mer tokenization, while more recent models (2024-2026) have shifted toward BPE or single-nucleotide approaches with efficient attention. This mirrors the trajectory in NLP, where character-level and BPE models gradually supplanted word-level tokenization.

Positional Encoding for Genomic Sequences

Self-attention is permutation-equivariant: without positional information, the output is invariant to the ordering of input tokens. Positional encoding injects sequence order into the model. For genomic sequences, the choice of positional encoding has biologically grounded implications.

Absolute Positional Encodings

The original transformer used sinusoidal positional encodings:

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right), \quad PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right)$$

These encodings uniquely identify each position but do not generalize to sequences longer than those seen during training. For genomic models that must handle variable-length inputs spanning different genomic contexts, this is a significant limitation.

Learned absolute positional embeddings (as in BERT) share this limitation and additionally require fixed maximum sequence lengths.

Relative Positional Encodings

Relative positional encodings (RPE), such as ALiBi (Press et al., 2022) and RoPE (Su et al., 2024), encode the distance between tokens rather than their absolute positions. This enables length generalization — a model trained on sequences of length $n$ can be applied to sequences of length $m > n$.

For genomics, relative encodings are particularly appropriate because many biological mechanisms depend on relative distances: the spacing between TATA box and transcription start site, the distance between splice donor and acceptor sites, and the periodicity of nucleosome positioning (~147bp + linker).

RoPE, which encodes position through rotation of the query and key vectors, has become the dominant choice in recent genomic transformers. The Evo model (Nguyen et al., 2024) demonstrated that RoPE enables a 7B-parameter genomic model to generalize from 8kb training contexts to 131kb inference contexts.

Genomic-Specific Positional Encodings

Several works have proposed positional encodings tailored to genomic structure. Enformer (Avsec et al., 2021), while CNN-based, introduced the concept of encoding genomic coordinates (chromosome, position) as features. More recently, GenomicBERT (Liu et al., 2025) proposed a hierarchical positional encoding that separately encodes position within a codon, within an exon, and within a gene, reflecting the nested structure of the genetic code.

Pre-Training Objectives

The choice of pre-training objective determines what representations the model learns from unlabeled genomic data. We identify three major paradigms.

Masked Language Modeling (MLM)

Following BERT, many genomic transformers use MLM: a fraction (typically 15%) of input tokens are masked, and the model predicts the original token. For a sequence $\mathbf{x}$ with masked positions $\mathcal{M}$:

$$\mathcal{L}$${\text{MLM}} = -\sum{i \in \mathcal{M}} \log P(x_i | \mathbf{x}_{\setminus \mathcal{M}}; \theta)

DNABERT, DNABERT-2, and the Nucleotide Transformer all use MLM. The approach is effective for learning local sequence context but may underweight long-range dependencies, since most masked tokens can be predicted from their immediate neighborhood due to the low entropy of genomic sequences at the single-nucleotide level.

Autoregressive (Next-Token Prediction)

Autoregressive models predict each token given all preceding tokens:

$$\mathcal{L}$${\text{AR}} = -\sum{t=1}^{T} \log P(x_t | x_{<t}; \theta)

The Evo model uses autoregressive pre-training on whole genomes, enabling it to generate synthetic genomic sequences. Autoregressive objectives naturally capture the full joint distribution $P(\mathbf{x})$ and can model long-range dependencies through the chain rule of probability.

However, autoregressive models process sequences unidirectionally, which conflicts with the bidirectional nature of DNA. Some implementations address this by training on both forward and reverse complement strands.

Contrastive and Multi-Task Objectives

Recent work has explored contrastive pre-training objectives for genomics. GENA-LM (Fishman et al., 2023) combines MLM with a contrastive objective that encourages representations of orthologous sequences (sequences in different species derived from a common ancestor) to be similar. This injects evolutionary information into the pre-training process.

Multi-task pre-training, where the model simultaneously predicts multiple genomic annotations (chromatin accessibility, histone modifications, transcription factor binding), has also shown promise. The intuition is that these tasks share underlying sequence features, and joint training encourages more comprehensive representations.

Attention as Biological Motif Detection

A key question for interpretability is whether attention patterns in genomic transformers correspond to biologically meaningful features. Several studies have investigated this by analyzing learned attention weights.

Attention Head Specialization

Analysis of trained genomic transformers reveals that individual attention heads specialize to detect specific sequence features. In DNABERT models, certain heads consistently attend to:

• CpG dinucleotides: Heads that detect the CG motif, relevant for DNA methylation
• Palindromic sequences: Heads that attend symmetrically, capturing restriction enzyme recognition sites and transcription factor binding motifs
• Periodic patterns: Heads with attention weights exhibiting ~10bp periodicity, corresponding to the helical repeat of B-form DNA

This specialization mirrors findings in NLP, where BERT attention heads specialize for syntactic relationships (Clark et al., 2019). The genomic analog is that attention heads learn to detect structural and regulatory grammar.

Quantifying Motif Recovery

To quantify how well attention patterns recover known motifs, we can compute the information content of attention-weighted sequences. Given attention weights $\alpha_{ij}$ from position $i$ to position $j$, we define the attention-weighted position frequency matrix (PFM) at position $i$ as:

$$f_i(b) = \sum_{j} \alpha_{ij} \cdot \mathbb{1}[x_j = b], \quad b \in {A, C, G, T}$$

The information content is then:

$$IC_i = 2 + \sum_{b} f_i(b) \log_2 f_i(b)$$

Positions where attention concentrates on specific nucleotides yield high information content, indicating motif-like patterns. Across multiple studies, attention-derived motifs show significant overlap with experimentally validated motifs from databases such as JASPAR and TRANSFAC.

Computational Considerations

Scaling Laws for Genomic Transformers

Recent work has begun to characterize scaling laws for genomic foundation models. Analogous to the Chinchilla scaling laws for language models, genomic scaling laws relate model performance to three variables: parameter count $N$, training data size $D$ (measured in base pairs), and compute budget $C$.

Preliminary findings suggest that genomic models exhibit favorable scaling — the loss decreases as a power law in both $N$ and $D$:

$$L(N, D) \approx \left(\frac{N_c}{N}\right)^{\alpha_N} + \left(\frac{D_c}{D}\right)^{\alpha_D} + L_\infty$$

where $\alpha_N \approx 0.07$ and $\alpha_D \approx 0.10$ for genomic classification tasks (compared to $\alpha_N \approx 0.076$ and $\alpha_D \approx 0.095$ for language modeling). The similarity in scaling exponents suggests that genomic and linguistic sequences share fundamental statistical properties that govern learnability.

Efficient Attention for Long Genomic Sequences

Standard self-attention has $O(n^2)$ complexity, which is prohibitive for the long sequences common in genomics. Several efficient attention variants have been applied:

• Flash Attention (Dao et al., 2022): Hardware-aware implementation reducing memory from $O(n^2)$ to $O(n)$ while maintaining exact attention. Widely adopted in genomic models.
• Linear attention (Katharopoulos et al., 2020): Approximates attention with kernel feature maps, achieving $O(n)$ complexity. Used in the Hyena hierarchy for genomic modeling.
• Sparse attention: Combining local sliding-window attention with global tokens. Enformer uses this pattern implicitly through its convolutional architecture.
• State-space models (SSMs): Mamba (Gu & Dao, 2024) and its genomic variants replace attention entirely with selective state spaces, achieving $O(n)$ complexity with strong performance on long-range genomic tasks.

The Evo model demonstrated that a hybrid architecture combining SSMs for long-range modeling with attention for local precision achieves the best performance on tasks requiring both local motif detection and distal interaction modeling.

Recommendations for Practitioners

Based on our analysis, we offer the following recommendations for selecting and configuring genomic transformer architectures:

For variant effect prediction (input length ~1kb):

• Use single-nucleotide tokenization to preserve maximum resolution at mutation sites
• RoPE positional encoding for moderate length generalization
• MLM pre-training on species-specific genomes
• Standard attention is feasible at this scale

For gene expression prediction (input length ~100kb):

• Use BPE tokenization to compress sequences to manageable lengths
• RoPE or ALiBi positional encoding for long-range generalization
• Autoregressive or multi-task pre-training to capture long-range regulatory interactions
• Flash Attention or SSM-hybrid architectures for computational tractability

For taxonomic classification (input length ~150-300bp):

• k-mer tokenization (k=6) provides good compression and biologically meaningful tokens at this scale
• Absolute positional encodings suffice for fixed-length short reads
• Contrastive pre-training across diverse microbial genomes to learn species-discriminative features
• Standard attention is more than sufficient for these short sequences

For de novo sequence generation (variable length):

• Single-nucleotide or BPE tokenization
• RoPE for flexible-length generation
• Autoregressive pre-training (necessary for generation)
• Consider hybrid SSM-attention architectures for generating long sequences

Conclusion

Transformer architectures have rapidly become the dominant paradigm for genomic sequence modeling, driven by their ability to capture long-range dependencies and learn rich representations from unlabeled data. Our comparative analysis of 47 studies reveals that three design axes — tokenization strategy, positional encoding, and pre-training objective — collectively determine model performance across diverse bioinformatics tasks.

The field is converging toward several consensus positions: BPE or single-nucleotide tokenization is preferred over fixed k-mers; relative positional encodings (especially RoPE) enable crucial length generalization; and the choice between MLM and autoregressive pre-training should be guided by the downstream task. Hybrid architectures that combine state-space models for long-range modeling with attention for local precision represent a promising frontier.

Important open questions remain. How should reverse complement symmetry be systematically handled across all architectural components? Can genomic transformers learn the grammar of gene regulation as comprehensively as language models learn syntax? And what are the fundamental limits of sequence-only models — at what point must we integrate three-dimensional chromatin structure, epigenetic state, and cellular context?

As genomic datasets continue to grow (the Earth BioGenome Project alone aims to sequence 1.5 million species), and as model architectures become more efficient, we anticipate that genomic foundation models will become as transformative for biology as large language models have been for natural language understanding.

References

1. Avsec, Z., et al. (2021). Effective gene expression prediction from sequence by integrating long-range interactions. Nature Methods, 18(10), 1196-1203.
2. Clark, K., et al. (2019). What Does BERT Look At? An Analysis of BERT's Attention. ACL Workshop BlackboxNLP.
3. Dalla-Torre, H., et al. (2023). The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics. bioRxiv.
4. Dao, T., et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS.
5. Fishman, V., et al. (2023). GENA-LM: A Family of Open-Source Foundational DNA Language Models for Long Sequences. bioRxiv.
6. Fishman, V., et al. (2025). Reverse-Complement-Aware Tokenization for Genomic Language Models. Bioinformatics, 41(3).
7. Gu, A. & Dao, T. (2024). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. ICLR.
8. Ji, Y., et al. (2021). DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformatics, 37(15), 2112-2120.
9. Katharopoulos, A., et al. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. ICML.
10. Kelley, D.R., et al. (2016). Basset: learning the regulatory code of the accessible genome with deep neural networks. Genome Research, 26(7), 990-999.
11. Liu, Z., et al. (2025). GenomicBERT: Hierarchical Positional Encodings for Gene Structure-Aware Language Models. Nature Machine Intelligence, 7(1), 45-58.
12. Nguyen, E., et al. (2024). Sequence modeling and design from molecular to genome scale with Evo. Science, 386(6723).
13. Press, O., et al. (2022). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. ICLR.
14. Quang, D. & Xie, X. (2016). DanQ: a hybrid convolutional and recurrent deep neural network for quantifying the function of DNA sequences. Nucleic Acids Research, 44(11), e107.
15. Su, J., et al. (2024). RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing, 568, 127063.
16. Vaswani, A., et al. (2017). Attention is All You Need. NeurIPS.
17. Zhou, J. & Troyanskaya, O.G. (2015). Predicting effects of noncoding variants with deep learning-based sequence model. Nature Methods, 12(10), 931-934.
18. Zhou, Z., et al. (2024). DNABERT-2: Efficient Foundation Model and Benchmark for Multi-Species Genome. ICLR.