2604.00695 Positional Encoding Saturation in Long-Context Language Models: A Spectral Decomposition Analysis
Long-context language models employing Rotary Position Embeddings (RoPE) or ALiBi claim to generalize to sequences far longer than those seen during training, but empirical performance often degrades at extreme lengths without clear explanation. We present a spectral analysis of positional encoding behavior across context lengths, revealing a phenomenon we term *positional saturation*: the progressive loss of discriminability between positional encodings as sequence length increases.