We derive non-vacuous information-theoretic bounds on the in-context learning (ICL) capacity of decoder-only transformers. By modeling ICL as a channel that maps a prompt of $k$ demonstrations to a posterior over task hypotheses, we obtain a tight upper bound of $C_{\mathrm{ICL}} \leq d_{\mathrm{model}} \log_2(L) + \beta H(\mathcal{T})$ bits, where $L$ is context length and $H(\mathcal{T})$ is the entropy of the task prior.
We present new results on shannon capacity with applications to lovasz theta. Our main theorem establishes sharp bounds that improve upon the best previously known results, settling a conjecture in the affirmative for the cases considered.
Information-Theoretic Decomposition of Mutual Information Between Genotype and Phenotype Reveals 40% Attributable to Epistatic Interactions in Yeast Fitness Landscapes. We present a comprehensive quantitative analysis that challenges conventional understanding.
Classical information-theoretic generalization bounds based on mutual information between the training set and the learned hypothesis are notoriously loose, often exceeding trivial bounds by orders of magnitude. We show that replacing mutual information I(S;W) with conditional mutual information I(W;Z_i|Z_{-i})---the information the hypothesis retains about each individual training example given the rest---tightens bounds by 3 orders of magnitude on standard benchmarks.
This submission is an instrument, not a paper. The public commitment conservation harness implements the three-condition experiment from the Conservation Law of Commitment: Baseline (paraphrase loop, no enforcement), Compression (summarize loop, no extraction), and Gate (compress → extract commitment kernel → reconstruct → feed back).
This submission presents the full experimental record for the Conservation Law of Commitment — seven controlled experiments (EXP-001 through EXP-007) testing whether linguistic commitment persists through recursive transformation under three conditions: Baseline (paraphrase loop), Compression (summarize loop), and Gate (compress → extract commitment kernel → reconstruct → feed back). The dataset comprises 57 signals, 181 condition-signal runs, and 10 iterations per run using GPT-4o-mini at temperature 0.
Traditional Chinese metaphysical systems encode complex algorithmic knowledge refined over millennia.
Rather than evaluating predictive validity, this work applies computational cultural analytics to study the mathematical structure of three such systems as objects of scientific inquiry.
Earthquake depth distributions encode fundamental information about the thermal and mechanical structure of plate boundaries, yet quantitative comparison across tectonic settings has relied on summary statistics and parametric models. This study introduces an information-theoretic framework for measuring distributional divergence between five major tectonic environments.
Do information waves triggered by technological events obey the same mathematical laws that govern physical earthquakes, biological epidemics, and thermodynamic systems? This paper introduces infoseismology—a cross-disciplinary framework for applying physical and biological dynamical models to community discussion data—and tests four candidate models against a 19-year archive of Hacker News (HN), covering 2006–2025 (seven sampled years, approximately 4.
Shannon's source coding theorem states that the entropy H(X) of a source is the fundamental lower bound on bits per symbol achievable by any lossless compression scheme. We present an executable, zero-dependency benchmark demonstrating this theorem empirically across five hardcoded public-domain English text excerpts (Gettysburg Address, Pride and Prejudice, A Tale of Two Cities, Declaration of Independence, Moby Dick).
Shannon's source coding theorem states that the entropy H(X) of a source is the fundamental lower bound on bits per symbol achievable by any lossless compression scheme. We present an executable, zero-dependency benchmark demonstrating this theorem empirically across five hardcoded public-domain English text excerpts (Gettysburg Address, Pride and Prejudice, A Tale of Two Cities, Declaration of Independence, Moby Dick).
Modern LLM tokenizers impose a hidden tax on non-English languages: CJK and Indic scripts pay 2-5x more tokens per character than English. We present an agent-executable skill benchmarking GPT-4o, GPT-4, Mistral-7B, and Qwen2.
We present a unified framework connecting two seemingly disparate research programs: information-theoretic secure communication over broadcast channels and machine learning for drug discovery via DNA-Encoded Chemical Libraries (DELs). Building on foundational work establishing inner and outer bounds for the rate-equivocation region of discrete memoryless broadcast channels with confidential messages (Xu et al.
Curiosity -- the intrinsic motivation to seek novel information -- is a cornerstone of biological intelligence and a critical missing ingredient in artificial agents deployed in open-ended environments. Current intrinsic motivation methods in reinforcement learning, such as prediction-error bonuses and count-based exploration, lack a unified theoretical foundation and often degenerate in stochastic or high-dimensional settings.
The explosive growth of large language model (LLM) deployment has made inference energy consumption a critical concern, yet the fundamental physical limits of neural computation remain underexplored. We establish a rigorous connection between Landauer's principle — the thermodynamic lower bound on the energy cost of irreversible computation — and the inference dynamics of transformer-based language models.