---
name: shannon-entropy-bound
description: >
  Empirically demonstrates Shannon's source coding theorem: entropy is the lower bound
  for lossless compression. Hardcodes 5 famous public-domain text excerpts (Gettysburg
  Address, Pride and Prejudice, A Tale of Two Cities, Declaration of Independence, Moby
  Dick) as Python string constants. Computes character-level H1, bigram H2_per_char, and
  trigram H3_per_char Shannon entropy for each text, compresses with zlib (stdlib), and
  verifies that H1 > H2_per_char > H3_per_char (monotonic convergence) and that zlib
  achieves below H1 bits/char. Zero pip installs, zero network, fully deterministic.
  Triggers: Shannon entropy, source coding theorem, entropy bound, compression ratio,
  n-gram entropy, information theory benchmark.
allowed-tools: Bash(python3 *), Bash(mkdir *), Bash(cat *), Bash(cd *)
---

# Shannon's Entropy Bound

Empirically tests Shannon's source coding theorem: the entropy H(X) of a source is the
theoretical lower bound on bits per symbol achievable by any lossless compressor.

For 5 famous public-domain English text excerpts, this skill computes:
- **H₁** (bits/char): character-level (unigram) Shannon entropy
- **H₂_per_char**: joint bigram entropy divided by 2 — per-character estimate from bigram model
- **H₃_per_char**: joint trigram entropy divided by 3 — per-character estimate from trigram model
- **zlib_bits_per_char**: actual compression ratio using `zlib.compress()` at level 9

Expected result: H₁ > H₂_per_char > H₃_per_char for all texts (monotonic convergence as
n-gram order increases), and zlib bits/char falls below H₁ (LZ77-based compression
outperforms the unigram entropy bound). All data is hardcoded — no network access required.

---

## Step 1: Setup Workspace

```bash
mkdir -p workspace && cd workspace
mkdir -p scripts output
```

Expected output:
```
(no terminal output — directories created silently)
```

---

## Step 2: Write and Run Entropy Analysis Script

```bash
cd workspace
cat > scripts/analyze.py <<'PY'
#!/usr/bin/env python3
"""Shannon entropy bound benchmark.

Computes character/bigram/trigram entropy for 5 hardcoded public-domain
text excerpts and compares against zlib compression ratios. Demonstrates
Shannon's source coding theorem empirically.

All texts are public domain in the United States and worldwide.
"""
import json
import math
import zlib
from collections import Counter

# ── Configurable parameters ────────────────────────────────────────────────────
OUTPUT_FILE = "output/results.json"

# ── Hardcoded public-domain text excerpts ─────────────────────────────────────
TEXTS = {
    "gettysburg": (
        "Four score and seven years ago our fathers brought forth on this continent, "
        "a new nation, conceived in Liberty, and dedicated to the proposition that all men "
        "are created equal.\n\n"
        "Now we are engaged in a great civil war, testing whether that nation, or any nation "
        "so conceived and so dedicated, can long endure. We are met on a great battle-field "
        "of that war. We have come to dedicate a portion of that field, as a final resting "
        "place for those who here gave their lives that that nation might live. It is "
        "altogether fitting and proper that we should do this.\n\n"
        "But, in a larger sense, we can not dedicate -- we can not consecrate -- we can not "
        "hallow -- this ground. The brave men, living and dead, who struggled here, have "
        "consecrated it, far above our poor power to add or detract. The world will little "
        "note, nor long remember what we say here, but it can never forget what they did "
        "here. It is for us the living, rather, to be dedicated here to the unfinished work "
        "which they who fought here have thus far so nobly advanced. It is rather for us to "
        "be here dedicated to the great task remaining before us -- that from these honored "
        "dead we take increased devotion to that cause for which they gave the last full "
        "measure of devotion -- that we here highly resolve that these dead shall not have "
        "died in vain -- that this nation, under God, shall have a new birth of freedom -- "
        "and that government of the people, by the people, for the people, shall not perish "
        "from the earth."
    ),
    "pride_and_prejudice": (
        "It is a truth universally acknowledged, that a single man in possession of a good "
        "fortune, must be in want of a wife.\n\n"
        "However little known the feelings or views of such a man may be on his first "
        "entering a neighbourhood, this truth is so well fixed in the minds of the "
        "surrounding families, that he is considered as the rightful property of some one "
        "or other of their daughters.\n\n"
        "\"My dear Mr. Bennet,\" said his lady to him one day, \"have you heard that "
        "Netherfield Park is let at last?\"\n\n"
        "Mr. Bennet replied that he had not.\n\n"
        "\"But it is,\" returned she; \"for Mrs. Long has just been here, and she told me "
        "all about it.\"\n\n"
        "Mr. Bennet made no answer.\n\n"
        "\"Do not you want to know who has taken it?\" cried his wife impatiently.\n\n"
        "\"You want to tell me, and I have no objection to hearing it.\"\n\n"
        "This was invitation enough.\n\n"
        "\"Why, my dear, you must know, Mrs. Long says that Netherfield is taken by a young "
        "man of large fortune from the north of England; that he came down on Monday in a "
        "chaise and four to see the place, and was so much delighted with it that he agreed "
        "with Mr. Morris immediately; that he is to take possession before Michaelmas, and "
        "some of his servants are to be in the house by the end of next week.\"\n\n"
        "\"What is his name?\"\n\n"
        "\"Bingley.\"\n\n"
        "\"Is he married or single?\"\n\n"
        "\"Oh! single, my dear, to be sure! A single man of large fortune; four or five "
        "thousand a year. What a fine thing for our girls!\"\n\n"
        "\"How so? How can it affect them?\"\n\n"
        "\"My dear Mr. Bennet,\" replied his wife, \"how can you be so tiresome! You must "
        "know that I am thinking of his marrying one of them.\"\n\n"
        "\"Is that his design in settling here?\"\n\n"
        "\"Design! Nonsense, how can you talk so! But it is very likely that he may fall "
        "in love with one of them, and therefore you must visit him as soon as he comes.\""
    ),
    "tale_of_two_cities": (
        "It was the best of times, it was the worst of times, it was the age of wisdom, "
        "it was the age of foolishness, it was the epoch of belief, it was the epoch of "
        "incredulity, it was the season of Light, it was the season of Darkness, it was "
        "the spring of hope, it was the winter of despair, we had everything before us, "
        "we had nothing before us, we were all going direct to Heaven, we were all going "
        "direct the other way -- in short, the period was so far like the present period, "
        "that some of its noisiest authorities insisted on its being received, for good "
        "or for evil, in the superlative degree of comparison only.\n\n"
        "There were a king with a large jaw and a queen with a plain face, on the throne "
        "of England; there were a king with a large jaw and a queen with a fair face, on "
        "the throne of France. In both countries it was clearer than crystal to the lords "
        "of the State preserves of loaves and fishes, that things in general were settled "
        "for ever.\n\n"
        "It was the year of Our Lord one thousand seven hundred and seventy-five. Spiritual "
        "revelations were conceded to England at that favoured period, as at this. Mrs. "
        "Southcott had recently attained her five-and-twentieth birthday, to the "
        "immense joy of a numerous sect, who had long ago -- among other marvels -- foretold "
        "her arrival as the Second Advent of a personage of greater importance than "
        "the Apostles. Mere messages in the earthly order of events had lately become "
        "the talk of the town: a prophetic private in the Life Guards had heralded the "
        "sublime appearance, by announcing that arrangements were made for the "
        "swallowing up of London and Westminster."
    ),
    "declaration_of_independence": (
        "When in the Course of human events, it becomes necessary for one people to "
        "dissolve the political bands which have connected them with another, and to assume "
        "among the powers of the earth, the separate and equal station to which the Laws "
        "of Nature and of Nature's God entitle them, a decent respect to the opinions of "
        "mankind requires that they should declare the causes which impel them to the "
        "separation.\n\n"
        "We hold these truths to be self-evident, that all men are created equal, that "
        "they are endowed by their Creator with certain unalienable Rights, that among "
        "these are Life, Liberty and the pursuit of Happiness. -- That to secure these "
        "rights, Governments are instituted among Men, deriving their just powers from the "
        "consent of the governed, -- That whenever any Form of Government becomes "
        "destructive of these ends, it is the Right of the People to alter or to abolish "
        "it, and to institute new Government, laying its foundation on such principles and "
        "organizing its powers in such form, as to them shall seem most likely to effect "
        "their Safety and Happiness. Prudence, indeed, will dictate that Governments long "
        "established should not be changed for light and transient causes; and accordingly "
        "all experience hath shewn, that mankind are more disposed to suffer, while evils "
        "are sufferable, than to right themselves by abolishing the forms to which they "
        "are accustomed. But when a long train of abuses and usurpations, pursuing "
        "invariably the same Object evinces a design to reduce them under absolute "
        "Despotism, it is their right, it is their duty, to throw off such Government, "
        "and to provide new Guards for their future security."
    ),
    "moby_dick": (
        "Call me Ishmael. Some years ago -- never mind how long precisely -- having little "
        "money in my purse, and nothing particular to interest me on shore, I thought I "
        "would sail about a little and see the watery part of the world. It is a way I "
        "have of driving off the spleen and regulating the circulation. Whenever I find "
        "myself growing grim about the mouth; whenever it is a damp, drizzly November in "
        "my soul; whenever I find myself involuntarily pausing before coffin warehouses, "
        "and bringing up the rear of every funeral I meet; and especially whenever my "
        "hypos get such an upper hand of me, that it requires a strong moral principle to "
        "prevent me from deliberately stepping into the street, and methodically knocking "
        "people's hats off -- then, I account it high time to get to sea as soon as I can. "
        "This is my substitute for pistol and ball. With a philosophical flourish Cato "
        "throws himself upon his sword; I quietly take to the ship. There is nothing "
        "surprising in this. If they only knew it, almost all men in their degree, some "
        "time or other, cherish very nearly the same feelings towards the ocean as I do.\n\n"
        "There now is your insular city of the Manhattoes, belted round by wharves as "
        "Indian isles by coral reefs -- commerce surrounds it with her surf. Right and "
        "left, the streets take you waterward. Its extreme downtown is the battery, where "
        "that noble mole is washed by waves, and cooled by breezes, which a few hours "
        "previous were out of sight of land. Look at the crowds of water-gazers there.\n\n"
        "Circumambulate the city of a dreamy Sabbath afternoon. Go from Corlears Hook to "
        "Coenties Slip, and from thence, by Whitehall, northward. What do you see? -- "
        "Posted like silent sentinels all around the town, stand thousands upon thousands "
        "of mortal men fixed in ocean reveries. Some leaning against the spiles; some "
        "seated upon the pier-heads; some looking over the bulwarks of ships from China; "
        "some high aloft in the rigging, as if striving to get a still better seaward peep. "
        "But these are all landsmen; of week days pent up in lath and plaster -- tied to "
        "counters, nailed to benches, clinched to desks. How then is this? Are the green "
        "fields gone? What do they here?\n\n"
        "But look! here come more crowds, pacing straight for the water, and seemingly "
        "bound for a dive. Strange! Nothing will content them but the extremest limit of "
        "the land; loitering under the shady lee of yonder warehouses will not suffice. "
        "No. They must get just as nigh the water as they possibly can without falling in."
    ),
}


def char_entropy(text):
    """H1: character-level Shannon entropy (bits per character).

    H1 = -sum_c  p(c) * log2(p(c))
    """
    n = len(text)
    counts = Counter(text)
    return -sum((cnt / n) * math.log2(cnt / n) for cnt in counts.values())


def bigram_entropy_per_char(text):
    """H2_per_char: per-character entropy from the bigram model.

    Computes the joint entropy H(C1, C2) = -sum p(c1,c2) * log2(p(c1,c2))
    then divides by 2 to get a per-character figure.
    """
    n = len(text)
    bigrams = [text[i:i + 2] for i in range(n - 1)]
    counts = Counter(bigrams)
    total = len(bigrams)
    h_joint = -sum((cnt / total) * math.log2(cnt / total) for cnt in counts.values())
    return h_joint / 2.0


def trigram_entropy_per_char(text):
    """H3_per_char: per-character entropy from the trigram model.

    Computes the joint entropy H(C1, C2, C3) = -sum p(c1,c2,c3) * log2(p(c1,c2,c3))
    then divides by 3 to get a per-character figure.
    """
    n = len(text)
    trigrams = [text[i:i + 3] for i in range(n - 2)]
    counts = Counter(trigrams)
    total = len(trigrams)
    h_joint = -sum((cnt / total) * math.log2(cnt / total) for cnt in counts.values())
    return h_joint / 3.0


def zlib_bits_per_char(text):
    """Compress text with zlib level 9; return bits per character.

    Uses UTF-8 encoding (all texts are pure ASCII, so byte count = char count).
    bits_per_char = len(compressed_bytes) * 8 / len(text_chars)
    """
    original_bytes = text.encode("utf-8")
    compressed_bytes = zlib.compress(original_bytes, level=9)
    return (len(compressed_bytes) * 8) / len(text)


def main():
    text_results = {}
    for name, text in TEXTS.items():
        n = len(text)
        h1 = char_entropy(text)
        h2 = bigram_entropy_per_char(text)
        h3 = trigram_entropy_per_char(text)
        zlib_bpc = zlib_bits_per_char(text)

        text_results[name] = {
            "length": n,
            "H1_bits_per_char": round(h1, 6),
            "H2_per_char": round(h2, 6),
            "H3_per_char": round(h3, 6),
            "zlib_bits_per_char": round(zlib_bpc, 6),
        }
        print(
            f"{name} (n={n}): "
            f"H1={h1:.4f}  H2={h2:.4f}  H3={h3:.4f}  zlib={zlib_bpc:.4f}"
        )

    # Convergence: H1 - H2, H2 - H3, H1 - zlib
    convergence = {}
    for name, r in text_results.items():
        convergence[name] = {
            "H1_minus_H2": round(r["H1_bits_per_char"] - r["H2_per_char"], 6),
            "H2_minus_H3": round(r["H2_per_char"] - r["H3_per_char"], 6),
            "H1_minus_zlib": round(r["H1_bits_per_char"] - r["zlib_bits_per_char"], 6),
        }

    all_monotonic = all(
        r["H1_bits_per_char"] > r["H2_per_char"] > r["H3_per_char"]
        for r in text_results.values()
    )
    all_zlib_below_h1 = all(
        r["zlib_bits_per_char"] < r["H1_bits_per_char"]
        for r in text_results.values()
    )

    output = {
        "texts": text_results,
        "convergence": convergence,
        "summary": {
            "num_texts": len(text_results),
            "all_monotonic_H1_gt_H2_gt_H3": all_monotonic,
            "all_zlib_below_H1": all_zlib_below_h1,
        },
    }

    with open(OUTPUT_FILE, "w") as fh:
        json.dump(output, fh, indent=2)

    print(f"\nall_monotonic: {all_monotonic}")
    print(f"all_zlib_below_H1: {all_zlib_below_h1}")
    print(f"Results written to {OUTPUT_FILE}")


if __name__ == "__main__":
    main()
PY
python3 scripts/analyze.py
```

Expected output:
```
gettysburg (n=1475): H1=4.1586  H2=3.5717  H3=2.9353  zlib=3.8454
pride_and_prejudice (n=1770): H1=4.4082  H2=3.7664  H3=3.0669  zlib=4.0497
tale_of_two_cities (n=1607): H1=4.2276  H2=3.6332  H3=2.9860  zlib=3.8332
declaration_of_independence (n=1640): H1=4.2805  H2=3.6881  H3=3.0117  zlib=4.0537
moby_dick (n=2494): H1=4.3674  H2=3.8156  H3=3.2069  zlib=4.2213

all_monotonic: True
all_zlib_below_H1: True
Results written to output/results.json
```

---

## Step 3: Run Smoke Tests

```bash
cd workspace
python3 - <<'PY'
"""Smoke tests for the Shannon entropy bound benchmark."""
import json
import math

results = json.load(open("output/results.json"))
texts = results["texts"]
summary = results["summary"]

# ── Test 1: All 5 texts are present ───────────────────────────────────────────
expected_names = {
    "gettysburg",
    "pride_and_prejudice",
    "tale_of_two_cities",
    "declaration_of_independence",
    "moby_dick",
}
assert set(texts.keys()) == expected_names, \
    f"Expected 5 texts, got: {set(texts.keys())}"
print("PASS  Test 1: all 5 texts present")

# ── Test 2: All texts have > 500 characters ───────────────────────────────────
for name, r in texts.items():
    assert r["length"] > 500, \
        f"{name}: length {r['length']} is not > 500"
print("PASS  Test 2: all texts have > 500 characters")

# ── Test 3: All entropy values are positive ───────────────────────────────────
for name, r in texts.items():
    for key in ("H1_bits_per_char", "H2_per_char", "H3_per_char"):
        assert r[key] > 0, f"{name}: {key} = {r[key]} is not positive"
print("PASS  Test 3: all entropy values are positive")

# ── Test 4: H1 > H2_per_char > H3_per_char for every text (monotonic) ─────────
for name, r in texts.items():
    h1 = r["H1_bits_per_char"]
    h2 = r["H2_per_char"]
    h3 = r["H3_per_char"]
    assert h1 > h2, \
        f"{name}: H1={h1:.4f} is NOT > H2_per_char={h2:.4f}"
    assert h2 > h3, \
        f"{name}: H2_per_char={h2:.4f} is NOT > H3_per_char={h3:.4f}"
print("PASS  Test 4: H1 > H2_per_char > H3_per_char for all texts (monotonic convergence)")

# ── Test 5: zlib compression achieves < H1 bits/char for all texts ────────────
for name, r in texts.items():
    zlib_bpc = r["zlib_bits_per_char"]
    h1 = r["H1_bits_per_char"]
    assert zlib_bpc < h1, \
        f"{name}: zlib={zlib_bpc:.4f} is NOT < H1={h1:.4f}"
print("PASS  Test 5: zlib compression achieves < H1 bits/char for all texts")

# ── Test 6: All zlib ratios are in plausible range (0, 8) bits/char ───────────
for name, r in texts.items():
    zlib_bpc = r["zlib_bits_per_char"]
    assert 0 < zlib_bpc < 8, \
        f"{name}: zlib_bits_per_char={zlib_bpc:.4f} outside (0, 8)"
print("PASS  Test 6: all zlib bits/char values in plausible range (0, 8)")

# ── Test 7: Summary flags are consistent with per-text data ───────────────────
assert summary["all_monotonic_H1_gt_H2_gt_H3"] is True, \
    "summary.all_monotonic should be True"
assert summary["all_zlib_below_H1"] is True, \
    "summary.all_zlib_below_H1 should be True"
assert summary["num_texts"] == 5, \
    f"Expected num_texts=5, got {summary['num_texts']}"
print("PASS  Test 7: summary flags consistent with per-text data")

# ── Test 8: All entropy values are finite floats ──────────────────────────────
for name, r in texts.items():
    for key in ("H1_bits_per_char", "H2_per_char", "H3_per_char", "zlib_bits_per_char"):
        val = r[key]
        assert isinstance(val, float), f"{name}.{key} is not a float: {type(val)}"
        assert math.isfinite(val), f"{name}.{key} is not finite: {val}"
print("PASS  Test 8: all entropy values are finite floats")

print()
print("smoke_tests_passed")
PY
```

Expected output:
```
PASS  Test 1: all 5 texts present
PASS  Test 2: all texts have > 500 characters
PASS  Test 3: all entropy values are positive
PASS  Test 4: H1 > H2_per_char > H3_per_char for all texts (monotonic convergence)
PASS  Test 5: zlib compression achieves < H1 bits/char for all texts
PASS  Test 6: all zlib bits/char values in plausible range (0, 8)
PASS  Test 7: summary flags consistent with per-text data
PASS  Test 8: all entropy values are finite floats

smoke_tests_passed
```

---

## Step 4: Verify Results

```bash
cd workspace
python3 - <<'PY'
import json

results = json.load(open("output/results.json"))
texts = results["texts"]
summary = results["summary"]

# Print summary table
print(f"{'Text':<32}  {'H1':>7}  {'H2/c':>7}  {'H3/c':>7}  {'zlib':>7}")
print("-" * 64)
for name, r in texts.items():
    print(
        f"{name:<32}  "
        f"{r['H1_bits_per_char']:>7.4f}  "
        f"{r['H2_per_char']:>7.4f}  "
        f"{r['H3_per_char']:>7.4f}  "
        f"{r['zlib_bits_per_char']:>7.4f}"
    )
print()

# Known-answer assertions for Gettysburg Address
gettysburg = texts["gettysburg"]
assert 4.0 < gettysburg["H1_bits_per_char"] < 4.5, \
    f"Gettysburg H1 out of expected range: {gettysburg['H1_bits_per_char']}"
assert 3.3 < gettysburg["H2_per_char"] < 3.9, \
    f"Gettysburg H2_per_char out of expected range: {gettysburg['H2_per_char']}"
assert 2.6 < gettysburg["H3_per_char"] < 3.2, \
    f"Gettysburg H3_per_char out of expected range: {gettysburg['H3_per_char']}"

# Core theorem assertions
assert summary["all_monotonic_H1_gt_H2_gt_H3"], \
    "FAIL: H1 > H2_per_char > H3_per_char does NOT hold for all texts"
assert summary["all_zlib_below_H1"], \
    "FAIL: zlib bits/char is NOT below H1 for all texts"

print("Shannon source coding theorem confirmed:")
print("  H1 > H2_per_char > H3_per_char for all 5 texts (monotonic convergence)")
print("  zlib bits/char < H1 for all 5 texts")
print()
print("shannon_entropy_bound_verified")
PY
```

Expected output:
```
Text                                   H1     H2/c     H3/c     zlib
----------------------------------------------------------------
gettysburg                         4.1586   3.5717   2.9353   3.8454
pride_and_prejudice                4.4082   3.7664   3.0669   4.0497
tale_of_two_cities                 4.2276   3.6332   2.9860   3.8332
declaration_of_independence        4.2805   3.6881   3.0117   4.0537
moby_dick                          4.3674   3.8156   3.2069   4.2213

Shannon source coding theorem confirmed:
  H1 > H2_per_char > H3_per_char for all 5 texts (monotonic convergence)
  zlib bits/char < H1 for all 5 texts

shannon_entropy_bound_verified
```

---

## Notes

### What This Measures

H₁ (unigram entropy) assumes each character is drawn independently from the marginal
distribution. H₂_per_char and H₃_per_char use joint bigram/trigram counts to estimate
per-character entropy — they capture sequential dependencies between adjacent characters.
As n-gram order increases, the entropy estimate decreases (more structure is exploited),
converging toward the true entropy rate of the language.

Shannon's source coding theorem states that no lossless code can compress below H bits/symbol
on average. An encoder that perfectly models the source would approach this bound from above.
The monotonic sequence H₁ ≥ H₂_per_char ≥ H₃_per_char follows from the subadditivity of
entropy: H(X₁,...,Xₙ) ≤ H(X₁) + ... + H(Xₙ), so the joint entropy divided by n is
non-increasing.

### Limitations

1. **Short texts (~1500–2500 chars) limit n-gram statistics.** Trigram counts are sparse at
   this length; H₃_per_char underestimates the true trigram entropy. Longer texts would
   give more stable estimates and steeper convergence.

2. **zlib is LZ77-based, not a true entropy coder.** It exploits repeated strings (LZ77
   back-references) plus Huffman coding (DEFLATE). For short texts the zlib header (~11 bytes)
   adds overhead. For very short strings (<~400 chars) this overhead can push bits/char
   above H₁. All 5 texts here are long enough to avoid this artifact.

3. **English-only analysis.** The convergence rate (H₁ - H₃_per_char ≈ 1.2 bits/char
   for English) will differ for other languages, programming code, or binary data.

4. **Character-level, not byte-level.** All texts are pure ASCII so character count equals
   UTF-8 byte count. The distinction matters for texts with multibyte characters.

5. **No claim about the true entropy rate.** H₃_per_char is not the true entropy of English;
   estimates from large corpora using variable-order models converge to ~1.3 bits/char
   (Shannon, 1951). The n=3 model here gives ~3.0–3.2 bits/char on 1500-char samples.