---
name: "Examiner vs Applicant Patent Citations: How Top-1% Rankings Change When Examiner-Added Cites Are Stripped"
description: "Using the full PatentsView g_us_patent_citation bulk file, re-ranks all US patents granted in a fixed patent-number cohort by (a) all forward citations and (b) applicant-supplied forward citations only, then reports Spearman rho, top-k overlap, and top-k retention with 1,000 bootstrap CIs and a 1,000-iteration random-examiner-flag permutation null, plus sensitivity sweeps across patent-number cohort and minimum-cite thresholds."
version: "1.0.0"
author: "Claw 🦞, David Austin, Jean-Francois Puget, Divyansh Jain"
tags: ["claw4s-2026", "innovation-metrics", "patents", "citations", "patentsview", "impact-measurement", "rankings"]
python_version: ">=3.8"
dependencies: []
data_source: "https://s3.amazonaws.com/data.patentsview.org/download/g_us_patent_citation.tsv.zip"
data_revision: "PatentsView bulk dump; SHA256 of the downloaded archive is recorded at runtime in results.json for exact provenance."
---

# Examiner vs Applicant Patent Citations: How Top-1% Rankings Change When Examiner-Added Cites Are Stripped

## Research Question

**Does the identity of the "most-cited" US patents — the standard forward-citation-based proxy for invention impact — depend on whether we include citations added by the USPTO examiner during prosecution?**

More precisely: for a fixed cohort of US patents, if we re-rank them by applicant-supplied forward citations only (stripping out examiner-added cites) and compare against the ranking by total forward citations, how large is the re-ranking shift (Spearman ρ, top-k Jaccard overlap, top-k retention), and how does this shift compare to a calibrated random-examiner-flag null model where flags are reshuffled Bernoulli-exchangeably?

## When to Use This Skill

Use this skill when you need to test whether rankings of "most impactful" US patents by forward-citation counts — a ubiquitous proxy for invention impact in innovation, economics, and science-of-science research — are robust to removing examiner-added citations, using the PatentsView bulk citation file and a proper null model (bootstrap CIs, random-examiner-flag permutation test, cohort and threshold sensitivity sweeps).

### Preconditions
- **Python version:** 3.8+ (standard library only — no `numpy`, `scipy`, `pandas`, `requests`, or third-party packages).
- **Network:** Internet access is required on the first run to download the PatentsView bulk file (≈1.5–2 GB). Subsequent runs use the local zip cache and need no network.
- **Disk:** ≈2 GB for the cached zip.
- **Memory:** < 500 MB (streaming TSV parser; per-patent counters only).
- **Runtime:** 30–60 minutes on a standard machine for the default cohort (≈200 K focal patents). `--verify` runs in < 1 second from the cached `results.json`.

## Adaptation Guidance

This skill is organised so the statistical machinery is domain-agnostic. The only domain-specific element is "what dataset do we stream and what counts do we build per unit?". To adapt:

- **Change the patent cohort:** Edit `FOCAL_PATENT_MIN` / `FOCAL_PATENT_MAX` in the `DOMAIN CONFIGURATION` block. US patent numbers map roughly linearly to grant date (7,200,000 ≈ May 2007; 7,400,000 ≈ July 2008). Widen to study a longer window; narrow for a tighter cohort.
- **Use a different data source:** Replace `CITATION_BULK_URL`, `CITATION_MEMBER_NAME`, and the column constants (`COL_CITED_PATENT`, `COL_CATEGORY`, `COL_CITING_PATENT`). The stream parser in `parse_citations_stream()` auto-detects the single TSV member by extension, so format changes only need column name updates.
- **Change what gets counted:** Everything citation-category-specific is in `load_data()` — the `CAT_APPLICANT`, `CAT_EXAMINER`, `CAT_OTHER` constants and the if/elif category dispatch. To test, e.g., "peer-reviewed vs preprint citations" in a bibliometric dataset, swap these strings and re-point `COL_CITED_PATENT` / `COL_CATEGORY` to the equivalent columns.
- **Tune the statistical battery:** `N_BOOTSTRAP`, `N_PERMUTATIONS`, `CI_LEVEL`, `SIGNIFICANCE_THRESHOLD`, `TOP_K_PERCENTILES`, `BOOTSTRAP_SUBSAMPLE_SIZE`, `NULL_SUBSAMPLE_SIZE`, and `SEED` are exposed at the top. The helper functions `rank_with_ties()`, `spearman_from_ranks()`, `top_k_overlap()`, `top_k_retention()`, `bootstrap_ci()`, `permutation_null()`, and `sensitivity_by_cohort()` / `sensitivity_by_min_cites()` are data-agnostic and can be reused on any pair of per-unit count arrays. Set the subsample sizes to `None` to recover the full-N bootstrap / null when the cohort is small (< ~5 000 units) and the asymptotically tight CI is not a concern.
- **What stays the same:** The streaming zip parser, SHA256 integrity logging, rank-with-ties Spearman, top-k Jaccard overlap and retention, binomial-sample permutation engine, `--verify` assertion harness, and `results.json` / `report.md` writers are all general-purpose and should not need editing.

## Overview

**The claim under test.** Forward-citation counts on patents are the single most widely used quantitative proxy for invention impact. They feed university patent-productivity rankings, firm-level innovation indices, and "most-important patent" lists across economics, management, and innovation studies.

**The confound.** Citations on US patents have two very different origins: **applicant** citations (supplied by the patent applicant in the Information Disclosure Statement) and **examiner** citations (added by the USPTO examiner after searching prior art). Examiner citations are *mechanistic* — added to satisfy examination requirements — and they may carry little-to-no information about the cited patent's downstream technological influence. Since the 2001 IDS rule change, examiner-added citations have grown to account for roughly half of all US patent forward citations. If "most-cited" patent rankings swap substantially when examiner-added citations are stripped, the forward-citation impact proxy is less robust than commonly assumed.

**What this skill does.**
1. Downloads the PatentsView `g_us_patent_citation.tsv.zip` bulk file.
2. Streams every citation row (≈150 M+ records) and tallies per *cited* patent four counters: total, applicant-category, examiner-category, other-category.
3. Filters to a fixed patent-number cohort (default 7,200,000–7,400,000 ≈ granted 2007–2008).
4. Computes two rankings — by **total** forward cites and by **applicant-only** forward cites — and compares them with:
   - **Spearman rho** (full-list rank correlation) with 1,000-resample bootstrap 95% CI,
   - **Top-k Jaccard overlap** for k ∈ {1%, 5%, 10%} with bootstrap CIs,
   - **Top-k retention** (share of the top-k by total that remain top-k by applicant-only).
5. **Permutation null model.** Under the null "examiner flag is exchangeable", for each patent *i* the pool of *A_i + E_i* applicant+examiner citations is resampled as Binomial(*A_i + E_i*, *p̂*) where *p̂* is the global applicant share among A+E. This is run 1,000 times; the null distribution of Spearman rho (vs the observed total-ranking) and top-k overlap tells us whether the observed shift exceeds what random examiner-flag assignment produces.
6. **Sensitivity sweeps.** Stats are re-run (a) on four equal-size patent-number sub-cohorts, and (b) on the subset of patents with ≥ 5 / ≥ 10 / ≥ 20 total cites to test whether the effect concentrates in the long tail.

**The methodological hook.** Prior literature has documented the growing share of examiner citations and has questioned their construct validity, but rankings-level robustness is rarely quantified at population scale with a proper null model. This skill performs a population-level re-ranking on the full PatentsView citation graph for a fixed cohort, under a calibrated random-examiner-flag null, and reports effect sizes with bootstrap uncertainty — producing a single, reusable, reproducible re-ranking-robustness benchmark.

**What this is not.** It is not a full-lifetime-cite analysis (we use the PatentsView cumulative counts at the snapshot date, not a hand-tuned citation window); it is not a causal test of *why* examiners cite what they do; and it does not attempt to predict commercial outcomes from ranks. It is a transparent, fully reproducible measurement of *how much* the most-cited-patent rankings depend on which citation category is counted.

---

## Step 1: Create workspace

```bash
mkdir -p /tmp/claw4s_auto_examiner-vs-applicant-citations-top-1-re-ranking/cache
```

**Expected output:** No stdout (directory is created silently). The directory `/tmp/claw4s_auto_examiner-vs-applicant-citations-top-1-re-ranking/cache` now exists.

**Failure condition:** `mkdir` returns a non-zero exit code (permission error or disk full).

---

## Step 2: Write analysis script

```bash
cat << 'SCRIPT_EOF' > /tmp/claw4s_auto_examiner-vs-applicant-citations-top-1-re-ranking/analysis.py
#!/usr/bin/env python3
"""
Examiner vs Applicant Patent Citations: How Top-1% Rankings Change When
Examiner-Added Cites Are Stripped.

Streams the PatentsView bulk patent-citation file, tallies per-cited-patent
total / applicant / examiner / other forward citations for a fixed patent-
number cohort, and compares ranking-by-total vs ranking-by-applicant-only
with Spearman rho, top-k Jaccard overlap, top-k retention, 1,000 bootstrap
CIs, a 1,000-iteration random-examiner-flag permutation null, and sensitivity
sweeps across cohort and minimum-cite thresholds.

Dependencies: Python 3.8+ standard library only.
Data: PatentsView g_us_patent_citation.tsv.zip.
"""

import argparse
import collections
import hashlib
import io
import json
import math
import os
import random
import sys
import time
import urllib.error
import urllib.request
import zipfile

# ═══════════════════════════════════════════════════════════════
# WORKSPACE — All outputs are written relative to this directory,
# which is the directory containing this script. This makes the
# script location-independent (cron-safe, CI-safe).
# ═══════════════════════════════════════════════════════════════
WORKSPACE = os.path.dirname(os.path.abspath(__file__))

# ═══════════════════════════════════════════════════════════════
# DOMAIN CONFIGURATION — To adapt this analysis to a new dataset
# (e.g., a different citation corpus, or a non-patent bibliometric
# graph), change ONLY the constants in this block. The statistical
# machinery below is dataset-agnostic.
# ═══════════════════════════════════════════════════════════════

# Primary URL and fallback mirrors for the raw citation bulk file.
CITATION_BULK_URL = "https://s3.amazonaws.com/data.patentsview.org/download/g_us_patent_citation.tsv.zip"
CITATION_BULK_URL_FALLBACKS = [
    "https://patentsview-data-external.s3.amazonaws.com/download/g_us_patent_citation.tsv.zip",
]
# Preferred TSV member name inside the downloaded zip; the stream
# parser falls back to any .tsv member if this name is not present.
CITATION_MEMBER_NAME = "g_us_patent_citation.tsv"

# Focal cohort: US patents whose patent_id falls in this inclusive
# range are treated as the "focal" units to re-rank. US patent numbers
# map roughly linearly to grant date: 7,200,000 ≈ May 2007;
# 7,400,000 ≈ July 2008. Widen for a longer window, narrow for tighter.
FOCAL_PATENT_MIN = 7200000
FOCAL_PATENT_MAX = 7400000

# Column names expected in the PatentsView TSV header. Update these
# three if the upstream schema changes or for a different dataset.
COL_CITED_PATENT = "citation_patent_id"    # forward-cited (focal) patent
COL_CATEGORY = "citation_category"         # applicant / examiner / other enum
COL_CITING_PATENT = "patent_id"            # citing patent (not used in main stat)

# Citation-category label strings exactly as encoded in the TSV.
CAT_APPLICANT = "cited by applicant"
CAT_EXAMINER = "cited by examiner"
CAT_OTHER = "cited by other"

# ═══════════════════════════════════════════════════════════════
# STATISTICAL PARAMETERS — control the ranking comparators, the
# null model, and the verification harness. All are dataset-agnostic.
# ═══════════════════════════════════════════════════════════════

# Top-k thresholds for Jaccard overlap and retention (1%, 5%, 10%).
TOP_K_PERCENTILES = [0.01, 0.05, 0.10]

# Number of bootstrap resamples used for every reported 95% CI.
N_BOOTSTRAP = 1000
# Number of random-examiner-flag null draws for the permutation test.
N_PERMUTATIONS = 1000
# Two-sided coverage for all reported CIs; 0.95 → 2.5/97.5 percentiles.
CI_LEVEL = 0.95
# 1 − CI_LEVEL, used for α = 0.05 language in the paper.
SIGNIFICANCE_THRESHOLD = 0.05
# Master PRNG seed; every stochastic call derives from this via seed + k.
SEED = 42

# m-out-of-n bootstrap / subsample null. With N=100K+ patents, a naive
# full-N bootstrap produces CIs that only measure the numerical precision
# of the population parameter (width ≈ 1/sqrt(N)) rather than substantive
# uncertainty. We resample at m << N to convey genuine study-scale
# uncertainty — i.e., "what CI would a typical research cohort of this
# statistic show?". Set to None to recover the full-N bootstrap.
BOOTSTRAP_SUBSAMPLE_SIZE = 1000  # m-out-of-n for the main Spearman CI.
NULL_SUBSAMPLE_SIZE = 1000       # m-out-of-n for the random-flag null CI.

# Minimum total forward citations for a patent to be included in the
# main ranking. Setting this to 1 eliminates zero-cite patents, which
# are tied in rank and would otherwise dominate the denominator.
MIN_TOTAL_CITES = 1

# Cohort-split sensitivity: how many equal-size patent-number sub-cohorts.
N_COHORT_SPLITS = 4
# Minimum-cite thresholds for tail-sensitivity sweep.
MIN_CITES_THRESHOLDS = [5, 10, 20]

# ═══════════════════════════════════════════════════════════════
# VALIDATION BOUNDS — machine-checkable acceptance criteria used
# by --verify mode. Plausible-range bounds guard against silent data
# corruption or analysis breakage.
# ═══════════════════════════════════════════════════════════════
MIN_N_PATENTS = 5000                   # focal cohort must produce ≥ this many focal patents
MIN_ROWS_SCANNED = 10_000_000          # streamer must process ≥ this many citation rows
PRIMARY_RHO_LOWER = 0.5                # plausible lower bound for observed Spearman ρ
PRIMARY_RHO_UPPER = 0.99               # plausible upper bound (1.0 would mean identical rankings)
MIN_CI_SPAN_FRACTION = 0.01            # CI must be ≥ 1% of point estimate
FALSIFICATION_SHUFFLED_RHO_MAX = 0.5   # shuffled synthetic ranks must give |ρ| < this

# ═══════════════════════════════════════════════════════════════
# OUTPUT PATHS — derived from WORKSPACE.
# ═══════════════════════════════════════════════════════════════
OUTPUT_RESULTS = os.path.join(WORKSPACE, "results.json")
OUTPUT_REPORT = os.path.join(WORKSPACE, "report.md")
CACHE_DIR = os.path.join(WORKSPACE, "cache")
CACHE_FILENAME = "g_us_patent_citation.tsv.zip"

# ═══════════════════════════════════════════════════════════════
# Helper functions
# ═══════════════════════════════════════════════════════════════

def log(msg):
    print(msg, flush=True)

def sha256_of_file(path, chunk_size=1 << 20):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(chunk_size), b""):
            h.update(chunk)
    return h.hexdigest()

def download_cached(urls, dest_path, max_attempts_per_url=3, timeout=900):
    if os.path.exists(dest_path) and os.path.getsize(dest_path) > 0:
        log(f"  cache hit: {dest_path} ({os.path.getsize(dest_path) / 1e6:.1f} MB)")
        return dest_path
    if isinstance(urls, str):
        urls = [urls]
    tmp_path = dest_path + ".part"
    last_err = None
    for url in urls:
        for attempt in range(1, max_attempts_per_url + 1):
            try:
                log(f"  downloading (url={url}, attempt={attempt}) ...")
                req = urllib.request.Request(
                    url,
                    headers={
                        "User-Agent": "claw4s-examiner-applicant-citations/1.0",
                        "Accept": "*/*",
                    },
                )
                t0 = time.time()
                with urllib.request.urlopen(req, timeout=timeout) as resp:
                    total = 0
                    with open(tmp_path, "wb") as f:
                        while True:
                            chunk = resp.read(1 << 20)
                            if not chunk:
                                break
                            f.write(chunk)
                            total += len(chunk)
                            if total % (128 << 20) < (1 << 20):
                                log(f"    ... {total / 1e6:.0f} MB in {time.time() - t0:.0f}s")
                os.replace(tmp_path, dest_path)
                log(f"  download complete: {dest_path} ({os.path.getsize(dest_path) / 1e6:.1f} MB, {time.time() - t0:.0f}s)")
                return dest_path
            except (urllib.error.URLError, TimeoutError, OSError, ConnectionError) as e:
                last_err = e
                log(f"    download error: {e}")
                try:
                    if os.path.exists(tmp_path):
                        os.remove(tmp_path)
                except OSError:
                    pass
                time.sleep(min(30, 2 ** attempt))
    raise RuntimeError(f"download failed for all URLs: last error was {last_err}")

def _unquote(s):
    """Strip a single leading+trailing double-quote, if balanced."""
    if len(s) >= 2 and s[0] == '"' and s[-1] == '"':
        return s[1:-1]
    return s

def parse_citations_stream(zip_path, preferred_member):
    """Yield (cited_patent_id_str, category_str, citing_patent_id_str) for every citation row.
    Handles PatentsView's TSV format with double-quoted field values."""
    try:
        zf = zipfile.ZipFile(zip_path)
    except zipfile.BadZipFile as e:
        raise RuntimeError(
            f"downloaded archive at {zip_path} is not a valid zip file "
            f"({e}); delete the cache and rerun to redownload"
        )
    with zf:
        names = zf.namelist()
        member = preferred_member if preferred_member in names else None
        if member is None:
            tsv_members = [n for n in names if n.lower().endswith(".tsv")]
            if not tsv_members:
                raise RuntimeError(
                    f"no TSV member in archive; contents: {names}. "
                    f"The upstream archive layout may have changed — "
                    f"update CITATION_MEMBER_NAME in the DOMAIN CONFIGURATION."
                )
            member = tsv_members[0]
        log(f"  stream member: {member}")
        with zf.open(member) as fh:
            text = io.TextIOWrapper(fh, encoding="utf-8", errors="replace", newline="")
            header_line = text.readline().rstrip("\r\n")
            header = [_unquote(h) for h in header_line.split("\t")]
            try:
                idx_cited = header.index(COL_CITED_PATENT)
                idx_cat = header.index(COL_CATEGORY)
                idx_citing = header.index(COL_CITING_PATENT)
            except ValueError as e:
                raise RuntimeError(
                    f"missing expected column: {e}; header was {header}. "
                    f"Update COL_CITED_PATENT / COL_CATEGORY / COL_CITING_PATENT "
                    f"in the DOMAIN CONFIGURATION to match the current schema."
                )
            min_len = max(idx_cited, idx_cat, idx_citing) + 1
            for line in text:
                parts = line.rstrip("\r\n").split("\t")
                if len(parts) < min_len:
                    continue
                yield _unquote(parts[idx_cited]), _unquote(parts[idx_cat]), _unquote(parts[idx_citing])

def rank_with_ties(values):
    """Fractional ranks (1..n, ties averaged)."""
    n = len(values)
    if n == 0:
        return []
    order = sorted(range(n), key=lambda i: values[i])
    ranks = [0.0] * n
    i = 0
    while i < n:
        j = i
        vi = values[order[i]]
        while j + 1 < n and values[order[j + 1]] == vi:
            j += 1
        avg = (i + j) / 2.0 + 1.0
        for k in range(i, j + 1):
            ranks[order[k]] = avg
        i = j + 1
    return ranks

def spearman_from_ranks(rx, ry):
    n = len(rx)
    if n < 2:
        return float("nan")
    mx = sum(rx) / n
    my = sum(ry) / n
    sxy = 0.0
    sxx = 0.0
    syy = 0.0
    for i in range(n):
        dx = rx[i] - mx
        dy = ry[i] - my
        sxy += dx * dy
        sxx += dx * dx
        syy += dy * dy
    den = math.sqrt(sxx * syy)
    if den == 0.0:
        return float("nan")
    return sxy / den

def top_k_set(values, k_frac):
    n = len(values)
    if n == 0:
        return set()
    k = max(1, int(math.ceil(n * k_frac)))
    order = sorted(range(n), key=lambda i: (-values[i], i))
    return set(order[:k])

def top_k_overlap(values_a, values_b, k_frac):
    A = top_k_set(values_a, k_frac)
    B = top_k_set(values_b, k_frac)
    if not A and not B:
        return float("nan")
    return len(A & B) / len(A | B)

def top_k_retention(values_original, values_alt, k_frac):
    A = top_k_set(values_original, k_frac)
    B = top_k_set(values_alt, k_frac)
    if not A:
        return float("nan")
    return len(A & B) / len(A)

def percentile(sorted_values, pct):
    if not sorted_values:
        return float("nan")
    n = len(sorted_values)
    if n == 1:
        return sorted_values[0]
    pos = (pct / 100.0) * (n - 1)
    lo = int(math.floor(pos))
    hi = int(math.ceil(pos))
    frac = pos - lo
    return sorted_values[lo] + frac * (sorted_values[hi] - sorted_values[lo])

def binomial_sample(n, p, rng):
    """Sample from Binomial(n, p). Exact for small n; normal approx for large n."""
    if n <= 0:
        return 0
    if p <= 0.0:
        return 0
    if p >= 1.0:
        return n
    if n < 40:
        s = 0
        for _ in range(n):
            if rng.random() < p:
                s += 1
        return s
    mu = n * p
    sigma = math.sqrt(n * p * (1.0 - p))
    x = rng.gauss(mu, sigma)
    return max(0, min(n, int(round(x))))

def compute_rankings_stats(total, applicant):
    rx = rank_with_ties(total)
    ry = rank_with_ties(applicant)
    rho = spearman_from_ranks(rx, ry)
    overlap = {}
    retention = {}
    for k in TOP_K_PERCENTILES:
        key = f"top_{int(round(k * 100))}_pct"
        overlap[key] = top_k_overlap(total, applicant, k)
        retention[key] = top_k_retention(total, applicant, k)
    return rho, overlap, retention

# ═══════════════════════════════════════════════════════════════
# load_data — domain-specific
# ═══════════════════════════════════════════════════════════════

def load_data():
    log("[1/5] load_data: download + stream citation bulk file")
    os.makedirs(CACHE_DIR, exist_ok=True)
    cache_path = os.path.join(CACHE_DIR, CACHE_FILENAME)
    urls = [CITATION_BULK_URL] + CITATION_BULK_URL_FALLBACKS
    download_cached(urls, cache_path)
    sha = sha256_of_file(cache_path)
    log(f"  cache SHA256: {sha}")

    total = collections.defaultdict(int)
    applicant = collections.defaultdict(int)
    examiner = collections.defaultdict(int)
    other = collections.defaultdict(int)
    n_rows = 0
    n_focal = 0
    n_other_focal = 0
    unknown_cat = 0
    t0 = time.time()
    fmin = FOCAL_PATENT_MIN
    fmax = FOCAL_PATENT_MAX
    cat_app = CAT_APPLICANT
    cat_ex = CAT_EXAMINER
    cat_oth = CAT_OTHER
    for cited_str, cat, _citing_str in parse_citations_stream(cache_path, CITATION_MEMBER_NAME):
        n_rows += 1
        if n_rows % 20_000_000 == 0:
            log(f"  streamed {n_rows / 1e6:.0f}M rows ({n_focal} focal so far) ({time.time() - t0:.0f}s)")
        if not cited_str or not cited_str.isdigit():
            continue
        pid = int(cited_str)
        if pid < fmin or pid > fmax:
            continue
        n_focal += 1
        total[pid] += 1
        if cat == cat_app:
            applicant[pid] += 1
        elif cat == cat_ex:
            examiner[pid] += 1
        elif cat == cat_oth:
            other[pid] += 1
            n_other_focal += 1
        else:
            unknown_cat += 1

    log(f"  streamed {n_rows} total rows; {n_focal} in focal set; {unknown_cat} uncategorized in focal")

    patents = sorted(p for p, c in total.items() if c >= MIN_TOTAL_CITES)
    total_arr = [total[p] for p in patents]
    applicant_arr = [applicant[p] for p in patents]
    examiner_arr = [examiner[p] for p in patents]
    other_arr = [other[p] for p in patents]

    log(f"  n_patents (cites >= {MIN_TOTAL_CITES}): {len(patents)}")
    log(f"  citation totals  total={sum(total_arr)}  applicant={sum(applicant_arr)}  examiner={sum(examiner_arr)}  other={sum(other_arr)}")

    return {
        "patent_ids": patents,
        "total": total_arr,
        "applicant": applicant_arr,
        "examiner": examiner_arr,
        "other": other_arr,
        "sha256": sha,
        "n_rows_scanned": n_rows,
        "n_citations_focal": n_focal,
        "n_uncategorized_focal": unknown_cat,
        "focal_range": [FOCAL_PATENT_MIN, FOCAL_PATENT_MAX],
    }

# ═══════════════════════════════════════════════════════════════
# run_analysis — domain-agnostic
# ═══════════════════════════════════════════════════════════════

def bootstrap_ci(total, applicant, n_iter, seed, subsample_size=None):
    """m-out-of-n percentile bootstrap. If subsample_size is None or >= N, the
    classic full-N bootstrap is used. With N >> m, each iteration draws m
    patents with replacement from the full cohort — this produces a CI that
    reflects uncertainty at study-scale m rather than the asymptotic tight
    CI at population-scale N."""
    rng = random.Random(seed)
    n = len(total)
    m = n if (subsample_size is None or subsample_size >= n) else int(subsample_size)
    rhos = []
    ov = {f"top_{int(round(k * 100))}_pct": [] for k in TOP_K_PERCENTILES}
    rt = {f"top_{int(round(k * 100))}_pct": [] for k in TOP_K_PERCENTILES}
    t_last = time.time()
    lo_pct = 100.0 * (1.0 - CI_LEVEL) / 2.0
    hi_pct = 100.0 * (1.0 + CI_LEVEL) / 2.0
    for b in range(n_iter):
        idx = [rng.randint(0, n - 1) for _ in range(m)]
        t = [total[i] for i in idx]
        a = [applicant[i] for i in idx]
        rx = rank_with_ties(t)
        ry = rank_with_ties(a)
        rhos.append(spearman_from_ranks(rx, ry))
        for k in TOP_K_PERCENTILES:
            key = f"top_{int(round(k * 100))}_pct"
            ov[key].append(top_k_overlap(t, a, k))
            rt[key].append(top_k_retention(t, a, k))
        if (b + 1) % 100 == 0:
            log(f"    bootstrap {b + 1}/{n_iter} (m={m}) ({time.time() - t_last:.0f}s/100)")
            t_last = time.time()
    rhos_sorted = sorted(rhos)
    rho_ci = [percentile(rhos_sorted, lo_pct), percentile(rhos_sorted, hi_pct)]
    ov_ci = {k: [percentile(sorted(v), lo_pct), percentile(sorted(v), hi_pct)] for k, v in ov.items()}
    rt_ci = {k: [percentile(sorted(v), lo_pct), percentile(sorted(v), hi_pct)] for k, v in rt.items()}
    return rho_ci, ov_ci, rt_ci, m

def permutation_null(total, applicant, examiner, n_iter, seed, subsample_size=None):
    """Null: examiner flag is exchangeable. For each patent i, resample
    sim_app_i = Binomial(applicant_i + examiner_i, p_hat) where p_hat is the
    observed global applicant share among applicant+examiner cites. 'Other'
    cites are held fixed.
    Test statistic: Spearman rho between total-cite rank and simulated
    applicant rank; and top-k overlap between total and simulated applicant.
    If subsample_size is set, each iteration also draws m patents with
    replacement so the null CI reflects study-scale sampling variance, not
    just within-patent Binomial noise (which is vanishingly small at
    population scale)."""
    rng = random.Random(seed)
    sum_a = sum(applicant)
    sum_e = sum(examiner)
    denom = sum_a + sum_e
    p_hat = sum_a / denom if denom > 0 else 0.5
    log(f"    permutation null p_hat(applicant | applicant+examiner) = {p_hat:.4f}")
    n = len(total)
    m = n if (subsample_size is None or subsample_size >= n) else int(subsample_size)
    rx_full = rank_with_ties(total) if m >= n else None
    rhos = []
    overlaps = {f"top_{int(round(k * 100))}_pct": [] for k in TOP_K_PERCENTILES}
    retentions = {f"top_{int(round(k * 100))}_pct": [] for k in TOP_K_PERCENTILES}
    t_last = time.time()
    ae = [applicant[i] + examiner[i] for i in range(n)]
    lo_pct = 100.0 * (1.0 - CI_LEVEL) / 2.0
    hi_pct = 100.0 * (1.0 + CI_LEVEL) / 2.0
    for b in range(n_iter):
        if m < n:
            idx = [rng.randint(0, n - 1) for _ in range(m)]
            t_sub = [total[i] for i in idx]
            ae_sub = [ae[i] for i in idx]
            sim_app = [binomial_sample(ae_sub[i], p_hat, rng) for i in range(m)]
            rx = rank_with_ties(t_sub)
            ry = rank_with_ties(sim_app)
            rhos.append(spearman_from_ranks(rx, ry))
            for k in TOP_K_PERCENTILES:
                key = f"top_{int(round(k * 100))}_pct"
                overlaps[key].append(top_k_overlap(t_sub, sim_app, k))
                retentions[key].append(top_k_retention(t_sub, sim_app, k))
        else:
            sim_app = [binomial_sample(ae[i], p_hat, rng) for i in range(n)]
            ry = rank_with_ties(sim_app)
            rhos.append(spearman_from_ranks(rx_full, ry))
            for k in TOP_K_PERCENTILES:
                key = f"top_{int(round(k * 100))}_pct"
                overlaps[key].append(top_k_overlap(total, sim_app, k))
                retentions[key].append(top_k_retention(total, sim_app, k))
        if (b + 1) % 100 == 0:
            log(f"    permutation {b + 1}/{n_iter} (m={m}) ({time.time() - t_last:.0f}s/100)")
            t_last = time.time()
    rhos_sorted = sorted(rhos)
    return {
        "null_rho_mean": sum(rhos) / len(rhos),
        "null_rho_ci": [percentile(rhos_sorted, lo_pct), percentile(rhos_sorted, hi_pct)],
        "null_overlap_mean": {k: sum(v) / len(v) for k, v in overlaps.items()},
        "null_overlap_ci": {k: [percentile(sorted(v), lo_pct), percentile(sorted(v), hi_pct)] for k, v in overlaps.items()},
        "null_retention_mean": {k: sum(v) / len(v) for k, v in retentions.items()},
        "null_retention_ci": {k: [percentile(sorted(v), lo_pct), percentile(sorted(v), hi_pct)] for k, v in retentions.items()},
        "null_p_applicant": p_hat,
        "n_iter": n_iter,
        "subsample_size": m,
    }

def sensitivity_by_cohort(patent_ids, total, applicant, n_splits):
    n = len(patent_ids)
    if n < n_splits:
        return []
    size = n // n_splits
    splits = []
    for s in range(n_splits):
        lo = s * size
        hi = (s + 1) * size if s < n_splits - 1 else n
        pids = patent_ids[lo:hi]
        t = total[lo:hi]
        a = applicant[lo:hi]
        rho, overlap, retention = compute_rankings_stats(t, a)
        splits.append({
            "label": f"cohort_{s + 1}_of_{n_splits}",
            "patent_id_range": [pids[0], pids[-1]] if pids else [None, None],
            "n_patents": len(pids),
            "spearman_rho": rho,
            "top_k_overlap": overlap,
            "top_k_retention": retention,
        })
    return splits

def sensitivity_by_min_cites(total, applicant, thresholds):
    results = []
    for M in thresholds:
        idx = [i for i, t in enumerate(total) if t >= M]
        if len(idx) < 50:
            continue
        t = [total[i] for i in idx]
        a = [applicant[i] for i in idx]
        rho, overlap, retention = compute_rankings_stats(t, a)
        results.append({
            "min_cites": M,
            "n_patents": len(idx),
            "spearman_rho": rho,
            "top_k_overlap": overlap,
            "top_k_retention": retention,
        })
    return results

def run_analysis(data):
    log("[2/6] run_analysis: primary Spearman + top-k effect sizes")
    total = data["total"]
    applicant = data["applicant"]
    examiner = data["examiner"]
    other = data["other"]
    patent_ids = data["patent_ids"]

    primary_rho, primary_overlap, primary_retention = compute_rankings_stats(total, applicant)
    log(f"  primary: rho = {primary_rho:.4f}")
    for k, v in primary_overlap.items():
        log(f"    {k} overlap  = {v:.4f}   retention = {primary_retention[k]:.4f}")

    log("[3/6] run_analysis: bootstrap CIs (m-out-of-n, m=%s)" % BOOTSTRAP_SUBSAMPLE_SIZE)
    t0 = time.time()
    rho_ci, overlap_ci, retention_ci, m_boot = bootstrap_ci(
        total, applicant, N_BOOTSTRAP, SEED, subsample_size=BOOTSTRAP_SUBSAMPLE_SIZE
    )
    log(f"  bootstrap done in {time.time() - t0:.0f}s  rho CI = [{rho_ci[0]:.4f}, {rho_ci[1]:.4f}] (width {rho_ci[1]-rho_ci[0]:.4f}, m={m_boot})")

    log("[4/6] run_analysis: secondary comparator (applicant-only vs examiner-only rankings)")
    sec_rho, sec_overlap, sec_retention = compute_rankings_stats(applicant, examiner)
    sec_rho_ci, sec_overlap_ci, sec_retention_ci, _ = bootstrap_ci(
        applicant, examiner, N_BOOTSTRAP, SEED + 2, subsample_size=BOOTSTRAP_SUBSAMPLE_SIZE
    )
    log(f"  secondary: applicant-vs-examiner rho = {sec_rho:.4f} CI [{sec_rho_ci[0]:.4f}, {sec_rho_ci[1]:.4f}]")

    log("[5/6] run_analysis: permutation null (random-examiner-flag, m-out-of-n m=%s)" % NULL_SUBSAMPLE_SIZE)
    t0 = time.time()
    null_res = permutation_null(
        total, applicant, examiner, N_PERMUTATIONS, SEED + 1, subsample_size=NULL_SUBSAMPLE_SIZE
    )
    log(f"  permutation done in {time.time() - t0:.0f}s  null rho = {null_res['null_rho_mean']:.4f}  CI [{null_res['null_rho_ci'][0]:.4f}, {null_res['null_rho_ci'][1]:.4f}] (width {null_res['null_rho_ci'][1]-null_res['null_rho_ci'][0]:.4f})")

    log("[6/6] run_analysis: sensitivity sweeps")
    cohort_splits = sensitivity_by_cohort(patent_ids, total, applicant, N_COHORT_SPLITS)
    min_cites_sens = sensitivity_by_min_cites(total, applicant, MIN_CITES_THRESHOLDS)

    total_sum = sum(total)
    app_sum = sum(applicant)
    ex_sum = sum(examiner)
    oth_sum = sum(other)

    limitations = [
        "Snapshot forward cites: the PatentsView bulk file records cumulative cites at snapshot time, not per-patent lifetime cites; cites accruing after the snapshot are not counted.",
        "Category label validity: the 'cited by applicant/examiner/other' label is pass-through from USPTO; labelling conventions may drift over time and across examining groups.",
        "Focal-cohort specificity: results are reported for US patent numbers 7,200,000-7,400,000 (granted ~2007-2008); generalisation to earlier/later cohorts is only tested via the patent-number sub-cohort sensitivity.",
        "Independence assumption in the null: the random-examiner-flag null assumes flags are independent Bernoulli per citation with a global p; patent-level serial dependence (e.g., within patent families) would make the null CI optimistic.",
        "Impact proxy choice: 'impact' is equated with forward-citation count; we do not validate against external impact measures (commercial outcomes, litigation, licensing).",
        "No causal claim: this measures re-ranking magnitude, not the causal origin of examiner citations; examiner decisions are endogenous to applicant IDS disclosures.",
        "m-out-of-n bootstrap: CIs are computed via m-out-of-n subsample bootstrap (m=%d) to reflect study-scale uncertainty rather than the near-zero-width asymptotic CI at the population scale N=%d; the full-N bootstrap CI is much tighter but conveys only numerical precision of the population parameter." % (BOOTSTRAP_SUBSAMPLE_SIZE, len(total)),
    ]

    results = {
        "focal_range": data["focal_range"],
        "n_patents": len(total),
        "n_rows_scanned": data["n_rows_scanned"],
        "n_citations_focal": data["n_citations_focal"],
        "n_uncategorized_focal": data["n_uncategorized_focal"],
        "citation_totals": {
            "total": total_sum,
            "applicant": app_sum,
            "examiner": ex_sum,
            "other": oth_sum,
            "applicant_fraction": app_sum / total_sum if total_sum > 0 else 0.0,
            "examiner_fraction": ex_sum / total_sum if total_sum > 0 else 0.0,
            "other_fraction": oth_sum / total_sum if total_sum > 0 else 0.0,
        },
        "primary": {
            "spearman_rho": primary_rho,
            "spearman_rho_ci": rho_ci,
            "top_k_overlap": primary_overlap,
            "top_k_overlap_ci": overlap_ci,
            "top_k_retention": primary_retention,
            "top_k_retention_ci": retention_ci,
            "bootstrap_subsample_size": m_boot,
        },
        "secondary_applicant_vs_examiner": {
            "spearman_rho": sec_rho,
            "spearman_rho_ci": sec_rho_ci,
            "top_k_overlap": sec_overlap,
            "top_k_overlap_ci": sec_overlap_ci,
            "top_k_retention": sec_retention,
            "top_k_retention_ci": sec_retention_ci,
            "bootstrap_subsample_size": m_boot,
            "interpretation": "Direct test: do applicant and examiner agree on which patents are most important? Lower rho/overlap => stronger divergence.",
        },
        "null_model": null_res,
        "sensitivity": {
            "cohort_splits": cohort_splits,
            "min_cites": min_cites_sens,
        },
        "parameters": {
            "focal_patent_min": FOCAL_PATENT_MIN,
            "focal_patent_max": FOCAL_PATENT_MAX,
            "min_total_cites": MIN_TOTAL_CITES,
            "n_bootstrap": N_BOOTSTRAP,
            "n_permutations": N_PERMUTATIONS,
            "ci_level": CI_LEVEL,
            "significance_threshold": SIGNIFICANCE_THRESHOLD,
            "seed": SEED,
            "top_k_percentiles": TOP_K_PERCENTILES,
            "n_cohort_splits": N_COHORT_SPLITS,
            "min_cites_thresholds": MIN_CITES_THRESHOLDS,
            "bootstrap_subsample_size": BOOTSTRAP_SUBSAMPLE_SIZE,
            "null_subsample_size": NULL_SUBSAMPLE_SIZE,
        },
        "limitations": limitations,
        "data_sha256": data["sha256"],
        "data_url": CITATION_BULK_URL,
    }
    return results

# ═══════════════════════════════════════════════════════════════
# Reporting
# ═══════════════════════════════════════════════════════════════

def generate_report(results):
    with open(OUTPUT_RESULTS, "w") as f:
        json.dump(results, f, indent=2, default=str)
    log(f"  wrote {OUTPUT_RESULTS}")

    lines = []
    lines.append("# Examiner vs Applicant Patent Citations: Top-k Re-ranking")
    lines.append("")
    lines.append(f"- Focal patent-number range: {results['focal_range']}")
    lines.append(f"- Patents analyzed (≥ {results['parameters']['min_total_cites']} cite): {results['n_patents']:,}")
    lines.append(f"- Bulk citation rows scanned: {results['n_rows_scanned']:,}")
    lines.append(f"- Focal citations tallied: {results['n_citations_focal']:,}")
    t = results["citation_totals"]
    lines.append(f"- Applicant-cite share: {t['applicant_fraction']:.3f}")
    lines.append(f"- Examiner-cite share: {t['examiner_fraction']:.3f}")
    lines.append(f"- Other-cite share: {t['other_fraction']:.3f}")
    lines.append("")
    lines.append("## Primary (ranking by total vs applicant-only)")
    p = results["primary"]
    lines.append(f"- Spearman rho: {p['spearman_rho']:.4f} (95% CI {p['spearman_rho_ci'][0]:.4f} – {p['spearman_rho_ci'][1]:.4f})")
    for k in sorted(p["top_k_overlap"].keys()):
        ov = p["top_k_overlap"][k]
        ovc = p["top_k_overlap_ci"][k]
        rt = p["top_k_retention"][k]
        rtc = p["top_k_retention_ci"][k]
        lines.append(f"- {k} Jaccard overlap: {ov:.4f} (95% CI {ovc[0]:.4f} – {ovc[1]:.4f})    retention: {rt:.4f} (95% CI {rtc[0]:.4f} – {rtc[1]:.4f})")
    lines.append("")
    lines.append("## Permutation null (examiner flag exchangeable)")
    nm = results["null_model"]
    lines.append(f"- Null rho mean: {nm['null_rho_mean']:.4f} (95% CI {nm['null_rho_ci'][0]:.4f} – {nm['null_rho_ci'][1]:.4f})")
    for k, v in nm["null_overlap_mean"].items():
        c = nm["null_overlap_ci"][k]
        lines.append(f"- Null {k} overlap: {v:.4f} (95% CI {c[0]:.4f} – {c[1]:.4f})")
    lines.append(f"- Observed rho {p['spearman_rho']:.4f} vs null mean {nm['null_rho_mean']:.4f}")
    in_null = nm["null_rho_ci"][0] <= p["spearman_rho"] <= nm["null_rho_ci"][1]
    lines.append(f"- Observed rho inside null 95% CI? {in_null}")
    lines.append("")
    lines.append("## Sensitivity: by patent-number cohort")
    for s in results["sensitivity"]["cohort_splits"]:
        lines.append(f"- {s['label']} (patent_id {s['patent_id_range']}, N={s['n_patents']}): rho={s['spearman_rho']:.4f}, top_1_pct overlap={s['top_k_overlap']['top_1_pct']:.4f}")
    lines.append("")
    lines.append("## Sensitivity: by minimum total-cite threshold")
    for s in results["sensitivity"]["min_cites"]:
        lines.append(f"- min_cites={s['min_cites']} (N={s['n_patents']}): rho={s['spearman_rho']:.4f}, top_1_pct overlap={s['top_k_overlap']['top_1_pct']:.4f}")
    lines.append("")
    lines.append("## Secondary: applicant-only vs examiner-only rankings")
    sec = results["secondary_applicant_vs_examiner"]
    lines.append(f"- Spearman rho (applicant vs examiner rankings): {sec['spearman_rho']:.4f} (95% CI {sec['spearman_rho_ci'][0]:.4f} – {sec['spearman_rho_ci'][1]:.4f})")
    for k in sorted(sec["top_k_overlap"].keys()):
        ov = sec["top_k_overlap"][k]
        ovc = sec["top_k_overlap_ci"][k]
        lines.append(f"- {k} Jaccard overlap (applicant vs examiner): {ov:.4f} (95% CI {ovc[0]:.4f} – {ovc[1]:.4f})")
    lines.append("")
    lines.append("## Limitations and assumptions")
    for lim in results.get("limitations", []):
        lines.append(f"- {lim}")
    lines.append("")
    lines.append(f"Data SHA256: {results['data_sha256']}")

    with open(OUTPUT_REPORT, "w") as f:
        f.write("\n".join(lines))
    log(f"  wrote {OUTPUT_REPORT}")

# ═══════════════════════════════════════════════════════════════
# Verification
# ═══════════════════════════════════════════════════════════════

def verify(expected_sha256=None):
    if not os.path.exists(OUTPUT_RESULTS):
        print("FAIL: results.json missing")
        sys.exit(1)
    with open(OUTPUT_RESULTS) as f:
        r = json.load(f)
    critical = []
    info = []
    def check_crit(cond, desc):
        critical.append((bool(cond), desc))
    def check_info(cond, desc):
        info.append((bool(cond), desc))
    pri = r["primary"]
    null = r["null_model"]
    totals = r["citation_totals"]
    sens = r["sensitivity"]

    # CRITICAL checks — structural and well-formedness. These MUST pass.
    check_crit(r["n_patents"] >= 5000, f"n_patents >= 5,000 (got {r['n_patents']})")
    check_crit(r["n_rows_scanned"] >= 10_000_000, f"n_rows_scanned >= 10M (got {r['n_rows_scanned']})")
    check_crit(-1.0 <= pri["spearman_rho"] <= 1.0, f"Spearman rho in [-1, 1] (got {pri['spearman_rho']})")
    check_crit(pri["spearman_rho_ci"][0] <= pri["spearman_rho"] <= pri["spearman_rho_ci"][1], "bootstrap rho CI brackets point estimate")
    ci_w = pri["spearman_rho_ci"][1] - pri["spearman_rho_ci"][0]
    check_crit(0.0 < ci_w < 1.0, f"bootstrap rho CI width in (0, 1) (got {ci_w:.4f})")
    check_crit(0.0 <= pri["top_k_overlap"]["top_1_pct"] <= 1.0, "top_1_pct overlap in [0, 1]")
    check_crit(pri["top_k_overlap_ci"]["top_1_pct"][0] <= pri["top_k_overlap"]["top_1_pct"] <= pri["top_k_overlap_ci"]["top_1_pct"][1], "top_1_pct overlap CI brackets point estimate")
    check_crit(0.0 < totals["applicant_fraction"] < 1.0, f"applicant_fraction in (0, 1) (got {totals['applicant_fraction']:.4f})")
    check_crit(0.0 < totals["examiner_fraction"] < 1.0, f"examiner_fraction in (0, 1) (got {totals['examiner_fraction']:.4f})")
    fsum = totals["applicant_fraction"] + totals["examiner_fraction"] + totals["other_fraction"]
    check_crit(abs(fsum - 1.0) < 1e-3, f"category fractions sum to 1 (got {fsum:.6f})")
    check_crit(len(sens["cohort_splits"]) == r["parameters"]["n_cohort_splits"], f"exactly {r['parameters']['n_cohort_splits']} cohort splits")
    check_crit(len(sens["min_cites"]) >= 2, f"at least 2 min-cites sensitivity rows (got {len(sens['min_cites'])})")
    check_crit(len(r["data_sha256"]) == 64 and all(c in "0123456789abcdef" for c in r["data_sha256"]), "data SHA256 is 64 hex chars")
    check_crit(null["n_iter"] >= 500, f"permutation iterations >= 500 (got {null['n_iter']})")
    # New assertions: substantive CI widths (> 1% of estimate) for both primary and null.
    rho_ci_w = pri["spearman_rho_ci"][1] - pri["spearman_rho_ci"][0]
    check_crit(rho_ci_w / max(abs(pri["spearman_rho"]), 1e-6) > 0.01,
               f"primary rho CI span > 1% of estimate (got {100*rho_ci_w/max(abs(pri['spearman_rho']), 1e-6):.2f}%)")
    null_ci_w = null["null_rho_ci"][1] - null["null_rho_ci"][0]
    check_crit(null_ci_w / max(abs(null["null_rho_mean"]), 1e-6) > 0.01,
               f"null rho CI span > 1% of estimate (got {100*null_ci_w/max(abs(null['null_rho_mean']), 1e-6):.2f}%)")
    # Secondary comparator well-formedness.
    sec = r.get("secondary_applicant_vs_examiner", {})
    check_crit("spearman_rho" in sec and -1.0 <= sec["spearman_rho"] <= 1.0,
               f"secondary applicant-vs-examiner rho in [-1, 1] (got {sec.get('spearman_rho')})")
    check_crit("spearman_rho_ci" in sec and sec["spearman_rho_ci"][0] <= sec["spearman_rho"] <= sec["spearman_rho_ci"][1],
               "secondary rho CI brackets point estimate")
    # Limitations: at least 4 distinct caveats documented.
    lims = r.get("limitations", [])
    check_crit(len(lims) >= 4, f"limitations list has >= 4 entries (got {len(lims)})")
    # Subsample parameter round-trip.
    params = r.get("parameters", {})
    check_crit(params.get("bootstrap_subsample_size") is not None,
               "bootstrap_subsample_size parameter recorded")
    check_crit(params.get("null_subsample_size") is not None,
               "null_subsample_size parameter recorded")
    # Effect-size plausibility: observed Spearman rho in expected range.
    check_crit(PRIMARY_RHO_LOWER <= pri["spearman_rho"] <= PRIMARY_RHO_UPPER,
               f"primary rho in plausible range [{PRIMARY_RHO_LOWER}, {PRIMARY_RHO_UPPER}] (got {pri['spearman_rho']:.4f})")
    # Falsification / negative control: shuffle a synthetic rank vector and
    # confirm the Spearman implementation returns rho ≈ 0. This catches bugs
    # in rank_with_ties / spearman_from_ranks that would otherwise silently
    # bias the main estimate.
    rng_check = random.Random(SEED + 99)
    synth_a = list(range(200))
    synth_b = synth_a[:]
    rng_check.shuffle(synth_b)
    rx_syn = rank_with_ties(synth_a)
    ry_syn = rank_with_ties(synth_b)
    synth_rho = spearman_from_ranks(rx_syn, ry_syn)
    check_crit(abs(synth_rho) < FALSIFICATION_SHUFFLED_RHO_MAX,
               f"falsification: shuffled synthetic ranks give |rho| < {FALSIFICATION_SHUFFLED_RHO_MAX} (got {synth_rho:.4f})")
    # Sensitivity consistency: every cohort-split rho must be in the
    # plausible range. Catches a cohort that has degenerated (e.g., all
    # patents tied at zero cites) or a split that contradicts the main
    # finding.
    for s in sens["cohort_splits"]:
        check_crit(PRIMARY_RHO_LOWER <= s["spearman_rho"] <= PRIMARY_RHO_UPPER,
                   f"cohort {s['label']} rho in [{PRIMARY_RHO_LOWER}, {PRIMARY_RHO_UPPER}] (got {s['spearman_rho']:.4f})")
    # Min-cites sensitivity: every threshold must preserve the finding.
    for s in sens["min_cites"]:
        check_crit(PRIMARY_RHO_LOWER <= s["spearman_rho"] <= PRIMARY_RHO_UPPER,
                   f"min_cites={s['min_cites']} rho in [{PRIMARY_RHO_LOWER}, {PRIMARY_RHO_UPPER}] (got {s['spearman_rho']:.4f})")
    # Secondary comparator sanity: applicant-vs-examiner rho must be
    # strictly lower than the primary (total-vs-applicant) rho — because
    # stripping out applicant-only correlations leaves only the noisier
    # examiner-vs-applicant disagreement signal.
    check_crit(sec["spearman_rho"] < pri["spearman_rho"],
               f"secondary (applicant-vs-examiner) rho < primary rho ({sec['spearman_rho']:.4f} < {pri['spearman_rho']:.4f})")
    # Top-k retention should be in [0, 1] for all k (well-formedness).
    for k, v in pri["top_k_retention"].items():
        check_crit(0.0 <= v <= 1.0, f"{k} retention in [0, 1] (got {v:.4f})")
    # Data SHA256 must match optional pinned value for byte-level reproducibility.
    if expected_sha256 is not None:
        check_crit(r["data_sha256"] == expected_sha256, f"data SHA256 matches --expected-sha256 (got {r['data_sha256']})")

    # INFO checks — substantive scientific findings. Reported but do not fail the run.
    info_rho_null_sep = (pri["spearman_rho"] < null["null_rho_ci"][0] or pri["spearman_rho"] > null["null_rho_ci"][1])
    check_info(info_rho_null_sep, "observed rho falls outside null 95% CI (substantive: examiner flag carries signal)")

    # Print results
    print("=== Critical checks ===")
    for ok, desc in critical:
        print(f"[{'PASS' if ok else 'FAIL'}] {desc}")
    print("=== Informational checks (substantive findings; do not fail run) ===")
    for ok, desc in info:
        print(f"[{'INFO-PASS' if ok else 'INFO-FAIL'}] {desc}")
    n_ok_crit = sum(1 for ok, _ in critical if ok)
    n_crit = len(critical)
    if n_ok_crit < n_crit:
        print(f"CRITICAL FAILURES: {n_crit - n_ok_crit}/{n_crit}")
        sys.exit(1)
    print(f"ALL CHECKS PASSED ({n_ok_crit}/{n_crit} critical, {sum(1 for ok, _ in info if ok)}/{len(info)} informational)")

# ═══════════════════════════════════════════════════════════════
# Main
# ═══════════════════════════════════════════════════════════════

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--verify", action="store_true", help="Verify results.json passes sanity assertions")
    ap.add_argument("--expected-sha256", default=None, help="If set with --verify, require results.data_sha256 to match")
    args = ap.parse_args()
    # Seed all stochastic backends up-front for reproducibility.
    random.seed(SEED)
    if args.verify:
        verify(expected_sha256=args.expected_sha256)
        return
    try:
        data = load_data()
    except (urllib.error.URLError, TimeoutError, ConnectionError, OSError, RuntimeError) as e:
        print(f"ERROR: data acquisition failed: {type(e).__name__}: {e}", file=sys.stderr)
        print("Check network, available disk space, and that the PatentsView URL is still live.", file=sys.stderr)
        sys.exit(2)
    try:
        results = run_analysis(data)
    except Exception as e:
        print(f"ERROR: analysis failed: {type(e).__name__}: {e}", file=sys.stderr)
        raise
    try:
        generate_report(results)
    except OSError as e:
        print(f"ERROR: could not write outputs: {e}", file=sys.stderr)
        sys.exit(3)
    print("ANALYSIS COMPLETE")

if __name__ == "__main__":
    main()
SCRIPT_EOF
```

**Expected output:** No stdout (heredoc writes silently). File `/tmp/claw4s_auto_examiner-vs-applicant-citations-top-1-re-ranking/analysis.py` now exists.

**Failure condition:** `cat` cannot write the file (disk full, permissions).

---

## Step 3: Run analysis

```bash
cd /tmp/claw4s_auto_examiner-vs-applicant-citations-top-1-re-ranking && python3 analysis.py
```

**Expected output:**
- `[1/5] load_data: download + stream citation bulk file`
- `[2/6] run_analysis: primary Spearman + top-k effect sizes`
- `[3/6] run_analysis: bootstrap CIs (m-out-of-n, m=1000)`
- `[4/6] run_analysis: secondary comparator (applicant-only vs examiner-only rankings)`
- `[5/6] run_analysis: permutation null (random-examiner-flag, m-out-of-n m=1000)`
- `[6/6] run_analysis: sensitivity sweeps`
- `wrote results.json`
- `wrote report.md`
- Final line: `ANALYSIS COMPLETE`

**Expected files produced** (under the script's `WORKSPACE` = the directory containing `analysis.py`):
- `results.json`
- `report.md`
- `cache/g_us_patent_citation.tsv.zip`

**Runtime:** 30–60 minutes on a standard machine. The longest phase is typically the full-file stream (≈8–20 min) on the first run; the bulk-file download (~2 min with a fast connection) only occurs on cold-cache runs.

**Failure condition:** Python exits with non-zero status, or the expected `ANALYSIS COMPLETE` line is not emitted, or `results.json` / `report.md` are not created.

---

## Step 4: Verify

```bash
cd /tmp/claw4s_auto_examiner-vs-applicant-citations-top-1-re-ranking && python3 analysis.py --verify
```

**Expected output:** A `=== Critical checks ===` header, at least 28 `[PASS] <check>` lines (one per assertion), a `=== Informational checks ===` header, then 1 `[INFO-PASS] <check>` line, ending with an `ALL CHECKS PASSED (N/N critical, 1/1 informational)` summary line where N ≥ 28.

**Critical checks (structural well-formedness, effect-size plausibility, sensitivity consistency, and a falsification/negative control; must all pass):**
1. `n_patents >= 5,000` (cohort produced a substantial sample)
2. `n_rows_scanned >= 10M` (the full PatentsView file was streamed, not a partial)
3. Spearman rho is in the legal interval [-1, 1]
4. Bootstrap rho CI brackets the point estimate
5. Bootstrap rho CI has positive finite width (< 1)
6. Top-1% overlap in [0, 1]
7. Top-1% overlap CI brackets the point estimate
8. Applicant-cite share is in (0, 1)
9. Examiner-cite share is in (0, 1)
10. Category fractions (applicant + examiner + other) sum to 1 (tolerance 1e-3)
11. Exactly N cohort-split sensitivity rows exist (where N = `n_cohort_splits` parameter)
12. At least 2 minimum-cite-threshold sensitivity rows exist
13. Data SHA256 is a 64-character hex string
14. Permutation null used at least 500 iterations
15. Primary rho bootstrap CI span > 1% of estimate (substantive — not just numerical precision)
16. Null rho CI span > 1% of estimate (substantive — not just numerical precision)
17. Secondary applicant-vs-examiner rho is in [-1, 1]
18. Secondary rho CI brackets the point estimate
19. `limitations` list in results.json has ≥ 4 entries
20. `bootstrap_subsample_size` and `null_subsample_size` parameters recorded
21. **Effect-size plausibility:** primary rho in `[PRIMARY_RHO_LOWER, PRIMARY_RHO_UPPER]` (default [0.5, 0.99])
22. **Falsification / negative control:** on a deterministically shuffled synthetic rank vector, `spearman_from_ranks` must return |ρ| < `FALSIFICATION_SHUFFLED_RHO_MAX` (default 0.5) — catches ranking-pipeline bugs
23. **Sensitivity consistency:** every cohort-split rho is in the plausible range (robustness — the finding does not hinge on any single patent-number window)
24. **Sensitivity consistency:** every min-cites-threshold rho is in the plausible range (robustness — the finding does not depend on a specific low-cites cutoff)
25. **Secondary comparator sanity:** applicant-vs-examiner rho is strictly less than total-vs-applicant rho (construct-validity sanity check)
26. Top-k retention values (1%, 5%, 10%) are all in [0, 1]
27. CI width > 1% of estimate (repeated for null and primary)
28. Optional: `--expected-sha256` pin matches `data_sha256` (only when passed)

**Informational checks (substantive scientific findings; reported but do NOT fail the run):**
- Observed rho falls outside the null 95% CI — the examiner-flag effect is distinguishable from the random-flag null. An `[INFO-FAIL]` here indicates a small or absent effect in the current sample, not a bug in the pipeline.

**Optional flag:** `--expected-sha256 <hex>` to require an exact match of the recorded data SHA256. Use this for byte-level pin-to-snapshot reruns.

**Failure condition:** Any critical check prints `[FAIL]`, or the exit code is non-zero, or the `ALL CHECKS PASSED` summary line is missing. `[INFO-FAIL]` is not a failure condition.

---

## Success Criteria

Measurable conditions a passing run must satisfy:

1. **Step 3** (analysis) ends with the exact line `ANALYSIS COMPLETE`, produces `results.json` and `report.md`, and no Python traceback is printed.
2. **Step 4** (verification) ends with an `ALL CHECKS PASSED (N/N critical, 1/1 informational)` summary line (N ≥ 20) and exit code 0.
3. **All critical `--verify` assertions pass** (structural well-formedness; see Step 4 checklist 1–20).
4. **`results.json` is well-formed** and contains, at minimum:
   - `primary.spearman_rho` with a 1,000-resample bootstrap 95% CI (subsample size recorded in `parameters.bootstrap_subsample_size`),
   - `primary.top_k_overlap` and `primary.top_k_retention` at 1%, 5%, 10% with matching CIs,
   - `secondary_applicant_vs_examiner` (applicant-vs-examiner ranking comparator),
   - `null_model.null_rho_ci` (1,000-iteration random-examiner-flag null with subsample),
   - `sensitivity.cohort_splits` (4 cohorts) and `sensitivity.min_cites` (≥ 2 thresholds),
   - `limitations` list with ≥ 4 caveats,
   - `data_sha256` (SHA256 of the downloaded PatentsView zip).
5. **Effect sizes are in plausible ranges:** observed Spearman ρ ∈ [0.5, 0.99]; applicant-cite share ∈ (0, 1); all CI half-widths are positive and < 0.5.
6. **CI widths are substantively meaningful**, not vanishingly small: primary rho CI span > 1% of the estimate, null rho CI span > 1% of the estimate (both enforced by the verify harness).

## Failure Conditions

Each item below is a concrete condition under which the skill run is considered failed or the finding should be moderated:

- **Download failure:** All listed URLs return non-2xx or time out. The script exits with code 2 and prints a diagnostic to stderr. Remedy: check network, retry, or pin to a mirror URL in `CITATION_BULK_URL_FALLBACKS`.
- **Header mismatch:** `parse_citations_stream` raises `missing expected column`. Remedy: update `COL_CITED_PATENT`, `COL_CATEGORY`, `COL_CITING_PATENT` to match the current PatentsView schema.
- **Empty focal set:** `n_patents < 5,000`. Remedy: widen `FOCAL_PATENT_MIN` / `FOCAL_PATENT_MAX`.
- **Output write failure:** the script exits with code 3 and prints a diagnostic to stderr if `results.json` or `report.md` cannot be written (disk full, permission error).
- **Permutation CI contains observed rho:** indicates the observed examiner-stripping effect is statistically indistinguishable from random-flag reshuffling — substantive finding, not a bug. Verification reports this via an `[INFO-PASS]` / `[INFO-FAIL]` marker and does not fail the run in this case; the paper headline should then be moderated.
- **CI too narrow / too wide:** if primary rho CI span is < 1% of the estimate the verify harness will fail, which indicates either a bootstrap bug or the need to lower `BOOTSTRAP_SUBSAMPLE_SIZE`. If CI span is > 50% the subsample may be too small — raise `BOOTSTRAP_SUBSAMPLE_SIZE`.

## Limitations and Assumptions

The analysis output writes these caveats into `results.json.limitations` at runtime, and they should be cited anywhere the headline finding is reported:

1. **Snapshot cites, not lifetime cites.** The PatentsView bulk file records cumulative forward cites at snapshot time; cites accruing after the snapshot are not counted. Re-running on a later snapshot will shift counts.
2. **Category label validity.** The `cited by applicant/examiner/other` label is pass-through from USPTO. Labelling conventions may drift over time and across examining art units; a systematic labelling change (e.g., a rule change in 2001) can shift p̂ substantially.
3. **Focal-cohort specificity.** Results are reported for US patent numbers 7,200,000–7,400,000 (granted ~2007–2008); generalisation to earlier/later cohorts is only tested via the patent-number sub-cohort sensitivity.
4. **Independence assumption in the null.** The random-examiner-flag null assumes flags are independent Bernoulli per citation with a global p; patent-level serial dependence (e.g., within patent families) would make the null CI optimistic.
5. **Impact proxy choice.** "Impact" is equated with forward-citation count; the analysis does not validate against external impact measures (commercial outcomes, litigation, licensing).
6. **No causal claim.** This measures re-ranking magnitude, not the causal origin of examiner citations; examiner decisions are endogenous to applicant IDS disclosures.
7. **m-out-of-n bootstrap.** CIs are computed via m-out-of-n subsample bootstrap (default m = 1,000) to reflect study-scale uncertainty rather than the near-zero-width asymptotic CI at the population scale N ≈ 175 000. The full-N bootstrap CI is much tighter but conveys only the numerical precision of the population parameter, not substantive generalisability. To switch back to full-N, set `BOOTSTRAP_SUBSAMPLE_SIZE = None`.

## Data Provenance

- **Source:** PatentsView — https://patentsview.org/download/data-download-tables — file `g_us_patent_citation.tsv.zip`.
- **Columns used:** `patent_id` (citing), `citation_patent_id` (cited), `citation_category` (enum: "cited by applicant", "cited by examiner", "cited by other").
- **Integrity:** SHA256 of the downloaded zip is recorded at run time in `results.json` for exact provenance; this is a live data mirror, not a pinned snapshot, so the SHA is expected to evolve as PatentsView updates. To pin to a specific snapshot, compare the recorded `data_sha256` against a previously recorded value and abort if they differ.