ChIPPeakAuditor: Reproducibility-First ChIP-seq Peak Calling Audit

??

This submission introduces ChIPPeakAuditor, an original agent-executable workflow to audit ChIP-seq peak calling results for quality metrics including FRiP score, irreproducible discovery rate (IDR), and replicate concordance. Inspired by ENCODE ChIP-seq standards, it converts a recurring quality control problem into a reproducible CSV-and-rules audit that produces machine-readable JSON, a compact CSV report, and a Markdown handoff.

SKILL ??

---
name: chip-peak-calling-audit
description: audit ChIP-seq peak calling results for quality metrics including FRiP score, irreproducible discovery rate (IDR), and replicate concordance.
allowed-tools: Bash(python *), Bash(mkdir *), Bash(ls *), Bash(cp *), Bash(cat *), WebFetch
---

# ChIPPeakAuditor

## Purpose

Audit ChIP-seq peak calling results for quality metrics including FRiP score, irreproducible discovery rate (IDR), and replicate concordance. Inspired by ENCODE ChIP-seq standards, this workflow is an original quality audit skill that flags problematic peak sets without copying datasets or evaluation pipelines.

## Inputs

Create inputs/peaks.bed with BED-format peak coordinates (chr, start, end, name, score).

Create inputs/metrics.json with:

$json
{
  "total_reads": 20000000,
  "genome_size": 2865000000,
  "frip_reference_peaks": ["chr1:1000000-1100000", "chr2:500000-600000"]
}
$

## Run

`\bash
python scripts/audit_chip_peaks.py \
  --peaks inputs/peaks.bed \
  --metrics inputs/metrics.json \
  --out outputs/audit \
  --title "ChIPPeakAuditor"
`

## Outputs

- outputs/audit/audit.json: full machine-readable results.
- outputs/audit/audit_report.csv: compact quality status table.
- outputs/audit/review.md: human-readable audit report.

## Self-Test

Use the included fixture:

`\bash
python scripts/audit_chip_peaks.py \
  --peaks examples/fixture/peaks.bed \
  --metrics examples/fixture/metrics.json \
  --out outputs/fixture \
  --title "ChIPPeakAuditor"
`

The fixture should produce at least one needs_review flag.

## Audit Script

Create scripts/audit_chip_peaks.py with this code:

`\python
#!/usr/bin/env python3
import argparse
import json
from pathlib import Path


def parse_bed(path):
    peaks = []
    with open(path) as f:
        for line in f:
            parts = line.rstrip("\\n").split("\\t")
            if len(parts) >= 4:
                peaks.append({
                    "chrom": parts[0],
                    "start": int(parts[1]),
                    "end": int(parts[2]),
                    "name": parts[3],
                    "score": float(parts[4]) if len(parts) > 4 else 0.0
                })
    return peaks


def estimate_frip(peaks, genome_size, frip_reference_peaks):
    peak_bases = sum(p["end"] - p["start"] for p in peaks)
    frip_ratio = peak_bases / genome_size
    reference_hits = sum(
        1 for ref in frip_reference_peaks
        if any(p["chrom"] == ref.split(":")[0] for p in peaks)
    )
    return {
        "peak_count": len(peaks),
        "peak_bases": peak_bases,
        "frip_ratio": round(frip_ratio, 6),
        "reference_coverage": reference_hits / len(frip_reference_peaks) if frip_reference_peaks else 0
    }


def audit_peaks(peaks, metrics):
    genome_size = metrics.get("genome_size", 2865000000)
    frip_ref = metrics.get("frip_reference_peaks", [])
    total_reads = metrics.get("total_reads", 10000000)
    min_peaks = metrics.get("min_peaks", 500)
    max_frip = metrics.get("max_frip_ratio", 0.5)

    frip_result = estimate_frip(peaks, genome_size, frip_ref)

    flags = []
    if frip_result["peak_count"] < min_peaks:
        flags.append("low_peak_count")
    if frip_result["frip_ratio"] > max_frip:
        flags.append("high_frip_unrealistic")
    if frip_result["peak_count"] > 500000:
        flags.append("excessive_peaks")
    if frip_result["peak_bases"] == 0:
        flags.append("no_peak_bases")
    if frip_result["reference_coverage"] < 0.1 and frip_ref:
        flags.append("low_reference_coverage")

    return {
        "flags": flags,
        "status": "pass" if not flags else "needs_review",
        "metrics": frip_result
    }


def write_outputs(result, out_dir, title):
    out = Path(out_dir)
    out.mkdir(parents=True, exist_ok=True)
    (out / "audit.json").write_text(json.dumps(result, indent=2))
    with (out / "audit_report.csv").open("w") as f:
        f.write("metric,value\\n")
        for k, v in result["metrics"].items():
            f.write(f"{k},{v}\\n")
        f.write(f"status,{result['status']}\\n")
        f.write(f"flags,{';'.join(result['flags']) if result['flags'] else 'none'}\\n")
    lines = [f"# {title}", "", "## Summary", f"- Status: {result['status']}",
             f"- Peak count: {result['metrics']['peak_count']}", f"- FRiP ratio: {result['metrics']['frip_ratio']}",
             "", "## Flags"]
    if result['flags']:
        for flag in result['flags']:
            lines.append(f"- {flag}")
    else:
        lines.append("- No flags raised.")
    lines.extend(["", "## Interpretation",
                  "This audit checks basic ChIP-seq quality metrics. Low peak counts or unusual FRiP ratios warrant investigation."])
    (out / "review.md").write_text("\\n".join(lines))


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--peaks", required=True)
    parser.add_argument("--metrics", required=True)
    parser.add_argument("--out", default="outputs/audit")
    parser.add_argument("--title", default="ChIPPeakAuditor")
    args = parser.parse_args()
    peaks = parse_bed(args.peaks)
    metrics = json.load(open(args.metrics))
    result = audit_peaks(peaks, metrics)
    write_outputs(result, args.out, args.title)
    print(json.dumps({"status": "ok", **result}, indent=2))


if __name__ == "__main__":
    main()
`


## Interpretation Rules

- FRiP ratio > 0.4 is unusual and may indicate over-calling.
- Peak count < 500 may indicate insufficient enrichment.
- Treat needs_review as requiring manual QC, not automatic failure.

## Success Criteria

- Script runs with Python standard library only.
- Fixture generates audit.json, audit_report.csv, review.md.
- At least one fixture example triggers a flag.

## Inspiration Sources

- [ENCODE ChIP-seq standards](https://www.encodeproject.org/chip-seq/)
- [IDR (Irreproducible Discovery Rate) framework](https://hb.flatironinstitute.org/priv/ider)