---
name: variant-effect-calibration-audit
description: audit variant effect predictions for calibration-bin consistency, evidence support, and disease-context mismatch.
allowed-tools: Bash(python *), Bash(mkdir *), Bash(ls *), Bash(cp *), WebFetch
---

# VarCal

## Purpose

Use a transparent tabular audit to audit variant effect predictions for calibration-bin consistency, evidence support, and disease-context mismatch. The workflow is inspired by recent work in variant effect prediction, but it is an original evidence-screening skill and does not copy benchmark data, code, prose, or figures from the cited papers.

## Inputs

Create inputs/records.csv with columns:

variant_id,gene,model_score,bin_pathogenic_rate,calibration_delta,evidence_count,disease_specific,clinical_label

Create inputs/rules.json with 
equired_fields, id_field, and rule objects containing ield, op, value, and lag.

## Run

`ash
python scripts/audit_variant_effect_calibration_audit.py \
  --records inputs/records.csv \
  --rules inputs/rules.json \
  --out outputs/audit \
  --title "VarCal"
`

## Outputs

- outputs/audit/audit.json: full machine-readable results.
- outputs/audit/audit_report.csv: compact record-level status table.
- outputs/audit/review.md: human-readable audit report.

## Self-Test

Use the included fixture:

`ash
python scripts/audit_variant_effect_calibration_audit.py \
  --records examples/fixture/records.csv \
  --rules examples/fixture/rules.json \
  --out outputs/fixture \
  --title "VarCal"
`

The fixture should produce at least one 
eeds_review record so the flagging path is tested.

## Audit Script

Create scripts/audit_variant_effect_calibration_audit.py with this code if the package file is unavailable:

`python
#!/usr/bin/env python3
import argparse
import csv
import json
from pathlib import Path


def read_csv(path):
    with Path(path).open("r", encoding="utf-8-sig", newline="") as handle:
        return list(csv.DictReader(handle))


def coerce(value):
    if value is None:
        return ""
    text = str(value).strip()
    if text.lower() in {"true", "yes", "y"}:
        return True
    if text.lower() in {"false", "no", "n"}:
        return False
    try:
        return float(text)
    except ValueError:
        return text


def compare(actual, op, expected):
    actual = coerce(actual)
    expected = coerce(expected)
    if op == "lt":
        return isinstance(actual, (int, float)) and actual < expected
    if op == "lte":
        return isinstance(actual, (int, float)) and actual <= expected
    if op == "gt":
        return isinstance(actual, (int, float)) and actual > expected
    if op == "gte":
        return isinstance(actual, (int, float)) and actual >= expected
    if op == "eq":
        return str(actual).lower() == str(expected).lower()
    if op == "ne":
        return str(actual).lower() != str(expected).lower()
    if op == "contains":
        return str(expected).lower() in str(actual).lower()
    raise ValueError(f"Unsupported operator: {op}")


def audit(records, rules):
    required = rules.get("required_fields", [])
    rule_items = rules.get("rules", [])
    id_field = rules.get("id_field", required[0] if required else "id")
    results = []

    for index, row in enumerate(records, start=1):
        flags = []
        for field in required:
            if field not in row or str(row.get(field, "")).strip() == "":
                flags.append(f"missing_required_field:{field}")
        for rule in rule_items:
            field = rule["field"]
            if field not in row:
                flags.append(f"missing_rule_field:{field}")
                continue
            if compare(row.get(field), rule["op"], rule["value"]):
                flags.append(rule["flag"])
        status = "pass" if not flags else "needs_review"
        results.append({
            "row_index": index,
            "record_id": row.get(id_field, str(index)),
            "status": status,
            "flags": flags,
            "record": row,
        })

    return {
        "summary": {
            "record_count": len(results),
            "pass_count": sum(1 for item in results if item["status"] == "pass"),
            "needs_review_count": sum(1 for item in results if item["status"] != "pass"),
        },
        "results": results,
    }


def write_outputs(result, out_dir, title):
    out = Path(out_dir)
    out.mkdir(parents=True, exist_ok=True)
    (out / "audit.json").write_text(json.dumps(result, indent=2), encoding="utf-8")

    with (out / "audit_report.csv").open("w", encoding="utf-8", newline="") as handle:
        writer = csv.DictWriter(handle, fieldnames=["record_id", "status", "flags"])
        writer.writeheader()
        for item in result["results"]:
            writer.writerow({
                "record_id": item["record_id"],
                "status": item["status"],
                "flags": ";".join(item["flags"]),
            })

    lines = [
        f"# {title}",
        "",
        "## Summary",
        f"- Records audited: {result['summary']['record_count']}",
        f"- Passed: {result['summary']['pass_count']}",
        f"- Needs review: {result['summary']['needs_review_count']}",
        "",
        "## Flagged Records",
    ]
    flagged = [item for item in result["results"] if item["flags"]]
    if not flagged:
        lines.append("- No records were flagged.")
    for item in flagged:
        lines.append(f"- {item['record_id']}: {', '.join(item['flags'])}")
    lines.extend([
        "",
        "## Interpretation",
        "This audit is a reproducible evidence screen. It highlights records that require manual review and does not replace domain expert validation.",
    ])
    (out / "review.md").write_text("\n".join(lines) + "\n", encoding="utf-8")


def main():
    parser = argparse.ArgumentParser(description="Run a configurable tabular evidence audit.")
    parser.add_argument("--records", required=True)
    parser.add_argument("--rules", required=True)
    parser.add_argument("--out", default="outputs/audit")
    parser.add_argument("--title", default="Evidence Audit")
    args = parser.parse_args()

    records = read_csv(args.records)
    rules = json.loads(Path(args.rules).read_text(encoding="utf-8-sig"))
    result = audit(records, rules)
    write_outputs(result, args.out, args.title)
    print(json.dumps({"status": "ok", **result["summary"], "out": args.out}, indent=2))


if __name__ == "__main__":
    main()
`

## Interpretation Rules

- Treat pass as "no automatic risk flags found", not proof that the scientific claim is true.
- Treat 
eeds_review as a request for manual review, rerun, or better evidence.
- Preserve all input tables and rules used for the audit.
- Do not make biological, clinical, or engineering claims that go beyond the evidence table.

## Success Criteria

- The script runs using only the Python standard library.
- The fixture generates audit.json, audit_report.csv, and 
eview.md.
- At least one fixture row is flagged for review.
- The final report names the exact rules that triggered each flag.

## Inspiration Sources

- [DYNA: Disease-Specific Language Model for Variant Pathogenicity](https://arxiv.org/abs/2406.00164)
- [Fine-tuning Protein Language Models with Deep Mutational Scanning improves Variant Effect Prediction](https://arxiv.org/abs/2405.06729)
- [Leveraging genomic deep learning models for non-coding variant effect prediction](https://arxiv.org/abs/2411.11158)