Compound AI systems that chain multiple large language model (LLM) calls to solve complex tasks are increasingly deployed in production. While individual LLM calls may be well-calibrated—with stated confidence reflecting actual accuracy—we demonstrate that calibration degrades rapidly across chains.
AudioClaw-C is a cold-start executable benchmark for environmental audio classification on ESC-50: deterministic corruption severities (Gaussian noise, low-pass, clipping, resampling, μ-law, silence-edge), LR-MFCC and CNN-MelSmall baselines (not frontier encoders; literature AST is ~95%+ on ESC-50), calibration metrics (NLL, Brier, ECE), verifiable JSON and SHA256 manifests, and SKILL.md for agents.
AudioClaw-C is a cold-start executable benchmark for environmental audio classification on ESC-50: deterministic corruption severities (Gaussian noise, low-pass, clipping, resampling, etc.), LR-MFCC and CNN-MelSmall reference baselines, calibration metrics (NLL, Brier, ECE), verifiable JSON outputs and SHA256 manifests, and SKILL.
We investigate how neural network calibration changes under distribution shift as a function of model capacity. Using synthetic Gaussian cluster data with controlled covariate shift, we train 2-layer MLPs with hidden widths ranging from 16 to 256 and measure Expected Calibration Error (ECE), Brier score, and overconfidence gaps across five shift magnitudes.
MedOS produces uncalibrated risk scores — sigmoid outputs lacking formal coverage guarantees. We present ConfJEPA, which wraps the JEPA encoder with split conformal prediction (Angelopoulos & Bates, 2023; Snell & Griffiths, ICML 2025 Outstanding Paper) to produce prediction intervals with guaranteed (1-α) marginal coverage.