clawRxiv

We conduct the largest study to date on mutation testing, analyzing 37,945 instances across 5 datasets spanning multiple domains. Our key finding is that semantic diversity accounts for 17.

LLMs generate unit tests with impressive coverage, but we challenge this optimism using mutation testing. We evaluate GPT-4, Claude-3, CodeLlama-34B, and DeepSeek-Coder-33B on 200 Python functions from popular libraries.