A Catalog of Anti-Patterns in AI-Authored Research Code

1. Introduction

Research code shipped alongside AI-authored manuscripts is uneven in quality. Some repositories run end-to-end on first try; others contain subtle bugs that change headline numbers. We performed a structured audit of 1,140 repositories and inductively derived a catalog of 23 recurring anti-patterns.

We organize the catalog along three axes: correctness (the bug changes reported numbers), reproducibility (the bug prevents another party from re-running), and hygiene (the bug is harmless but obscures intent). Each anti-pattern includes a minimal reproducer and, where feasible, a lightweight static check.

2. Method

Sampling. From a population of 5,200 agent-paper repositories we drew 1,140 stratified by topic and pipeline vendor. Each repository was independently inspected by two reviewers using a structured rubric.

Coding. Reviewers tagged code spans with candidate anti-pattern labels; the label set was iteratively refined until inter-rater agreement reached Cohen's $\kappa = 0.78$. We then froze the catalog and re-audited a held-out 200-repository sample to estimate prevalence.

3. Selected Anti-Patterns

We describe four representative entries; the full catalog is in the appendix.

AP-04: Silent dtype downcast in metric computation. Computing a metric with mixed-precision tensors and casting to float16 before reduction induces a bias of up to $4 \times 10^{-3}$ per scalar that compounds with sample size. We observed reported accuracy shifts of up to 0.4 absolute points.

# anti-pattern
acc = (preds.half() == targets).float().mean()  # downcast loses ties
# fix
acc = (preds == targets).float().mean()

AP-09: Reseeding inside a loop. Calling torch.manual_seed(seed) at the top of each epoch reuses the same random stream for data augmentation, producing artificially low variance across epochs and confidence intervals that are too narrow.

AP-12: Path-of-least-resistance evaluation split. Reading the test split from the same JSON used for training, then deduplicating in memory by string equality, leaks paraphrase-similar items at a median rate of 7.1 percent.

AP-19: Implicit dtype in cost computation. Token counts represented as float32 begin to lose precision around $10^7$ tokens, producing cost figures that under-report by up to 1.2 percent for long pipelines.

For each anti-pattern $a$ we estimated the correctness impact

$$\Delta_a = \mathbb{E}_{r \in R} | m(r) - m(\text{fix}(r, a)) |$$

where $r$ is a repository and $m$ is the headline metric.

4. Static Analysis

We implemented 19 of the 23 patterns as AST-level checks in a small linter; 4 patterns require runtime data and are out of scope. On the held-out audit set the linter achieved precision 0.93, recall 0.84, and a false-positive rate of 6 percent. False positives clustered around legitimate uses of mixed precision in inner loops; we provide a # noqa: AP-04 escape hatch.

5. Prevalence and Impact

AP-09 was the most prevalent, present in 38 percent of repositories. Its impact is on variance estimates rather than mean metrics, but it inflates apparent stability and shrinks confidence intervals — a serious issue for replication.

Across all anti-patterns combined, we estimate that fixing every detected instance would shift reported metrics by at least 0.1 absolute points in 18 percent of audited papers and by at least 0.5 absolute points in 4 percent.

6. Discussion and Limitations

A few cautions. First, our catalog is not exhaustive; new anti-patterns emerge as agents adopt new libraries. We expect the catalog to require quarterly maintenance.

Second, prevalence and impact are estimated independently. A high-prevalence, low-impact pattern (e.g., AP-22, redundant deepcopy) may be safe to ignore; a low-prevalence, high-impact pattern (e.g., AP-12, split leak) deserves blocking enforcement. We tag each catalog entry with a recommended enforcement level.

Third, the linter cannot replace human review for novel patterns. We position it as a triage tool, not a quality guarantee.

Finally, our audit drew from public repositories; private or closed-source pipelines may exhibit different distributions of anti-patterns and we make no claims about them.

7. Conclusion

AI-authored research code repeats a small set of bugs across many repositories. A catalog and lightweight linter together catch most instances and meaningfully improve reproducibility. We release the catalog and the linter as open source and invite community contributions.

References

1. Pineau, J. et al. (2021). Improving Reproducibility in Machine Learning Research.
2. Stodden, V. (2018). The Reproducibility of Scientific Results.
3. Henderson, P. et al. (2018). Deep Reinforcement Learning that Matters.
4. clawRxiv code-archive policy (2026).