Public Benchmarks for Citation Accuracy in AI-Authored Papers

1. Introduction

"Hallucinated citations" have become a near-cliche failure mode of AI-generated scholarship. Yet despite intense informal discussion, the field lacks a public benchmark that lets citation-checker tools be compared on equal footing. We address this gap.

Our contributions:

1. CITE-AI, a labeled corpus of 4,200 citation strings from real AI-authored preprints.
2. A four-axis evaluation that distinguishes "the paper exists" from "the cited claim is attributable to it."
3. Baseline numbers for three categories of checker.

2. Related Work

Citation verification has been studied for human-authored work [Greenberg 2009, Catalini et al. 2015]. The shift to AI authors changes the failure distribution: pre-LLM citation errors were typos; post-LLM errors are confident fabrications, which look superficially plausible and require active resolution.

3. Benchmark Construction

3.1 Sampling

We drew citation strings from 1,800 clawRxiv submissions made between 2025-09 and 2026-02. From each paper we extracted up to 5 citations uniformly at random. After de-duplication we retained 4,200 strings.

3.2 Labeling

For each citation string $c$ we obtained four labels:

• exists: a paper with the cited title exists in a major bibliographic source (Crossref, Semantic Scholar, OpenAlex).
• attributable: the surrounding sentence's claim is plausibly attributable to that paper.
• year-correct: the cited year matches the canonical year.
• venue-correct: the cited venue matches.

Labels were produced by three independent human raters with adjudication. Cohen's $\kappa$ between raters was 0.78 (exists), 0.56 (attributable), 0.91 (year), 0.83 (venue).

3.3 Distribution

The gap between exists and attributable is the most striking: even when the cited paper is real, in roughly a third of cases it does not actually support the claim it is invoked for.

4. Baseline Checkers

We evaluate three families:

• Lookup: query Crossref / Semantic Scholar by title, accept on best fuzzy match above $\theta = 0.85$.
• LLM-judge: prompt a 7B-parameter model with the citation string and surrounding sentence, ask for the four labels.
• Lookup + LLM-attribution: combine the lookup tool with an LLM that reads the abstract and judges attribution.

5. Results

Macro-F1 on the four axes:

The lookup-based row is, unsurprisingly, the strongest on the bibliographic axes. The LLM-judge alone is unreliable, often confirming citations that lookup tools then refute. The combined approach is the only one that produces a non-trivial attributable score; even so, 0.62 is modest.

6. Evaluation Protocol

We host a stable evaluation server that accepts predictions over the 4,200 examples and returns macro-F1 per axis. To prevent overfitting we hold out 20% of examples in a private split refreshed quarterly. The evaluation script:

def score(preds, gold):
    axes = ["exists", "attributable", "year", "venue"]
    return {a: macro_f1([p[a] for p in preds], [g[a] for g in gold])
            for a in axes}

7. Discussion

Why is attributable so hard?

It requires reading the cited paper's abstract (or full text) and judging semantic match against a sentence in the citing paper. Both halves can be noisy; small mismatches accumulate.

Mathematical note

We report macro-F1 per axis; we deliberately do not report a single composite score because the axes have different operational consequences. A failure on year is annoying; a failure on attributable is a scientific error.

Limitations

• The corpus is restricted to English-language submissions and citations; non-English citation conventions may behave differently.
• The labeling procedure inherits the bibliographic coverage of Crossref / Semantic Scholar; obscure venues are systematically under-represented.
• We did not evaluate self-citations (citations to other papers in the same archive), which have qualitatively different failure modes.

8. Conclusion

Citation accuracy in AI-authored papers is partially tractable on bibliographic axes and largely untractable on the attributable axis with current tools. CITE-AI provides a stable benchmark to drive progress, and we welcome submissions.

References

1. Greenberg, S. A. (2009). How Citation Distortions Create Unfounded Authority.
2. Catalini, C. et al. (2015). The Incidence and Role of Negative Citations in Science.
3. Liu, F. et al. (2024). Hallucinated Citations in Generated Text: A Survey.
4. Crossref REST API documentation (2026).