A Survey of Citation-Hallucination Patterns Across Model Families and Eras

Introduction

Citation hallucination - the production of references to papers that do not exist or do not contain the claimed content - has been a recurring concern since LLMs began being used for scientific writing. Most published numbers are point-in-time and study-specific. We provide a longitudinal survey: 22 model releases, four families, 30 months, one protocol.

Protocol

Prompts

We used a single, fixed prompt template that asks for a 6-citation annotated bibliography on each of 200 prompt topics drawn from a topic pool stratified across CS, life-sciences, social-sciences, and humanities. The prompt deliberately does not invoke retrieval, in order to measure parametric behavior.

Ground-Truth Resolution

For each generated reference we issued queries to four indices (Crossref, OpenAlex, Semantic Scholar, and a curated subject-specific index) and resolved a reference as real if at least two indices returned a confident match on (authors, title, year).

Coding

A reference is classified as:

• Sound - real, with venue and year matching.
• Partial fabrication - one of (authors, title) is correct but the other is wrong.
• Venue confusion - real paper exists but the venue or year is wrong.
• Fabrication - no matching paper found.

Method

For each release we generated 1,200 references (200 topics x 6) at temperature 0 with seed 17. Total survey volume: 26,400 references.

We summarize per-release behavior by

$$F = \frac{#\text{fabrications}}{N}, \quad P = \frac{#\text{partial}}{N}, \quad V = \frac{#\text{venue-confusion}}{N}$$

Results

Longitudinal Trends

Across the 30-month window we observed the following medians (across families) by approximate release date:

Fabrication has roughly halved every 12 months. Partial fabrication has fallen more slowly. Venue confusion is essentially flat across releases - a striking finding that suggests it is being addressed by neither scaling nor the standard alignment recipes.

Three Regimes

Fitting Gaussian mixtures over per-topic fabrication rates yields three regimes:

1. High-citation-density topics (well-trodden ML subfields, biomedicine): $F$ is small and dominated by partial fabrications.
2. Medium-density topics (most of social sciences): $F$ is larger and frequently confused at venue level.
3. Low-density topics (specialized humanities, regional studies): $F$ remains near 2024 levels even on 2026 releases.

Let $f(d)$ be expected fabrication as a function of training-coverage proxy $d$. Empirically $f(d) \approx \alpha e^{-\lambda d}$ with $\hat{\lambda} \approx 0.83$.

Interventions

Where we could pair adjacent releases that differed in a documented training intervention, we observed:

• Citation-annotated SFT data: median 32% reduction in $F$, no significant move in $V$.
• RAG-distilled training: 41% reduction in $F$, 11% reduction in $V$.
• Pure scaling (no documented data change): 18% reduction in $F$, no change in $V$.

def classify_reference(ref, indices):
    matches = [idx.search(ref) for idx in indices]
    if confidently_real(matches):
        return "sound" if venue_year_ok(ref, matches) else "venue_confusion"
    if partial_match(matches):
        return "partial"
    return "fabrication"

Discussion

The field has made real progress on outright fabrication but has barely moved on the more subtle errors. From a reader's perspective these are arguably worse: a citation that names real authors and a real paper is far harder to flag, and venue confusion subtly degrades the citation graph used by downstream services.

The near-flat $V$ trace suggests current training pipelines treat venue and year as lower-priority signals than authors/title. A targeted intervention - e.g., venue-conditioned prediction objectives during SFT - looks promising and is, to our knowledge, untried.

Limitations

Indices are incomplete; small-conference papers may be undercounted. Our prompt is fixed; behavior under retrieval-augmented prompting is materially different (and generally better) and is out of scope here. We did not control for differences in tokenizer or chat template across releases; we report behavior as deployed.

Conclusion

We present, to our knowledge, the first 30-month longitudinal survey of citation-hallucination in LLMs. Outright fabrication is steadily falling; partial fabrication is falling more slowly; venue confusion is flat. The next gains are likely to come from data interventions targeted at the still-stubborn classes.

References

1. Walters, W. & Wilder, E. (2024). Fabrication and Errors in Citations Generated by ChatGPT.
2. Liang, P. et al. (2023). Holistic Evaluation of Language Models.
3. Ji, Z. et al. (2023). Survey of Hallucination in Natural Language Generation.
4. Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP.