Authorship Attribution in AI-Co-Authored Manuscripts

1. Introduction

As journals adopt mandatory AI-disclosure policies (e.g., Nature, IEEE, NeurIPS 2024), a quantitative question follows: given a published manuscript, can we attribute each paragraph to a level of human involvement? The question is not academic: tenure committees, funding agencies, and integrity offices increasingly need defensible measurements rather than self-reports.

We formulate paragraph-level attribution as a three-class classification problem with classes

• $H$: human-drafted, AI not used
• $E$: AI-drafted, human-edited substantively
• $P$: AI-drafted, human polish only

Our contributions are:

1. A 1{,}204-paragraph dataset with adjudicated labels.
2. A hybrid attribution model combining three signal families.
3. A calibration analysis showing where the estimator should and should not be used.

2. Threat Model and Constraints

We assume the analyst has access to: (i) the published Markdown/PDF, (ii) optionally, the version-control history of the manuscript, (iii) optionally, a provenance commitment of the form described in [provenance literature, 2024]. We do not assume access to the original LLM logits.

The relevant adversary is an author who wishes to under-report AI involvement. We therefore design our estimator to be robust to lossy intermediate steps such as paraphrasing through a second model.

3. Method

3.1 Stylometric features

For each paragraph $p$ we compute a 312-dimensional feature vector $\phi(p)$ comprising token-length distribution moments, function-word frequencies (top-150), Yule's K, sentence-length entropy, and 12 syntactic-tree shape statistics.

3.2 Edit-trace features

If a Git history is available, we compute per-paragraph edit signatures: the ratio of insertions to deletions, the burstiness of edits in time, and the median latency between LLM API calls and the next commit. Let $\tau_p$ be this 18-dim vector.

3.3 Provenance features

If a provenance commitment is published, we extract per-token attestation flags. Let $\pi_p \in {0, 1}^*$ be the resulting bitmask. We summarize via the fraction of attested tokens and the longest contiguous attested run.

3.4 Hybrid model

We train a gradient-boosted decision-tree classifier on the concatenation $[\phi(p), \tau_p, \pi_p]$ with missingness indicators. Class imbalance is addressed via inverse-frequency weighting.

import lightgbm as lgb
model = lgb.LGBMClassifier(
    objective="multiclass", num_class=3,
    class_weight="balanced", n_estimators=400
)
model.fit(X_train_with_missing_flags, y_train)

4. Dataset

We assembled 1{,}204 paragraphs from 142 manuscripts donated by authors who consented to label disclosure. Two trained annotators reviewed each paragraph alongside the version history and assigned an $H$/$E$/$P$ label; a third adjudicated disagreements ($\kappa = 0.74$). The class balance is 41% / 33% / 26%.

5. Results

5.1 Headline performance

5.2 Robustness to paraphrase laundering

We stress-test by routing $P$-class paragraphs through a second LLM tasked with paraphrase. Stylometry-only F1 collapses from 0.61 to 0.43. The hybrid retains F1 = 0.71, primarily because edit-trace and provenance features are unaffected by laundering.

5.3 Calibration

Using the empirical-Bayes calibration plot, the hybrid is well-calibrated for posteriors above 0.7. Below this threshold, predictions are best treated as flags for human review rather than as decisions.

6. Discussion

We emphasize two findings. First, no single signal family suffices: the relative ordering of stylometry and metadata flips depending on whether laundering is suspected. Second, when provenance commitments are present, the analyst's job is largely consistency-checking rather than detection, suggesting that the most cost-effective integrity intervention is at the publishing-platform layer.

A novel concern is adversarial co-author personas --- humans who imitate LLM stylometry to deflect attribution. Pilot data ($n = 30$) suggests this is feasible at modest skill cost; future work should quantify it.

7. Limitations

Our dataset over-represents English-language ML manuscripts. Translation across stylometric registers (e.g., Mandarin academic prose) is unstudied. The labeled categories collapse what is plausibly a continuum into three bins.

8. Conclusion

Reliable per-paragraph attribution is achievable when the analyst can combine stylometric, edit-trace, and provenance signals, and when posteriors are reported with calibrated thresholds. We do not recommend that the estimator be used to make adverse career decisions in isolation.

References

1. Stamatatos, E. (2009). A Survey of Modern Authorship Attribution Methods.
2. Gao, C. et al. (2023). Comparing Scientific Abstracts Generated by ChatGPT to Original.
3. Nature Editorial (2024). Tools such as ChatGPT threaten transparent science.