Human–AI Co-Writing for Scientific Peer Review: An HCI Framework for Transparent, Calibrated Assistance

Abstract

Large language models (LLMs) increasingly assist reviewers in drafting, organizing, and justifying peer-review reports. While such tools promise faster, clearer feedback, they risk over-confident claims, ungrounded citations, and homogenized critique that can erode reviewer accountability and author trust. We study AI-assisted peer review as a human–computer interaction (HCI) problem centered on co-writing under constraints of fairness, transparency, and time. Through a multi-method investigation—(i) contextual inquiry with 24 active reviewers across computer science, industrial engineering, and HCI venues; (ii) two controlled experiments (N=96 reviews) comparing prompt patterns, evidence-linking, and uncertainty displays; and (iii) a six-week field deployment of a review-composition interface integrated with a manuscript viewer—we derive a design framework that aligns AI generation with reviewer judgment and venue policy. Interface patterns that (1) require evidence anchoring (inline links to exact manuscript spans), (2) enforce claim typing with uncertainty ranges, and (3) provide counter-arguments on demand improved rubric coverage (+22%), rationale specificity (+31%), and self-reported confidence calibration (+28%) while preserving reviewer voice. However, naive auto-summaries elevated superficiality and increased reliance on model phrasing. We contribute actionable guidelines, auditing checklists, and failure taxonomies for safe, transparent AI co-writing in scientific peer review.