AI-generated portal message drafts often missed clinical context and required physician editing in a Dartmouth analysis.
Artificial intelligence-generated draft responses to patient portal messages may not consistently reduce clinicians’ documentation burden and, in some cases, could add editing work, according to a Dartmouth-led analysis of more than 146 000 primary care portal conversations presented at the 2026 Annual Meeting of the Association for Computational Linguistics and published in the conference proceedings.1,2
“We find that AI can sound like a doctor but not think like one,” Sarah Preum, PhD, MSc, assistant professor of computer science at Dartmouth and co-corresponding author of the study, said in the release.1 The finding is clinically relevant as health systems increasingly deploy large language models (LLMs) to address growing volumes of asynchronous patient messages, a workflow already linked to substantial after-hours electronic health record work in primary care.3
Key Facts
The Dartmouth team described the project as the first large-scale evaluation of an online patient portal using AI to draft physician responses to patients. Investigators developed a comparison tool using a dataset of real responses created with health care professionals from Dartmouth Health, then analyzed 146 000 conversations between 10 105 patients and their primary care physicians at the rural health system. The study was approved by the Dartmouth Health Institutional Review Board, with anonymization and other privacy protections applied as required, according to the release.1
Researchers evaluated AI-generated draft responses from the portal system and from Claude, Gemini, ChatGPT, Llama, Aloe, and Qwen. Across platforms, they reported frequent misalignment between AI drafts and what clinicians would actually write. Common problems included overly long replies, failure to ask relevant follow-up questions, and inclusion of irrelevant or inaccurate medical details. In one example cited by the investigators, an AI draft suggested dietary adjustment for a 32-year-old woman taking an acid reflux medication who reported persistent nausea; the physician instead asked whether pregnancy was possible.1
The study’s central outcome was not traditional diagnostic accuracy or patient outcome improvement, but the degree to which clinicians would need to edit an AI-generated message before sending it. That distinction is important for clinical implementation: an AI-generated draft that appears polished may still require substantial physician review if it misses the key clinical question or introduces extraneous information.
The investigators also developed a training approach called Thematic Agentic Direct Preference Optimization for Learning Enhancement, or TADPOLE, which used a hybrid model built from physician- and AI-generated responses. When applied to the 6 commercial LLMs, TADPOLE improved alignment with physician standards for precision and information quality, increasing accuracy by 33% and reducing editing by 26%, according to the release.1
Tim Burdick, MD, MSc, associate professor of community and family medicine at Dartmouth’s Geisel School of Medicine and a family medicine physician at Dartmouth Health, said in the release that efficient, high-quality message generation could improve workflow if it “asks the right things.” He added, however, that he does not foresee patient portals responding without clinician review, underscoring the continued need for physician oversight.1
The findings add nuance to earlier enthusiasm around chatbot-generated medical communication. In a 2023 cross-sectional study, licensed health care professionals rated chatbot responses to public patient questions as higher in quality and empathy than physician responses, but that study used public forum questions rather than live patient portal workflows and did not evaluate integration into clinical documentation burden.4 The Dartmouth analysis instead focused on how much editing physicians might need to perform in real-world portal conversations, a practical determinant of whether AI reduces or redistributes workload.
The patient population also has implications for portal design. Investigators reported that 65% of portal messages came from people older than 55 years, and 24% came from patients older than 65 years. Preum said those data suggest patient portals should be designed with older adults in mind.1
Several limitations remain. The data came from a single large rural health system, and the release did not report patient outcomes, adverse events, response times, or clinician time measured prospectively. The reported accuracy and editing reductions reflect alignment with physician response standards, not evidence that AI-drafted messages improve clinical outcomes or safety. The investigators also noted that future work will evaluate actual editing time, clinician and patient ratings, and whether the TADPOLE model meaningfully reduces workload in practice.1
References
