Artificial intelligence (AI) is increasingly being integrated into healthcare with the promise of alleviating administrative burdens for physicians and nurses, allowing them more time for direct patient care. However, a recent study from Dartmouth University suggests that AI-generated responses, particularly in patient portals, can introduce errors and irrelevant details, potentially increasing the time physicians spend editing rather than writing messages from scratch.
The research, presented at the 2026 Annual Meeting of the Association for Computational Linguistics, highlights a critical gap: while AI can mimic the language of a physician, it may not possess the clinical reasoning or contextual understanding necessary for accurate patient communication. This means doctors might find themselves correcting AI-generated messages, negating the intended efficiency gains.
AI’s Pitfalls in Clinical Communication
The Dartmouth study conducted a large-scale analysis of an online patient portal that utilizes AI to draft physician responses. Researchers developed a tool to compare AI-generated replies against a dataset of actual physician responses, compiled with input from healthcare professionals at Dartmouth Health. This analysis encompassed 146,000 conversations involving 10,105 patients and their primary care physicians within a large rural health system. The study adhered to strict privacy protocols, including data anonymization, and received approval from the Dartmouth Health Institutional Review Board.
The team evaluated responses drafted by several prominent AI models, including Claude, Gemini, and ChatGPT, alongside smaller commercial platforms like Llama, Aloe, and Qwen. Their
