AI Medical Scribes in Ontario: Auditor General Warns of ‘Hallucinations’ and Patient Safety Risks
The integration of artificial intelligence into clinical workflows promises to reduce administrative burdens and allow physicians to focus more on patient care. However, a recent report from Ontario’s Auditor General, Shelley Spence, has raised significant alarms regarding the reliability of these tools. During a provincial procurement process for AI medical systems designed to transcribe doctor-patient conversations, evaluators discovered that some programs demonstrated “hallucinations”—a phenomenon where AI generates incorrect or entirely fabricated information.
The ‘Hallucination’ Problem in Medical Transcription
In the context of medical AI, a “hallucination” occurs when a system produces information that was never discussed during the actual clinical encounter. To test the capabilities of various AI medical scribe systems, evaluators conducted two test conversations between hypothetical doctors and patients. The results were concerning.
According to the report, evaluators identified serious errors in transcripts generated by 20 different programs. Most notably, nine of these systems experienced hallucinations. These errors included the AI fabricating information or making medical suggestions that the doctor never actually made. Specific examples of these inaccuracies included the AI incorrectly referring a patient for therapy or ordering blood tests that were not part of the physician’s clinical plan.
Why Accuracy is a Matter of Patient Safety
For healthcare providers, the medical record is the foundation of clinical decision-making. When an AI tool introduces errors into these notes, the consequences can extend far beyond simple clerical mistakes. The Auditor General’s report highlights the direct link between transcription accuracy and clinical outcomes.
“Inaccuracies in medical notes generated by AI scribe systems could potentially result in inadequate or harmful treatment plans that may potentially impact patient health outcomes,” the auditor’s report stated.
If a physician relies on a note that falsely claims a specific test was ordered or a specific symptom was discussed, it can lead to a cascade of medical errors, including misdiagnosis or inappropriate interventions.
The Debate Over Testing and Oversight
The findings have sparked a debate regarding the rigor of the provincial government’s oversight. Auditor General Shelley Spence noted that the evaluation of these AI tools was “inadequate,” suggesting that more robust testing is required to ensure these systems provide the necessary assurances regarding the quality of their generated notes.
The province has responded to these findings by stating that the errors identified occurred only during the testing phase of the procurement process. This distinction suggests that the issues were caught before the tools were deployed for widespread clinical use, though the Auditor General maintains that the testing protocols themselves must be improved to minimize the risk of inaccuracies.
Key Takeaways
- Significant Errors Found: Evaluators identified serious errors in transcripts from 20 AI medical transcription programs.
- AI Hallucinations: Nine systems fabricated information, such as suggesting therapy or blood tests that the doctor did not order.
- Safety Risks: The Auditor General warned that inaccurate notes could lead to harmful treatment plans and impact patient health.
- Testing Concerns: While the province notes errors were caught during testing, the Auditor General described the testing of these systems as “inadequate.”
Frequently Asked Questions
What is an AI “hallucination” in healthcare?
An AI hallucination occurs when a language model generates information that is factually incorrect or not present in the source data. In medical scribing, this means the AI might “invent” a diagnosis, a medication, or a patient symptom that was never actually mentioned by the doctor or the patient.
Are these AI tools currently being used on patients in Ontario?
The report indicates that these specific errors were uncovered during a provincial procurement and testing process. The province has maintained that these inaccuracies were identified during the testing phase.
How can doctors protect themselves from AI errors?
Medical professionals must treat AI-generated notes as drafts rather than final records. It is essential to review, verify, and manually correct any AI-generated transcriptions to ensure they accurately reflect the clinical encounter before they are finalized in a patient’s medical record.