Microsoft AI System Diagnoses Complex Cases in New Study
An artificial intelligence diagnostic system developed by Microsoft correctly diagnosed complex medical cases in a simulation more than twice as often as generalist physicians, according to a study published Thursday in Science. The software, called MAI-DxO or DxO, simulates a panel of physicians with different roles to track differential diagnoses, challenge leading hypotheses, and weigh the cost of each test before ordering it.
Mustafa Suleyman, Microsoft AI’s CEO, called the result a step toward medical superintelligence in an interview with TechCrunch. Dominic King, vice president of Health at Microsoft AI, told TechCrunch that there is something very substantial and exciting here.
Accuracy Comparisons Between DxO and Physicians
In simulations using complex cases from the New England Journal of Medicine, DxO was correct 83% of the time. Primary care doctors and general internists working individually with reference materials averaged 22.3% accuracy without reference tools and 36.1% with them on clinicopathological cases. The highest-scoring individual physician reached 51.8% accuracy, while the best-performing physician on emergency department cases reached 62.5%.
The study found that the AI and the physicians worked differently. Physicians asked an average of 5.6 questions before ordering their first test, whereas DxO asked 25.5 questions, which the authors described as more extensive, structured history-taking. When DxO made errors, they typically involved a closely related diagnosis at the wrong level of specificity, while physician errors landed further from the correct answer.
Testing Costs Under Different Configurations
Study results regarding costs varied based on how the system was configured. Without budget limits, DxO’s average testing cost per case was higher than the physicians’ at $5,431 versus $4,725 on clinicopathological cases, and $4,979 versus $3,400 on emergency department cases. When researchers imposed budget constraints, DxO cut costs while reducing accuracy, reaching 80.9% accuracy at $2,241 per case and 79.6% at $1,303 on clinicopathological cases.
The authors acknowledged several limitations in their paper. The comparison was tested against individual generalist physicians rather than specialists or multidisciplinary teams. The benchmark was text-only and omitted imaging, waveforms, pathology slides, and real-world uncertainties such as patients omitting information. All study authors are Microsoft employees and hold equity in the company, with eight named as inventors on a related patent application.
Microsoft is working with the Mayo Clinic to train a foundation model for healthcare with a release planned for next year, Suleyman told TechCrunch. Through a partnership with Beth Israel Deaconess Medical Center in Boston, Microsoft plans to evaluate the clinical reasoning capabilities of DxO across different hospital settings. This technology is ready for real-world evaluations, and you can imagine the complexity and diversity of issues that come from a Boston emergency department on a Friday evening,
King told TechCrunch.
Suleyman also stated that he believes the industry will be on the verge of a coding style breakthrough for medical records by 2027. The company aims to establish state-of-the-art medical clinical decision support products over the next few years.
Worth a look