Study: AI can outperform doctors on diagnosing cases

Artificial intelligence that can "reason" is now capable of diagnosing real-life medical scenarios as well as or better than physicians, according to the results of a study published Thursday in Science.

The researchers used previously unknown clinical cases to test OpenAI's reasoning model o1 against the company's older model, GPT-4, as well as physicians and medical residents in training.

In a range of experiments, the o1 model often improved significantly on GPT-4's diagnostic ability and bested physicians, too. When tested with the electronic health records of random emergency department cases from a Boston hospital, the o1 model was diagnostically accurate more than two-thirds of the time at initial triage. Two expert attending physicians had correct diagnoses roughly half of the time.

Dr. Robert Wachter, professor and chair of the Department of Medicine at the University of California, San Francisco, described the study's findings as "important" and suggested it's now "indisputable" that modern AI will outperform older large language models and doctors when asked to identify the right diagnosis and next step. He was not involved in the study.

However, Wachter, author of "A Giant Leap: How AI is Transforming Healthcare and What That Means for Our Future," added that more research is necessary before AI is fully implemented in clinical practice.

"The question is how closely this replicates real life, and the answer is moderately well but not perfectly," Wachter wrote in an email.

As the study's authors acknowledge, the experiments were limited to text-only input and didn't include the visual and auditory clues and cues that doctors often rely on for diagnosis. These can include a patient's level of distress and medical imaging.

"GenAI can probably begin to integrate these inputs but for now, a test of a written, and often artificially 'clean' clinical case scenario is not the same as going into an ER and dealing with the chaos," Wachter said. "Just watch The Pitt."

Based on their findings, the study's authors highlighted an "urgent" need for further studies and prospective clinical trials to determine how AI systems can improve clinical practice and patient outcomes.

"The rapid pace of improvement in LLMs has substantial implications for the science and practice of clinical medicine," wrote the authors, many of whom are based at Boston's Beth Israel Deaconess Medical Center, where the study was conducted.

An accompanying article, also published in Science and written by two experts at Flinders Health and Medical Research Institute in Adelaide, Australia, who were not involved in the study, agreed with its urgent implications. They also argued against replacing doctors with AI, instead envisioning a style of collaboration that provides oversight, contextual judgment, and accountability.

"Without robust demonstrated effectiveness, equity, and safety, many AI systems will remain insufficient for clinical use," the experts wrote.