AXY News Global

AXY News Global · technology

Microsoft Says Its Diagnostic AI Beats Doctors on Case Tests

The Diagnostic Orchestrator outscored physicians on written medical cases in a Science study, but it has not been tested with patients in real clinics

By AXY Tech Desk ·

The west campus of Microsoft, Redmond, Washington, taken from about 1,500'. The photo is looking south-east. The west campus is bounded on the west by 148th Ave, on the east by SR 520, and on the nort
The west campus of Microsoft, Redmond, Washington, taken from about 1,500'. The photo is looking south-east. The west campus is bounded on the west by 148th Ave, on the east by SR 520, and on the nort

Key points

Microsoft says it has built an artificial intelligence tool that can diagnose medical conditions up to four times more accurately than human doctors. The claim is laid out in a study published Thursday in Science, one of the world's most prestigious scientific journals. Microsoft AI chief Mustafa Suleyman told The Verge that the technology is, in his words, "finally here."

The catch is in how the test was run. The system was not used on patients. It was tested against doctors on written scenarios drawn from medical case reports and emergency department data. Microsoft says the next step is to prove the tool works in real clinical care, and it says that step is still ahead.

For readers, the question is what this could mean if it holds up. A tool that reliably suggests the right diagnosis could shorten the time a patient waits for an answer, especially in busy emergency rooms or in areas where specialists are hard to reach. But a tool that is wrong in a hospital can cause harm, so the bar for proof is high. This story looks at what the study tested, how the numbers should be read, and what would need to happen before any hospital could use something like it.

How the Diagnostic Orchestrator Copies a Doctor's Workflow

The tool is called the Diagnostic Orchestrator. Rather than acting as a single chatbot that answers a question in one go, it is designed to work like a panel of medical experts. It uses multiple AI agents, which are separate software components that each handle one part of a task. Together they read symptoms and medical histories, decide which tests to order, and arrive at a diagnosis.

That sequence is meant to follow the way a physician investigates an unclear case. A doctor asks follow-up questions, weighs several possible explanations, orders blood work or imaging, and then narrows the list. Microsoft says the Diagnostic Orchestrator follows the same steps in order. The company argues that this makes its tests a closer match to real clinical reasoning than a simple question-and-answer exam would be.

Microsoft also built its own yardstick for the comparison. The cases were compiled into a benchmark called SDBench. A benchmark is a standard set of problems that lets researchers compare different systems on the same material. Microsoft says it plans to open-source SDBench, which means other researchers would be able to download it and run their own tools against the same cases.

Reading the 83 Percent and 22 Percent Figures

The study used two main groups of cases. The first was drawn from hundreds of reports published in The New England Journal of Medicine, one of the most widely read medical journals in the world. On those cases, Microsoft's system reached a diagnostic accuracy of around 83 percent. Generalist physicians who were allowed to use reference materials scored roughly 36 percent. Generalist physicians working without those references scored 22 percent.

The second group was emergency department cases taken from a large database of patient admissions at the Beth Israel Deaconess Medical Center in Boston. In those cases, the Diagnostic Orchestrator scored 80 percent accuracy. Human physicians averaged 47 percent.

The four-times figure depends on which doctors are used as the comparison. In the emergency cases, where the system scored 80 percent and physicians averaged 47 percent, the gap is smaller again. Readers should keep that in mind when they see the four-times number repeated. It describes a comparison against doctors working without references, on one set of cases.

The study also found that the system tended to reach its diagnoses at a lower estimated cost than human counterparts. Cost could matter a great deal to hospitals and health systems, but it is an estimate produced in a test setting, not a measured saving in a real budget.

Why GPT-5 Is the Engine and What Newer Models Could Change

The Diagnostic Orchestrator is powered by OpenAI's GPT-5. The researchers say the system also outperformed frontier AI models working on their own. Frontier models are the most advanced general-purpose AI systems available at a given time, and they can answer medical questions without any extra structure around them.

The peer-reviewed paper does not include several newer models. The coverage names GPT-6 Astra, Meta's Muse Wonder Full, Google's Gemini 4 Argon and Anthropic's Opus 5.5 as systems that were not part of the study. That means the comparison is a snapshot of the models that were tested. If those newer systems were run against the same cases, the results could look different.

The Diagnostic Orchestrator was first developed when GPT-4.1 was the most advanced model available. The researchers say later testing suggests that "improvements in model capability do not necessarily diminish the benefits" of having multiple agents work together. In plain terms, the team is arguing that a group of AI agents can still add value even as the single models underneath get stronger. That is a reasonable claim, and it is worth testing. The study, however, does not prove it for the newest models on the market.

Why Past Case Tests Are Not Hospital Trials

This is the most important limit in the story. The system has only been tested on past medical cases. A written case describes a patient in a tidy sequence of facts, often after the answer is already known. A real patient in a clinic may describe symptoms vaguely, leave out details they do not think matter, or develop new signs over several days. None of that messiness is fully captured in a case report.

Microsoft acknowledges this directly. The Verge reports that the company accepts there is a long road ahead before the technology can be widely deployed. It also remains unclear how well the system would perform in real clinical settings, which is the question that matters most for patients.

Suleyman put the next step plainly. "It needs clinical validation," he says. "That is a leap, but this is a major milestone along that journey."

What Clinical Validation Would Have to Prove

Clinical validation means testing a tool in real care settings, with real patients, to see whether it helps doctors reach correct decisions without causing new harms. That is a far heavier job than running a benchmark. A tool might score well on written cases and still slow down a busy emergency department, flood doctors with unhelpful test suggestions, or miss the cases that matter most.

Several questions sit outside the published study. The study does not show how doctors would use the system day to day, whether they would trust it, or how often they would override it. It does not show who would be responsible if the system pointed a doctor toward the wrong diagnosis. It also does not show how patients would be told that a tool had played a part in their care.

In the United States, medical software that informs diagnosis is generally reviewed by regulators before it is sold for clinical use, and health systems have their own rules for adopting new tools. Those are general facts about how medical technology reaches patients. The Verge's report does not describe Microsoft's regulatory plans for this system, so any path to hospitals is still open.

Suleyman's Prediction About the Next Two Years

Suleyman went further in his interview with The Verge. He said Microsoft is going to build something that is "indistinguishable from human performance" in the next couple of years. That is a forecast about the pace of development. It is not a finding of the study, and it describes a goal rather than a result that has been measured.

For a patient, the practical question is simpler. If a tool like this reaches a hospital, what changes at the bedside? A doctor might get a second opinion sooner, a list of tests to consider, or a prompt that a rare condition deserves a closer look. Each of those uses would still need a clinician to weigh the output, decide what to do and take responsibility for the outcome.

The next tests will show whether the results hold up outside the study. Researchers who are not part of Microsoft's team would need to run the system on SDBench once it is released. The team would also need to show results against the newer models that were left out of the peer-reviewed paper. And clinical studies would have to show whether the tool actually improves patient outcomes. Until then, the four-times figure describes performance on selected written cases, not a measured improvement in care.

Questions and answers

How accurate is Microsoft's AI diagnosis tool?

On New England Journal of Medicine cases, Microsoft says its Diagnostic Orchestrator reached around 83 percent accuracy. On emergency department cases from Beth Israel Deaconess Medical Center in Boston, it scored 80 percent, compared with an average of 47 percent for human physicians.

Is Microsoft's Diagnostic Orchestrator being used in hospitals?

No. The system has only been tested on past medical cases, and it remains unclear how well it would perform in real clinical settings. Microsoft says it needs clinical validation before it can be widely deployed.

What is the Diagnostic Orchestrator?

It is an AI tool that uses multiple AI agents to read symptoms and medical histories, decide which tests to order and reach a diagnosis. It is powered by OpenAI's GPT-5 and was tested against physicians in a study published in Science.

Why does the study matter?

If results hold up in real care, a tool that suggests accurate diagnoses could help doctors reach answers faster. The study does not yet show that it would help patients in a clinic, which is the test that matters most.

Topics: Microsoft, Mustafa Suleyman, Diagnostic Orchestrator, AI in healthcare, Medical diagnosis, Science journal, SDBench, OpenAI GPT-5

Sources: The Verge

https://axynewsglobal.com/article/microsoft-says-its-diagnostic-ai-beats-doctors-on-case-tests