Nature Medicine Compared the Big Chat Models to OpenEvidence and UpToDate

Opening

A June 2026 Brief Communication in Nature Medicine reported that three general-purpose frontier models outscored two specialized clinical AI tools on every benchmark it ran. Six weeks later the study has drawn a retraction demand from one of the named vendors, an on-the-record methodological rebuttal from the other, a vendor-backed counter-study, and a detailed technical argument among named clinicians and informaticians.

“General-purpose large language models outperform specialized clinical AI tools on medical benchmarks”

1 · What the study measured

Vishwanath, Alyakin, Ghosh and 13 co-authors, with Eric Oermann as senior author, published “General-purpose large language models outperform specialized clinical AI tools on medical benchmarks” on 12 June 2026 in Nature Medicine — the Nature Portfolio’s clinical and translational title, a separate journal from Nature itself and one of the most cited in medicine [1] (DOI 10.1038/s41591-026-04431-5, Nat Med 2026;32(7):2405–2409) [3]. Sixteen authors, mostly NYU Langone Health, under NYU Langone IRB i23-00510. It runs five pages, in the journal’s Brief Communication format, with a companion Research Briefing in the same issue.

Two products were tested: OpenEvidence and Wolters Kluwer’s UpToDate Expert AI. Three frontier models: GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6. Google Search AI Overview was included as a sixth system, deliberately, because it is “routinely encountered by clinicians.” That is the summary panel at the top of an ordinary Google results page, not Google’s Gemini model, which is one of the three frontier systems above.

Two facts about the products matter for reading what follows, and both come from the vendors’ own documentation [20]. Both support multi-turn refinement, where a clinician narrows a question across several exchanges. And OpenEvidence accepts uploaded documents and, at Cedars-Sinai, reads the patient chart; UpToDate Expert AI does neither by design, and states that it is “not intended for use with personal data, including protected health information.” Fuller descriptions of both are in the appendix.

1.1 · The three benchmarks

Both MedQA and HealthBench are public and widely used. MedQA has been a standard reference point for medical question answering for years, which is what makes the contamination argument later in this piece possible. HealthBench is newer and was built by OpenAI [2].

RCQ was built for this study. Its 100 de-identified questions came from physicians querying NYU Langone’s HIPAA-compliant general-purpose GPT instance — not from physicians using OpenEvidence or UpToDate. Each of the six systems was put each question once, and 12 clinicians rated the answers blind, three to an answer.

The results table reports one number per benchmark. RCQ was scored on four dimensions, plus two binary flags for harm and hallucination. The RCQ column is the aggregate mean across clinical correctness, completeness, safety and clarity.

No single system led everywhere: Gemini led MedQA and RCQ, GPT-5.2 led HealthBench.

On RCQ the six systems separated into two tiers. All nine significant pairwise differences fell between tiers rather than within them, after correction for multiple comparisons. Both clinical products landed in the same tier as the free, auto-generated Google result, and the study’s authors put that more sharply than a tier does: Google AI Overview “scored as well or better than OpenEvidence and UpToDate AI across all dimensions.” The whole spread from top to bottom is 0.45 points on a four-point scale.

The paper also reports the gap a second way, using a model that adjusts for how harshly each rater scored. On that measure the separation is larger, not smaller. Both readings are the study’s own, published in the same paper.

Two things qualify all of it. Thirty-two of the 600 model–question items were refusals and were dropped, so 1,704 of the 1,800 ratings were analysed, and the head-to-head comparisons used only the 74 questions every system answered. And the raters disagreed a good deal about individual answers, agreed strongly on whether an answer was acceptable, and ranked the six systems in much the same order. Vishwanath and colleagues report that low item-level agreement themselves and describe it as fair.

1.2 · Three findings in the details

One — UpToDate Expert AI declined to answer 19% of queries, against 1–3% for the frontier models.

Two — OpenEvidence scored lowest of all six systems on clarity, at a mean of 2.84. The study’s authors read that as “suggesting its weakness was communication, not knowledge.”

Three — no safety difference was detected between any of the six systems. On both binary flags, harmful content and hallucination, the difference was not statistically significant (p = 0.55 and p = 0.42). That is a failure to detect a difference rather than a demonstration that none exists. With 100 questions and events this rare the study was not powered to rule out safety differences, and its authors do not claim it was. What it measured as different was usefulness.

2 · How it was run, and what it could not settle

2.1 · What the study’s authors say about the limitations of their testing

They state plainly that they could not determine what the clinical products are built on: “As the architecture of proprietary clinical AI tools is inaccessible, it is impossible to definitively assess a mechanistic understanding for their underperformance against general-purpose models.”

They also flag, in their own words:

• that the blinded RCQ evaluation is their primary evidence and HealthBench is supplementary;
• possible training-data contamination may improve the MedQA and HealthBench scores of the frontier models as they “may have been exposed to MedQA or HealthBench during training”;
• that HealthBench is OpenAI’s, so GPT-5.2’s score “may be influenced by potential benchmark–developer overlap”;
• grading bias, since frontier models served as both evaluated systems and judges, mitigated by using a three-model panel;
• that the result is “a snapshot of a rapidly evolving landscape rather than a permanent ordering of approaches.”

One claim sits alongside those caveats, and it is the one the critics have concentrated on: that “scale, alignment and cross-domain reasoning may outweigh domain-specific tuning as determinants of medical competency for particular tasks,” with implications for “procurement, reimbursement and regulatory oversight.”

2.2 · How each system was tested

The frontier models were driven through public APIs, at fixed settings and with web search switched on. The two clinical products were used by hand through their browser interfaces, because, as the paper states, they lack public application programming interfaces.

MedQA and HealthBench ran separately, on 500 questions each, across five systems rather than six, and were graded automatically rather than by clinicians.

2.3 · What the study’s published code shows

The study’s evaluation code is published under AGPL-3.0 [4]. The pipeline drives the three frontier models only: OpenEvidence, UpToDate Expert AI and Google AI Overview are registered in the code as manual-import-only, their answers were collected by hand and read in from files, and the generation code raises an error if asked to query them. The facts below describe how the frontier models were run.

A full read of the roughly 2,500-line pipeline adds three facts the paper does not state.

• No system prompt, persona or clinical framing was sent to any of the three frontier models. Each generation call carried the raw benchmark question and nothing else.
• No PubMed, literature, retrieval or citation-checking integration of any kind is present.
• No medical model variants or health-specific API features were used.

3 · What the arguments are about

The dispute began on publication day and has not resolved. Three separate arguments are running at once, and they are often confused for each other. A dated account of how it unfolded is in the appendix.

3.1 · The research doesn’t support the headline

This started within hours, among researchers and clinicians who do not sell either product, though several work in clinical AI. Six distinct objections are in play. They are not all compatible with each other, nobody has adjudicated between them, and this piece does not either.

The frontier models and the clinical products were not accessed the same way. Abel Torres-Espin, on publication day [5]: the clinical tools were queried through their interfaces while the frontier models went through APIs, and “because the Clinical AI is queried through the interface, there are many extra layers that direct API calls don’t have (e.g., hidden prompts).” He also noted that search was enabled on the frontier side while both public benchmarks sit online. The study’s authors state the same limitation themselves.

The three benchmarks do not test what the title claims. David Talby, CTO of John Snow Labs, published the fullest version on 12 July [6]. “The study is fine. The methods are reasonable and the statistics are careful. The problem is the title. It states a general conclusion that the study did not test and does not support.” All three benchmarks test one capability, in his words “answering a single-turn, general medical question, in isolation, with no patient record attached.” Real clinical work, in his list, is “summarizing a chart, drafting a discharge instruction, extracting a tumor stage from a pathology report, reconciling a medication list, catching an error in a note, turning a clinician’s question into a database query.” Talby competes in clinical AI and markets his own benchmark results in the same piece.

The differences are too small to carry the conclusion. Torres-Espin again, and he is the only commentator to have made this point: the RCQ scale is ordinal, 1 to 4, and “the difference between all models is less than 0.5 point in such scale.” Sunita Mohanty reads the numbers as the products tying the frontier models on knowledge, harm and hallucination and losing on readability, which she calls “a UX observation, not a verdict on clinical usefulness” [14].

UpToDate Expert AI’s refusals may be correct behaviour rather than a failure. Max Topaz took its 19% refusal rate and read it as a safety feature [15]. Wolters Kluwer’s chief medical officer makes the stronger version of the same argument, in §2.2.

The paper could not identify which models the products run on. Its authors say so, and Sergei Polevikov pressed the point publicly [16].

Peer review had already caught some of it. Gilles Frydman read the published peer-review file and found reviewers had flagged the single-judge grading, which became a three-model panel, and a missing safety analysis, which was added [17].

Sources 14 through 17 were retrieved agentically rather than read at the post; the URLs and dates are held and the wording is as retrieved.

4 · Where the parties stand

4.1 · The vendors’ answer — two companies, two different responses

The retraction letter and the on-record responses in this section come from Giles Bruce’s reporting for Becker’s Hospital Review [7]. The letter itself has not been published.

OpenEvidence disputed the paper itself. The product is free and ad-funded, valued at around $12B, and marketed on how many physicians use it [20]. It responded publicly on 13 June alleging methodological flaws and an undisclosed conflict of interest, and on 15 June wrote to Nature Medicine demanding retraction, a public apology, and an independent review [7]. Its stated ground is contamination: MedQA and HealthBench are public and may sit in the frontier models’ training data. Peer reviewers raised the same concern during review, and the study’s authors state it as a limitation. The company went on to back a counter-study, launch a product, and publicise a second study measuring which tool physicians choose. Both are compared in the appendix.

The company’s account of the conflict is sharper than the disclosure question. In its 13 June thread it said the study’s authors run a competing in-house medical AI at their hospital, asked OpenEvidence for API access to power it including rights to build a competing product, and were declined [18]. Farah Deshmukh, a physician writing on 15 June, reported the same three objections independently — contamination, a rubric she says favoured the general models’ communication style, and the API-access allegation [21]. It remains OpenEvidence’s characterisation and it is unadjudicated. On the disclosure itself, the paper carries a competing-interests declaration that includes consulting for Google by the senior author, and Google’s Gemini led two of the three benchmarks, so “undisclosed” is contested rather than established. That declaration has not been read in the primary back matter for this piece; it is reported by Talby, who read the paper, and reflected in the published record.

Wolters Kluwer disputed the measurement. UpToDate is a subscription business, roughly 80% enterprise, sold on editorial rigour [20]. Peter Bonis, chief medical officer of Wolters Kluwer Health, told Becker’s the study “confused clinical quality and complete-sounding answers,” and on the refusal rate argued that the paper counted declining to answer as a deficit without establishing whether those refusals were appropriate. In clinical decision support, he said, declining an underspecified or risky prompt “may be the safer behavior.” No retraction demand, no counter-study — one statement from its chief medical officer, one LinkedIn post, and a validation framework the company had already published.

4.2 · The study’s authors — they have not moved

Eric Oermann, the senior and corresponding author, told Becker’s on 16 June that the team stands behind the work: “We continue to stand by our study and its results.” [7] Their substantive answer to the contamination charge is in the paper itself. They designate the blinded clinician evaluation as their primary evidence precisely because its 100 questions were written for the study and appear in no public benchmark, so contamination cannot reach them — and the frontier models led there too.

That 16 June quote is the last public statement from the study’s team that this piece has found. Searches on Vishwanath, Oermann and each of the other 14 co-authors surfaced no post-publication interview, podcast or further post, and neither vendor has filed the Matters Arising the journal has pointed them toward. Absence of a later statement is not proof of silence, but it is what the record shows.

4.3 · Nature Medicine’s position — use the process

Chief editor João Monteiro declined to address the retraction demand directly, said the journal “welcomes scientific debate,” and pointed to its formal Matters Arising process [8], where concerns are assessed editorially alongside a response from the original authors. As of the date of this piece, no Matters Arising submission from either vendor has been published.

Final take