Doctors and medical students gave higher marks to a clinical AI grounded in a curated medical library than to ChatGPT, in a paired evaluation published in Scientific Reports on 4 November 2026. The ratings measure what participants thought of the answers, not whether the answers were correct. And the company behind the higher-rated system chose the questions.
What the study compared
Researchers from Marburg University ran the test at the German Society of Internal Medicine congress in Wiesbaden, 18 to 21 April 2026. Sixty-four attendees took part: 46 physicians and 18 medical students. Each picked one of 15 internal medicine questions and rated two stored answers to it.
One answer came from ChatGPT on GPT-5.3 Instant. The other came from LiSA 1.0, the model behind the AMBOSS AI Mode clinical tool. Grounding means the system answers from a fixed library rather than the open web. AMBOSS says in its November 2025 launch post for AI Mode that it limits sources to its own knowledge base, curated US clinical guidelines and a drug database.
What participants rated higher
On the primary outcome, a quality score from 1 to 5, LiSA averaged 3.96 and ChatGPT 3.48 (p = 0.002). Trust, on a 1 to 7 scale, showed a wider gap: 5.11 against 3.67. LiSA also scored higher on completeness, actionability and perceived medical accuracy. Clarity ratings did not differ significantly.
Participants flagged possible safety problems in 17 of 64 ChatGPT answers (26.6%) and 10 of 64 LiSA answers (15.6%). That gap was not statistically significant (p = 0.143), and the authors say the study was not powered to detect it. So the paper does not show that one system was safer. It shows that a sizable share of answers from both drew a safety flag.
Who shaped the test
The paper’s competing interests section says AMBOSS provided the questions, helped design the study, selected the final set and gave access to LiSA. It also says AMBOSS paid nothing and had no access to the data or the analysis. The questions were real ones its users had previously submitted to AMBOSS.
The audience leaned the same way. Forty of the 64 participants (62.5%) were current AMBOSS users, and among them LiSA’s quality lead was 0.86 points. Among the other 24 it was minus 0.17, so ChatGPT came out slightly ahead. The authors call this a post hoc analysis and say familiarity may explain it, without proving that it does.
The test was not blinded either. Participants saw each answer in its own interface, with the brand and any source references visible. The authors write that the differences “cannot be attributed specifically to knowledge-base grounding.”
Why it still matters
Larger work points the same way. The NOHARM benchmark preprint tested 20 language models and 4 clinical AI tools that retrieve from medical sources. The clinical tools did better than general models, but recommendations still carried potential for severe harm in up to 24.6% of cases. It is a preprint, so those figures are the authors’ own.
For teams outside medicine, the lesson is about vendor evidence. When the vendor picks the questions, raters see the brand and most already use the product, the result tells you about preference, not error rates. For the consumer side of the same question, see our explainer on what health AI assistants do and where your data goes.
What to do this week
If a vendor sends you a study backing its AI tool, check who chose the test questions, whether raters knew which system they were judging, and whether answers were checked against a reference standard, or only scored on impressions. If it was impressions, run your own blind comparison on 10 to 20 of your real questions before you buy.
Sources
Get the weekly email
The week's most useful AI news, a tutorial and an interview.
Martin Scott
Martin Scott is the editor for business and policy at AI Observer, covering startups and deals, regulation, health and science, and AI agency profiles. He follows the money and the legal text, and separates what a rule says from what people claim it says. His style is analytical and plain-spoken.
Spotted a mistake? Tell us on the corrections page.