Why ChatGPT hallucinates clinical facts (and how to constrain it)

Three real hallucinations

Example 1. A pharmacology lecture covered mechanism of action of beta blockers in heart failure. ChatGPT's "study guide" added that bisoprolol has the highest cardioselectivity (true, but the lecturer never said this and it wasn't on the syllabus for this exam) and recommended starting at 1.25 mg daily (true, but again, lecturer didn't cover it).

Example 2. A renal lecture on diabetic nephropathy. ChatGPT inserted ACE inhibitor + SGLT2 inhibitor combination therapy as standard of care. The lecturer covered ACE/ARB; SGLT2 wasn't on the slides at all (this lecture was from 2019, before SGLT2 was integrated into renal protection guidelines).

Example 3. An anatomy lecture on the brachial plexus. ChatGPT added a paragraph about the triangle of Calot. The triangle of Calot is in the abdomen, not the upper limb. The model confused two unrelated anatomy regions because both are commonly tested.

Why this happens

LLMs are trained to produce complete, helpful, fluent answers. "Complete" is rewarded by their training objective; "faithful to a specific input" is not, unless the prompt makes it the loss function.

Generic chat-style prompting tells the model to be helpful. It does what it was trained to: fill in gaps with high-probability facts from its training data. For a clinical study guide, that's a feature when the user is asking a general question and a bug when they're asking about a specific lecture.

The constraint that fixes it

Three rules, all enforced in the system prompt:

1. The transcript is the source of truth. Every fact in your output must derive from the transcript. Do NOT add classic teaching points, eponyms, named signs, histology findings, numeric thresholds, mnemonics, drug names, or anatomic details that are absent from the transcript, even if they are well-known board facts.

2. If you find yourself reaching for outside knowledge to fill a gap, OMIT it instead.

3. Under-coverage of the lecture is a bug. Over-coverage beyond the lecture is a worse bug because it dilutes the testable facts.

That's the literal text from SlydeMe's study-guide prompt. It works because it gives the model a different loss function: completeness loses to fidelity, and the model knows it.