NeurIPS 2026Position Paper Track

Alignment Is Not Enough for Safe Medical LLM Evaluation

Zhiwen You1* Yikun Han1 Jana Diesner1,2,3,4 Yue Guo1

1University of Illinois Urbana-Champaign 2Technical University of Munich
3Munich Center for Machine Learning 4Munich Data Science Institute

* Contact: zhiweny2@illinois.edu

Paper BibTeX
01 The position
LLJ–expert alignment should not be reported as standalone evidence of validity or safety.

LLM-as-judges (LLJs) are increasingly used to scale medical AI evaluation and cut clinician review costs. This creates a second-order safety problem: medical AI systems may be benchmarked, selected, or trusted based on judgments produced by other models.

What studies typically report

High agreement with clinician ratingsOften the only evidence offered that a judge can be trusted.

What agreement alone does not establish

  • Reliability

    The same verdict across prompts, repeated runs, and judge models.

  • Clinical validity

    Judgments grounded in clinical evidence and task-specific medical rules.

  • Safety

    Fitness for the downstream decision the score is used to make.

Why medicine needs a stricter standard than general NLG

Open-ended

Notes, answers, and treatment plans are free text, not a single option to match.

Context-dependent

Whether an output is acceptable depends on the patient, the source record, and the setting.

Safety-critical

A judge error can pass an unsafe answer, reward unsupported content, or reject a valid diagnosis.

No single ground truth

Evidence can be incomplete and criteria ambiguous, so clinicians themselves may disagree.

02 Key findings

Re-running judges from prior medical studies exposes how fragile their scores are

Same outputs, same clinician references — only the evaluation setup changes.

66.2–83.8%

of ratings are reproduced when the same judge scores the same output twice with the same prompt.

Risk 1
11–40%

of physician-vs-model preferences reverse when only the order of the two answers is swapped.

Risk 1
18.8 pp

change in measured diagnostic accuracy from switching the ICD-10 matching rule — with identical predictions.

Risk 1
20 of 42

prior studies validating LLM judges against humans omit human–human reliability, leaving judge agreement without a reference point.

Protocol step 4
03 Four risks

What high agreement can hide

We revisit LLJ evaluation across four medical tasks and audit clinician-rated outputs to see where judge scores and clinical judgment come apart.

1
Risk 1

Unstable medical evaluation

Same judge, same input, same prompt — called twice

Share of 1–5 ratings that match exactly across two independent calls.

Same ratingRating changed

Swapping only the order of a physician and a model answer also reverses the preferred answer on 11–40% of pairs. GPT-5.4 Nano reaches 40% under a prompt that explicitly tells it to ignore order.

One sentence of system prompt flips the verdict

GPT-5.6 Luna decides whether a predicted diagnosis matches the gold standard (MedCaseReasoning prompt).

System prompt: “You are a professional medical expert that answers only ‘y’ or ‘n’.”

  • Dx1GoldUrticaria multiformePredAcute urticaria
    Subtype vs. broader diagnosis: acceptance depends on whether the task permits broader diagnoses.
  • Dx2GoldOculomotor mononeuropathyPredPost-vaccinal acute oculomotor nerve palsy
    Same ICD-10 category (H49.0), but post-vaccinal palsy is one rare cause — not another name for it.
  • Dx3GoldCholecystitisPredPerforated gallbladder
    Accepted by the MIMIC-CDM benchmark rule, yet rejected by the judge either way.

A more detailed prompt with alias examples fixes Dx1 and Dx2, but Dx3 is still labeled incorrectly: a judge can be consistently wrong when the prompt does not encode task-specific medical rules.

Takeaway High accuracy under a single judge–prompt configuration does not establish reliable medical evaluation; the judge, prompt, and scoring rules should be validated together, with robustness tested across models, prompts, and repeated runs.
2
Risk 2

Opaque scores hide clinically meaningful errors

Averages look fine. Individual criteria do not.

Exact agreement with individual human raters on the MedHELM multi-annotator subsets.

Human–humanMean of three LLJs
Clinical notes, by criterion
GPT-5.4 Nano GPT-5.6 Luna Claude Sonnet 5 Human–human

Each judge has two dots: agreement with annotator 1 and with annotator 2. Judges reproduce how a note reads (clarity) far more often than what it leaves out (completeness).

A fluent note that earns a perfect score

Clinical-note generation (MedHELM). Note written by Gemini 1.5 Pro from a doctor–patient conversation about light-headedness and shortness of breath after a long bike ride; judged by the original GPT-4o.

Expert ratingLLJ rating
  • Unsupported finding: shortness of breath “even at rest,” which the conversation does not clearly state.
  • Missing negatives: no hematemesis, no bleeding elsewhere, no bruising or gum bleeding, no recent emergency-room visit.
  • Both can change how a clinician interprets the case — yet the score is perfect on every criterion.

Both human annotators gave this note 3 / 3 / 4.

Takeaway Aggregate agreement can conceal failures on clinically important criteria; reliable medical evaluation requires criterion-level validation and source-grounded analysis of the errors that LLJ scores detect or miss.
3
Risk 3

Weak validity under clinical uncertainty

Judge agreement rises and falls with physician consensus

Original GPT-4o judge on 300 CQA-Eval answers, each rated by three physicians. Exact judge–physician agreement.

All three physicians agree All three differ Physician–physician

On risk disclosure, physicians agree with each other only 26.3% of the time — so “agreement with a clinician” has no stable reference to be measured against.

An unqualified answer scored as fully correct

Free-form medical QA (CQA-Eval). Answer written by GPT-4; average of three physicians vs. the original GPT-4o judge.

QuestionDoes allopurinol affect fertility?
Answer“There is no evidence to suggest that allopurinol has a direct effect on fertility…”
Physician averageLLJ rating
  • Animal studies show no impaired fertility at the tested doses, but the FDA label lists male infertility among reported adverse reactions.
  • A reported adverse reaction does not establish causality — but the unqualified statement does not reflect all available evidence.
Takeaway LLJ–expert agreement must be interpreted in light of uncertainty in the clinical reference; establishing LLJ validity requires examining expert disagreement, weighing the underlying evidence, and adjudicating clinically consequential discrepancies.
4
Risk 4

Downstream safety and data contamination

Switching the judge changes which answers get flagged

Risk-disclosure ratings on CQA-Eval, compared against the median of three physicians. Same answers, same physicians — four different judges.

Potential missed disclosure Potential false alarm

Flags are screening signals, not independently adjudicated clinical errors.

A missed disclosure…

may reward an answer with incomplete warnings. If scores like these filter synthetic data, incomplete or unsupported examples can stay in a training set.

A false alarm…

may penalize an adequately cautious answer or add unnecessary review work.

Errors point in different directions

GPT-4o leans toward missed disclosures; Nano, Luna, and Sonnet lean toward false alarms. The counts motivate targeted expert review — they do not establish which judge is safest.

Takeaway When LLJ scores guide model selection or synthetic-data filtering, evaluation errors can propagate into future models and clinical decisions; validation and clinical oversight should therefore scale with the consequences of those uses.
04 The protocol

Six steps that put clinicians where they matter most

Rather than scoring every output, clinicians define and validate the evaluator: they write a task-specific codebook, annotate a calibration sample, and adjudicate uncertain or consequential disagreements. The LLJ then scores the remaining outputs — only within the scope that validation supports.

Protocol overview. Risk 1, unstable medical evaluation, is addressed by protocol steps 1 and 4. Risk 2, opaque scores hiding clinically meaningful errors, by steps 3 and 6. Risk 3, weak validity under clinical uncertainty, by step 2. Risk 4, downstream safety and data contamination, by steps 4, 5 and 6. Click to enlarge
From risks to safeguards. Each risk is paired with concrete safeguards and the protocol steps that address it. Download figure (PDF) ↓
1Define

Define the evaluation construct and codebook

Before prompting any judge, specify what quality means for the task: dimensions, rating scales, examples, and exclusion criteria — including how to treat diagnosis aliases, ICD-10 specificity, unsupported findings, and acceptable paraphrases.

Domain experts + researchers
2Annotate

Build a human-annotated validation set

Sample outputs across quality levels, task types, and expected edge cases. Experts annotate independently with the codebook; report inter-annotator agreement and resolve meaningful disagreements into a reference label or an explicit “ambiguous” category.

≥ 2 qualified experts
3Design

Design the LLJ prompt from the human codebook

Mirror the codebook’s criteria, scales, decision rules, and allowed evidence. Ask for criterion-specific judgments instead of “is this answer good?”, cite the supporting source span, and avoid judges from the generator’s model family.

Researchers
4Validate

Validate LLJ–expert alignment before scaling

Report judge–human and human–human agreement per criterion. Stress-test runs, prompt variants, and judge models; probe order, format, scale, and verbosity effects; and check whether model rankings or filtering decisions change.

Researchers
5Analyze

Conduct clinical error analysis

Classify confirmed errors by type — missed safety issues, unsupported claims, important omissions, over-penalized aliases, overvalued fluency — and their frequency. Unresolved high-severity failures limit how the judge may be used.

Clinicians
6Report

Report and release evaluation artifacts

Publish the codebook, annotation procedure and qualifications, agreement and adjudication, prompt, judge version, run count, alignment report, and error analysis — together with the judge’s intended role and risk level.

Researchers
05 Readiness levels

Assurance proportional to the cost of being wrong

The evidence a judge needs depends on how its scores will be used. Each level states what an LLJ may be used for, the minimum assurance required, and the restrictions that apply.

L0
Exploratory
Permitted use

Error analysis, prompt comparison, hypothesis generation.

Minimum assurance
  • Report criteria, prompt, judge version, settings
  • Report run-to-run variability
  • State that the judge is unvalidated
No safety, superiority, or deployment claims.
L1
Validated reuse
Permitted use

Reuse within a benchmark’s validated scope, or a low-risk task that is easy to check.

Minimum assurance
  • Confirm task, population, criterion, judge, and prompt are still in scope
  • Report repeatability
  • Add expert review if the scope or conclusion changes
L2
Benchmark / model selection
Permitted use

Building benchmarks, ranking models, release decisions.

Minimum assurance
  • Expert codebook
  • Multi-clinician reference on a sample
  • Pre-specified margin against expert–expert agreement
  • Prompt, run, and judge stress tests
  • ≥ 95% bootstrap winner stability
  • Adjudication of high-severity cases
L3
Real-world use
Permitted use

Procurement, deployment, workflow decisions, safety-critical filtering.

Minimum assurance
Everything in L2, plus
  • Prospective or external validation
  • Audit trails and monitoring
  • Revalidation
  • Human override
Unresolved high-severity failures block use.

Requirements scale with intended use rather than study context. The numerical gates are provisional starting points, not universal standards.

06 Scaling responsibly

Scalable — but not automatically cheaper

LLJs do not inherently reduce total evaluation cost. The hybrid design spends clinician effort on calibration and escalation instead of on every output — and adds its own costs for inference, testing, and monitoring.

Input
N model outputs

Everything that needs to be evaluated.

Calibration sample · m
Rated by clinicians

Validates the judge against the codebook.

Remaining · N − m
Scored by the validated LLJ

Only within the scope that was tested.

Escalated · fraction e
Back to clinicians

Uncertain or high-severity cases are adjudicated by experts.

Try the cost model

Each output needs k expert ratings under full review.

Default 400: the median rated sample among prior studies that rate generated outputs (our structured review).
Expert ratings needed
—
Expert ratings avoided
—
Clinician ratings Sent back ratings
Full clinician reviewN · k
Validated LLJ + targeted reviewm·k + e(N − m)·k

This counts expert ratings only. The hybrid design is cheaper overall only if LLJ inference, prompt testing, robustness checks, monitoring, and revalidation cost less than the avoided review, (N − m)(1 − e)·k·cH, where cH is the cost per expert rating. It is a cost-accounting condition, not proof of savings.

600 s→22 s

Per clinical summary: physician review vs. a validated judge that reached ICC 0.818 with clinicians (Croxford et al.).[1]

USD 50→0.05

Estimated physician labor vs. API cost per summary — different expense types, and validation overhead is not included.

262 physicians

wrote 48,562 criteria across 5,000 conversations for HealthBench: expert-curated criteria are what make reliable scaling possible.[2]

  1. Croxford, E., et al. (2025). Evaluating clinical AI summaries with large language models as judges. npj Digital Medicine, 8(1), 640.
  2. Arora, R. K., et al. (2025). HealthBench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775.
07 Citation

BibTeX

If you find this work useful, please cite:

@inproceedings{you2026alignment,
  title     = {Alignment Is Not Enough for Safe Medical {LLM} Evaluation},
  author    = {Zhiwen You and Yikun Han and Jana Diesner and Yue Guo},
  booktitle = {The Fortieth Annual Conference on Neural Information Processing Systems Position Paper Track},
  year      = {2026},
  url       = {https://openreview.net/forum?id=Tju86JH3mo}
}