A new AI model is out. What should a doctor test first?

Separate the announcement, access and your result

A new GPT, Claude, Gemini or Grok announcement can describe several things: a model, a consumer-app feature, a preview or a change in availability. Start at the provider’s official announcement and record the exact name, publication date, supported inputs and stated limitations. A social post about a launch is not enough to identify what you will be testing.

Then check MedSafeAI’s current model catalog and the model selected for your task. A provider announcement does not by itself mean that the model is already available in every application or plan. Finally, keep your own observations separate from the provider’s claims. This article is a reusable test plan, not a report that any particular model was released or won a comparison.

Keep a small task set you can repeat

Choose three tasks whose source material you already understand. Save the exact prompts, documents, expected output format and checks before comparing models. Avoid patient-identifiable material in a general evaluation set. The goal is to notice a meaningful change without spending the afternoon inventing new prompts for every model.

Hold the workflow constant, then inspect the differences

Use the same documents, prompt and requested length. Record the model name, date, available reasoning setting and whether web search was used. Tool access and supplied context are part of the setup; if they differ, say so instead of attributing every difference to the model. Note response time and the manual corrections needed, without pretending one run establishes a universal ranking.

In MedSafeAI, Ask All AI sends one question to multiple models and Consensus brings the responses together. Use that comparison to find a missed condition, a different source or a clearer explanation. Then inspect the original answers. Consensus gives you a focused place to investigate; it does not turn majority agreement into proof.

Use one output format for the first pass

‘Using only the supplied document, complete [task] for [audience]. Return: 1) the requested output, 2) a table mapping each substantive factual claim to its source location, and 3) missing information or unresolved points. Preserve numbers and qualifications. Do not add information to make the output look complete. Maximum length: [limit].’

Keep an observation sheet with columns for correct, needs correction and could not verify. Record a concrete example in each populated column. If a result would affect which model you choose, repeat that task and examine whether the improvement persists. Your practical question is whether this model reduces correction work while preserving the information that matters.

Turn the findings into a useful model note

A good model note links the official release, states what was available when tested, describes the exact task and shows one checked example. Include what improved, what did not, and what you have not tested. A benchmark result can suggest a question to investigate; it is not a substitute for evaluating your documents, language and workflow. That is the difference between repeating launch publicity and helping another doctor decide what to try.