By Nnamdi Okorafor
Draft revision for review. This version has not replaced the published article.
A useful AI review starts with a concrete question: what would happen if this output were wrong?
Consider a clearly hypothetical scheduling example. An automated system creates two meetings in the same time slot and reports each as confirmed. Each record looks reasonable on its own; the conflict appears when someone checks the whole day. The point is to inspect the workflow around an output, rather than treating a successful software response as proof that the task was done correctly.
The failure that looks like success
For an AI-generated clinical summary, try following one statement back to its source. Is the information present? Is it current? Has an important qualification been lost? Then check where the summary goes and whether someone could rely on it before those questions are answered.
Different tools call for different checks. A text generator, an imaging device and a scheduling system do not have the same intended use or risks. Visible errors and unnoticed errors both need safeguards.
Questions worth taking to a demonstration
What happens when the tool cannot support an answer? Ask for a demonstration of its refusal, uncertainty or escalation behavior. If the vendor cannot show it, treat that capability as unverified. Ask what the user sees and what action is expected next.
Where does the output go? Trace draft notes, flowsheets and device-fed observations separately. Ask when information becomes visible, who reviews it and how authentication works. Do not assume that a signature requirement proves an AI draft was substantively checked.
Which inputs matter? Ask which information the intended use requires, what the tool can access and how missing or outdated inputs are communicated. Test a case with a relevant input missing.
Who owns the decisions and the monitoring? Ask the vendor and institutional leaders to document review responsibilities, escalation routes and ongoing monitoring. An answer about contractual responsibility cannot, by itself, establish how much automation is appropriate.
What do the misses look like? Request examples of failures and the evidence behind performance claims. Ask which users and settings were represented in evaluation, and which situations remain untested.
Shared oversight
Nurses can contribute questions about the workflows they use. Patients, other clinicians, developers and institutional teams also have perspectives worth including. The FDA's transparency principles for machine-learning-enabled medical devices emphasize human-AI team performance and clear information for users.[1] These are guiding principles, not a certification checklist for every AI application.
Before introducing a tool to reduce work, assess the checking work it adds. Who has the time, information and authority to intervene? What happens if a reviewer is unavailable? Those questions belong in the implementation plan, rather than becoming an unstated expectation that an individual will catch every problem.
When something goes wrong
Address an immediate patient-safety concern through the appropriate clinical escalation pathway. Then document and report the event under organizational policy. Distinguish what was observed from what is still uncertain.
For a qualifying US device user facility, medical-device reporting rules apply to information reasonably suggesting that a device has or may have caused or contributed to a patient's death or serious injury.[2] Under 21 CFR 803.30, death reports go to the FDA and the manufacturer, if known, as soon as practicable and no later than 10 work days after the day of awareness.[2] Serious-injury reports go to the manufacturer within that deadline, or to the FDA if the manufacturer is unknown.[2] Follow the institution's reporting procedures and confirm which requirements apply to the event.
The useful question at the end of a demonstration is simple: show what happens when this tool is wrong, including what the people around it can actually do.
Sources
[1] FDA: Transparency for Machine Learning-Enabled Medical Devices — Guiding Principles.
[2] 21 CFR 803.30: Device user facility reporting requirements.