Preventing AI-Invented Allergies and Vitals: Drafts, Evidence and Structured Records
Keep generated proposals separate from verified clinical fields; preserve unknown status, source, units and timing, and test downstream effects of a wrong value.
An AI suggestion should not become an active diagnosis, prescription or patient instruction merely because generated text appeared or was inserted into a note. Keep the proposed content distinguishable from the authorised clinical decision, expose its basis and let the responsible clinician review the exact order. Evaluate each automation by what it can change and who can authorise that change.
Indian doctors and clinic owners evaluating AI assistance, clinical decision support and prescription automation in outpatient software.
“AI prescribing” can describe very different products: a dictation tool that copies a medicine name, a calculator, a rule-based warning, a suggested order set or a system that selects treatment. Grouping these together hides the decision the clinic actually needs to make.
Start by identifying what the feature changes. Does it propose text, populate an editable draft, modify a structured field, create an order or issue instructions to a patient? The consequences grow when output moves from something a clinician can inspect to something another person may act on.
WHO warns that generated systems can produce inaccurate information and encourage automation bias. This supports careful scrutiny of suggestions, but it does not mean every calculator or clinical algorithm has the same failure mechanism. The evaluation should match the actual feature and intended use.
This article proposes a clinician-controlled workflow for outpatient software. It does not claim that every form of medical automation is universally prohibited by one law, or that a review button alone establishes safe practice.
Source: WHO: AI ethics and governance guidance, 18 January 2024
A proposed order needs an inspectable basis appropriate to its purpose. That may include the source facts, relevant rules or evidence, missing information and the system’s intended scope. A polished explanation generated by the same model is not independent proof that the recommendation is correct.
Check the patient inputs that materially affect the decision. A stale medicine list, wrong encounter, uncertain allergy history or unit mismatch can make a technically consistent suggestion inappropriate. The clinician should be able to inspect and correct those inputs before authorising the action.
Keep missing data visible. A system should not fill a required measurement with a typical value or convert an unknown condition into a negative finding simply to complete a recommendation. A complete-looking input set can be less trustworthy than an honest unresolved one.
Where the tool uses a deterministic calculation, distinguish input verification from method verification. A correct formula applied to the wrong units is still wrong. The clinic’s assessment should address the calculation’s intended population and use through appropriate clinical expertise, not just whether the arithmetic runs.
A nominal clinician approval can become a formality if the proposed order is hidden, the alternatives are unclear or a busy workflow rewards accepting defaults. Display the complete actionable content and allow correction or rejection without forcing the user through an unrelated task.
Review should cover the actual version that will be issued. If the software regenerates instructions after approval, the final content may differ from what the clinician saw. Test changes to medicine selection, strength, directions and destination after the draft review.
The SAFER order-entry guidance includes design, implementation and monitoring of clinical decision support. In this article, that becomes a practical requirement to inspect the whole ordering workflow rather than rely on a broad “doctor in control” claim.
Record what the clinician did without claiming more than the event proves. An acceptance event identifies an action; it does not certify the clinical judgment. Preserve appropriate accountability and a route for later correction without describing the software as transferring responsibility automatically.
Source: ASTP/ONC: SAFER Computerized Provider Order Entry with Decision Support
Use a fictional history saying that a patient previously stopped a medicine. Ask whether the feature merely drafts that history or creates a new active order. The expected result should be defined by the actual clinician decision, not by the presence of the medicine name in the input.
Use a fictional ambiguous product name or missing unit. The feature should expose the uncertainty through its supported process rather than choose a clinically specific answer to complete the form. The test is about preserving uncertainty, not about asking an assistant to recommend a substitute.
Test an unreviewed suggestion followed by printing, sharing and pharmacy handover. Confirm that a downstream output cannot acquire apparent authorisation simply because staff used another screen. A draft label inside the editor is insufficient if the printed copy looks final.
Finally, test a changed patient, changed source or late response after sign-off. The system should not reuse an earlier approval for different content. These cases are original acceptance tests and do not amount to a complete clinical validation study.
Procurement evidence should state the feature’s intended use, what it can modify, the authorising role and the limitations found. Separate demonstration results from vendor statements and from clinical validation evidence supplied for the product.
If the clinic limits a feature to drafting, confirm that configuration across roles and outputs. A restricted clinician view may still leave an administrative route that issues a proposed instruction. Review the actual permissions rather than the name of a setting.
The outline below makes the decision reviewable without pretending to replace specialist evaluation or applicable regulatory requirements.
If the feature is adopted after appropriate review, begin within the agreed scope and observe how clinicians use it. Look for acceptance without inspection, repeated corrections, unsupported field changes and outputs that reach patients before review. Do not equate a high acceptance rate with quality.
Review near misses and rejected suggestions as well as final records. A feature that is frequently corrected may still have value, but its total burden and error patterns need to be understood. A low complaint count alone cannot establish that the workflow is reliable.
Reassess when the model, prompt, catalogue, rule set or clinical form changes. A previous evaluation applies to a defined configuration and use case; it is not permanent approval of every future release.
The useful buying question is what the software can cause to happen without a meaningful clinician decision. Make that boundary visible, test it with fictional cases and obtain the professional evidence appropriate to the feature before using it in care.
A fictional test note says that the patient reports having stopped an earlier medicine. A generation feature recognises the medicine name and adds it to the current order draft. The clinician rejects the proposal, but the test continues through print and sharing to check that no active instruction remains elsewhere. This exposes a change in intent from reported history to prescribing. Correct spelling would not have caught the problem; the important issue is what action the system created.
Use these prompts in your clinic’s own notes. Record the result, responsible person and next action for each item.
No. They have different inputs, purposes and effects. Evaluate what the feature can change and how the resulting action is authorised. A drafting aid and a treatment-selection system should not be accepted on the basis of the same demonstration.
No. Plausible wording is not clinical evidence. The responsible clinician assesses the patient and the basis of the statement, preserves uncertainty where appropriate and determines what belongs in the authorised record.
It describes use, not necessarily correctness or benefit. Examine meaningful review, corrections, rejected suggestions, downstream effects and missed problems. A high rate can coexist with default acceptance or automation bias.
Ask about intended scope, input reliability, what the feature can modify, authorisation, validation evidence and change monitoring. Use fictional boundary cases and inspect the complete patient-output route, with appropriate clinical and regulatory review for the actual product.
Updated on . Workflow and procurement analysis, not a treatment recommendation or a universal legal classification of AI products. Clinical decisions and applicable product requirements need qualified review. No independent clinical or legal review is claimed.
Primary discussion of inaccurate outputs and automation bias. The clinic examples and proposed controls here are original analysis, not WHO validation of a product.
PDF linked from the 2025 collection; its footer is dated August 2024 and the resource page was updated April 2026. Supports design, implementation and monitoring review, not a universal acceptable alert rate.
Keep generated proposals separate from verified clinical fields; preserve unknown status, source, units and timing, and test downstream effects of a wrong value.
Distinguish dictation, conversation summaries, OCR and translation; label transformations and review status without implying that generated text is a verified diagnosis.
Measure alert burden with meaningful denominators, inspect false triggers and missed concerns, and test rule changes before reducing or suppressing warnings.
Ask Bractis to show exactly what its AI-assisted features can and cannot change. Use fictional history, missing-input and draft-to-print cases, and verify the authorisation boundary for the demonstrated configuration.
Request a clinic demo