7. The Patient Is Not the Prediction

The person on the other side of the screen

Imagine a woman arriving at a clinic after several weeks of unusual fatigue. This is an invented case, but the choices are ordinary enough. The clinician has a crowded schedule. Her records come from different facilities. A program assembles a summary, notices a pattern worth investigating, and places a warning beside her name. The clinician asks a question that otherwise might have been missed. An investigation begins earlier. Something useful has happened.

Now imagine the same woman encountering a different use of the same broad technology. A program places her in a low-priority category. The appointment offered is weeks away. Nobody tells her that a prediction shaped the delay. When she calls, the receptionist can see the category but cannot change it. There is no person assigned to hear why her circumstances might be different.

Neither scene requires a machine that hates people. The difference lies in what an institution allows a prediction to do. In the first case, it directs attention toward a person. In the second, it can become an obstacle between the person and attention. The question is not settled by asking whether the program is intelligent. We need to know whose judgment it strengthens, whose choices it restricts, and how an error becomes visible.

Healthcare makes these questions unusually clear because the person affected often cannot afford to walk away. A shopper can sometimes choose another shop. A frightened patient may have one nearby hospital, one insurance network, one affordable consultation, or one appointment before a condition worsens. That dependence does not make technological assistance illegitimate. It increases the responsibility of those offering it.

The promise is substantial. A useful system might help a clinician find a relevant passage in a long record, translate instructions, identify an image requiring attention, or spend less of a consultation typing. None of these purposes requires pretending that every medical problem has become a computation. They require careful tools for particular difficulties. The World Health Organization’s 2021 guidance recognizes potential gains in diagnosis, treatment, research, and public health while placing ethics and human rights within the design and use of these systems.¹

A good starting point is to take the patient’s problem seriously before admiring the technology. Someone wants an answer, relief, a chance of recovery, or help living with an illness that cannot be cured. They may also want to remain at home, keep working, attend a family event, avoid a burdensome treatment, or understand what uncertainty means. A technically impressive answer can fail them if it answers the wrong question.

Suppose a tool estimates the probability of surviving a particular intervention. That estimate may be valuable. It does not itself tell the patient whether the intervention is worth its burdens. Two people with the same predicted outcome might reasonably choose differently. One values every possible extension of life; another places greater weight on comfort or time outside a hospital. Respecting the patient means making room for that difference, including when the patient’s priorities surprise the professionals.

Dignity enters medicine here as something practical. It is the standing of the person whose life the decision concerns. It means that intelligence about someone does not erase the need to speak with them. An excellent prediction can inform their choice without becoming the owner of it.

Evidence that reaches the bedside

An algorithm can perform well on a collection of past records while failing to improve the care of people who arrive tomorrow. Those are different achievements. The first shows something about pattern recognition under specified conditions. The second concerns a changing chain of people, equipment, habits, resources, and decisions.

Consider a hypothetical system that flags patients who might need urgent review. To evaluate it, an institution must ask more than whether flagged patients later became ill. Were the warnings early enough to help? Did they arrive where someone could act? Did responding to them improve outcomes? Did the extra warnings overwhelm a small team? Did the tool overlook people whose records contained less information? The hospital could have an accurate prediction system and an ineffective response system.

There is a documented reason to resist treating widespread adoption as proof. In a 2021 external evaluation at Michigan Medicine, Andrew Wong and colleagues found that the version of the proprietary Epic Sepsis Model they studied performed substantially worse than the developer’s reported performance. Their retrospective study examined 38,455 hospitalizations and identified problems with discrimination and calibration.² This was an evaluation of a particular model in a particular setting, not a verdict on all sepsis prediction or on later versions. Its enduring lesson is narrower and stronger: an institution needs evidence about the tool it actually proposes to use.

A 2026 prospective evaluation of the updated Epic Sepsis Model version 2 covered 227,091 encounters at four US health systems. The results were encouraging for risk prediction, but performance varied by site and many alerts did not identify a sepsis case. The models had been adjusted locally. These results cannot be read as a controlled before-and-after comparison with the 2021 study, and neither study established a mortality benefit from acting on alerts.³ A later version deserves fresh evaluation; a familiar product name cannot supply it.

Calibration is a modest sounding word with a serious meaning. If a system routinely assigns a risk of one in five to a group of patients, events should occur at roughly that frequency in sufficiently large comparable groups. A ranking can place higher-risk patients above lower-risk patients while still misstating how much risk either group faces. Different decisions require different kinds of accuracy. A score useful for sorting a queue might not be reliable enough to communicate as a person’s likelihood of survival.

Encouraging evidence should receive the same precision. The randomized MASAI mammography trial in Sweden compared an AI-supported reading workflow with standard double reading. Its 2023 interim analysis met the early safety criterion and reported 44.3 percent fewer screen readings.⁴ The 2025 full-cohort report analyzed 105,915 participants. Screen-detected cancers numbered 338 with AI support and 262 with standard reading, or 6.4 and 5.0 per 1,000 participants. The number of readings fell by 44.2 percent, with no statistically significant increase in false positives or recalls.⁵

The primary outcome, reported in January 2026, concerned interval cancers, diagnosed between scheduled screening rounds. There were 82 with AI support and 93 with standard reading. The uncertainty range excluded the trial’s prespecified 20 percent worsening margin but still included no difference and some worsening. This met the non-inferiority criterion; it did not establish that AI reduced interval cancers.⁶

These are reports from one trial, not independent replications. Its results support a particular workflow, including human readers, at four Swedish sites. They do not establish a mortality benefit, remove the question of overdiagnosis, or make every imaging product interchangeable. A reduction in the number of readings also does not mean the same reduction in total clinical time or staff required.

Scroll sideways to see all columns. Keyboard: focus the table and use the left and right arrow keys.

Text equivalent: Screening: benefit and uncertainty

Part of figure	Value or meaning
Title	Screening: benefit and uncertainty
Study	MASAI • one randomized Swedish trial
Participants	Participants analyzed: AI-supported: 53,043 • standard: 52,872
Primary measure	Cancers between screens Primary outcome • January 2026 Cases per 1,000 participants Scale: 0 to 2.5
AI interval cancers	AI-supported: 1.55 (82 cases)
Standard interval cancers	Standard: 1.76 (93 cases)
Relative comparison	Ratio: 0.88 • 95% interval: 0.65–1.18
Ratio scale and markers	Axis: 0.50 to 1.30 Solid rule: 1.00 = no difference Dashed rule: 1.20 = trial threshold
Interpretation	The interval permits some worsening, but excludes the trial’s 20% margin. It does not establish a reduction.
Screening measure	Cancers found at screening Secondary outcome • 2025 report Cases per 1,000 • scale: 0 to 8
AI detection	AI-supported: 6.4 (338 cases)
Standard detection	Standard: 5.0 (262 cases)
Workload measure	Screen readings Secondary outcome • 2025 report Total readings • scale: 0 to 120,000
AI readings	AI-supported: 61,248
Standard readings	Standard: 109,692
Workload interpretation	44.2% fewer readings This is not a staffing reduction.
Sources	Sources: Hernström et al. (2025); Gommers et al. (2026).
How to read the figure	Interval cancers were diagnosed after a negative screen, before the next round or within two years. A ratio below 1 favors AI-supported screening.

Figure 1. One trial, different outcomes and units. These results do not establish a mortality benefit.

The case for progress becomes more convincing when described this way. Real benefit is more valuable than an inflated promise because a health service can decide what to build around it. A reduction in reading workload might shorten a backlog, free staff for complex cases, or extend screening capacity. Whether those benefits materialize depends on choices made after the performance result. Time saved in one part of a service is an opportunity, not a complete account of what patients receive.

We should also reject an impossible standard under which AI must never make a mistake before it may assist medicine. Human practice includes mistakes, delays, uneven access, and exhaustion. The relevant comparison is with the available alternatives under realistic conditions. A tool can deserve adoption because it reduces a serious existing failure, even though it leaves other risks to manage.

That comparison must remain honest in both directions. It should not set an imperfect machine against an imaginary perfect physician. Nor should it set a polished demonstration against the worst anecdote about human care. Fair evaluation asks what happens when a particular service uses a particular tool for a stated purpose, compared with a credible alternative. The people experiencing that difference matter more than the prestige of either side.

The clinician needs room to disagree

The phrase human oversight can describe a real protection or an empty ritual. A clinician who has three seconds to approve an output they cannot investigate does not meaningfully control the decision. A clinician who can investigate but is punished for slowing a queue may possess formal authority without practical freedom. A signature is evidence that someone signed. It is not evidence that they had the conditions needed to judge.

The 2021 good machine learning practice principles issued jointly by the FDA, Health Canada, and the United Kingdom’s MHRA specifically emphasize the performance of the human and AI team, representative data, appropriate testing, and monitoring after deployment.⁷ The focus on the team matters. Responsibility belongs to a working arrangement, not merely to a model’s performance in isolation.

Imagine a hospital introducing an automated summary of patient records. At first, clinicians use the draft as a starting point and consult the underlying notes. The hospital then reduces appointment lengths because summaries are supposedly saving time. Over several months, schedules become so tight that opening the original record is treated as an avoidable delay. The hospital can still say that every summary receives human review. Yet its own decisions have made serious review harder.

In this invented example, the problem is not that assistance inevitably destroys judgment. The problem is that management has spent the safety margin before measuring it. If a tool saves ten minutes on average, those minutes cannot all be promised simultaneously to the patient, the clinician, and the budget. Deciding where they go is part of implementation. It deserves an explicit decision rather than an assumption hidden in scheduling software.

Meaningful oversight therefore needs time, access, competence, and authority. The reviewer needs enough time to notice trouble, access to information that can reveal it, training relevant to the task, and the right to change the course of action. The organization must also make disagreement ordinary enough that staff can raise it before harm occurs. None of this requires physicians to understand every mathematical detail of a model. It requires them to understand what they are relying on and when that reliance should stop.

A good interface can help. A summary can link a statement to the record from which it came. A warning can distinguish a missing measurement from a normal result. A system can display that it lacks sufficient information instead of turning absence into reassurance. These are design proposals, not promises that a particular product delivers them. They show how a concern about accountability can become a concrete improvement.

The clinician should not become a convenient destination for every responsibility in the chain. Developers choose training methods and document limitations. Hospitals select products, configure thresholds, and organize staffing. Professional users decide how to interpret outputs within their role. Public authorities establish requirements and investigate failures. If only the person nearest the patient is answerable, the organizations with the greatest ability to prevent recurring problems may escape scrutiny.

This is a moral allocation of responsibilities, not a claim that every jurisdiction assigns legal liability in the same way. The practical point is straightforward: the party able to fix a problem must remain visible. A mistaken input, a poorly validated model, an unsafe local configuration, and a clinician’s failure to respond are different failures. Treating them as one vague case of AI error obstructs improvement.

After a serious incident, the patient should not have to solve that institutional puzzle alone. They need an explanation of what happened, a route to correction, and access to the appropriate review. An institution that adopts a system should accept the work of answering for its use.

Consent has to concern the actual choice

A proportionate ethical approach does not require a separate ceremony of consent for every calculation inside a hospital. The information and choice owed to a patient should reflect the material use and applicable law. Routine scheduling software raises different issues from automated treatment advice, data reuse, or experimental tools. Major choices should not be buried in a sentence so broad that nobody could understand what they had accepted.

The useful question is what a person needs to know in order to exercise the choice that genuinely belongs to them. A clinician using software to draft a letter raises different issues from a service allowing a conversational system to give treatment recommendations directly to patients. A tool that examines a scan for a narrowly defined purpose differs from a program that collects intimate conversations for unrelated product development. Calling all of them AI conceals more than it explains.

The FDA and its Canadian and British counterparts addressed this practical problem in 2024 guiding principles on transparency for machine learning medical devices. The document emphasizes information suited to the audience, the context of use, and the stage at which information is needed.⁸ Transparency should help someone make a decision. An overwhelming technical disclosure can fail that test as thoroughly as a reassuring slogan.

In practice, a patient might reasonably want plain answers to several questions in the course of a conversation. What will the tool do? Will a professional review what it says? Does it affect access to care? Will the information be retained or used beyond this episode? Who can explain a result? Can the patient ask for another route, and what would that route involve? These are proposals for useful communication, not a claim that every use requires an identical form.

The possibility of declining deserves special care. A nominal alternative may be so expensive or delayed that refusal is unrealistic. That does not automatically establish coercion in every case; healthcare always operates with constraints. It does mean that providers should describe the constraint honestly. A patient should not be told that a choice is freely available when the institution has made one branch practically inaccessible.

People can also reasonably choose more automation. A patient might prefer a convenient screening tool, an automated reminder, or an interface available in their language. Respect is not measured by how successfully professionals discourage these preferences. It is measured by whether the person understands the relevant tradeoffs and has a service capable of responding when something goes wrong.

Consent to receiving care and consent to helping develop a commercial system should not be casually fused. They involve different interests. A person may welcome assistance with an immediate health problem while objecting to indefinite reuse of their information. Organizations can seek permission for research, explain its possible value, and design forms of participation that are neither exploitative nor impossible to administer. The fact that information could help others is a reason to make a fair proposal, not to pretend that the information has no relationship to the person who supplied it.

Nor does consent exhaust ethics. Someone can agree to a poorly designed service because they have no better option. An institution still has duties concerning evidence, security, professional competence, and truthful claims. A signature should not convert preventable institutional failure into the patient’s personal responsibility. The person who agrees to treatment has accepted a properly explained course of care, not granted moral immunity to everyone involved in delivering it.

Better care must become available care

The strongest argument for medical AI may be made by someone who cannot currently obtain enough medical attention. A specialist’s appointment can be distant, expensive, or unavailable. A small health service may be unable to staff every kind of expertise at every hour. In such circumstances, assistance that helps local professionals detect problems or obtain advice could be a serious gain. It deserves more than the dismissive observation that machines are imperfect.

Yet scarcity creates a temptation to offer disadvantaged people a weaker standard and call it innovation. If wealthy patients retain access to professionals while everyone else is directed into a system with little evidence and no effective appeal, efficiency has become a vocabulary for unequal treatment. The right comparison includes both the shortcomings of current care and the standard the new service could reasonably achieve.

Consider a hypothetical island clinic evaluating an imaging tool developed mainly using data from large urban hospitals overseas. The clinic should not assume that the product will fail locally. It should ask whether its equipment, referral patterns, population, and staffing differ in ways relevant to performance. A partnership with another health service might make evaluation affordable. A narrow pilot could identify the conditions under which the tool is helpful. Local expertise would contribute to the assessment, rather than merely receive instructions from elsewhere.

The subscription price is only one part of that decision. The clinic needs connectivity, maintenance, secure handling of records, training, and a plan for interruptions. If a useful result creates a need for follow-up that the clinic cannot provide, the discovery can become another unresolved burden. The purpose of adoption is to improve a path of care. Purchasing a prediction is only one possible step along that path.

This also exposes a quieter form of responsibility: deciding what problem receives investment. A service may have a celebrated AI pilot while patients still struggle to make appointments or obtain transport. The pilot might nonetheless be worthwhile. But novelty should not excuse it from comparison with the other uses of limited money and staff. A less glamorous improvement may sometimes save more time, prevent more harm, or reach more people.

The choice need not be permanently binary. Better administrative tools could release funds for patient transport. More efficient screening could increase the usefulness of an additional nurse. Technology and ordinary care can strengthen each other when planned as parts of the same service. The important distinction is between a demonstrated connection and an attractive story about one.

An ethical health system must also remember the people a predictive tool cannot help. Someone with a rare condition, an unusual history, or incomplete records remains entitled to attention. Uncertainty is a reason to investigate appropriately, not a reason to disappear a person from the system. The edge of a model’s competence should be the beginning of an institution’s response.

Healthcare gives the question of playing God its most compassionate form. We want greater power over disease because suffering is real. Refusing useful assistance can carry a human cost. But the power to recognize a pattern in a person’s body does not grant ownership of their future. The proper achievement is a system in which knowledge becomes care, care remains answerable, and the patient can still be heard when the prediction says something else.

Sources and notes