In 2019, a research team led by the physician and economist Ziad Obermeyer reverse-engineered a commercial algorithm already running quietly across the United States health system. The tool, sold by Optum, helped decide which patients — out of a population of roughly 200 million a year — would be flagged for extra medical attention. To estimate who needed that help, its designers had made a choice that looked entirely reasonable: they used each patient’s past health-care costs as a stand-in for future clinical need. Costs are measurable, structured and available at scale. The logic seemed sound. It was also, the researchers showed, systematically wrong — and wrong in a way that harmed the sickest.
A medical algorithm becomes dangerous precisely when it mistakes an administrative trace for a clinical truth. Health is not the same as expenditure. When two patients have the same underlying illness but one has historically received less care — because of poorer access, weaker coverage, delayed diagnosis or systemic inequality — the cost record does not represent the disease. It represents the inequality. Obermeyer’s team found exactly this pattern: the system spent, on average, some 1,800 dollars less per year on Black patients than on White patients with the same level of need. Because it had been trained to predict cost, the algorithm read that lower spending as lower illness. Black patients assigned the same risk score were in fact considerably sicker. The bias cut the number of Black patients referred for additional care by more than half. The algorithm had not corrected an injustice; it had learned it, formalized it, and returned it to the health system wearing the mask of technical neutrality.
The result was not a minor statistical imperfection but a structural error, and the distinction matters. A model trained to predict cost can grow highly accurate with respect to billing while remaining wrong with respect to medicine. This is why the choice of proxy is never a secondary design detail in clinical AI: the proxy defines what the system is actually learning. If the target is cost, the model is not learning illness, urgency or vulnerability. It is learning a financial residue produced by a health-care system with its own exclusions and asymmetries — and then projecting that residue forward as if it were biology.
This matters because algorithmic outputs arrive with institutional force. A risk score may decide who is reviewed, who enters a care-management programme, who receives extra attention and who stays invisible. When the underlying target is misaligned, the system can redirect care away from those who need it most while appearing perfectly objective. The physician sees a number; the number may carry a history of unequal access disguised as prediction. The error is then technical, clinical and ethical at once — and worse, self-concealing, because no patient ever receives a notice explaining that an algorithm underestimated their need. The harm is real but diffuse, invisible to the very people it touches.
The remedy, tellingly, lay in the same place as the fault. When the researchers reformulated the model so that it no longer used cost as the proxy for need, the racial bias fell by roughly 84 percent. The mathematics had not failed; the question had. This is the lesson a serious architecture for medical AI must absorb before it celebrates predictive performance: ask first what is being predicted. Accuracy against the wrong target is not medical intelligence — it is disciplined confusion. Clinical AI must hold apart health from cost, need from expenditure, risk from billing, evidence from institutional residue. If a system is meant to support care, its variables, its targets and its outputs must remain answerable to care. Otherwise medicine is quietly reduced to whatever the data happened to record, and the patient is reduced to the administrative shadow left by their previous access to the system. In health care, the wrong proxy is not merely a modelling choice. It is a way of deciding who counts.
Authors
Lloret, E., Juan Antonio
Liked this post? Follow this blog to get more.