A medical algorithm becomes dangerous when it mistakes an administrative trace for a clinical truth. One of the clearest documented failures in health-care artificial intelligence came from a commercial risk-management algorithm that used past medical cost as a proxy for future clinical need. At first glance, that choice may appear practical: costs are measurable, structured and available at scale. But health is not the same as expenditure. If two patients have the same clinical need, but one has historically received less care because of poorer access, weaker coverage, delayed diagnosis or systemic inequality, the cost record will not represent the disease burden. It will represent the inequality. When that record is then used as the target of prediction, the algorithm does not correct the injustice; it learns it, formalizes it and sends it back into the health system with the appearance of technical neutrality.
The result is not a minor statistical imperfection. It is a structural error. A model trained to predict cost may become accurate with respect to billing and still be wrong with respect to medicine. It may identify the patients who are expected to generate higher expenditure, while missing patients whose needs are greater but whose access to care has historically been lower. This is precisely why proxy selection is not a secondary design decision in clinical AI. The proxy defines what the system is actually learning. If the proxy is cost, the model is not directly learning illness, urgency, vulnerability or medical need. It is learning a financial residue produced by a health-care system with its own inequalities, exclusions and asymmetries.
This matters because algorithmic outputs often arrive with institutional force. A risk score may decide who is reviewed, who is enrolled in a care-management programme, who receives additional attention and who remains invisible. If the underlying target is misaligned, the system can redistribute care away from those who need it most while appearing objective. The physician may see a number, but the number may contain a history of unequal access disguised as prediction. In such cases, the error is not only technical. It is ethical, clinical and administrative at the same time. The system does not merely fail to see the patient; it may help the institution continue not seeing the patient.
A serious architecture for medical AI must therefore ask a prior question before celebrating predictive performance: what is being predicted? Accuracy against the wrong target is not medical intelligence. It is disciplined confusion. Clinical AI must distinguish health from cost, need from expenditure, risk from billing, and evidence from institutional residue. If a system is intended to support care, its variables, targets and outputs must remain accountable to care. Otherwise, medicine is reduced to what the data happened to record, and the patient is reduced to the administrative shadow left behind by previous access to the system. In health care, the wrong proxy is not just a modelling choice. It is a way of deciding who counts.
Liked this post? Follow this blog to get more.