In 2016, the European Union's General Data Protection Regulation had to acknowledge something uncomfortable: a person's right not to have an automated decision be the last word on their life. Eight years later, the European Artificial Intelligence Act went further and required human oversight for high-risk systems.
That it took legislation to get here says more than any consulting report. Putting a human being at the end of a process does not guarantee that this human being is judging anything. There can be a formally responsible person, a confirmation screen, a signature, while the real decision was already made earlier, when the system classified the case and drafted the explanation that is later presented as evidence.
The boundary between efficiency and prudence
Thomas Aquinas had already drawn the exact boundary of this problem in the thirteenth century, with a precision that much of today's vocabulary on technology ethics still has not reached. He distinguished efficacy from prudence. A system can find the most efficient means of cutting a cost or raising a conversion rate. What it cannot do on its own is judge whether the end it pursues was well chosen, whether the sacrifice it demands is proportionate, whether the metric it uses to represent the good is the right one. A person can reason with enormous skill about a flawed objective and be, in that field, highly competent without being prudent. Artificial intelligence optimizes within the world we build for it. When that world contains a badly formulated end, more precision does not fix the problem. It makes the problem faster and less visible.
Leo XIV carried that same distinction to the center of his first encyclical, Magnifica Humanitas, signed in May of this year. He warns that the use of artificial intelligence is never a purely technical matter, and that when it intervenes in work, credit or access to opportunities, it can end up selecting who is considered worthy without anyone bearing the weight of that selection. He does not say this to demand sentimentality from business. He says it because a person can legitimately exceed the category a system used to describe them, and no algorithm is designed to recognize that excess.
Mary Parker Follett, who first worked as a social worker in Boston's immigrant neighborhoods before becoming one of the most original voices in management thought, had anticipated the same boundary a century earlier from a different angle. In 1926 she argued that authority does not reside in the office of whoever gives the orders, but in the concrete situation an order is meant to resolve. She proposed depersonalizing command, studying that particular situation together, and obeying what she called the law of the situation, not the person who states it. A system can greatly help us understand a situation better. What it cannot do is discover that law in the sense Follett gave the expression, because it is not a statistical pattern. It is a judgment about what this singular case demands, morally and not only technically, and only someone able to answer for it can make that judgment.
An algorithm that turned inequality into a number
In 2019, a team led by Ziad Obermeyer published in the journal Science something that shows this problem already doing real harm, without anyone having intended it. An algorithm used by hospitals and insurers to prioritize patients, which influenced health decisions for around two hundred million people a year in the United States, used past spending as an indicator of how sick each patient was. It seems reasonable: spending more usually correlates with needing more. The problem is that Black patients, because of barriers to access that predate the algorithm, generated less spending than white patients with the same real level of illness. At the same risk score, they were considerably sicker. Correcting the bias would have more than doubled the share of Black patients identified for extra help, from seventeen to forty-six percent. Nobody programmed that prejudice. The system inherited a real inequality and turned it into a seemingly neutral number, and that apparent neutrality was precisely what allowed it to reproduce the harm without any discriminatory intent being needed.
Aquinas distinguished efficacy from prudence. Leo XIV warned that no algorithm can bear the weight of deciding who is worthy. Follett showed that legitimate authority comes from studying a situation together, not from an office or a calculation. The three, separated by seven hundred years and by vocabularies that could not be more different, point to the same exact boundary: there is a judgment about people that no machine can make on behalf of whoever leads, because making it is, precisely, the definition of leading.
