In 2016, the European Union’s General Data Protection Regulation had to recognize something uncomfortable: a person’s right not to have an automated decision be the final word on their life. Eight years later, the European Artificial Intelligence Act went further and required human oversight for high-risk systems.

The fact that this had to be written into law says more than any consulting report. Placing a human being at the end of a process does not guarantee that this person is judging anything. There may be a formal responsible party, a confirmation screen, a signature, while the real decision has already been made by the system that classified the case and drafted the explanation later presented as evidence.

Thomas Aquinas had already drawn the exact border of this problem in the thirteenth century. He distinguished efficiency from prudence. A system can find the most efficient means to reduce a cost or increase a conversion. What it cannot do by itself is judge whether the end being pursued is rightly chosen, whether the sacrifice it demands is proportionate, or whether the metric used to represent the good is the correct one.

Artificial intelligence optimizes within the world we build for it. When that world contains a badly formulated end, greater precision does not correct the problem. It makes it faster and less visible.

Leo XIV brought that same distinction to the center of his first encyclical, Magnifica Humanitas. He warns that the use of artificial intelligence is never a purely technical fact. When it intervenes in work, credit or access to opportunities, it may end up selecting who is considered worthy without anyone assuming the moral weight of that selection.

Mary Parker Follett anticipated the same boundary from another angle. In 1926, she argued that authority does not reside in the position of the person who gives an order, but in the concrete situation that the order attempts to resolve. She proposed studying the situation together and obeying what she called the law of the situation, not merely the person who states it.

A system can help us understand a situation better. What it cannot do is discover that law in Follett’s sense, because the law of the situation is not merely a statistical pattern. It is a judgment about what this singular case morally and practically requires. That judgment can only be made by someone capable of answering for it.

In 2019, a team led by Ziad Obermeyer published in Science a case that shows this damage already occurring without anyone intending it. An algorithm used by hospitals and insurers to prioritize patients used historical spending as a proxy for how sick each patient was. The problem was that Black patients, because of previous barriers to access, generated less spending than white patients with the same real level of illness.

At the same risk score, Black patients were substantially sicker. Correcting the bias would have more than doubled the proportion of Black patients identified for additional help. No one programmed that prejudice directly. The system inherited a real inequality and turned it into an apparently neutral number.

Aquinas distinguished efficiency from prudence. Leo XIV warned that no algorithm can bear the weight of deciding who is worthy. Follett showed that legitimate authority is born from studying a situation, not from a rank or a calculation. Separated by centuries and by very different vocabularies, all three point to the same boundary: there is a judgment about persons that no machine can make on behalf of the person who leads, because making that judgment is precisely what leadership means.