2. Capability Is Not Permission

The machine that gives the right answer

Suppose a town builds an AI system that can forecast local flooding with remarkable accuracy. For this thought experiment, grant the technology its strongest plausible case. The measurements are reliable. Independent researchers have tested the system. Its warnings consistently outperform the process the town used before. We need not weaken the machine in order to ask a serious question about its authority.

The system identifies two neighborhoods facing danger. The town has enough equipment to protect one immediately. Protecting the commercial district will preserve more buildings by monetary value. Protecting the residential district will prevent more families from being displaced. The first option may preserve businesses that support many livelihoods. The second may protect people who lack insurance and alternative housing. A forecast cannot make these considerations become the same consideration.

Engineers could specify a single objective and instruct the system to optimize it. They could combine property values, expected injuries, disruption, and displacement into a weighted score. That might help officials compare options. But deciding the weights would already involve decisions about what counts and how much. The moral choice would not disappear. It would move into the specification.

Even a correct recommendation therefore leaves a question: correct according to what aim, established by whom? If the town authorized the system to minimize deaths, and it reliably helps officials do that, its recommendation has a strong claim on their attention. If a contractor quietly programmed it to protect the most valuable property, the same technical excellence may support a decision the public never authorized.

This distinction is sometimes obscured by the language of intelligence. A very intelligent adviser may understand a problem better than a mayor. That gives the adviser reasons to persuade the mayor and the public. It does not automatically give the adviser a right to assume office. Expertise can make a person’s contribution indispensable without making that person’s authority unlimited.

The distinction also protects us from romanticizing human discretion. A mayor should not ignore reliable evidence merely to demonstrate that a human remains in charge. If the evidence clearly shows that an evacuation route is unsafe, responsible authority requires taking it seriously. Human responsibility includes the duty to recognize when a tool knows something useful that its operator does not.

The proper relationship is demanding on both sides. The system must be worthy of reliance for its intended task. The institution must be authorized to make the decision. The purpose must be defensible. The people affected must have the protections appropriate to the stakes. Failure in one area is not repaired simply by excellence in another.

This is why the phrase “the AI decided” often tells us less than it appears to tell us. It may describe the immediate mechanism that produced an output. It does not identify who defined the objective, selected the evidence, accepted the risk, purchased the service, or gave the output practical force. Those decisions establish the human arrangements through which an output becomes someone’s fate.

A right to build and a right to rule

The person who creates something normally has legitimate interests in it. Building a useful system may require imagination, time, investment, and considerable risk. A serious ethics of AI should acknowledge this. Treating every successful invention as if it had simply appeared from nowhere would be unfair and would weaken the reasons people have to undertake difficult work.

Yet ownership of a tool and authority over everyone affected by it are different matters. Owning a building does not settle every question about how one may use it. Owning a database does not make every possible inference from it appropriate. A system’s creator may have a legitimate claim to payment and a legitimate desire to protect confidential methods without acquiring unlimited rights over users, workers, or bystanders.

Consider a hypothetical landlord who buys a system for scheduling maintenance. The system notices recurring plumbing problems and helps prevent expensive damage. The landlord’s decision improves the building. Now suppose an update begins estimating which tenants are most likely to complain, organizing inspections to discourage them, and suggesting how to raise their costs without provoking a formal dispute. No new consciousness has emerged. The moral problem lies in the authorized purpose and the power relationship.

The landlord might say the tool is merely advising, leaving every final action to a person. That statement does not settle the issue. Advice can be designed around an improper goal. Human approval can faithfully implement that goal. A person can remain in the process while the process remains abusive.

Now reverse the example. A tenants’ association uses AI to interpret maintenance records and prepare clear requests for repairs. The same general capability, finding patterns and generating useful language, can help people with less institutional power participate more effectively. It would be perverse to condemn both deployments merely because each uses AI. The direction of power, the purpose, and the availability of recourse change the assessment.

There is a further distinction between legal permission and moral justification. A practice may be permitted because the law has not addressed it, because enforcement is weak, or because the relevant jurisdiction deliberately allows it. None of those circumstances proves that the practice treats people well. Equally, moral discomfort alone does not establish that a practice should be prohibited. A prohibition needs reasons concerning harms, rights, alternatives, enforceability, and the costs it creates.

The space between these conclusions is where mature public judgment operates. It can recognize something as distasteful without criminalizing it. It can recognize something as beneficial while placing conditions on its use. It can acknowledge an inventor’s achievement while limiting what a product may do in a workplace, classroom, or public agency.

UNESCO’s 2021 Recommendation on the Ethics of Artificial Intelligence places human dignity and rights at its center and calls for proportionality, appropriate transparency, oversight, and accountability.¹ These principles do not write a complete rulebook for our hypothetical landlord. They establish the kind of reasons a rulebook ought to consider. The practical work remains ours.

What a yes can authorize

Consent offers one powerful source of permission. If an adult freely asks an AI assistant to help organize personal notes, the request matters. Respecting the person means taking their choices seriously, including choices others would not make. A society that invokes dignity to prevent adults from using harmless tools would be protecting an abstraction at the expense of actual freedom.

But consent has a shape. A yes to one action is not a yes to every adjacent action. Agreeing that an assistant may summarize a private letter does not automatically authorize its use in advertising, disclosure to an employer, or permanent inclusion in unrelated profiles. Permission has a purpose, a scope, and sometimes a duration. A responsible institution should be able to say what its permission covers.

Imagine that a community center offers a voluntary AI tutor. Participants understand that it is a machine, receive clear information about its limitations, and can choose ordinary lessons instead. That arrangement provides a reasonable starting point. Now imagine that the center silently makes use of the tutor a condition of receiving food assistance. The interface still contains an accept button, but the moral meaning of pressing it has changed.

Necessity does not make every agreement invalid. People regularly make choices under constraints. The important questions concern the source of the constraint, the importance of the service, the relevance of the condition, and the availability of alternatives. Requiring information needed to administer food assistance is different from requiring unrelated emotional profiling. The existence of a click should not prevent us from examining that difference.

Consent can also fail to cover other people. A person may freely publish a photograph containing friends who did not agree to its use. A manager may purchase software that monitors employees who had no role in the purchase. A household may install an assistant that records guests. Calling the buyer a willing user does not answer the claims of everyone else drawn into the system.

Nor can consent alone settle every collective effect. Thousands of individually reasonable choices may alter a shared environment. A platform could persuade many users to disclose personal details, allowing inferences about people who never joined. A city could become difficult to navigate without entering a proprietary system. Each voluntary transaction may appear modest while the cumulative arrangement creates dependence.

These observations need not lead to a world of endless permission forms. More forms can make consent worse when people become exhausted by documents they cannot reasonably assess. A useful response is to reduce the number of unexpected choices imposed on them. Collect less information where less is sufficient. Make the ordinary purpose clear. Separate optional uses from essential services. Require a fresh decision when the purpose changes materially.

Consent is strongest when refusal remains a livable option and understanding is reasonably possible. Where those conditions cannot be secured, institutions need another legitimate basis for action and stronger duties toward affected people. A government cannot govern only those who click yes. It therefore owes citizens public reasons, fair procedures, and limits on its power. Consent and legitimate public authority are related answers to different situations; neither should be used as an excuse to avoid the other.

The freedom to disagree with the good

The most troubling uses of power are not always presented as selfish. Some arrive with a sincere promise to make people healthier, wiser, calmer, or more cooperative. An AI system might infer that a user would benefit from particular habits and begin steering the person toward them. The intervention could work. Its success would still leave questions about whether the user knew what was happening and whether the system respected the user’s own priorities.

For a thought experiment, imagine an assistant that can substantially reduce impulsive spending by making certain purchases harder to find. A user who explicitly requests this help may welcome it as self-government. An employer who quietly imposes the same intervention to reduce wage demands is exercising a different kind of power. The observable outcome, fewer impulsive purchases, does not tell the whole moral story.

A society that takes people seriously must allow room for choices that competent observers regard as unwise. Someone may value adventure over security, privacy over convenience, religious observance over income, or artistic work over a more profitable career. The right to pursue a life would be thin if it protected only choices that an approved adviser would recommend.

This does not make every preference binding. One person’s freedom can collide with another person’s safety or equal standing. The point is that disagreement needs a real justification before it becomes a reason to override someone. A prediction that a person will later regret a choice is not by itself an unlimited warrant to prevent the choice.

Freedom of thought, conscience, and religion is recognized in Article 18 of the International Covenant on Civil and Political Rights, including protection against coercion that impairs a person’s freedom to adopt a religion or belief.² That legal principle is not a complete theory of AI persuasion. It does, however, remind us that the inner life is not simply an unfinished project waiting for more efficient administration.

The same protection must extend across conviction. A religious user should be able to ask an assistant for help understanding a faith tradition. A nonreligious user should be able to investigate arguments against it. A person uncertain about belief should not be covertly assigned to a conversion campaign. Providers may set reasonable boundaries on what their services offer, but they should not misrepresent a system’s purpose or disguise a sponsor’s campaign as neutral personal guidance.

Sometimes pluralism requires different tools rather than one supposedly neutral tool for everyone. A literary assistant, a Catholic study assistant, and a secular philosophy assistant may each be legitimate if their purposes are clear and their claims are honest. Problems grow when a system becomes unavoidable and quietly treats one conception of the good life as if it were an objective fact of nature.

Neutrality itself requires care. Choosing which questions to answer, how to handle uncertainty, and what language to use already shapes a service. Complete absence of perspective is not a realistic standard. The more practical standard is candor about important commitments, fairness toward alternatives, accuracy about evidence, and room for users to examine the reasons offered.

Dissent has a further value beyond individual freedom: it can reveal a mistake. The person who refuses an apparently sensible arrangement may notice something its designers overlooked. A system that treats all resistance as irrational deprives itself of a source of correction. It becomes least able to learn precisely where its authority is most disputed.

A responsibility that can be found

Suppose our flood system produces a recommendation that officials follow, and people are harmed. It would be unreasonable to conclude that harm proves misconduct. Emergencies can force choices among terrible alternatives. The relevant inquiry asks what was known, what should have been known, whether the system was used appropriately, and whether the decision-maker acted within a defensible mandate.

It would be equally unreasonable for every participant to disappear into the chain. The contractor says it only supplied software. The agency says it relied on experts. The operator says it followed instructions. Senior officials say the process was automated. If every statement were accepted as a complete defense, the arrangement could exercise power while leaving no one answerable for its design or operation.

Responsibility should be divided according to real control and real obligations. A developer may be responsible for concealing a known defect. A purchaser may be responsible for deploying a system in conditions for which it was never assessed. A manager may be responsible for making review impossible. An operator may be responsible for ignoring a clear warning. These are different failures, and identifying them requires evidence rather than a search for the easiest person to blame.

This is why merely keeping a human near the final decision is insufficient. A junior employee might have neither the information nor the authority to reject the output. If reviewing one case properly takes fifteen minutes but the employee is expected to approve fifty cases an hour, the institution has created a performance of oversight. Its signatures record compliance more than judgment.

Madeleine Clare Elish calls a related failure a “moral crumple zone”: the nearby operator absorbs blame even when important choices were beyond that person’s control. Her analysis of accidents and their public interpretation is a warning about how responsibility is assigned, not a measure of how often all automated systems fail.³ In our example, the employee’s signature should lead investigators to the workload target, the information available, and the managers who could change both. It should not allow the inquiry to end with the easiest person to identify.

A meaningful reviewer needs an intelligible task, adequate time, access to relevant evidence, permission to disagree, and an effective route for escalation. Where the system works faster than meaningful intervention is possible, control must also be exercised beforehand through the conditions of deployment and the limits of permitted action. Human responsibility can exist at several levels. It should be located where it can actually affect events.

This approach has costs. Review takes time. Documentation requires work. Restrictions can prevent some useful actions as well as harmful ones. A serious case for accountability must admit these burdens and scale them to the stakes. A spelling assistant does not require the same institutional machinery as a system that determines access to essential support. Treating every output as equally consequential wastes attention that high-stakes decisions need.

Nor does accountability require everyone to understand every mathematical detail. A passenger need not master an aircraft’s engineering to have a claim to safe transport. A person affected by an AI decision may need to know the relevant evidence, the rule applied, the responsible institution, and how to challenge an error. Different audiences need different forms of explanation. Technical disclosure without practical recourse can leave a person as powerless as before.

The ambition should be to make responsibility findable. Someone can identify the institution making the decision. That institution can explain its authority. The relevant people can examine whether the system was appropriate. An affected person can seek correction without having to reconstruct an entire technical supply chain. Where harm occurred, the process can distinguish bad luck, reasonable uncertainty, negligence, and abuse.

Greater capability makes this ambition more urgent because it can compress the distance between a choice and its consequences. A policy that once moved slowly through offices might be implemented across many cases at once. That reach is valuable when the policy is sound. It can also spread a mistaken assumption before the institution realizes what it has done. The right response is to make good action easier to sustain and error easier to discover and repair.

Return to the town facing the flood. We should want its forecasting system to be excellent. We should want officials to listen to it carefully. We should also want residents to know which values govern the allocation of protection, what authority made those choices, and how the town will learn afterward. The machine’s intelligence and the community’s judgment are not enemies. They become partners when the distinction between knowing how and being entitled to decide remains clear.

Sources and notes