15. Power That Can Answer Back

Six questions before permission

A principle becomes useful when it changes what someone must do. Dignity matters in the design of a service when it affects how a person can be heard. Accountability matters when an identifiable institution must investigate a failure and provide a remedy. Human oversight matters when the human has time, information, authority, and a workable alternative to accepting the machine’s output.

The difficulty is making these commitments concrete without creating a ritual of paperwork. A responsible institution needs something clear enough to use and demanding enough to expose weak decisions. It also needs a method that permits beneficial work to proceed. A process that can only say no risks being bypassed, especially where the existing service is failing people.

I propose six questions. What legitimate purpose are we pursuing? What standing do affected people have to consent, participate, or challenge? What evidence supports the use? What boundaries constrain it? Who must answer for its operation? What happens when it goes wrong?

These are the book’s proposed questions, not a papal formula, a new legal standard, or a claim to have invented the principles behind them. They organize the argument developed here around an institution’s duty to give an answer. That answer must concern the whole arrangement, including staff, incentives, contracts, and remedies, rather than merely the model at its center.

Established frameworks provide a foundation. The US National Institute of Standards and Technology’s voluntary AI Risk Management Framework 1.0, published in 2023, organizes work around governing, mapping, measuring, and managing risk. It treats governance as continuous and recognizes the importance of context and the system’s lifecycle. Its categories support disciplined organizational work; completing them does not itself confer legal permission or demonstrate that a particular use is justified.¹

The six questions add a reader’s route through that institutional work. Purpose asks whether the organization has described a human benefit rather than a technology purchase. Standing asks whether people have a meaningful position in decisions affecting them. Evidence asks what is known, including what is known about the current alternative. Boundaries ask what the system may never do under its present authorization. Answerability asks who has both responsibility and the means to act. Remedy asks how people and services recover.

Each answer must be connected to the others. A noble purpose cannot compensate for a missing remedy. A capable owner cannot authorize an unlawful use. Public participation cannot establish technical reliability. Excellent performance cannot justify quietly changing the purpose. The point is not to add six scores and declare success when the total is high. Some weaknesses prevent a use from being justified at all.

The level of detail should follow the stakes. An individual using a writing assistant for a birthday invitation does not need an institutional review. A public agency introducing a consequential service does. Between those cases lie many gradations. Proportionate governance should make ordinary beneficial use easier while requiring stronger evidence and control where the consequences justify them.

To see what this means, consider a fictional housing assistance agency serving a fictional town after a serious flood. It offers repair grants under rules already authorized by the town’s lawful public process. The agency has too few staff, confusing forms, and a growing queue. Some applicants have urgent safety needs. Others need ordinary repairs. Fraud is possible, but delay can also expose families to harm. Every person, figure, rule, and result in the following case is invented to examine choices. The example describes no actual government program.

Define the service before buying the system

The agency’s first proposal is attractive: buy AI to process repair grants faster. It is also too vague to guide a responsible purchase. Faster at which task? Faster for whom? Does processing end when an application leaves an employee’s desk, when a decision reaches the household, or when the household receives the money?

The team studies the existing journey. Applicants repeatedly provide the same information. Staff retype details from documents. Questions move between offices. Some cases wait because the eligibility rule itself is unclear. Payment occurs in a separate process. The agency discovers that no single technology can remove every delay.

It first simplifies the form and clarifies the rule through the proper authority. It then defines a narrower purpose for AI: help applicants understand the published requirements, extract proposed details from documents for confirmation, and prepare a source-linked summary for an authorized case officer. Grant decisions and urgent safety determinations remain outside the system’s authority. The service will be judged by whether it improves a household’s experience and the accuracy of the whole process.

This boundary is a design choice for the fictional pilot, not a universal claim that automated decisions can never be legitimate. A different use, evidence base, and legal setting might justify a different allocation. Here, the agency lacks enough knowledge about complex cases to delegate more. Narrowing the first use lets it obtain useful assistance while learning under controlled conditions.

It compares three approaches: improving the existing system without AI, adding document extraction and structured search, and purchasing a broader conversational service. The comparison includes continuing staff costs, review time, accessibility, support, security, and exit. A cheap subscription is not necessarily the least expensive service. A more sophisticated model is not necessarily the best fit.

Applicants and frontline staff then examine a prototype. One applicant explains that the tool’s apparently helpful question assumes everyone owns the home they occupy. Another points out that the request for a digital document is impossible for someone whose papers were destroyed. Staff explain how shared households and informal arrangements complicate the form. These contributions lead to a supported route for missing documents and a clear distinction between asking for evidence and declaring a person ineligible.

Participation does not authorize the agency to change the grant rules casually. Where a policy choice is needed, it goes to the body empowered to make it. Where the problem is the interface, the service team can correct it directly. Keeping these decisions separate prevents a software supplier from becoming an unnoticed legislator.

The agency also rejects a misleading consent screen. People seeking an essential public service cannot meaningfully approve every part of the institution’s data processing merely by clicking a box. The agency must establish the appropriate lawful basis and protections with competent legal review. For this pilot, it offers an equivalent staff-assisted route that does not require conversational AI. Choosing that route does not reduce a person’s eligibility or place in the ordinary queue.

Some applicants genuinely prefer automation. They can ask questions outside office hours and correct a draft without feeling rushed at a counter. Their preference matters too. Protecting choice should not mean obliging everyone to use the least convenient method. The task is to provide workable options and disclose the differences without turning the human route into a punishment.

The procurement names an accountable owner. In this fictional agency, Director Ana Vale accepts responsibility for the service as deployed. Her name, office, and contact route appear in the public record. She is supported by a casework lead, a technical lead, and an independent review function. Naming her does not erase the duties of the supplier, the governing body, or other officials. It establishes who must coordinate action rather than let responsibility disappear between them.

She receives authority to restrict or suspend the system, funds for evaluation and correction, and a reporting duty to the governing body. Without those powers, naming an owner would merely identify the person most likely to be blamed for decisions made elsewhere.

Make evidence stronger than the demonstration

The supplier’s demonstration is fluent. The agency asks for something less theatrical: evidence under its own conditions.

Before using personal files, the team tests with invented documents representing the range of situations it expects to encounter. It includes damaged scans, contradictory dates, incomplete forms, ambiguous wording, and requests outside the program. It checks whether the system invents eligibility rules, exposes information, follows instructions hidden in an uploaded document, or presents uncertainty as certainty. Security specialists conduct authorized testing within a defined scope.

These tests do not prove that real operation will be safe. They reveal obvious weaknesses before people bear their consequences. A system that fails here returns for correction. A system that performs well advances to a more informative stage.

The agency then evaluates a version of the service on appropriately governed historical cases, with privacy protections and access restricted to people who need it. Reviewers compare proposed summaries with the original records and the applicable rules. They look for missing facts, misplaced confidence, inconsistent treatment, and extra work created for staff. They examine the entire service, including how the officer sees and corrects the summary.

The interface matters. A polished summary displayed above a buried source document invites trust. The agency instead shows the extracted fact next to its source and makes uncertainty visible. Officers can reject a proposed field without writing an elaborate defense of their disagreement. The system cannot penalize them for spending longer on a complex case.

The contract supports this arrangement. It specifies permitted data uses, security responsibilities, retention, access, incident reporting, and rights to obtain necessary operational records. It limits reuse of applicants’ information for unrelated purposes. It establishes notice and review for significant changes, including a replacement of the underlying model. It provides usable data export, transition support, and a route for resolving disputes. The agency obtains legal and technical advice appropriate to its jurisdiction rather than copying these ideas as ready-made contract clauses.

The pilot has a fixed scope and a review date. It begins with staff support rather than autonomous decision making. The ordinary service remains available. The agency does not knowingly withhold a needed safeguard in order to produce a cleaner experiment. Its evaluation plan receives appropriate ethical, legal, and operational review before live use.

In the invented results, 200 applications proceed through the conventional route and 200 through the assisted route under the approved evaluation design. Median time from a complete application to a decision falls from eighteen days to twelve. That looks promising. But median time from the household’s first attempt to apply until payment changes only from twenty-six days to twenty-five. The payment bottleneck remains, and some applicants spend longer completing the new intake process.

Independent checking finds eight material record errors in the conventional group and six in the assisted group. Those small counts do not establish that the new service is reliably more accurate. Differences in the case mix and ordinary sampling uncertainty also matter. The evaluation must report what the design supports, with uncertainty, rather than turn a modest numerical difference into a headline.

A subgroup review raises another concern. The assisted route has particular difficulty with applications involving lost documents and nonstandard household arrangements. The number of such cases is small, so the agency cannot confidently estimate the rate of failure. It nevertheless has enough evidence to investigate a recurring mechanism: the intake assistant is steering some people toward a document requirement that has a valid alternative.

This is a useful result because it changes the next decision. The agency retains the summary tool for trained staff under review, corrects the intake process, and delays expansion of the conversational feature. It also directs effort toward the payment stage, where faster summaries alone cannot improve the household’s outcome.

The pilot has not failed merely because it did not justify the entire original sales pitch. It has identified a useful component, exposed a weakness, and shown where another kind of improvement is required. Evidence has earned a narrower permission. That is a successful function of governance.

Failure must have an address

During the next limited phase, an applicant named Rosa contacts the office. The system told her that she could not proceed without a particular document. She has since learned that the program permits an alternative. Her repair need has become more urgent during the delay.

The institution now faces a revealing choice. It can explain that the assistant offered information rather than a formal decision, direct her to the general website, and count the incident as resolved. Or it can recognize that official guidance changed her access to the service even though no grant was formally denied.

The agency’s proposed duty of remedy follows the effect on Rosa. A trained officer receives the complaint, explains the available route, and arranges urgent review under the program’s existing criteria. Rosa does not have to prove which component malfunctioned. She needs to describe what happened in a form the agency can understand.

The service has promised a rapid response for urgent concerns and a stated response period for ordinary complaints, supported by actual staffing. The exact periods were chosen for this fictional program’s needs and resources; a number written into a policy without capacity behind it would offer little protection. Unresolved complaints can reach a reviewer outside the original decision chain.

The technical team preserves relevant records under the established retention and privacy rules. It determines that an updated reference document was interpreted incorrectly. The failure is not a mysterious act of an independent intelligence. It arose in a chain involving content management, testing, interface design, and supervision. More than one actor may have contributed, but the director remains responsible for coordinating the response.

The agency temporarily removes the affected guidance while keeping other useful functions available. Staff receive a clear correction. The team searches for similarly affected cases using the minimum access necessary, contacts people where appropriate, and restores a fair opportunity to complete their applications. It does not wait for every household to discover the error independently.

Where the agency has authority to remedy a financial loss, it uses the applicable procedure. Where another body must decide, it helps the person reach that route. It does not promise unlimited compensation it cannot lawfully provide. It also does not treat the correction of software as complete redress for a person whose situation worsened during the delay.

The governing body receives an incident report distinguishing known facts from unresolved questions. The public receives an appropriate account of what failed, what functions were affected, what the agency did, and what remains under review. Private household details stay private. Legitimate security restrictions can limit technical disclosure, but they should not become a general excuse to conceal whether the service injured people.

Staff also need protection when they raise concerns. A case officer who notices that summaries omit an important fact should not have to choose between silence and insubordination. The agency establishes an internal route for reporting problems, permits urgent escalation, and checks whether performance targets discourage challenge. An instruction to exercise judgment means little when an employee is punished for taking the time judgment requires.

The supplier may dispute the agency’s diagnosis. That possibility is addressed through evidence and the contract, not by suspending support for affected people until commercial responsibility is settled. Citizens should not become involuntary parties to a technical dispute between institutions more powerful than they are.

Failure has an address when someone must receive the complaint, act within a known process, preserve the evidence, and explain the result. It does not require pretending that one individual caused everything. It requires preventing complexity from becoming immunity.

At the counter

At the counter
she unfolds the letter.
The clerk turns the screen
so both can see.
A blank line waits
for what the form left out.

Renew permission, preserve the way out

At the review date, the agency asks again whether the service should continue. Its early authorization was limited, and its continued use is a decision rather than a default.

The evidence now includes costs that the original proposal overlooked. Staff spend time checking summaries. The system saves time on some routine tasks. Complaints have revealed weaknesses in published guidance that predated AI. The payment process has improved through separate administrative work. Applicants differ in which route they prefer. The agency must judge this whole picture.

A useful evaluation separates several questions. Has the service improved the outcome it was bought to improve? Are the benefits reaching the people with the greatest difficulty using the old service? What serious errors remain? How often do officers override the system, and what do those overrides reveal? Is the supplier meeting its obligations? Can the agency afford to maintain the service without degrading other essential work?

No single measure can answer all of them. A high satisfaction rating may omit people who abandoned the service. A low complaint count may indicate a difficult complaint process. Faster case completion may reflect staff choosing easier files first. The agency therefore studies the path into the system, the experience inside it, and what happens after a case leaves the dashboard.

The director proposes continuing the staff summary function, with specific corrections, while keeping more consequential delegation outside the authorization. The governing body accepts part of that proposal and asks for further evidence on accessibility. It publishes its reasons. The decision neither declares the technology safe in all settings nor condemns the broader possibility of automation. It states what has been justified here.

Continued operation includes periodic checks and review after material changes. A new model, a new data source, a changed eligibility rule, or a proposed secondary use may alter the basis of permission. The agency should not quietly turn a repair-grant assistant into a fraud-profiling system because the supplier offers an additional feature. A new purpose requires its own justification.

The way out is tested before it is needed. Records are exported into a usable format. Staff rehearse operating the essential process without the AI component. The agency checks that important decisions remain intelligible from its own records. It knows which functions can stop temporarily and which require immediate fallback. It budgets for transition rather than assuming that the last day of a contract will also be the last day of responsibility.

Reversibility has limits. A disclosed private fact cannot simply be undisclosed. A missed opportunity may not be fully restored. Time lost during a crisis remains lost. That is why some boundaries must operate before harm rather than rely entirely on repair afterward. A workable exit reduces dependence; it does not make every experiment acceptable.

The same approach can support more ambitious uses as evidence improves. A later system might handle a carefully defined routine task with less direct review if law, performance, safeguards, and experience justify the change. The six questions do not demand that institutions freeze their first allocation of responsibility forever. They demand that expanded authority be earned and remain answerable.

This fictional procurement ends with neither a miracle nor a prohibition. The agency has gained a useful capability. It has also improved its forms, clarified its duties, learned from a failure, and retained the ability to change suppliers. Some of the most valuable work happened outside the model. That is exactly why responsibility belongs to the institution using it.

The next official who takes over can inspect a record of purpose, permission, evidence, limits, incidents, and remedies. The next applicant can find a route to a human answer. The next supplier can understand what it must prove. In those ordinary arrangements, a moral claim becomes a service that people can question and improve.

Sources and notes