Skip to reading
Forecaster
Complete research draft6 chapters published
Contents Chapter 4

The limits

What to approve and what to automate

Design checkpoints around evidence and consequences, then test whether a person can catch the relevant mistake.

By Kevin L. Michel6 minute read

A request for approval takes time and thought. It should help the person make a real decision.

Imagine the workshop agent asks, “May I proceed?” The person sees a green button but no room, date, price or action. Proceed with what? Search again? Send an inquiry? Pay a deposit? The button gives the appearance of control while withholding the information needed to use it.

A better request names the act and its consequences. It shows the recipient, the proposed message or change, the amount if money is involved, and the evidence behind the choice. It makes clear what will happen if the person agrees. If an important term changes afterward, the old approval no longer answers the new question.

That is a design recommendation. We should test it rather than assume that better-looking approval screens make work safe. The evidence on human oversight contains both reasons for hope and reasons to be careful.

Attention is part of the system

People have to divide their attention. Raja Parasuraman and Dietrich Manzey reviewed research on how they do so when they use automated tools. People may check a trusted tool less as other demands grow. Wrong advice can then lead to missed problems or wrong actions. Skill and warnings do not always prevent this. The authors also caution against labeling an error complacency without evidence about what the person actually monitored. 2010 review

That caution matters. A worker asked to review hundreds of agent actions while meeting the same old deadline has been given a second job. “A human was in the loop” says little about whether they had time to understand it. Repeated interruptions can also teach people that most alerts are routine. We should investigate the work conditions before deciding that the reviewer simply failed to care.

One tempting response is to add friction. Make the person wait. Make them answer first. Make them request the agent’s suggestion instead of seeing it at once.

Zana Buçinca, Maja Barbara Malaya and Krzysztof Gajos tested such approaches in a nutrition-choice task with a simulated AI aid. The results covered 199 people. The extra steps helped people respond when the aid was wrong. Some choices and people showed clearer gains. But on the full task, the study found no clear gain over the usual explanations. The task was brief and low stakes. It does not justify a universal delay before every agent action. 2021 experiment

An extra step should help the person make a judgment. Asking the organizer to check the event date against their calendar can reveal a date error. Making them wait thirty seconds before clicking the same unexplained button may reveal nothing.

Explanations can persuade or help you check

An agent can often give a fluent reason for what it proposes. That reason may make the proposal easier to accept. It does not follow that it makes an error easier to find.

Gagan Bansal and colleagues compared explanation designs in tasks involving sentiment judgments and law-school entrance-test questions. Human–AI teams could outperform the individual partners, but the tested explanations did not significantly improve that combined performance over showing how confident the AI was. The reasons given could lead people to accept wrong advice as well as right advice. These were short, chosen tasks. The finding does not cover every kind of explanation. 2021 study

There is important counterevidence. Helena Vasconcelos and colleagues studied people using simulated AI to solve mazes. People were less likely to follow wrong advice when an explanation made it easier to check the answer. A visible path can let a person spot a blocked route more cheaply than tracing the whole maze unaided. The setting differs from open-ended agent work, where a neat explanation may omit a crucial fact. Still, it identifies a useful design aim: lower the cost of checking. 2023 experiments

For the workshop, “This is the best venue because it balances your needs” is mostly a conclusion. A short comparison that links the access statement, shows the full price and flags an unconfirmed date gives the person more to work with. The evidence should make disagreement possible.

Avoid asking the same agent to be the sole witness to its own success. It can help gather evidence, summarize a change or point to a receipt. When possible, seek the record from the source that can confirm the result: the venue’s response, the calendar entry, the payment state, the actual file difference. Those records can also require interpretation. A separate source helps with one problem. It does not remove every doubt.

Match the checkpoint to the consequence

A reasonable review policy begins with the damage a mistake could cause, how soon it could be noticed and how hard it would be to repair. Those questions are more useful than a single rule to approve everything or automate everything.

A private list of venue candidates can usually be inspected after it is made. An inquiry sent to a known address may deserve a preview until the workflow has earned trust. A payment or a message containing private information needs stricter control because the consequences can leave the system at once.

This is not a universal ranking of tasks. A calendar change can be trivial or serious. Moving a private reminder differs from moving a medical appointment or a large meeting. The task’s context determines the consequence.

For work with small, easy-to-fix errors, try a small test first and check a sample later. For work with serious or hard-to-reverse effects, ask before the action. Novel situations, changes in scope and uncertain outcomes should return to a person even when familiar actions are usually automated.

These are proposed rules of thumb. The research reviewed here does not establish an ideal approval rate or a safe sample size for every agent. Sampling also misses rare events. If one missed case would be unacceptable, a policy based only on occasional review is a poor fit.

At a good checkpoint, the person can do more than say yes. They should be able to edit the proposed action, narrow the permission, ask for evidence, decline it or stop the task. If refusing means losing all completed work, the interface creates pressure to accept. Saving partial progress makes a careful no less costly.

Fewer clicks can mean different things

Anthropic’s February 2026 study of agent autonomy reports that more experienced Claude Code users more often enabled full auto-approval. They also interrupted the agent more often. These are records of product use from one company, mostly about coding. The researchers did not randomly assign people to be new or experienced users. The study does not show that either pattern made work safer. It suggests that supervision can move from individual approvals toward other forms of intervention. Provider study

Do not turn that observation into advice to switch off approvals. Long-time users may have chosen easier jobs. They may have learned warning signs or used checks the records did not capture. The result tells us that approval counts alone are an incomplete picture.

A rising approval rate could mean the agent is improving. It could mean the person is tired. It could mean the task has become routine, or that difficult cases are being excluded. To know which explanation fits, inspect outcomes and a sample of the decisions themselves.

One practical test is to ask what the reviewer could have detected. Did the approval show the complete price? Could the recipient be checked? Was the source accessible? Was there enough time? If the necessary fact was absent, blaming the person for missing it will not improve the next decision.

Another test is to inspect cases that were approved and later needed correction. Was the evidence wrong, the rule unclear, the interface misleading, or the check skipped? Different causes need different changes. More warnings are not a general repair for missing information.

The best checkpoint is sometimes an automatic constraint. A correctly built control can reject an amount above a fixed cap on the actions it covers. It need not ask a tired person to do the arithmetic again. The person should spend attention on matters that require judgment, such as whether the proposed purchase serves the event’s purpose. Software can handle a clear boundary; people can decide when a new boundary is justified.

A well-designed division of work gives each side a task it can actually perform. Approval then becomes a choice backed by evidence. The last person to click should not have to carry the blame for a system they could not control.