Skip to reading
Forecaster
Complete research draft6 chapters published
Contents Chapter 5

The review

Make mistakes visible and reversible

Use records, narrow changes and planned recovery to handle errors and uncertain outcomes without making them worse.

By Kevin L. Michel6 minute read

A task can fail after something has already happened.

Suppose the workshop agent is now allowed to send an approved inquiry. It submits the message, then loses the connection before receiving a clear reply. The person sees an error. Was the message sent?

The error alone cannot answer. It may describe a lost reply rather than a failed action. Sending again could create a duplicate. Reporting success could hide a failure. The first step is to check the message in the service’s own records.

This problem is older than language models. The HTTP standard defines some actions that can be repeated with the same intended effect. Other actions do not have that property. For other requests, it warns against an automatic retry unless there is reason to know that doing so is safe or that the first request was not applied. Idempotent is the technical word for the repeated-effect property. It does not mean harmless: deleting a record can be idempotent. HTTP Semantics, section 9.2

For the person managing an agent, the plain rule is to check an uncertain action before repeating it. Some services accept a unique key with a request. If the same request arrives again with that key, the service can recognize it as a repeat. The feature must exist and be used correctly. An instruction to “avoid duplicates” does not create it. A documented retry design

Keep evidence of the action

A useful record links the request, the permission, the action and the result. It should let you ask: what was meant to happen, what was allowed, what was tried, and what did the outside system confirm?

For the inquiry, that may mean the approved message, the intended address, the time, the service’s message identifier and its recorded status. The agent’s final summary can point to these records. It should not replace them.

Logs need care. Keeping everything can expose private facts and hide the details that matter. A payment record may need an amount and transaction reference, not full payment credentials. A message record may need a controlled link rather than a copy spread across several systems. Keep the evidence needed for review, limit who can read it, and set retention to match the purpose.

NIST’s control catalogue calls for audit records that identify events and outcomes and for protection of those records. This gives a useful foundation. It does not make any particular log complete or trustworthy. If a program can alter its own evidence without leaving a trace, the record may be weak precisely when it matters most. Audit controls in SP 800-53

The record should distinguish states rather than collapse them into a green success label. A request may be prepared, approved, submitted, confirmed or unresolved. A file may have been changed locally but not published. A payment may be authorized but not settled. These examples are not one universal service model; the actual states depend on the system. The point is to use the service’s meaning instead of guessing from the agent’s wording.

Keep partial progress when a task stops. If two venues have answered and one inquiry remains unresolved, preserve the two answers and the uncertain message reference. Restarting the whole task can repeat completed work and conceal the original failure.

Undo has several meanings

Changing a draft back to its previous version is one kind of recovery. Canceling a reservation is another. Asking someone to disregard a message is a third. They should not share one reassuring label.

An original file and a version history may allow a technical rollback. A cancellation may require a new request, incur a fee or arrive too late. A correction can change what recipients believe, but it cannot make them unsee information already disclosed.

Database researchers Hector García-Molina and Kenneth Salem described sagas in 1987: long activities broken into steps, with compensating actions for steps that need to be unwound. A compensating action can put the business back in an acceptable state. It need not restore the exact past. Other work may have happened in the meantime. Their paper also recognizes failures that require further recovery or manual repair. Sagas

This is a useful way to plan an agent’s job. Before a step occurs, ask what its repair would require. Can the file be restored? Can the order be canceled? Who must approve the cancellation? What happens if the repair also fails?

A repair step also needs permission. Permission to make one purchase does not automatically grant permission to buy a replacement, agree to a cancellation charge or send a public explanation. The repair can create a new commitment. It deserves the same clarity as the original action.

The limit is practical. Some mistakes cannot be repaired well after the fact. That strengthens the case for a narrower permission or a review before the action. Recovery planning helps identify where prevention must do more of the work.

Make a change small enough to inspect

A narrow change is easier to compare with the intended result. If the agent edits one section of a document, a reviewer can inspect that section and its immediate effects. If it rewrites the whole document, moves files and changes sharing settings at once, the review becomes harder.

There can be good reasons for a large change. Where you can, separate changes that serve different aims. Keep the original and show what changed. A preview should reflect the action that will actually occur. If the preview and execution use different targets, the preview offers little protection.

Research on how people use AI supports tools that let them correct errors, dismiss help and see the effects of their choices. Saleema Amershi and colleagues developed 18 guidelines and had 49 practitioners assess them across 20 products. This work supports the relevance of those design needs. It did not measure how much a rollback feature reduces agent incidents. Nor does a button labeled undo establish that all effects are reversible. Human–AI interaction guidelines

A useful rehearsal can be small. On a harmless test file, make a known change, restore it and check the result. For a recurring workflow, walk through an unresolved outcome before trusting an unattended run. Such checks do not certify the whole system. They can reveal that a promised recovery route is missing, inaccessible or poorly understood.

The person should also know how to stop further work. Stopping the agent may prevent new actions without canceling requests already sent. Revoking access may block future calls without retracting data already delivered. A clear stop control should state these limits so the person can investigate the remaining effects.

Repair the cause, not just the result

Once the immediate problem is contained, examine why it occurred. Was the task ambiguous? Was the permission too broad? Did the agent use the wrong source? Did the service return a misleading status? Did the reviewer lack the fact needed to spot the error?

Keep these possibilities separate. The model can make a mistake while an access rule fails too. Fixing the text of the prompt may reduce one without closing the other. An error that escaped review may call for better evidence rather than more blame.

Madeleine Clare Elish uses the term “moral crumple zone” for a pattern in which responsibility falls on a nearby human who had limited control over an automated system. Her work is a qualitative argument, not an estimate of how often agent failures follow that pattern. It warns against treating a human name on the workflow as the end of the inquiry. 2019 article

The warning applies directly to the word manager. If a person cannot inspect the relevant actions, restrict the tools or obtain help after a failure, we should ask what responsibility the provider and organization have retained. Management requires power, information and time. It cannot be created by a disclaimer.

Keep an incident record short enough to use: the intended task, the observed result, the unresolved facts, the recovery and the change that should prevent a repeat. Avoid turning a near miss into a success story merely because someone caught it. The catch may be good evidence that a control worked, while the initiating mistake still needs attention.

Then test the changed process on the relevant case. A repair that has not been checked is another claim awaiting evidence.

Visible mistakes are uncomfortable. Hidden mistakes are harder to learn from and often more costly to repair. A good system gives people an early, accurate account of trouble, preserves what is known and makes the next safe step easier to choose.