At the review meeting, the new system appears to be working. Most applications move quickly. The charts are tidy. Then a staff member opens a small group of cases that have gone nowhere.
One person has a name the form will not accept. Another has moved but cannot update the record without information sent to the old address. A third has supplied the requested document, which the system has filed under the wrong case.
These people are present in the database. They are absent from the picture of success.
The team can describe them as unusual cases and move on. Or it can ask whether a system meant to serve people has been designed to recognize enough of the ways people actually live.
There will always be unusual cases. That is a reason to design a route for them.
A category does a job
Institutions need categories. A library has to distinguish a borrowed book from a returned one. A school needs to know which classes a student is taking. An agency needs a consistent way to apply eligibility conditions. Removing categories would not produce fairness; it could make decisions more arbitrary and harder to explain.
The difficulty is that a category is a tool for a purpose. It is never the whole person.
Suppose a form asks whether someone is employed. For one task, a yes-or-no answer may be enough. For another, the institution may need to understand variable hours, several small jobs or a recent interruption. The useful categories depend on what the decision requires.
A designer who begins with the available database fields can reverse that order. Instead of asking what the institution needs to know, they ask what the system is already able to store. People then have to fit the old structure, even when it does not match the new decision.
This can happen without anyone being indifferent. Teams inherit systems. Budgets are limited. A change to one field can affect several processes. Those are reasons to plan carefully. They are not reasons to pretend the mismatch does not exist.
The first question should be practical: what decision is this category meant to support, and what important distinctions does it miss?
An answer can be accurate and still fail the form
Imagine a resident entering an address for a home reached by a lane that is missing from the available list. The system requires a selection from that list. The person cannot submit the correct address because correctness has been defined as agreement with the menu.
A quick workaround might be to select a nearby address and explain later. That creates another problem. The record now contains something the person knows is wrong. A later check may treat the inconsistency as evidence against them, even though the interface produced it.
A better system would offer a safe way to say that the list is incomplete, collect the necessary detail and send the case for review. It would preserve the distinction between an unverified address and a false one.
The labels matter because they guide later action. Unknown, incomplete, disputed and incorrect are different states. If they all become invalid, the institution loses information about what needs to happen next.
A person should not have to choose between abandoning the task and making a false statement to pass a required field.
That is a design principle I would defend strongly. It does not require the institution to accept every claim. It requires a route for receiving a claim accurately before deciding whether it is supported.
The same principle applies to names, dates and relationships that do not fit a narrow format. A field can enforce a useful syntax, but the institution should know whether that syntax excludes valid real-world answers. The burden of finding the mismatch should not remain entirely with the person rejected by it.
Faster processing can make an error travel faster
Automation can reduce repetitive work and improve consistency. It can also spread a mistake if later systems treat an earlier output as settled fact.
Suppose an address is incorrectly marked as unverified. A second system uses that status to pause a service. A third sends a notice explaining the pause, but sends it to the address that has not been updated. Each step follows its local rule. The whole process becomes difficult for the person to escape.
The speed of the sequence does not tell us whether it is good. We have to examine what information moves, what uncertainty is retained and where a correction can enter.
Wu and Meyer studied Indiana’s 2007–09 welfare-processing overhaul using its staged county rollout and administrative records. Their nonrandomized analysis found enrollment losses. People deterred at entry were relatively less needy on measured indicators, while those losing benefits at renewal were needier. This was one bundled, flawed implementation; it does not establish that automation generally fails. It shows why entry and renewal, and the people affected by each, need separate scrutiny.1
The broader design lesson is to inspect the point at which a person can correct a record. A process may work well for a new application and badly for someone whose circumstances change. It may handle standard renewals quickly while treating a missing response as if all relevant facts had changed.
A reliable institution should be able to explain how a record became decisive. If a service is paused because of a status in another system, the person needs a way to learn what that status means and where it can be corrected. Being sent back and forth between organizations is a sign that responsibility has not followed the data.
Human review needs authority
The phrase human in the loop can sound reassuring. It leaves important questions unanswered.
Which human? With how much time? Can they see the evidence? Are they allowed to change the result? What happens if they disagree with the system?
A worker who can only read a message aloud is providing contact, which may be useful, but not necessarily review. A worker who can override any rule without explanation creates a different risk: similar cases may receive different treatment, and the result may depend on persuasion or personal bias.
The task is to give a reviewer bounded authority and a duty to give reasons.
For the missing address, the reviewer might check specified evidence, update the record and document why it was accepted. If the evidence remains insufficient, they could explain what is needed and identify a further review route. The aim is a traceable decision, rather than a favor granted by someone unusually sympathetic.
Staff also need a way to report that the rule itself is producing repeated failures. Otherwise they become permanent repair workers for a design no one changes. Their quiet effort can make a broken system appear more successful than it is.
A good exception route should produce information for the people who maintain the main route. If many cases need the same exception, perhaps the exception is pointing to a missing category or an outdated requirement.
This does not mean that every exception should become automatic. Some situations need careful individual judgment. The institution should be able to explain why and fund the time that judgment takes.
Accessibility belongs in the task
A person may understand the questions and have the right information while being unable to operate the process.
They may need a larger display, a keyboard route, more time or a different way to receive instructions. They may be using a shared device in a setting where private information is difficult to enter safely. A design that assumes one kind of body, device or environment can create obstacles unrelated to the decision itself.
WCAG 2.2 includes criteria concerning repeated information and accessible authentication, with defined conditions and exceptions. These offer concrete places to inspect a process rather than treating accessibility as a general promise.2
The institution still has to test the complete task. Can a person begin, understand a request, correct an error and receive confirmation? Does an alternative route preserve access to the same service, or does it become a slower and less reliable second-class path?
A phone alternative can be valuable. So can in-person assistance. But calling something an alternative does not make it usable. Its hours, waiting time, communication options and staff knowledge matter.
There is a temptation to solve every gap by asking the person to bring someone to help. Trusted assistance can be welcome when the person chooses it. It should not become a routine condition that forces them to reveal private information or surrender control of the task.
The goal is to increase the person’s ability to act, including their ability to choose help.
More information is not always the answer
When a form fails to understand people, the obvious remedy may seem to be more fields.
Sometimes an additional field is necessary. Sometimes the better remedy is to ask less, allow a wider range of valid answers or provide a review route. Collecting a detailed personal history can create risks without improving the decision.
A useful test is whether the institution can explain how a piece of information changes what it must do. If the answer is that it might be interesting later, that is a weak basis for requiring it now.
Consider the address problem. The institution may need enough information to verify the location and deliver a notice. It does not automatically need an account of why the person moved, who they live with or what happened in their previous home. Those details may be sensitive and irrelevant.
People should be able to correct a narrow error without turning the encounter into a broad disclosure of their lives.
This is especially important when a public complaint becomes the easiest route to attention. A person may feel pressure to tell a painful story in order to persuade an organization to fix an ordinary administrative mistake. The organization should offer a private, evidence-based path that does not require a public performance of hardship.
There may be cases in which circumstances genuinely affect the decision. Then the institution should explain why it needs the information, limit who can access it and handle it through appropriate secure channels. A desire to be helpful does not remove the duty to protect privacy.
The denominator should include people who could not finish
A service can report excellent completion times among completed cases and still leave a serious problem unmeasured.
Think about the person whose name is rejected before an application is created. If the system counts only submitted applications, that failure may never appear in its performance report. The organization sees a quick process because it has excluded some of the people for whom the process did not work.
This is a measurement problem as well as a design problem.
A more honest review asks who tried to begin, where attempts stopped and what can be learned without collecting unnecessary personal detail. It distinguishes a person who chose to leave from a person who could not continue, while admitting where the available evidence cannot tell them apart.
Small numbers can matter. An uncommon failure may have severe consequences for the people affected. A team should not need a large share of users to be harmed before fixing a clear defect with a reasonable remedy.
At the same time, resources are limited. Institutions have to prioritize. That choice should consider the seriousness and duration of harm, the number affected and the feasibility of repair. It should not rely on frequency alone, or on which group is best able to attract attention.
A report that includes unresolved cases can be less flattering and more useful. It gives leaders a chance to act before the person outside the report has to become a public controversy.
An exception is evidence
Back at the review meeting, the team adds a second page to its report. It lists cases that the ordinary route could not handle and the reasons they stopped. No private stories appear. The information is enough to show where the system needs work.
The name field can be corrected quickly. The address problem requires a change to a shared record. The misplaced document reveals a handoff that needs a clearer owner. None of these improvements requires the team to abandon consistency or accept unsupported claims.
It requires them to take the failed cases seriously as information about the service.
This is a more hopeful view of exceptions than treating them as annoying people at the edge of a finished design. They can teach an institution where its picture of the world is incomplete. Some will remain rare and need careful review. Others will reveal a repair that benefits many people who never complained.
A form cannot contain a whole life. It does not need to. It needs to ask what matters for its purpose, preserve uncertainty honestly and leave a workable route when the answer does not fit.
The person is already there. The institution’s task is to become able to see them.
Sources and limits
Footnotes
-
Derek Wu and Bruce Meyer, “Certification and Recertification in Welfare Programs: What Happens When Automation Goes Wrong?”, AEJ: Economic Policy, forthcoming; author manuscript, 28 November 2025. Staged rollout comparison, not random assignment; need measured through available administrative indicators. ↩