From answers to actions

A program that drafts a message gives you something to judge. A program that sends the message has changed another person’s day. The words may be identical. The decision is not.

This is the practical meaning of an AI agent in this book: software that can choose steps and use tools to pursue a task. Those tools may search files, change a calendar, run code or place an order. Systems differ in how much they can do without returning to a person. Calling all of them agents does not tell us which actions they can take, whose authority they use, or how well they work.

The interesting promise is time. You could hand over a job and return to something you value. Yet a fast draft can leave slow work behind it: checking claims, finding a missing change, fixing an error, explaining what happened. If the person spends the saved hour inspecting the machine’s hour, the promised freedom has not arrived.

My argument is that useful delegation needs a manageable job around it. Someone must define a good result, limit the means, decide what needs approval and inspect enough evidence to know what happened. The software provider must supply the controls and records that make those duties possible. An employer must allow time for them. A person cannot sensibly manage work they are unable to see or stop.

The word manager describes these decisions. It does not mean a program has human needs, judgment or responsibility. It also does not mean every person should become a full-time supervisor of software. The point is to reduce the work of getting a reliable result, including the work of supervision.

A result has several parts

Imagine an agent asked to find three venues for a community workshop. These examples are hypothetical, not reports of people observed for this book. The first version of the job ends with a comparison. The next version allows the agent to email venues. A third permits a deposit.

Each change adds a question. Are the facts right? Was the recipient allowed? Was the message accurate? Did anyone make a commitment? Was the total price within the limit? A polished comparison answers none of those questions about the later actions.

It helps to separate four things. Quality concerns whether the work meets the need. Authority concerns whether the agent may take the action. Execution concerns what the outside system actually did. Recovery concerns what can happen if the result is wrong or incomplete.

An agent can get one right and another wrong. It could find a suitable room but pay without permission. It could stay within permission but choose the wrong date. It could make the right request and receive no clear reply. It could correctly report that a deposit was paid while offering no way to cancel it.

These are different failures. Giving the model a longer instruction may help with some. It cannot, by itself, provide a bank’s refund process or force a calendar service to keep a change history. The surrounding tools belong in the assessment.

Researchers have made a related point about evaluation. Sayash Kapoor and colleagues, in AI Agents That Matter, argue for comparing agents on cost as well as accuracy, improving test design and accounting for human involvement. A system tested alone can miss the value of help from a person. A generous test can also make it look more capable than it is. The appropriate comparison depends on the work the system is meant to do. Accessible 2024 paper

For the workshop, the comparison might be an ordinary search and a short phone call. The agent does not earn its place by producing more material. It earns its place if the whole process gives you a suitable, checked choice with acceptable effort and risk.

What four attempts can reveal

Repeated work poses a further question. Can the system do the same kind of job dependably, or did we happen to see a good attempt?

To make this question concrete, a public archive from the original τ-bench project was reanalyzed for this book. Its name is pronounced “tau-bench.” It contains simulated airline customer-service tasks. An agent talks to a simulated customer and uses tools in a test environment. The saved scores reflect checks of expected database states or required answer text. They do not establish that every task rule was followed. This is research software, not a record of actual airline customers. Original benchmark paper

The archive used here contains 50 tasks with four stored attempts each: 200 records. It was added to the repository in November 2024 and carries a GPT-4o label. The file does not establish the exact model snapshot or when each run occurred. Its authors now describe the original tasks as outdated. No model or fresh benchmark was run. The calculation uses the success or failure value already saved in each record. Pinned archive and project documentation

Across the 200 attempts, 84 have a stored success value. That is 42%. But only 10 of the 50 tasks succeeded on all four saved attempts. That is 20% of tasks. The full pattern is more helpful than either number alone:

The table answers a narrow historical question. It shows uneven results in these saved records. Twenty-six tasks had a mix of success and failure. It does not tell us the chance that a current product will mishandle a real booking. Nor does it show that a human approval step would have prevented any particular failure.

There is an important arithmetic trap here. Multiplying 42% by itself four times gives about 3.1%. That is not the observed 20%. The multiplication treats every attempt as if it had the same underlying chance, independent of the others. The archive contains different tasks. Some are easier under its scoring rules than others. We counted the actual four-attempt outcomes for each task instead of treating all attempts as interchangeable.

The score itself also deserves inspection. Earlier work by Zhu and colleagues showed how weak task targets could let a constructed do-nothing agent pass some airline tasks. That published audit is separate from the recount here; it is not a value to subtract from this archive’s score. Benchmark audit

In the archive examined for this book, nineteen tasks have no required answer text and reference actions limited to reading records, handing off, or doing nothing. Nine of the ten tasks with four stored successes fall in this group. That does not establish that the agent behaved safely. It shows why we should ask what a successful score required before praising a high score.

The analysis kept all 50 tasks. It did not remove this group and rename the remainder the “true” result. Five failed records lack a detailed reward explanation; they remain failures in the main count because that is how the archive records them. The public method note gives the inputs, calculations, checks and limitations. The result is a checked reanalysis of stored scores, not an independent review of every conversation. Analysis and reproducibility note

For a manager, the lesson is a question rather than a forecast: what evidence shows that this job works repeatedly, under the conditions in which I will use it? A demonstration can establish that something is possible. It cannot, alone, establish that it is dependable.

A person in the process is not a guarantee

Adding a person can help. It can also create another place for a mistake.

Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone reviewed 106 experiments that compared people, AI systems and combinations of the two. On average, combined systems did better than people alone but worse than whichever standalone partner performed better. Results varied greatly by task. The studies largely predate current agents, and their performance measures do not settle questions of permission or responsibility. They do establish a reason to test a division of work rather than assume that mixing human and machine must improve it. Meta-analysis

Suppose the agent is good at finding room capacities and the person is good at judging whether a venue will suit the group. That could be a productive split. If the person must recheck every search because the sources are missing, the split is less promising. If the system requests approval with no visible date, price or recipient, human involvement may amount to guessing.

I want software that gives people more room to live. That requires a modest kind of ambition: make the job genuinely easier, then expand it when evidence supports doing so. We should not confuse the number of actions a system can take with the amount of work it can responsibly remove.

The first management decision comes before any action. Decide what you are handing over, and what would let you recognize a job done well.