The Task Is Not the Whole Job

The report is finished. The homework is complete. The answer is correct.

Three signs of success. Three reasons to keep asking questions.

Was the report accurate enough to use? Can the student solve the next problem without assistance? Did the correct answer reach the person who needed it, or did it create another task for someone else?

A tool can improve the visible product while leaving the larger purpose untouched. It can also produce a modest change in the visible product that has a large effect on someone’s life. We need to know which outcome we are measuring before a percentage can tell us much.

The research on AI assistance contains enough evidence to resist easy dismissal. It also contains enough variation to resist a universal promise. The most useful findings do not tell us simply to believe or disbelieve in AI. They tell us where a particular form of help worked, how the comparison was made and what the study did not measure.

That is a better basis for action than a contest between demonstrations.

An hour with fewer unresolved problems

Erik Brynjolfsson, Danielle Li and Lindsey Raymond studied the introduction of a generative-AI assistant among 5,172 customer-support agents. Their final 2025 paper estimated an average increase of 15 percent in issues resolved per hour. Gains varied across workers. The study used a staggered rollout in a working organization, rather than a randomly assigned experiment across the economy.¹

The outcome is concrete. More customer problems were resolved in the same amount of agent time, on average, in this setting. That is a reason to take assistance seriously.

It is not evidence that every worker became 15 percent better at every task. It does not establish an equivalent rise in pay or a particular change in total employment. Those are different outcomes with additional causes.

The distinction matters most for the worker. If a tool helps a new employee handle a difficult question, the benefit may include less frustration and a better chance to become competent. The employer may also see a way to process more calls. What happens to staffing, pay and the pace of work depends on decisions outside the model.

A technology that makes skill easier to acquire could expand opportunity. The same technology could be used to make experienced workers easier to replace. Neither result follows from the productivity number alone.

We should not require a study to answer questions it was not designed to answer. We should require the public argument to stop where the study stops.

Here is a small workplace illustration. A team handles difficult customer requests. The new assistant helps workers find relevant information, so each request takes less time. If demand is waiting, the team can serve more people. If the number of requests is fixed, management has choices about how to use the time released. If quality improves, customers may have fewer reasons to call again. If workers rely on bad suggestions, some work may return later as complaints.

The gain has to be followed through the service. A stopwatch placed only beside the first reply can miss the second call.

This is why the best question for an organization is rarely just whether employees use AI. It is whether the process produces a better result for the people it serves, at an acceptable cost, with workers able to detect and correct failure. Tool adoption is an input. It is easy to count because a login leaves a record. The human result often takes more care to see.

A dashboard full of adoption figures can therefore be a strange kind of ignorance. It knows who opened the tool and may know very little about who was helped.

The percentage that changes when you name it

In a randomized online experiment reported in Science in 2023, Shakked Noy and Whitney Zhang studied 453 college-educated professionals doing short, occupation-related writing tasks. Access to ChatGPT reduced average completion time by 40 percent and raised assessed quality by 18 percent.²

That is strong evidence for a benefit on those tasks under the study conditions. The use of random assignment helps separate the effect of access from the possibility that people who chose AI were already different from those who did not.

The result does not measure an entire working week. It does not include every meeting, correction, negotiation or responsibility that surrounds writing in an actual job. Nor does it show what happens to skill or earnings over years.

A writing task is real work. It is just not all the work.

Even the arithmetic needs attention. A 40 percent cut in time does not mean a 40 percent rise in output per hour. Suppose, only to illustrate the calculation, that a task used to take ten minutes and now takes six. A person could complete ten such tasks in an hour instead of six, assuming tasks can follow one another without other demands. That would be about 67 percent more tasks per hour. Real jobs contain interruptions and other work, so neither number can be applied to the whole day without further evidence.

The time and quality percentages are also not quantities to add together. They describe different things. Adding them would produce a bigger number and a less useful account.

These errors are easy to make because a percentage feels complete. It seems to carry its own meaning. In fact it needs a noun, a baseline and a setting. Percent of what? Compared with whom? Under which conditions?

People should be able to ask those questions without being accused of refusing good news. An improvement is more persuasive when we can explain it properly.

A measured task gain also gives us a practical starting point. Instead of redesigning an entire organization around an assumption of universal competence, test the part for which the evidence is strongest. Look at the work that remains around it. If writing becomes easier, perhaps review becomes the limiting step. If more reports can be produced, perhaps fewer reports should be requested.

That last possibility deserves more attention than it receives. A tool that makes output cheap can encourage an organization to demand more output whether or not anyone needs it. Ten people can each produce a long memo, and nine of them must then find time to read the other nine.

The machine may have saved the writer an hour while creating a meeting for everyone else.

Progress sometimes consists of producing less material and making the necessary material clearer. A system should be judged against the purpose of the work, not the number of pages it can help fill.

The answer and the learner

Education makes the difference between a product and a purpose especially clear. A completed exercise can be part of learning. It can also conceal the absence of learning.

In a classroom-randomized study at a Turkish high school, Hamsa Bastani and colleagues examined AI assistance during mathematics practice. Nearly a thousand students took part. A basic GPT interface improved performance while assistance was available but was followed by poorer performance on unassisted exams. A teacher-informed tutoring version improved assisted practice and did not show a statistically detectable exam penalty relative to the control group. It did not establish an unassisted learning gain in that comparison.³

The finding changes the question we should ask of a homework tool. How good is the completed work? How much of the ability remains with the learner? Those questions can have different answers.

There is no contradiction in a student doing better with help and worse without it. The help may remove the very effort through which the skill is learned. A tool can make a difficult step unnecessary today while leaving the learner less prepared to perform it tomorrow.

That does not mean all difficulty is educational. Confusing instructions, inaccessible material and a lack of feedback can obstruct learning without building useful skill. Good teaching makes some things easier so that a student can do the work that matters. The design problem is deciding which burden to remove and which effort to support.

A tutor that asks a learner to explain a step is doing something different from a tool that supplies the finished answer. Both may use a language model. Their effect depends in part on what the surrounding system asks the learner to do.

A separate randomized study by Greg Kestin and colleagues used a carefully designed AI tutor for two undergraduate physics lessons at Harvard. The tutor condition produced higher immediate post-test results than the in-class active-learning comparison. Expert-prepared prompts, solutions and videos were part of the intervention.⁴

This is evidence that a well-designed tutoring system can help in a particular educational setting. It is not proof that teachers are generally unnecessary, or that short-term test gains establish long-term mastery.

The two studies should not be treated as a direct contest. They involved different students, subjects, systems and conditions. Together they show why the design of assistance and the choice of outcome deserve as much attention as the model’s name.

If our goal is learning, the evaluation must include learning. A system that keeps a student engaged may be valuable, but engagement alone cannot tell us whether the student understands. A pleasant exchange may encourage a learner to continue. It may also allow a misunderstanding to remain unchallenged.

We should ask what the learner can explain, remember and use after the assistance is removed or changed. We should ask who benefits, who struggles, and whether the tool works for students whose needs differ from those of the group first studied. We should also notice when a tool allows a person to take part who otherwise would have been excluded. Access and mastery are both important, and they require different evidence.

A school needs the freedom to experiment carefully with such tools. It also needs the capacity to refuse a bad purchase, inspect a claim and change course without losing the records on which teaching depends. The promise of a tutor for every learner becomes thin if the school cannot find out what the tutor teaches.

When the clock disagrees

People can be mistaken about how much time a tool saves them. That possibility applies to enthusiasts and critics alike.

A 2025 randomized study by the independent research organization METR involved 16 experienced developers working on 246 tasks in familiar, mature open-source projects. Tasks for which AI tools were allowed took 19 percent longer on average. The reported uncertainty interval ran from 2 to 39 percent longer.⁵

That result is a warning against replacing observation with the feeling of speed. It is not a timeless verdict on programming tools. In a February 2026 update, METR described later work suggesting improvement but said selection and measurement issues prevented a reliable estimate of the current effect.⁶

Both dates belong in an honest account. The earlier result cannot simply be waved away because the technology is changing. The later uncertainty cannot be turned into a precise reversal that the researchers themselves did not claim.

Why might a tool feel fast while the job takes longer? There are several possible mechanisms. A draft appears quickly, but checking it takes time. The user pursues a suggestion that seems promising and later abandons it. The system explains something at length when the worker already knows a shorter route. These possibilities must be tested in a given setting; they are not findings about every developer in the METR study.

The experience of constant output can also make idle moments feel productive. Something is always happening on the screen. Yet the relevant clock runs until the task is done well enough to use.

An employer who wants a truthful answer should make it safe for workers to report both gains and losses. If every team is expected to demonstrate the success of a new purchase, people have a reason to choose flattering measures. If using the tool becomes a sign of loyalty, its failures become harder to discuss. The same problem occurs when a workplace treats refusal as proof of professional virtue. Either culture turns evidence into a test of belonging.

A useful trial allows the result to be disappointing.

That requires a clear starting point. What counts as finished? How will quality be checked? Which tasks are included? How much training and support does the trial supply? Are the people who use the tool the same people whose time is counted? What happens when work comes back later?

The answers should be written before the organization sees the result. Otherwise there is always another outcome to emphasize, another unsuccessful task to exclude and another reason the promise should be judged by next year’s version.

Improvement is possible. Endless exemption from judgment is also possible. We should know which arrangement we have purchased.

A target can be wrong without looking foolish

Sometimes the system performs the chosen task well and still causes harm because the task was poorly chosen.

Ziad Obermeyer and colleagues studied a health-management algorithm and reported their findings in Science in 2019. At the same risk score, Black patients were sicker than White patients. The system used medical spending as a stand-in for health need. Because spending reflected unequal access as well as illness, that target helped reproduce an unequal allocation of support.⁷

This was a predictive risk-scoring system, not a generative chatbot. It belongs here because it exposes a problem that greater technical skill alone cannot solve: a model can become better at predicting the wrong thing for the purpose at hand.

The error is understandable enough to be dangerous. Spending is recorded. Need is harder to measure. A convenient number can become the official answer to a more difficult human question. Once it enters a system, the number may acquire the appearance of objectivity simply because it is applied consistently.

Consistency does not repair a poor substitute.

Imagine an office trying to identify residents who need help completing an application. It measures the number of times each person contacts the office. That may reveal some difficulties. It may miss people who have no credit on their phone, cannot take time away from work or have already given up. If the office allocates help only to the loudest record of demand, its data can make silence look like ease.

The lesson is broader than bias in one model. We must inspect the relationship between the thing measured and the thing we value. Sometimes they track each other well. Sometimes the link breaks precisely for the people who need attention most.

This gives affected people a practical role in evaluation. They may know where the record fails to describe the life. A worker can explain that a quick resolution code conceals a customer sent elsewhere. A student can show that a completed answer masks confusion. A resident can explain why a missing form reflects an inaccessible process rather than a lack of need.

These observations do not replace systematic research. They help researchers and decision-makers ask a better question.

That is also why a complaint should not automatically be treated as resistance to change. It may be information about a failure in the measure. An institution determined to learn will want to hear it early, before a tidy result is repeated at scale.

Across these examples, there is a practical case for acceleration. Some tools help people write, resolve problems, learn and communicate. We should expand the work of finding, testing and delivering those benefits instead of treating the whole field as guilty until the distant future has been settled. There is also real cause to insist on tests that follow the benefit beyond the first output. Neither side of that account is optional.

The task on the screen is only part of the work. The purpose is outside it, in the person who can now do something they need to do.

Sources and notes