Skip to reading
Forecaster
Complete research draft10 chapters published
Contents Chapter 4

Part II: Making a claim

The Work Behind the Output

A fair account of AI gains follows completed work through checking, handoffs and repair. It recognizes shared contributions without turning workers into permanent subjects of surveillance or making ownership depend on a perfect record of individual effort.

By Kevin L. Michel12 minute read

By Wednesday, the training company has twenty new course packs.

They look excellent. Each has an introduction, exercises, a summary, and a short quiz. The manager opens three at random and likes what she sees. Last month, producing twenty would have taken much longer. The new tool appears to have changed the business in a week.

Then the tutors open the files.

One course refers to a feature the customer’s software does not have. Another gives the right procedure in the wrong order. Several exercises require material that learners will not receive until the following session. The language is clear. The lessons cannot yet be taught.

The tutors correct them. A coordinator checks that the revised files carry the right dates. Someone rebuilds the answer sheets. On Friday, the company delivers eight packs and schedules the rest for another review.

Perhaps this is still a substantial gain. Perhaps the old process would have delivered only four. We do not know yet, because the first report counted files produced and the customer needed lessons ready to use.

Before we divide a gain, we need an honest account of what happened. Otherwise, the most visible output will collect the credit while the work that makes it dependable disappears into the background.

Decide what finished means

A finished task should leave the next person able to do something.

For the training company, a usable pack must fit the agreed course, work with the customer’s software, and include exercises the tutor can teach. A draft can be valuable on the way to that result. It is still a draft.

This sounds obvious until a dashboard begins counting whatever the new system can produce most easily. If the tool generates documents, document counts rise. If it generates messages, message counts rise. The numbers look busy even when the person receiving the work has acquired a larger reading burden.

Choose the unit of output before comparing methods. In this example, the unit might be a course pack accepted for teaching under a short set of agreed standards. The standards should be demanding enough to protect the customer and plain enough for the team to apply. They should also apply to the old method. A comparison that inspects AI work closely while accepting human work without review will answer a different question.

Now choose the boundary around the work. Begin when the company has enough information to start. End when the pack is ready for its intended use. Include the people who prepare the brief, draft the materials, check the content, correct errors, and arrange delivery.

The boundary is a choice, and we should explain it. If the customer must spend time repairing the pack after delivery, a company-only measure may count a saving that has merely moved elsewhere. If a colleague answers questions in the evening, paid-time records may miss part of the cost.

We do not need perfect information to begin. We need a boundary that makes sense and an honest note about what remains outside it. Then we can improve the measure as we learn.

A particularly useful question is: whose work becomes visible only when it is not done? In the training company, the tutor’s knowledge may appear as silence. The class proceeds. No one complains. The fact that a failure did not reach the customer can conceal the effort that prevented it.

Keep two clocks

A customer cares how long it takes to receive the course. The firm also cares how many staff hours it takes to prepare it. These are different measures.

A pack can spend two days waiting for a short review. Reducing that wait improves delivery without necessarily reducing the hours spent working. Equally, a process can require fewer staff hours and still take longer to reach the customer because work is held for a weekly approval meeting.

Consider a hypothetical comparison across two periods. In the first, a team completes forty accepted requests using two hundred staff hours. In the second, it completes fifty using two hundred and fifty staff hours. Both periods average one accepted request per five staff hours. Completed output has risen by a quarter. Measured output per staff hour has not changed.

Suppose the typical customer wait also falls from three days to one. That is a real service improvement. Calling it a labor-productivity gain would obscure what improved. Calling the whole effort a failure because hours per request stayed level would obscure it too.

The right account can hold both facts. The company served more people and served them sooner; it used proportionately more staff time. We can then ask whether the shorter wait improves retention, whether extra staffing is sustainable, and whether a further process change could reduce effort.

Parallel AI systems make these distinctions more important. A person may start several runs and work on something else while they continue. Adding every machine minute to human work time would exaggerate the person’s effort. Ignoring the later time spent comparing, checking, and joining the results would understate it. Elapsed time, human work time, and machine use each answer a different question.

Keep the units separate. You can combine their financial costs later, using stated assumptions. You cannot sensibly add an hour of unattended computation to an hour of concentrated human review and call the sum “two hours saved.”

Quality needs its own place beside the clocks. A quicker first delivery that creates repeated corrections may be worse than a slower clean delivery. Track the serious returns or failures that matter to the work, and allow enough time for them to appear. A process declared successful at the instant of handoff has not yet met its next user.

Our impressions need a comparison

A tool can feel fast because its activity is visible. Text arrives. A plan takes shape. Something that would have started with a blank page now starts with a full one. That experience can be pleasant even if the full page takes longer to repair than expected.

In a 2025 randomized study, METR researchers followed sixteen experienced open-source developers doing 246 tasks in familiar projects. With access to the tested early-2025 AI tools, tasks took about 19 percent longer on average. The developers nevertheless estimated afterward that AI had made them faster. The finding concerns those developers, tasks, and tools; it does not establish that AI generally slows software work.1

The later story is instructive too. In February 2026, METR reported that a follow-up experiment had serious selection and measurement problems. Some developers did not want to work without AI, some tasks were withheld, and concurrent agent use complicated time reports. The researchers thought gains had probably improved, but judged their new data unreliable for estimating their size.2

That is what evidence sometimes looks like while the practice itself is changing. A careful answer may become more conditional rather than more dramatic.

For a workplace, the lesson is to compare actual completed work instead of relying only on enthusiasm or frustration. Staff impressions still matter. They can reveal fatigue, awkward steps, and benefits a time measure misses. They are one kind of evidence, not a substitute for every other kind.

A modest trial can help. Use comparable cases, keep the acceptance standard steady, and record important differences. If experienced staff get the new tool while beginners use the old process, the result mixes experience with tool access. If the new method handles only straightforward requests, do not compare it with an old queue full of exceptions.

Where feasible, assigning similar eligible work to different methods can make the comparison more informative. In some settings, a formal trial will require approval or careful design. No one should withhold a necessary safeguard to create a cleaner experiment. Ordinary business learning can begin with a transparent comparison and appropriately modest conclusions.

The question is not whether your office can publish a research paper. It is whether the decision rests on something better than two memorable successes and an uncounted pile of corrections.

The work that makes the tool work

A new system often arrives with preparation work attached.

Someone must identify the current instructions and remove obsolete versions. Someone must decide which examples are good enough to reuse. Someone must test whether the tool handles a customer’s usual terms. People need time to learn the process and a place to report failures without being treated as obstacles to progress.

Some of this work creates an asset for the business: cleaner records, a better process, a reusable set of checks. It may be easy to overlook because it does not resemble a machine delivered in a box.

Erik Brynjolfsson, Daniel Rock, and Chad Syverson’s work on the productivity J-curve explains how new general-purpose technologies can require complementary investments, including changes in organization and skills. These investments can be poorly captured in conventional accounts, affecting the pattern of measured productivity over time.3

The practical implication is to count transition work without assuming it will continue forever. If a tutor spends a day building a reusable review guide, that day belongs in the cost of adoption. It would also be misleading to charge the full day against every future pack. Record the initial investment separately from recurring work.

The reverse mistake is more convenient for a hurried manager: treat all checking as temporary, then stop counting it while staff continue doing it. A promise that the system will improve does not complete today’s review. Until the need actually changes, the work is still there.

The team can distinguish what is being built from what is being repeated. It can revisit that distinction after enough use. Perhaps a shared checklist removes a recurring problem. Perhaps a new version of the model creates a different one. Measurement should help people find these changes, not defend the original purchase decision indefinitely.

This is a pro-progress discipline. It directs effort toward the parts that improve the result. A company that calls every concern resistance may conceal the very information needed to make its investment pay.

Give the team its proper name

The person who enters the prompt is easy to identify. The people who make the output useful may be harder to see.

In the training company, one worker prepares the first draft. A tutor knows the customer’s equipment. A coordinator has kept the course records consistent. Another colleague developed examples that the team now uses repeatedly. The final pack depends on all of them.

A fair contribution record should make those dependencies visible. Otherwise, AI can intensify an old workplace habit: credit follows the person presenting the finished object, while support work is treated as part of the furniture.

This does not require dividing every result into exact personal percentages. Often there is no honest way to do so. If one person supplies a necessary correction and another creates a necessary structure, both contributions can be indispensable. Assigning a tidy fraction to each may add confidence without adding knowledge.

Name the contributions in ordinary language. Who designed the new process? Who tested it? Who trained others? Who caught a recurring failure and changed the instructions so it stopped recurring? Include the people whose work made the improvement repeatable, not only the person associated with the best-looking demonstration.

Keep the tool’s contribution visible as well. A worker should not have to pretend to have written every sentence unaided to receive credit for sound judgment, execution, and responsibility. We should recognize the work actually done. False claims help neither a fair pay discussion nor a reliable account of the process.

At the same time, using a tool does not erase the human contribution. A tutor who detects a dangerous misunderstanding in a lesson has done something valuable even if the file was generated in seconds. The speed of the draft says nothing by itself about the value of the correction.

This is where an ownership discussion gains substance. Workers may help create a reusable capacity that produces value after the original effort is over. A wage can compensate the hours spent building it. A continuing ownership claim can give them a share in what it goes on to produce. The form of that claim needs explicit terms; the contribution helps explain why the discussion belongs on the table.

A record small enough to use

A contribution record can be brief. For one improvement, keep the problem, the change, the evidence, and the people involved together. Link to an existing approved work item where possible. Avoid copying confidential customer material into a personal file.

The training team might record that course packs had repeatedly required late corrections because draft instructions assumed the wrong software version. Two tutors developed a version check; a coordinator added it to the intake process; the drafter applied it. Across the next agreed sample, the team recorded whether that particular correction returned. Other quality questions remained open.

That account is more useful than “I used AI on thirty documents.” It identifies a problem someone needed solved, explains the contribution, and gives a reviewer a way to check the claim. If the sample is small, the record should say so. If several changes happened together, it should not attribute the entire improvement to one of them.

Use this record for a decision: recognition, resourcing, training, a change in responsibility, or a discussion about sharing gains. Agree who can see it and when it will be reviewed. A record without a purpose has a way of becoming another obligation that no one is paid to maintain.

We should be particularly wary of turning this advice into permanent personal surveillance. Keystrokes, screenshots, or minutes marked “active” cannot by themselves tell us whether a tutor noticed a faulty assumption. Constant monitoring may generate a huge record while missing the contribution that matters most.

Workers should have a voice in the measures used to judge them. They should be able to explain unusual cases and correct errors. A team handling harder work should not be punished because its simple completion count falls. A person who asks for help early may be protecting quality rather than demonstrating weakness.

The aim is enough evidence to make a fairer decision. More data is not automatically more understanding.

The claim beyond the record

These records can strengthen a worker’s position. They can also reveal how much production depends on cooperation. That is a reason to build fairer claims, not a reason to demand a complete ledger of human worth.

Universal capital ownership must not become a prize for people who can document the most impressive contribution. The unpaid carer, the person between jobs, and the person unable to work do not lose their place in society because they lack a workplace dashboard. Their claim is part of the political design we should choose for shared prosperity.

Within a firm, careful records can help workers negotiate the transition and receive credit for what they build. Across society, ownership must reach further. We need both levels of action, and we should not confuse their tests of eligibility.

Back at the training company, the manager has a better report now. It counts accepted packs, shows the time spent preparing and checking them, and names the people who improved the process. It also shows what the company still has to learn.

The team can stop arguing over whether twenty generated files were a miracle or a disappointment. They can discuss the capacity they actually created and the terms on which it will be used. That is a much more promising meeting to enter.

Sources and notes

Footnotes

  1. Joel Becker, Nate Rush, Beth Barnes, and David Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” July 2025. Randomized task assignment among sixteen developers and 246 tasks; the observed time increase was 19 percent, while developers perceived a speedup. This is a historical result for a specific sample and tool period, not a current benchmark for all coding. Research paper. ↩

  2. Joel Becker, Nate Rush, Tom Cunningham, David Rein, and Khalid Mahamud, “We are Changing our Developer Productivity Experiment Design,” February 24, 2026. The authors explain participant and task selection, concurrent work, and other obstacles to interpreting the follow-up estimate. This chapter does not treat that estimate as a clean reversal of the earlier result. METR research update. ↩

  3. Erik Brynjolfsson, Daniel Rock, and Chad Syverson, “The Productivity J-Curve: How Intangibles Complement General Purpose Technologies,” American Economic Journal: Macroeconomics 13(1), 2021, 333–372, DOI 10.1257/mac.20180386. A model and empirical analysis of intangible investments and productivity measurement, not a guarantee that any particular AI purchase will eventually succeed. The chapter’s training-company examples and record format are the author’s proposals. Published paper. ↩