# Four saved attempts: methods and results

Supporting analysis for Kevin L. Michel's *Your Agent Needs a Manager*. Completed and independently checked on 16 October 2026.

## What was done

This is a descriptive reanalysis of stored scores in one historical τ-bench archive. No agent, language model, benchmark harness or upstream executable was run. No policy-compliance labels were assigned to the conversations. These results are not current-model performance, real-world failure rates or evidence that a management intervention works.

The [official archive](https://github.com/sierra-research/tau-bench/blob/59a200c6d575d595120f1cb70fea53cef0632f6b/historical_trajectories/gpt-4o-airline.json) contains 50 simulated airline tasks with four saved attempts each. Its filename is `gpt-4o-airline.json`. It was added on 23 November 2024 at commit `7362a1708d720a4d697d042f21fd8637ea144c3b`; that is an upload date, not an established execution date. The exact model snapshot, run date, generator commit and full configuration are not recorded. The [pinned project README](https://github.com/sierra-research/tau-bench/blob/59a200c6d575d595120f1cb70fea53cef0632f6b/README.md) describes the original tasks as outdated.

Acquired bytes: 4,114,038. SHA-256: `e9e6c0297660c537f83d4fd9c476ce7a9a86ecd2784874b7bfc13be598e37bfa`. Git blob SHA-1: `a5f0d91569e740b949b37a181993b3f1822a86f4`.

The archive's root license is [MIT, copyright Sierra 2024](https://github.com/sierra-research/tau-bench/blob/59a200c6d575d595120f1cb70fea53cef0632f6b/LICENSE). A copy accompanies the derived data. No transcript text or realistic customer-like identifiers are reproduced here. A public repository's stated license is not proof of provenance for every upstream contribution.

## Counts and calculation

All 200 expected task/trial pairs are present exactly once: tasks 0–49 and trials 0–3. All rewards are present and binary. No record was excluded. All trajectories are nonempty and distinct; task definitions are stable across repeats. This does not establish independent sampling.

| Saved successes out of four | Tasks |
| --- | ---: |
| 0 | 14 |
| 1 | 12 |
| 2 | 10 |
| 3 | 4 |
| 4 | 10 |

There are 84 stored successes out of 200 attempts (42%). Ten of 50 tasks succeed on all four attempts (20%). Twenty-six tasks have mixed outcomes. Trial labels 0, 1, 2 and 3 have 21, 22, 20 and 21 successes respectively.

For each task with s stored successes, the all-success fraction for k attempts is C(s,k) / C(4,k). C means the number of unordered combinations; it is zero when s is smaller than k. Average these fractions equally across the 50 tasks. This follows the finite-sample metric in the [original benchmark paper](https://arxiv.org/html/2406.12045v1).

| k | All-success subsets | All subsets | Fraction |
| ---: | ---: | ---: | ---: |
| 1 | 84 | 200 | 42.00% |
| 2 | 82 | 300 | 27.33% |
| 3 | 44 | 200 | 22.00% |
| 4 | 10 | 50 | 20.00% |

Overlapping subsets are a way to calculate the metric, not new independent observations. These values reproduce the pinned README row when rounded to three decimals. They do not reproduce the original June 2024 paper's different experiment. Raising the pooled 42% to the fourth power gives 3.11%, which answers a different question under different assumptions. Four repeats support the displayed k=1..4 summaries, not observed pass^8. No population confidence interval is claimed.

## Separate structural sensitivity

Nineteen embedded tasks have an empty required-output list and reference action names drawn only from `get_user_details`, `get_reservation_details`, and `transfer_to_human_agents`; empty action lists also qualify. Static inspection of the upload-era tool and grader source establishes that these reference actions do not write the database. Transfer ends a dialogue and is skipped during reference-state replay. Every saved record in this group has the same target database hash as empty-reference tasks.

| Group | Tasks | Saved successes / attempts | Tasks with four successes |
| --- | ---: | ---: | ---: |
| All tasks: primary result | 50 | 84 / 200 | 10 / 50 |
| No-write reference and no required output | 19 | 55 / 76 | 9 / 19 |
| Remaining structure-selected tasks | 31 | 29 / 124 | 1 / 31 |

This grouping describes reference definitions, not what the agent actually did. It does not prove correct refusal, successful handoff or full compliance. The remaining group is not a corrected or unbiased capability estimate. No do-nothing agent was run in this analysis, and no saved trajectory was regraded.

The distinction has prior literature. The original τ-bench paper recognizes that a successful outcome score can miss a policy violation. [Zhu and colleagues' benchmark audit](https://arxiv.org/html/2507.02825v5) tested constructed baseline agents against weaknesses in task definitions. Their results are different evidence, not values to subtract from this archive. This analysis makes no claim to discover the general reward-versus-compliance problem.

## Missing details and version limits

Five records have null reward diagnostics but retain an explicit zero reward: task/trial pairs (2,1), (9,2), (9,3), (33,0), (46,3). They remain in the denominator with their recorded value. No failure cause is inferred from the missing diagnostic.

All 50 embedded task definitions match the upload-era task file, ignoring an annotator field absent from the archive. Thirty-seven task instructions differ from the later pinned repository snapshot. The upload-era grader checks specified answer text when present, otherwise database state; the later source checks state and any required text. Neither source was executed or established as the exact run-time generator. Stored scores remain unchanged.

The sample is one published archive, not a random sample of deployments. Its selection, simulated users and reward design limit what can be inferred. Consistent scores alone do not establish safe actions or sound human oversight.

## Reproduction and independent check

Download the pinned archive through its official source, preserve its bytes, and run the accompanying standard-library-only [reproduction script](reproduce.py) with the file path as its sole argument. The script verifies the SHA-256 before processing, checks all cells and recomputes the primary and structural results. It makes no network requests and never runs upstream code.

The separately executed full analysis also verified Git object hashes, historical source definitions, missingness, table/README agreement and chart values. An independent implementation checked 49 assertions, with no discrepancies. The public data are [all 200 derived cells](task_trial_cells.csv), [task-level counts](task_success_counts.csv), and a [histogram](successes_histogram.svg). Counts and arithmetic have been checked; the underlying conversations have not received an independent compliance audit.

This note, the figure, data and code are research support. Their words do not count toward the book's newly published manuscript total.