Do the eleven sources in "Learning While the Tools Change" support the readings it gives them?
For version 1 of the essay, as filed at b4610b0: yes, on every claim that can be checked against a number or an attribution: all seven of the
essay's quantitative statements are exact, and no source is made to say something it does not
say. The failures are of reach and of position, not of fact. One reading runs past its source's
own boundary condition (the chess result). One source is renamed in a way that loses what it
named. And twice the essay reaches for a historical analogy where a 2026 study of software
engineers, sitting inside the evidence base of a source it already cites, measures the same
thing directly — that omission is the largest single improvement available to it. Its central
distinction is also not new to Harbour: docs/ladder.md made the same move two days earlier,
and the essay does not cite it.
Findings
Every checkable number is right. Each was read at the publisher's own page or record.
| Essay's claim | At the source |
|---|---|
| Cui et al.: 4,867 developers, Microsoft, Accenture "and another large company" | 4,867 across three experiments; Microsoft, Accenture, an anonymous Fortune 100 company ✓ |
| "increased completed tasks by about 26 percent" | 26.08% (SE 10.3%) increase in completed tasks ✓ |
| "larger gains among less-experienced developers" | "Less experienced developers had higher adoption rates and greater productivity gains" ✓ |
| Shen and Tamkin: 52 mostly junior engineers, the Trio library | 52 software engineers, mostly junior, learning Trio ✓ |
| "50 percent … compared with 67 percent … a difference of 17 percentage points" | 50% and 67% ✓ — and the essay is more precise than its source, which says "17% lower" |
| "The time advantage was not statistically significant" | "about two minutes faster, although the difference was not statistically significant" ✓ |
| Anthropic: auto-approval more common among experienced users, interruption rates also higher | ~20% of sessions under 50 sessions, over 40% at ~750; interruptions ~5% of turns against ~9% ✓ |
| Dhanorkar et al.: 17 experienced developers, four forms of oversight | 17 experienced developers; four forms ✓ (but see below on their names) |
| Lee and Lim: stage-skipping in DRAM and automobiles, path creation in CDMA | ✓, and the authors' reason — a predictable technological trajectory — is the one the essay gives |
| Tripsas: the typesetter industry 1886 to 1990; typeface libraries as the surviving asset | ✓; the font library is the paper's own example of an asset resistant to the technical change |
The essay's hedges are honest in both directions. It says the Cui experiments "were noisy" rather than quoting the standard error, and the standard error is 10.3% on a 26.08% effect — the hedge is doing real work, not covering a weak result. It says the Shen and Tamkin interaction patterns "were not randomly assigned", which the source also says.
One reading runs past its source. On Bilalić, McLeod and Gobet the essay says "stronger experts were less susceptible" — true, and the authors' own words are that the more expert the player, "the less prone they were to the effect". But the paper does not stop there: grandmasters showed no Einstellung effect on the two-solution problems used, and did show it on problems where the better solution was harder to find. The authors' conclusion is that inflexibility of experts is "both reality and myth". Expertise raises the difficulty at which fixation appears; it does not remove it. The essay's next sentence — "the capacity of deeper expertise to overcome it" — reads a moderation as an escape, and §4's hypothesis that the person who understands why each step was necessary "may be better placed to simplify the sequence" inherits that optimism. The correction is small and it cuts the essay's way: if the same expert fixates again as soon as the alternative is harder to see, then §3's instruction to revisit the classification of a skipped step as capability changes is not good practice but the only thing that works.
One source is renamed in a way that loses what it named. The essay lists Dhanorkar et al.'s four forms as "preparation, co-planning, monitoring and review". The paper's are a priori control, co-planning, real-time monitoring and post hoc review. Three are fair shortenings. "Preparation" is not: a priori control is configuring what the agent may do before delegating — an act of control, which is why it belongs in a taxonomy of oversight. Calling it preparation turns the one pre-emptive control into a warm-up, and §7's use of the taxonomy to distinguish permissive prototype work from strict review of connected systems is precisely about how much control is set in advance.
The essay's central distinction is already in docs/ladder.md, dated two days before it.
The essay's §1 and §2 read as a correction to the ladder: it "captures a recognisable personal
journey", but "a claim that everyone passes through those practices in order requires further
evidence". The ladder doc, revised 19 September 2026, already says so in John's own words —
"the rungs are my personal journey, which seems to be happening to everyone, though of course as
people start these journeys at different points in time the models' behaviour is actually
different" — and draws the same two conclusions the essay draws: "Each cohort starts on a higher
rung because the tools moved between their start dates, and later entrants arrive beside a
colleague rather than by climbing", and "'where developers are' is a distribution over cohorts,
not a queue". That is the adoption-cohort/career-cohort split and the calendar clock. The essay
cites three papers about the ladder and never the ladder. Nothing here is wrong; the essay is
positioned against a version of the document that the archive has already revised, which makes
its opening weaker than the material behind it.
Twice, a historical analogy stands where a direct contemporary measurement was available inside
the essay's own reading list. Source 2's evidence base contains the Microsoft CLI-agent rollout
(arXiv 2607.01418; tens of thousands of engineers, January–April 2026), summarised in
developer-adoption-ladder.md as its strongest disconfirming source. It measures two of this
essay's load-bearing claims directly, on the essay's own population:
- §5 argues from Barley's two radiology departments that practices spread through everyday interaction. The rollout puts a number on it: +216% odds of trying the agent where more than a quarter of skip-level peers had adopted, +82% where the manager had; tenure "barely mattered".
- §4 argues from chess that procedural familiarity and understanding of purpose transfer differently. The rollout has the sign: prior IDE-assistant use raised the odds of trying the CLI agent by 49–83% and lowered 14-day retention by 12–15%.
An essay is entitled to prefer a mechanism to a correlation, and Barley explains why while the rollout only shows that. But the rollout is about software engineers, in 2026, at scale. Its retention result needs care: the authors read it as substitution — an engineer with a working IDE habit has "a familiar alternative to fall back on", so the new habit never forms — not as skill failing to transfer. It supports the essay's claim that time on one practice does not simply buy the next, without settling why. Leaving it out still costs the argument more than it saves it.
What the essay has that Harbour does not. Three things, and they are why it is worth filing
rather than folding into the ladder. First, the three clocks as a measurement claim rather than
a vocabulary: a cross-section of newcomers mixes a change in people with a change in tools, so the
instrument has to follow the same people through comparable work — developer-adoption-ladder.md
reports that no such instrument exists and does not say why one survey could never be enough.
Second, classifying a skipped step by function — compensates for a tool limitation, teaches a
concept or exposes a failure mode, coordinates people or controls an irreversible action — which
is a testable rule for pruning a prompt template or a gate, and the archive has no other. Third,
"more autonomy is a description of how work proceeds, not a complete measure of capability", which
contradicts nothing in the ladder and is absent from its What to measure section, where the
proposed Harbour analogue of Anthropic's auto-approve curve is permission mode by session count.
Three bibliographic errors reached the archive, and are corrected in this PR. The supplied .docx carried no punctuation in any title — an artifact of how it was generated — so the filed version's title punctuation is the transcriber's, and two words were wrong with it: Cui et al. is "High-Skilled", Lee and Lim is "catching-up", and Tripsas 1997 is the Summer Special Issue of volume 18. The Management Science DOI is added now that the Cui paper is published, so that entry lands on the published article rather than only the 2025 summary. This check touched no argument text. John later invited the checker to revise the essay as its second author, which the standard allows for an essay; version 2 is therefore a document this check does not cover.
Method
Each of the eleven sources was reached from the URL in the essay's own reading list, on
2026-09-21, and each numeric or attributional sentence in the body was compared with what that
source says. Seven were verified at the publisher's own page or record: Cui et al. (Microsoft
Research record and the Management Science listing), Shen and Tamkin and the autonomy report (both
Anthropic), Dhanorkar et al. (arXiv), Bilalić et al. (the Cognitive Psychology record and the
authors' own 2010 restatement), Lee and Lim, Tripsas. Four could not be read as text — two PDFs
returned unparseable binary, one returned HTTP 503, one is behind a cookie wall — and were checked
against indexed bibliographic records and secondary summaries instead: David 1990, Barley 1986,
Bainbridge 1983, and Tripsas's body text. The Harbour claims in §1 were checked against
developer-adoption-ladder.md and docs/ladder.md in the working tree. Version 1 of this check
took the Microsoft rollout's figures second-hand from developer-adoption-ladder.md; version 2 read
them at arxiv.org/html/2607.01418 on 2026-09-22. Every figure held. The reading did not: see the
rollout finding above.
Limits
The four sources that could not be read as text are the weak point, and the bias runs toward confirmation: a secondary summary of a much-cited classic and an essay drawing on the same classic can share one received reading, so agreement between them is weaker evidence than the paper agreeing would be. All four are the essay's historical sources, which is where its analogies do the most work.
An abstract is not a method. Nothing here tests whether Lee and Lim's stage-skipping classification survives their own data, or whether Barley's two cases support the inference drawn from them. This check asks only whether the essay reports its sources correctly.
This check tests the readings the essay gives the sources it has. It cannot in general test the sources it does not have — the Microsoft rollout was visible only because it was already inside the archive. A reader should assume other omissions of that kind exist and are invisible from here.
Next
Two lines go to proposals.md with this change. The first is a question for John rather than a
study: docs/ladder.md and this essay now make the same cohort argument independently, and one of
them should absorb the other — either the ladder gains the three clocks and the function
classification, or the essay gains the ladder's paragraph and stops correcting a claim the
document no longer makes. The second is the retention result: the Microsoft rollout found prior
IDE-assistant use lowered agent retention, and Harbour has the records to ask whether the same
sign shows up in its own operators' first bounded runs.
The three lines the essay itself filed stand unchanged; this check does not discharge them.