Bloom’s Two-Sigma Problem: The Claim Rests on Two Unreplicated Dissertations, and 96 Tutoring Trials Never Reproduced It

If you have read anything about AI tutoring in the last three years you have met this number. One-to-one tutoring, the claim goes, moves the average student two standard deviations - from the fiftieth percentile to roughly the ninety-eighth. Benjamin Bloom called the gap between that and what classrooms achieve the “two-sigma problem”, and an entire product category is now sold as its solution.

The figure is real in the sense that Bloom published it. What it rests on is smaller than almost anyone quoting it realises.

What Bloom claimed

Bloom (1984), The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring, in Educational Researcher, asserted that tutoring could raise achievement “by two full standard deviations” and that this represented “the best learning conditions we can devise.”

The paper’s structure is a challenge to the field: if tutoring achieves two sigma and classrooms do not, find a group method that closes the gap.

What the two sigmas came from

Paul T. von Hippel, writing in Education Next on 7 March 2024, went back to the evidence base. His finding is the reason this article exists.

Bloom’s claim rested almost entirely on two PhD dissertation studies:

Study	Subjects	Topic	Duration
Joanne Anania	4th and 5th graders	probability	three weeks
Arthur J. Burke	8th graders	cartography	three weeks

Both found two-sigma effects on post-tests. Von Hippel’s observation about the authorship is pointed: “Because Anania and Burke provided essentially all the empirical evidence that backed Bloom’s claim of two-sigma tutoring, it’s a little shocking that Bloom didn’t credit them as coauthors.”

Three problems follow from that design, and von Hippel sets them out.

The tests were narrow. These were short units on unfamiliar topics with post-tests written to match. “It’s easier to produce a large effect when students are starting from zero.” A two-sigma gain on three weeks of probability is not the same quantity as a two-sigma gain in mathematics.

It was never tutoring alone. The effect “wasn’t obtained by tutoring alone” but by a bundle: “tutoring; training and coaching in effective instructional practices; extra time; and frequent testing.” Attributing the whole result to one-to-one instruction attributes a package to one of its parts.

Nobody reproduced it. These forty-year-old findings, von Hippel writes, “have never been replicated.”

What tutoring actually achieves

There is now a substantial modern evidence base, and it is not ambiguous.

Nickow, Oreopoulos and Quan (2020), NBER working paper 27476, conducted the first systematic review and meta-analysis of randomised tutoring evaluations. Their pooled result: 0.37 standard deviations.

Their moderator findings are practically useful in a way the two-sigma figure never was. Effects are stronger for teacher and paraprofessional tutors than for non-professionals or parents. They are strongest in earlier grades. Reading tutoring works better early, mathematics tutoring better later. And programmes run during the school day outperform after-school ones.

Von Hippel’s summary of the field, placing this beside the older work: Cohen, Kulik and Kulik (1982) found about 0.33 standard deviations; Saga Education’s 2021 evaluation raised maths scores by 0.16 to 0.37; and “About one-third of a standard deviation seems to be the typical effect of an intense, well-designed program.”

Then the sentence that closes the question. Of the 96 tutoring studies reviewed in the 2020 meta-analysis, von Hippel reports: “none produced a two-sigma effect.”

Why this one matters now

A third of a standard deviation is an excellent result. Intensive, well-designed tutoring is among the best-evidenced interventions in education, and the 2020 meta-analysis is a genuinely encouraging document.

The problem is what the two-sigma figure is being used for. It is the benchmark in the pitch for AI tutoring - the promise that software can deliver, at scale and near-zero marginal cost, what Bloom found human tutors delivering. That promise inherits a number produced by two unreplicated three-week dissertations on narrow topics, using a bundled intervention, which ninety-six subsequent randomised trials never reached.

There is a further problem the figure conceals. Because Bloom’s effect came from a bundle - tutoring plus mastery techniques plus extra time plus frequent testing - a system that automates only the tutoring component is not attempting the thing that produced the result, even on the result’s own terms.

This is the same pattern as the largest randomised evaluation of putting laptops in children’s hands, where the hardware arrived, was used, and did not move attainment because the teaching did not reorganise around it. And it belongs beside the ten-thousand-hour rule for the same structural reason: a specific number, taken from a small and unusual sample, doing enormous rhetorical work a long way from where it was measured.

Limitations and disputes

• Bloom (1984) was not read in the original, nor were the Anania and Burke dissertations. All three are reported as described in von Hippel’s 2024 analysis.
• Education Next is an edited policy journal, not a peer-reviewed one. Von Hippel is a university professor writing in his field, and the piece is labelled here as expert analysis rather than peer-reviewed research.
• The “96 studies, none reached two sigma” count is von Hippel’s, from his reading of the 2020 meta-analysis. The meta-analysis abstract we verified gives the pooled figure of 0.37 SD but does not state the study count.
• Nickow, Oreopoulos and Quan (2020) is an NBER working paper at the version cited; it has since appeared in peer-reviewed form and the figures may differ marginally.
• This is not an argument against tutoring. Tutoring is one of the better-evidenced interventions in education; the objection is to a specific, unreplicated magnitude.
• This article does not evaluate any AI tutoring product. No such product’s own evidence is assessed here.
• A two-sigma effect is not impossible in principle. No study in the reviewed set produced one, which is different from proving none could.