The Search Engine Manipulation Effect: The Famous 20% Is a Summed Metric, Not a Share of Voters
The claim that search rankings shift 20% of undecided voters rests on a summed metric. Averaged, one critique puts the effect nearer 3% of all voters.

You have probably encountered the claim: biased search rankings can shift the voting preferences of undecided voters by twenty per cent or more, and by up to eighty per cent in some demographics. It has been cited in congressional testimony, in antitrust argument and in a great deal of writing about platform power.
The experiments behind it are real. The number is not what it sounds like.
What the experiments did
Epstein and Robertson (2015), in PNAS, coined the term Search Engine Manipulation Effect (SEME) and reported five experiments in two countries. Participants read about candidates, then used a mock search engine whose result rankings had been deliberately ordered to favour one candidate, then reported their voting preference. The design was randomised, counterbalanced and double-blind.
The reported headline: biased rankings can shift the preferences of undecided voters by 20% or more, with much larger figures in some subgroups.
We could not retrieve the 2015 paper from the publisher. What we can report is how the same research group describes its own numbers in a later, accessible paper. Epstein, Robertson, Lazer and Wilson (2018), in Proceedings of the ACM: Human-Computer Interaction 1, article 42, state that the earlier experiment produced a 37.1% shift and that their own replication produced 39.0% - and then test whether warnings suppress it:
| Condition | Shift |
|---|---|
| No alert | 39.0% |
| Basic alert | 22.1% |
| Detailed alert | 13.8% |
Alerts cut the effect substantially but did not remove it. The authors report that SEME “could be completely eliminated only by alternating search results.”
What the percentage actually measures
Here is the part that changes how the whole literature reads.
Katharina Zweig, professor of computer science and head of the Algorithm Accountability Lab at Kaiserslautern, examined the arithmetic in a piece for AlgorithmWatch on 7 April 2017. Her central finding is about the outcome variable, not the experiments.
The headline figures are not the percentage of participants who changed their vote. They are a quantity Epstein and Robertson call vote manipulation power (VMP). Zweig’s objection:
“The results are strongly exaggerated because they are based on a measure which sums up the effects of biasing the results for all candidates instead of averaging them.”
If you run the experiment favouring candidate A and observe a shift toward A, then run it favouring candidate B and observe a shift toward B, VMP adds those shifts together rather than averaging them. The resulting number is roughly the sum of separate manipulations, not the proportion of any single electorate that would move.
Recomputed on an averaged basis, Zweig puts the Indian study’s effect at “more likely to be 6%, i.e., 3% of all voters.”
Her account of the other two famous figures:
- The 20% headline involved excluding certain voters after the fact.
- The 80% subgroup claim came from examining demographic subgroups - which she compares to “throwing a fair dice long enough to find eight sixes”, the standard multiple-comparisons problem.
What is left
A great deal, and it is more interesting than the headline.
Position matters. Something moves when result order is manipulated. Three per cent of voters is not nothing - it is larger than the margin in many real elections, which is Epstein’s substantive point and it survives the correction.
People do not notice. The 2018 study’s whole design assumes the effect operates below awareness, and the fact that explicit alerts only partly suppress it supports that.
Warnings help and do not fix it. Detailed alerts cut the measured shift by roughly two-thirds. Only altering the results themselves eliminated it.
What does not survive is the specific claim as usually stated. “Search results can shift twenty per cent of undecided voters” is not what the studies measured, because VMP is not a share of voters. Anyone repeating that sentence is repeating a summed metric as though it were a proportion.
The structural problem underneath
There is a second issue that neither side disputes and that a reader should weigh.
Almost the entire positive evidence base for SEME comes from one research group. Epstein is an author on the 2015 paper, the 2018 replication, and the subsequent work. Independent replication by unaffiliated teams is scarce. That is not an accusation - the effect may be perfectly real - but a decade-old finding with policy implications this large, still resting substantially on its originators’ own studies, is in a fragile evidential position.
The experiments also use a mock search engine with artificially ordered results and, in two of the three original studies, unfamiliar foreign politicians. Both choices are defensible for experimental control. Both are a long way from a real voter forming a view about a candidate they have heard about for months from a dozen sources.
The honest summary is: a real ordering effect, of uncertain magnitude, measured with a metric that overstates it, in artificial conditions, replicated mainly by its discoverers.
Which is a boundary worth stating against the neighbouring articles here. Whether algorithmic curation isolates people ideologically has a much larger and better-controlled evidence base, including an experiment that switched 23,000 people to a chronological feed. Whether a platform can move emotions by altering what it shows is a different experiment with a famously tiny effect size. This article is about search result ordering specifically, and about one number.
Limitations and disputes
- The 2015 PNAS paper could not be retrieved. Its figures are reported as restated in the authors’ own 2018 follow-up and as described by Zweig. No figure is printed here that we could not attribute to an accessible source.
- Zweig’s analysis is an expert critique published by a research organisation, not a peer-reviewed reply, and is labelled as such. It is a named computer-science professor working through the paper’s own reported numbers.
- Epstein and colleagues dispute the criticism. We have not located a formal published response to Zweig specifically, and their subsequent work continues to use the same measure.
- The 2018 study’s 39.0% is the same VMP-style metric, so it carries the same interpretive caveat as the 2015 figures.
- Almost all the evidence comes from one group. Independent replications by unaffiliated teams could not be located.
- The experiments use mock search engines, not live Google results, and this article makes no claim about what any real search engine has done.
- Nothing here is a claim about any election outcome.
Human Operating System
Human Operating System is a research-led publication about the human mind under digital pressure. We report what the evidence does - and does not - support.
About Human Operating SystemSources & Further Reading
3 sourcesThese are the sources used for this article. Where a study's limits matter to the claim, those limits are kept in the citation.
- Epstein, R., & Robertson, R. E. (2015). The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. PNAS. https://doi.org/10.1073/pnas.1419828112 — five experiments in two countries; the origin of the 20% and 80% figures. Not retrievable from the publisher; reported here via sources 2 and 3.
- Epstein, R., Robertson, R. E., Lazer, D., & Wilson, C. (2018). Suppressing the search engine manipulation effect (SEME). Proceedings of the ACM: Human-Computer Interaction, 1, article 42. — restates the earlier shift as 37.1%; replication 39.0%; basic alerts 22.1%; detailed alerts 13.8%; only alternating results eliminated the effect.
- Zweig, K. A. (2017, 7 April). Watching the watchers: Epstein and Robertson’s “Search Engine Manipulation Effect”. AlgorithmWatch. https://algorithmwatch.org/en/watching-the-watchers-epstein-and-robertsons-search-engine-manipulation-effect/ — identifies vote manipulation power as a summed rather than averaged measure; recomputes the Indian study at approximately 3% of all voters; identifies post-hoc exclusion behind the 20% figure and multiple testing behind the 80% subgroup claim.
The Weekly System
The Weekly System has not launched yet. Join the launch list to hear when the first issue of this planned, research-led email is ready. No launch date or schedule has been announced.
