A Randomized Experiment Assessing the Accuracy of Microsoft’s “Bing It On” Challenge

Abstract

In advertisements associated with its "Bing It On" campaign, Microsoft claimed that "people preferred Bing web search results nearly 2:1 over Google in blind comparison tests." We tested Microsoft's claims by way of a randomized experiment involving U.S.-based Amazon's. Mechanical Turk ("MTurk") subjects and conducted on Microsoft's own www.bingiton.com website. We found that (i) a statistically significant majority of participants preferred Google search results to Bing search results (53% to 41 %); and (ii) participants were significantly less likely to prefer Bing results when randomly assigned to use popular search terms or self-selected search terms instead of the search terms Microsoft recommends test-takers employ on its website. Our findings suggest that some of the claims implicit in Microsoft's advertisements warrant legal scrutiny.