The Importance of Search Result Location

Editor's Note: The following is a guest post from Susan Athey, a Professor of Economics at Stanford University Graduate School of Business and a long-time Microsoft consultant.

Search engines are a technological marvel: In milliseconds, they can bring order to the vastness of the Internet. In fact, it’s estimated that about two billion people turn to a search engine on a regular basis to find their way around the Web.

As this technology has evolved, search engines have become more than simply virtual answer desks. Today they have become “commerce platforms,” pointing people to the products and services they seek by connecting potential customers to a growing number of online business.

But what happens if a search engine decides to place links to its own products and services at the top of the search results page, crowding out results identified by its algorithm as the best? Similarly, what about so-called “sponsored links,” where websites pay for the privilege to appear at the top of the page? How much does it really matter where a website appears on search results page? And can a search engine really have that much power to influence which links users click?

As a search engine user myself, I like to think that I am savvy, and won’t be misled by the location of a link on a search results page. And if location has very little impact, then perhaps we shouldn’t worry too much about it.

Unfortunately, it turns out that a website’s location on a search results page does matter—and it matters a lot.

Even though the basic fact that the prominence of results is important is widely known throughout the Internet ecosystem—anyone who designs a Web page and compares alternatives will quickly discover it—it is still worthwhile to quantify just how important it is. Much of the evidence that is available in the public domain concerns associations: For example, the links in the first position get so many more clicks than those in the second position. That kind of evidence doesn’t adequately separate out “correlation” from “causality.” Did the top link get clicked because it was the best link for the search query, or because it was in the top position? What would have happened if the links in the top position and a lower position were reversed?

With a randomized experiment, where different users see different rankings of links or different layouts on the screen, it is possible to address this question more definitively, and to avoid trying to generalize from specific examples that may not be representative. Search engines regularly run experiments to test out the performance of new algorithms. In these experiments, user searches are randomly assigned to either receive the “control” treatment—the baseline search experience—or one of a number of experimental “treatments,” where results are ranked or presented differently.

In order to answer the question about the potential effects of manipulation, I worked with the Bing team to design a special experiment to quantify the extent to which search order matters. The experiment studied the impact of several “treatments” in which we moved the best search result—the one that our algorithms would otherwise place first—to various lower positions on the search results page. The test ran for a few weeks, in the United States and overseas.

The data spoke very clearly about the impact of the treatments: A search engine can divert traffic from one website to another by manipulating the order of search results. In particular, moving the best result down just two positions (from first to third) reduced traffic to that site by half. The diversion effect becomes much more pronounced as a site is moved further down the page. A site that is moved from the first position to the tenth position typically will lose about 85 percent of its traffic. A site that is moved from the second position to the ninth loses about 75 percent of its traffic. And the results were similar for all users, regardless of the amount of time they spent searching on the site.

If you look at the same results from the perspective of the site that gets promoted from a lower position to first in the rankings, the effects are even more pronounced. (This is because the site appearing further down the page has so few clicks to start with). A site promoted from fifth to first gets a 340 percent increase in visitors from search, and the results are similar when you focus only on users who go to the site and stick around for a period of time. Imagine telling a business that they can more than quadruple their customer base overnight! That is a very tempting thing to do for a search engine if the site it is promoting is its own affiliated website.

In fact, the manipulation of results to preference a search engine’s own products and services is one of four areas of concern identified by competition authorities investigating Google’s business practices in Europe, where the world’s largest Internet company controls more than 90 percent of the search market. According to Joaquin Almunia, the European Commission vice president responsible for competition policy, “Google displays links to its own vertical search services differently than it does for links to competitors.” Vice President Almunia explained that “[w]e are concerned that this may result in preferential treatment compared to those of competing services, which may be hurt as a consequence.”

As a party that has formally raised concerns about Google’s business practices, Microsoft has submitted the results of the Bing study to the Commission for review. While the results of this study quantify the impact of what most people know intuitively, they demonstrate that the order and manner in which a search engine presents results powerfully affects how much traffic websites will receive.

Interestingly, the European Commission’s focus on this issue comes at a critical time: The impartiality of search results will become all the more important in the years to come given that screen sizes on smartphones and tablets are smaller than on traditional PCs. Smaller screens mean there is even less room for competing services to appear in Google’s mobile search results.