Hitting the search relevance wall

There is a curious fenomenon going on in the search arena today - something I call the "hitting the search relevance wall". The thing is that most search companies are getting so much better that it is hard to distinguish between them in terms of quality. The days when Google had the best search relevance are over.

It's no wonder why. There are only a few links that you can display in the first page of your search results. And, no matter how good your relevance technology is, you will have to display the same links over and over, no matter what search engine you use. Therefore, if all search engines are getting to the "minimum bar" of displaying correctly the first links, then it is almost impossible to distinguish between them, if all you need is a few good links in the first page. And isn't this the most frequent way we are using these search engines? It's like hitting a "wall" when spending money on more and more research in improving search relevance - at some point you get less and less benefit no matter how much money you pour out in the technology, because the first links should be always almost the same.

The latest survey results from Keynote Systems are confirming this fact (the emphasis in bold is mine):

However, Keynote said it found that actual search results returned by the five search engines do not differ significantly. Actual consumer success in doing complex searches showed that the performance of Lycos, Ask Jeeves and Microsoft's MSN was as good as Google and Yahoo.

The article also mentions that MSN Search is getting visibly better in the last few months.

So, if search relevance won't be anymore a differentiator between search engines, then what's left? Plenty of cool things. Local search is one - and here I personally think that Virtual Earth rocks. Of course I am a MS employee, but, hey, am I allowed to love this site :-)? (Of course, Google Maps rocks too, and they have all my sympathy for inventing the draggable maps in DHTML). Also, new features like the integration between White/Yellow pages and local search are cool too. MSN Spaces is also something that spreads like wildfilre. Flickr is making inroads too.

There is plenty of room for innovation, and right now, the competition between Google, Yahoo, MSN and others is higher than ever. Good times to live in...

Comments

• Anonymous
  August 19, 2005
  You've left out one of the most important differentiators, which is user experience. The Google UI is clean, uncluttered, and fast. Good products without equally good interfaces are an ongoing problem.

• Anonymous
  August 19, 2005
  Agree - I was only talking about search relevance. User experience is absolutely crucial too.
  
  Yahoo came up with a clutter-free page at http://search.yahoo.com, and MSN with http://search.msn.com
  
  There is an additional experiment with a shorter URL at start.com (right now at http://start.com/3)

• Anonymous
  August 19, 2005
  I understand what you are saying but when I want to find something I still end up wading through a lot of content I don't want to.
  
  Perhaps, you could try some new directions. Things like profile based searching where a user builds a profile and all results are screened based on the type of results a person in that profile would want and past searches. The idea is to get more information without forcing the user to make more complex queries and therefore get better results.
  
  Another idea, is offline based filtering. In this paradigm, the search engine returns a first shot at results and the user can use advanced tools to isolate even better results using searches that might be impractical on a server. These searches could also use information the user might use to determine good results but not want in their profile.
  
  Most search engines have already implemented 'Show me more results like' options but there are a lot of additional things that can still be done.
  
  
  Oh, and my search engine also occasionally returns results that I can't read. Could someone fix that?
  
  More and better input = more and better output
  
  The question is how to get more input without being a burden to the user.
  
  Shaun Bedingfield
  shaunbed@swbell.net
  blogsb.blogspot.com

• Anonymous
  August 19, 2005
  I don't disagree with the thrust of this post but it's interesting that you can still find queries where one search engine does a much better job. (I'm sure this is something that's well known but I haven't seen studies myself.)
  Compare for instance the results of a Google search for 'http' with the same from Yahoo. The
  Google results are completely irrelevant; the Yahoo results are (on the first page at least) mostly relevant.
  
  I wonder if anyone has done a systematic survey.

• Anonymous
  August 19, 2005
  It looks like Google is not using 'http' as a search word for indexing. This might be due less to the quality of the engine and more to the quality of test for exceptional cases.
  
  The results I get look like they are probably some of the most often hit sites or things I would expect to get if I could search for nothing in particular.

• Anonymous
  August 19, 2005
  Shaun: I agree with your observation that the hits that you get from googling for 'http' appear to be some of the most popular websites.
  
  If you google for "Hypertext Transfer Protocol" on the other hand, the results, not surprisingly, are relevant. Perhaps Yahoo is equating the two.
  
  Could it be something so simple?

• Anonymous
  August 19, 2005
  It looks like Yahoo indexes 'http' but stores the url without the protocol before indexing. The engine appears to definitely be looking for the string not the equivalence. However, common equivalence classes is another possible search optimization.
  
  For example, if a user searches on 'CGI', the engine could look for "CGI", "Common Gateway Interface", and "Computer Generated Imagery". Just another example of ways to improve current search technology.
  
  Google on the other hand seems to take http as a special word like "a" or "the" and not index on it period.

• Anonymous
  August 19, 2005
  And if you think these optimizations are that advanced, look at some of the great things SQL Server Reporting Services can do with neural networks, etc.
  
  
  With some of the technology out there, our current search engines are very primitive. Maybe MS Search should look at seeing if some of the members from Reporting Services would consult on Search.
  
  Shaun Bedingfield
  shaunbed@swbell.net
  blogsb.blogspot.com

• Anonymous
  August 19, 2005
  >> And if you think these optimizations are that advanced, look at some of the great things SQL Server Reporting Services can do with neural networks, etc.
  
  Shaun, funny that you mention neural networks. MSN Search is already using some sort of neural nets to improve relevance: http://blogs.msdn.com/adioltean/archive/2005/07/19/440311.aspx
  
  But I didn't know about SQL Server using them! So I did a quick search, and indeed SQL Server 2005 has some neural network technology too. Interesting.

• Anonymous
  August 19, 2005
  Now that I think about it, I think it was analysis services. They had a good write up showing some examples of what the tool could do a couple months ago in MSDN magazine.
  
  It is good stuff and there might be some interesting possibilities for collaboration in that arena.
  
  Shaun Bedingfield
  shaunbed@swbell.net
  blogsb.blogspot.com
  
  Do you need a programmer who is really good? Email me.

• Anonymous
  August 19, 2005
  >> Do you need a programmer who is really good? Email me.
  
  Sure we need. I think I already got an email from you but I noticed that the recruiting guys are aware about your application so I presume that you are already on the right track.
  
  https://blogs.msdn.com/jobsblog/articles/88832.aspx#438197
  
  Thanks, Adi
  

• Anonymous
  January 24, 2008
  PingBack from http://websitescripts.247blogging.info/antimail-hitting-the-search-relevance-wall/

• Anonymous
  April 03, 2008
  PingBack from http://drinksandbirthdaysblog.info/antimail-hitting-the-search-relevance-wall/