Core Improvements in Live Search

This post will provide some more info on the September 26, 2007 Searchification event that Microsoft put on. In particular, this post will focus on the core search engine improvements part of the presentation. I will provide some brief comments on each announcement, as well as pictures of the speakers.

1. Introduction by Brad Goldberg. Brad Goldberg, general manager of the Windows Client Product Management Group at Microsoft, who manages the search team from a business perspective, started things off for the event:

Brad Goldberg

Brad spoke about data that indicates some of the basic problems with search. For example, 40% of search queries fail to provide an answer, and 50% of these queries require refinement before an answer is found. People find that getting what they want requires a high level of cost and commitment.

One of the more interesting things he spoke about was the search market data share data from comScore:

Engine Users User share Query share
Live Search 69M 37% 11%
Yahoo 104M 56% 23%
Google 142M 77% 56%

Based on this data, he stated that Microsoft’s focus is on getting more repeat queries from their user share, or doing a better job of delighting their current customers.

2. Overview by Satya Nadella. Satya Nadella, Group VP, Search and Advertising Platform Group, was up next and dug into a bit more detail about Microsoft’s areas of focus for this update:

Satya Nadella

Note that one of the comments Satya made in a pre-show discussion I had with him was that the Microsoft infrastructure is finally getting caught up, and this is enabling them to do much more with their search product.

The last major update that Microsoft did was in September of 2006. Microsoft has been doing rolling updates through the year, including a number of performance and relevance changes. In this release, several of Microsoft’s search products were affected:

  1. A major update was done to the web search index
  2. Maps
  3. Mobile Search
  4. Shopping Search
  5. Health Search
  6. Image Search
  7. A Microsoft Video Search Product was Announced

Here is a screen shot of one of Satya’s slides:

Diagram of Updated Areas

Next up was a summary of customer feedback:

Microsoft Customer Feedback

The data in the above pie chart was based on an analysis of user click behavior. The sidebar point about relevance was based on an analysis of over 10,000 feedback submissions.

Given the preponderance of concerns about relevance, Microsoft did some further research to get a better understanding of the nature of the relevance concerns. This broke out as follows:

Microsoft Relevance Concerns

Based on what Microsoft learned from these analyses, they invested in 6 major areas:

  1. Coverage – They increased their index size from about 5B pages to about 20B pages
  2. Query intent – Making a better determination of what the user is really looking for
  3. Query refinement – Determining how to refine a query to provide the user with better results
  4. RankNet improvements – A variety of tweaks to Microsoft’s Neural Net algorithm to improve results
  5. Structured information extraction – Doing a better job of using structured data bases to improve relevance
  6. Rich answers – Incorporation of blended data from verticals, such as image and video search

For those of you who want a brief definition of what RankNet is, you can see it here:

RankNet Overview

Demos of Improvements. At this point in the presentation, Ramez Naam joined Satya to run some live demos:

Ramez Naam and Satya Nadella

Ramez demoed a variety of search queries and their results. Here are some of the queries that were demonstrated:

  1. EPRML – Microsoft did not show acronym based answers before, and now does. They also only showed about 1,700 results for this query previously, and now show more than 10,000.
  2. the office – search engines normally strip off “the” from this query, but many users are actually looking for the TV show. This now comes up in Live Search results.

  3. c.n.n. – Live search used to look for “c n n” after seeing this query, but now it knows to look for CNN.
  4. IL soccer – The new Live Search understands that in this query that “IL” means Illinois.

  5. Groig Freiderich Nicolai – Live Search now auto-corrects this query to “Groig Friederich Nicolai”.
  6. China – This search shows off some of the rich media integration, as well as the “Related Searches” functionality at the top right of the results screen.
  7. Volkswagen Kaefer – Shows a German page, which probably is the best result for most users. The page can be translated on the fly, and in fact, Live Search offers a mode in which you can see the original German version and the English version side by side. You can also modify how this works using on screen controls.
  8. San Jose weather – now shows the weather right on the screen. This feature will be live to the public soon.
  9. San Jose traffic – You now can near real time traffic info right there on the screen.
  10. MSFT – Provides an intraday stock chart, along with pricing and volume information on the search results page.
  11. Barack Obama – News results are integrated in for those queries where that would be relevant.
  12. space shuttle videos – Video results are incorporated directly in the results, and you can play the videos inline on that page.

This is a sampling of some of the more interesting queries demonstrated during this part of the presentation.

Ultimately, the objective of this effort was to increase their search results relevance. Microsoft then did some testing with live human subjects to assess relevance. Each participant was trained on how to assess relevance. They had this group do a large number of searches and presented them with results in format where they did not know which search engine the results were from.

Net-net, the results of this testing showed a dramatic improvement in Microsoft’s relevance scores. Here is a chart showing the details:

Search Relevance Scores

In summary, it is clear that Microsoft made a lot of improvements to their core search results. The real tale of the tape will emerge from tens of millions of searches done by real users. That said, a number of different issues were presented, and Microsoft has addressed them, so this presents good progress.

The other concern I would have would be is whether the underlying strategy of trying to capture more market share from their existing users will work. After all, is the reason they capture the initial search, but not the follow-on searches, simply because users know which search engine they already trust?

This could provide some resistance to getting comfortable with Live Search results. However, I can tell you that I am more intrigued by Microsoft’s improvements and Live Search then I have ever been. I know I will be doing some testing of it on an ongoing basis.

Comments

  1. says

    Hi

    I read a really fantastic article, thanks Eric, if you are more detail and the screen shots are more clear it is better.
    You have mentioned 7 points of where Live.com increase their work – fine but you have give data this is not clear to me >

    Engine Users User share Query share
    Live Search 69M 37% 11%
    Yahoo 104M 56% 23%
    Google 142M 77% 56%

    Still thank you for your good article
    Deb

  2. says

    Hi Deb,

    Regarding the chart, let me use the line for Live Search as an example, and explain what it says about Live Search. The first thing that I should clarify is that these are the comScore statistics for the search engine market in August of 2007.

    The first column, Users, says that 69 million people used Live Search in August. The second column, User Share, indicates that 69M users represent 37% of all people who conducted a search on a search engine in August (in other words, 186M people used a search engine in August.

    Lastly, Live Search had only 11% of all search queries. What this means is that even though 37% of the people who did any searches in August did at least on search on Live Search, many of those users did most of their searches on other search engines.

    It’s this information that has led to Microsoft’s strategy of trying to do a better job of making those 69M people happier with the results they receive, in order to capture more of their total search volume.

    Hope that helps!

  3. says

    It’s great to hear from you that it will be an ongoing process. If I can ask a question here…. are you going to restart the other search operators such as site: link: etc…too?

  4. says

    Hi Web Design and SEO,

    Microsoft did also announce some of the details of their Webmaster Tools products, and said that the link reporting would be revitalized when that is released later this year.

  5. says

    Dear Eric,

    Thanks for your response regarding my mail, yeah it bit clear to me that you said in your article, one interesting information I want to let you know, my client’s site http://www.superjumbopro.com indexed one page two months before but suddenly it is disappear from msn.com and now it is indexed only one page which more than 20 pages in the site.

    Thanks
    Deb

Leave a Reply

Your email address will not be published. Required fields are marked *

*