Eric Enge

Power Adwords Tools with Google’s Frederick Vallaeys

1/25/2012 by Eric Enge 1 Comment

Frederick Vallaeys is a Product Evangelist for Google AdWords. In this role, he helps advertisers learn about which Google products can best solve their marketing needs. He also represents the needs of advertisers with the engineering and product management teams. His main product focus is on ads quality and bulk tools like the AdWords Editor and the AdWords API.

Prior to Google, Frederick was an engineer at Sapient and a part-time wedding photographer who found new customers through AdWords. He joined Google in 2002 to help bring AdWords to the Dutch and Belgian markets. He earned his B.S. degree in electrical engineering from Stanford University in 2000.

Key Points from Interview with Frederick Vallaeys

ValueTrack is the AdWords feature that allows advertisers to tag their URLs with parameters. The resulting URL can then be used within the advertiser’s own tracking systems.
Too many advertisers settle for global level reporting and do not look further. Even if your top level metrics are OK, you can still get great gains in overall campaign performance by digging into more detailed reports.
Segmentation is the biggest power reporting feature that is not used by many advertisers.
Types of segmentation can include times of day, days of week, device type, social signals, and more.
(Fred): “… no matter from which channel the +1 comes in, it all aggregates at the URL level.”
(Fred): “In the social segmentation, you can actually see what the impact is of having each of these different variations.”
You can run multiple segments via the downloadable reports or the API.
(Fred): “… at the end of 2011, half of American consumers had a Smartphone in their pocket.”
Google has a site at howtogomo.com that you can use to see how your site renders on a mobile device.
(Fred): “Google Analytics offers multichannel funnels, and what these allow you to do is see what touch points people have with your online campaigns before a conversion happens.”
(Fred): “One tool that we have is the AdWords Campaign Experiments. That’s a great way for an advertiser to explore how to improve their ROI. They can send 10%, 20%, 30%, whatever percentage they want of their traffic to that experiment.”
(Fred): These (new ad formats) were a big thing for us in 2011, and will continue to be a big thing in 2012.
The Bid Simulator tool will show you what to expect for different types of increases (or decreases) in bids.
The Ad Preview Tool allows you to see whether or not your ads are running. It also allows you to test geotargeting in areas other than your current location, or various types of mobile devices.
Top of Page bid estimates show you what your bid would have to be to show up in the space above the organic results.
Impression share is a way to see what percentage of the time your ads are running. Tuning your campaign to increase impression share can be one of the best ways to get additional traffic.
Google Analytics is planning to expand its social reporting to include more than just the data from Google owned properties – i.e. data such as Facebook Likes.

Full Interview Transcript

Eric Enge: Can you tell me some great power reporting features in the AdWords interface that people rarely use?

Frederick Vallaeys: When you look at AdWords, there are three high-level types of reports that we make available for our customers. You can go into the campaign management interface and pull reports right there in your campaigns. Then, we also have Google Analytics which goes a little bit deeper into some of the data, for example, with real-time reports, social reports and cross-channel reports that look at how ALL your campaigns are contributing to your success and your ROI. The third one is making reporting available for people who prefer using APIs or building their own reporting systems using our URL tagging feature, ValueTrack.

That is a way for us to attach some additional information to each click that comes to your website so that your own reporting software can capture that and then process it. If you look specifically at what is available in the AdWords interface, it’s really gotten very sophisticated in terms of segmentation. And, I think one of the biggest mistakes that advertisers make is they look at their reports at too high a level.

There are probably all of these micro-segments within your campaign where things are performing fantastically well …

There are probably all of these micro-segments within your campaign where things are performing fantastically well, but you don’t know it because you looked at things are an aggregated level. On the flip side you also have elements of your campaign that just aren’t working well. Examples of segments that you could be looking at are the specific time of the day, and the specific day of the week. You may for some reason find people just aren’t buying your product at certain times of day or days of the week.

Eric Enge: What are some of the other segments you offer have?

Frederick Vallaeys: One of my favorite ones is the social segments. When we introduced Google+, part of that was the +1 button, and now the +1 button will show up next to ads. +1s are being collected from the ads, from the organic results, but also from having the +1 button on your own website.

… no matter from which channel the +1 comes in, it all aggregates at the URL level …

Eric Enge: The +1 is associated with a web page, and not the ad or organic results, isn’t that right?

Frederick Vallaeys: Exactly, no matter from which channel the +1 comes in, it all aggregates at the URL level, and will show up in any channel where the consumer is then looking for your business. As an advertiser, when you put the +1 button on your site, you start getting some +1s, and now when somebody is looking for your business or your service, they see your ad as you would have seen it in the past but now there is also a count of +1s next to it.

It might indicate that Eric and five other friends have +1d this page, or, you have the more generic one where it just says 500 people have +1d this. In the social segmentation, you can actually see what the impact is of having each of these different variations.

If you knew that this could drive a lot more clicks, a lot more conversions, then you could make a bit of an effort within your company to get more +1s for those specific URLs …

As an advertiser, what you can start to see is that probably the most powerful results are the ones that have the personal recommendations, and you could start finding some URLs on your sites in your campaigns that you are advertising for, that don’t have a lot of these personal recommendations. If you knew that this could drive a lot more clicks, a lot more conversions, then you could make a bit of an effort within your company to get more +1s for those specific URLs that you have in your ad that need more of the personal +1 recommendations.

Eric Enge: Very cool. What are the sub-flavors that go into the social reports then?

Frederick Vallaeys: When we moved all of reporting into the campaign management interface, the whole notion that we had was to make it really easy for people to immediately act on the information that we show them. Just imagine your traditional campaign management page, it’s down to the ad group level, you can see the results for that. And any time horizon of course that you want as well. Ryan, is there anyway to run multiple segments at the same time?

Ryan:Ryan Voccola: Not in the UI, but you can do it with a downloaded report, or with the API where you would have to add on additional segments that will come out in the CSV export.

Frederick Vallaeys: So you can see how social is effecting your performance, but then correlate that to how certain times of day, or the day of the week, is also impacting your results. That’s where you really get small micro segments where you could start figuring out some pretty interesting things. I think most advertisers will stay at that first level of segmentation, because that’s really going to give you some pretty good returns and that’s also where they have enough data to make statistically sound decisions.

Eric Enge: You can also segment by device, right?

… at the end of 2011, half of American consumers had a Smartphone in their pocket.

Frederick Vallaeys: Yes, that’s a huge one right now because at the end of 2011, half of American consumers had a Smartphone in their pocket. There is a lot of web usage occurring on these mobile devices. One interesting thing we see is that mobile device usage really spikes early in the morning and late at night, so literally the first thing people do in the morning is take out their mobile devices and check their email or research something and it is also the last thing they do before they go to sleep.

As that behavior becomes more common and the usage numbers go up dramatically, it’s really important for an advertiser to look at how they are performing differently on these different devices. Your performance could vary based on whether or not searchers are on a Smartphone, tablet device, desktop or laptop. Perhaps you should make a mobile website to drive up your conversions.

One really cool tool that we launched a little while ago that maybe not many people know about is howtogomo.com. On that site we do an evaluation of people’s websites and how they would render on a mobile device. It’s a really easy for someone to see if somebody came to the site from a mobile phone, would it even make sense to them, would they be able to click the links or are they too small, or how does the page render on these smaller screen devices.

Eric Enge: Have you seen examples of people where there are drastic differences in time of day in terms of conversion rates?

Frederick Vallaeys: That is a little bit industry specific for the most part, but in the travel space in the morning people tend do a lot of research, and then during their lunch break they call their spouse or significant other, and in the afternoon you might see a little bit more booking behavior. Obviously, you do have to be careful with this because those clicks and those visitors that were doing research may be just as important to getting the final conversion.

… a large percentage of conversions involve multiple clicks.

You probably look at Google Analytics to see the multiple steps that happen in your conversions. For the past 10 years, we have been looking at last click conversion in the industry, but a large percentage of conversions involve multiple clicks. It is important to understand the whole cycle for your site.

You might see really generic searches in the beginning, and then as people start to figure out exactly what they want, they get to a very specific search, and maybe the last search they do is a branded search. But all of the searches before that are often really important in convincing that customer that your company is a player in this space, someone they could trust to do business with.

Eric Enge: Can you talk about that a little bit about the problem of attribution?

Frederick Vallaeys: Google Analytics offers multichannel funnels, and what these allow you to do is see what touch points people have with your online campaigns before a conversion happens. Before we had this, we could tell you which keywords assisted in terms of search campaigns. This takes that one step further and tracks display campaigns and social media, so you can follow customers as they go through the funnel of conversion. Maybe you have three touch points through the display network and then you have two different searches happening, and then they bought something.

Where it becomes challenging is you have to figure out how to assign value to each of these actions, as they are all involved in the conversion. You have to start modeling that for yourself, and you have to experiment with it to see what makes sense.

Eric Enge: For display ads you have this concept of a view through conversion, right?

Frederick Vallaeys: Yes, but what is more powerful here is we can start showing you how your typical person who converts saw your email marketing campaign first, then maybe they saw a tweet, then they saw your display ad three times, and then they did seven searches. You can actually see how all these events contribute to lead to that conversion. Maybe there are 500 people who took a path that was similar to that, and then there are other people who go directly to search because they know exactly what they want.

In the past, if you just looked at last click conversion, you would eliminate these keywords because they had never given you a conversion.

Now you have the data and now you can start figuring out why. If you were to cut out this list of keywords would that have an impact on your campaign? In the past, if you just looked at last click conversion, you would eliminate these keywords because they had never given you a conversion. That could be a big mistake, because maybe that is the keyword everybody always ends up searching, one search before they do the final one that at leads to the conversion. If you got rid of these searches, then people might not even realize your company existed or had this service available, and you wouldn’t get these last click conversion anymore.

Eric Enge: Unfortunately, there really is no science to how you attribute value across multiple clicks or views.

Frederick Vallaeys: Exactly, at some point maybe we will have some more insight into that, but for now the point is to give advertisers the data, and then they can start making decisions off of that.

Eric Enge: Can you talk about the experiments segmentation?

Frederick Vallaeys: One tool that we have is the AdWords Campaign Experiments. That’s a great way for an advertiser to explore how to improve their ROI. They can send 10%, 20%, 30%, whatever percentage they want of their traffic to that experiment. This shows up in your campaign reports, so you can see how the experiment compares to the rest of your campaign. If the experiment is not working well then turn it off and try a different variation.

Eric Enge: It is an A/B test mode you can setup right within the interface.

Frederick Vallaeys: Exactly, in the past if you wanted to experiment, you would take two weeks of traffic and do one thing and then the next two weeks do something else. But the problem with that is, you are not comparing apples to apples because there might be outside factors during those two different periods that caused the numbers to change. With Campaign Experiments, you can actually split your traffic so all of the experiment is happening at the same time as the control and you get much more reliable data about how your changes impact your ROI.

Eric Enge: What about some of the new ad formats?

… we have seen tremendous success with advertisers who run Sitelinks.

Frederick Vallaeys: These were a big thing for us in 2011, and will continue to be a big thing in 2012. For example, we have seen tremendous success with advertisers who run Sitelinks. These are the additional portal links that you can have in addition to your headline in your ad. In the reports, you can segment on that so you can see how many clicks did you got from headline clicks and how many from your Sitelinks.

This will prove the value for the majority of advertisers and we have seen that these Sitelinks actually do work and have good click through rates and good conversion rates. You can start seeing how much of an impact this is causing and for those campaigns where you’re not using it, how much you are potentially losing as a result.

Eric Enge: Can you talk a bit about the bid simulator?

Frederick Vallaeys: It takes historical auction data and if you have bid x amount of dollars or y amount of dollars, where would you have come out in terms of the typical ad rank and what would that have done for your CTR and the number of clicks that you would’ve gotten. Instead of having to do an experiment and changing your bids around to get that data, we take whatever new number you put in and we run it against the past auction data, and model what would’ve happened in those cases. If you go from bidding a dollar for a click to a dollar fifty, is that going to give you a significant increase in the number of clicks.

What you can figure out from this is your incremental cost per click. Incremental cost per click is a number, by the way, that too few advertisers understand and leverage. And basically, the notion of incremental cost per click is simple. It is the cost of the incremental clicks I get by bidding higher. When you know this number, you can figure if the additional clicks that resulted from an increased bid cost more than what it was worth or does it cost me less. The problem is that most people when they look at an AdWords account only look at the big picture.

If you look at an average, what you are not seeing is how did that increase in my bids change the cost of individual clicks. So on average, you might still be under your desired cost per click to meet your ROI goal, but what you are not seeing in that average is the fact that your last ten clicks, the additional ten clicks that you got by bidding higher, actually cost you $2 per click, higher than the $1.50 average, and maybe $1.50 is the maximum you can afford to spend for a click to still be profitable.

Eric Enge: Can you talk about the Ad Preview Tool?

Frederick Vallaeys: The Ad Preview Tool is lets you find and click on your ads in a test mode without paying for them. For example, you can see if the ad you have for people in Milwaukee is going to the right page, and what would someone from Milwaukee see. You put in your keywords and the location you want to test, you can see if your ad would’ve shown up in that case.

Eric Enge: The diagnosis part also allows you to get more visibility into why it is not showing, right?

Frederick Vallaeys: Exactly, so if it is not showing up it will give you some ideas why that might be.

Eric Enge: Can you talk about top of bid page estimates?

What we do now is we also tell you how much you have to bid to show in the paid results above the organic results.

Frederick Vallaeys: In the past, we had first page bid estimates, which tell you how much you need to bid, on average, to be on the first page of search results. That’s the page where most people are going to click on ads, because most people do not go to the second page of results. What we do now is we also tell you how much you have to bid to show in the paid results above the organic results. We also offer segmentation in the reports between top ads and the side of the page ads.

Frederick Vallaeys: I did this in one of my test accounts yesterday and it was amazing. On the right hand side I was seeing a much lower click through rate than on the top of the page. That could be different for other people; but it tells you that this is a lot of potential clicks that I gave up by being on the right hand side as opposed to having bid a little bit more and showing up on the top of the page.

Eric Enge: What about impression share data?

You may find that you can get 30% more traffic just by tuning your bids because you only have 70% impression share.

Frederick Vallaeys: Impression share tells you what percentage of the available impressions your ads are being shown for. It tells you how many clicks you are missing out on by having bids too low, or by not having the right keywords. You may find that you can get 30% more traffic just by tuning your bids because you only have 70% impression share.

Eric Enge: I think few people realize that getting a hundred percent impression share is actually very hard, even for your brand terms. There are cases where people are leaving significant amount of the traffic on the table and that they are busily trying to add new keywords to diminishing returns when there is actually can be 20% and 30% gains by just going through and finding places where they are getting low impression share.

Frederick Vallaeys: That’s a great point. Where you should start is with your exact match impression share, because that is when somebody types in your exact keyword. You probably want to show up on a hundred percent of those. Sometimes your impression share could be lower because you are just not able to afford as high a bid. But even if you are in that situation, maybe it is a great time to go and work on your landing page. Somebody is apparently able to bid higher than you are in those instances and that is probably because they do a better job at converting the customer once they come to that site.

That’s where you can then connect on to Google Analytics and take a look at its flow visualization tools and see if there is some road block somewhere on your site that is causing a huge drop off in terms of conversions. If you can fix these types of things, you may be able to afford to spend more for that click and your 70% impression share goes up to a 100%.

Eric Enge: This is particularly powerful when you start with your high ROI keywords, as it can be easy money. Can you also tell us about the social platform integration in Google Analytics?

Frederick Vallaeys: What people on the web are starting to realize is that a lot of activity around your website, around your content is actually not happening on your own website anymore, and it is happening through social platforms. We are working right now to include some of that data such as likes, and +1s, and thumbs up, and votes and all that stuff that you get on third-party sites and bring it into Google Analytics so then you will have an even better view into how people engage with your brand and your site on the internet today.

Eric Enge: This is an expansion beyond what you talked about before with the social reporting

Frederick Vallaeys: Exactly, it is taking it beyond just the Google properties in these cases, so when it comes to +1s, we have all of that data, we can share it with our advertisers. There are a number of social properties that would be interesting to get some data about how people are interacting with your site and brand. We are building an API so that those other companies can plug into the Google tools and then hopefully they will be able to show the benefit of their platforms to advertisers, because those businesses will start seeing these metrics inside Google Analytics.

Eric Enge: Of the things we have discussed, what are the priorities, where do I start, what do I do first?

Frederick Vallaeys: I would definitely go to all of the segmentations that we have talked about, that is the number 1 thing, just look at those segmentations for your account and start looking for big differences. So, if you see there is a big discrepancy between your mobile performance and your desktop performance or your tablet performance, then that’s a good indicator that you need to focus on that.

Eric Enge: Great! Ryan, any extra thoughts from your side?

Ryan:Ryan Voccola: One minor thing I did want to touch on, we talked briefly about Ad Diagnosis, there is a bulk ad diagnosis feature in the account and that’s under the ‘More Actions’ button which will allow you to bulk diagnose a set of keywords and gain insights without having to go to the ad preview tool.

Eric Enge: Excellent. Thanks Fred and Ryan!

Frederick Vallaeys: Thank you!

Ryan:Ryan Voccola: Yes, thanks Eric!

Filed Under: Google, PPC, Search Engine Marketing

Mobile SEO Tips and Tricks with Cindy Krum

1/10/2012 by Eric Enge 1 Comment

photo of Cindy Krum Cindy Krum is the CEO and Founder of Mobile Moxie, based in Denver, CO. She brings fresh and creative ideas to her clients, speaking at national and international trade events about mobile web marketing, social network marketing and international SEO. Cindy is the author of Mobile Marketing: Finding Your Customers No Matter Where They Are, published by Que Publishing, and hosts a weekly radio show about mobile marketing called Mobile Presence. She writes for industry publications, and has been published in Website Magazine, Advertising & Marketing Review, Search Engine Land, ODG Intelligence, and quoted by many respected publications including PC World, Internet Retailer, TechWorld, Direct Magazine and Search Marketing Standard.

Cindy also served as the co-chair of the SEMPO Emerging Technologies Mobile Web Task Force, and is an active member of the search community. She is passionate about bringing creative online marketing solutions to clients, and working with clients to develop high level, integrated mobile marketing strategies.

Cindy Krum December 5, 2011>Your browser does not support inline frames or is currently configured not to display inline frames.

Mobile Search and Mobile SEO – Full Transcript

Eric Enge: Hi, this is Eric Enge. I am the president and founder of Stone Temple Consulting an online marketing firm, and I am here today with Cindy Krum the founder of MobileMoxie. Thanks for joining us today!

Cindy Krum: Thank you, happy to be here.

Eric Enge: Cindy, you have a lot of background knowledge on the world of mobile search and mobile SEO, so I am hoping to talk about that a bit today. Maybe you could share some of your thoughts on mobile market growth.

Cindy Krum: Sure, so there is obviously been loads of mobile market growth both in the US and internationally and especially in the past years some important things have happened, such as Android overtook iPhone in terms of Smartphone penetration. Then, beyond that Smartphone penetration overall has grown to a really healthy rate where, as much as half of the marketplace would be controlled by Smartphones.

It used to be, we were dealing with feature phones, or mid-level phones that had a Smartphone capability, without a true web rendering experience. So that’s all good. I don’t talk a whole lot about the statistics because it’s difficult in the mobile world to talk about statistics because they come from a lot of self-interested reporting bureaus who have an interest in showing particular kinds of growth.

That’s true of most statistics, but also just because in the world of mobile we don’t have a lot of clean definitions, with the midlevel Smartphones, or the Smartphones that don’t have true web browsing. You would market differently to a Smartphone that doesn’t have a true web browsing experience like a Sidekick or BlackBerry Curve or things like that where it says that it’s a Smartphone and it does email and it does lots of things, but it still isn’t very good at rendering the web.

So, there are three or four subcategories within mobile, you have feature phones, you have the midlevel Smartphones, you have the Smartphones that have true web rendering. And then, some people include tablets in the mobile world. When you are looking at statistics that say there is a massive growth in Smartphones or there is a massive growth in mobile, you have to ask those questions before you can really understand what those statistics mean for your marketing campaign.

Eric Enge: Understand. So, that makes it very challenging to understand where things are really at, especially when you try to compare one set of stats from one source with another set of stats from a different source.

Cindy Krum: Exactly, so if people try and drill me and question me about stats where one set says this is this and another set says this is that, but how could those things both be true? They could both be true depending on the methodology they used in their reporting and what they were considering as mobile and what they weren’t. So, there are a lot of questions to ask to get a meaningful set of statistics.

Eric Enge: Right, if we talk about the difference between mobile and desktop search, there are a number of interesting aspects to that, isn’t there?

Cindy Krum: The most obvious difference is the screen size, where you get a whole different experience between how much real estate you can get and the importance of that, and then also the interaction between organic and paid results. And then, of course there are differences in the results themselves because the search engines and especially Google, when you submit a query from your phone, they look at the phone that you are on in the user agent string that passes through their servers and so they can see what’s phone you are on, and they will adapt the results, sometimes it will make slightly to fit your phone.

So in some cases, you will get the results that are in a different order if you do the same search from two different phones, or results that are presented differently like with a map or without a map, or with a video thumbnail or without a video thumbnail. So those things are different as well. There are so many different things to consider that it can be very difficult to predict what the results will look like on all the different phones so I have actually build a tool to help people with that.

Then, on top of that, there is the inclusion of location in your queries. So if you have a phone that has a GPS or quasi GPS capabilities in it, they will try and add your location into the search query and use that to again change, or update, or inform how the query result should be presented to you.

Eric Enge: Right. And also, users because they are searching from a mobile device, there is evidence that they are closer on average to conversions, isn’t that so?

Cindy Krum: Yes, but a different conversion, one that’s harder to measure. If you are used to being an online marketer, you know what’s really easy to measure is traffic that converts online, but for the most part when you are doing mobile marketing, you are driving a lot of offline sales actually and you are driving foot-traffic which makes it difficult to measure. One of the things that a lot of companies are struggling with is to prove to management the value of mobile SEO for driving foot-traffic or for driving offline conversion because it’s hard to show that in your reporting.

Eric Enge: Right. I mean no better or worse than your average TV ad too?

Cindy Krum: Well, that’s not necessarily true. There are things that you can do that make it slightly better than a T.V., just an average T.V. or radio ad, but essentially it’s the same hurdles. Yes, it’s a bit more of a broadcast message that you push out, but it’s not as much, like in T.V. and radio I feel like a lot of the times you are closing your eyes and crossing your fingers, but there is more you can do with mobile to get slightly better measurements.

Eric Enge: If we are trying to rank well in mobile SEO, it seems to me that there is a few basic questions to start with and one of those is what platforms you intend to support where you know, at a very high level choices of things like feature phones, Smartphones and tablets. What are your thoughts on that decision making process?

Cindy Krum: Well, there are a lot of things you can do to have a good experience across all of them. It more depends on what your resources are in terms of development. When you are thinking about SEO, you can focus more on the tablets and the Smartphone than on the feature phones. Now, a lot of mobile marketers hate to hear me say that because the feature phones still make up a big portion of phone owners even in the US, but studies show that people with feature phones are less likely to search and less likely to be on the web even if they have web capabilities on their feature phone, they are just not as active because it’s such a bad experience. So, you want to focus your energy and focus your planning on the Smartphones because the better the phone the more likely the user is to really engage.

Eric Enge: Yes, understand and I guess you can just use your analytics to see whether you are getting feature phone visitors, right?

Cindy Krum: Sometimes, but sometimes no. There is this big trap in analytics where the older feature phone can’t execute JavaScript and thus can’t do cookies. So in some cases, you will be in a situation where you look at your analytics and you say oh, we have no feature phones at are all or we don’t have any of us x, y, z phone so we are not going to worry about that at all. Well, it could be that you have none of those or it could be that they just aren’t setup to track, they can’t execute the JavaScript necessary to be tracked in your analytics program.

Eric Enge: Right.

Cindy Krum: So you don’t, I never say write them off entirely, but you can definitely prioritize your efforts towards the Smartphones and the true web browsing experiences and true web browsing search.

Eric Enge: Right. And, Google has a separate bot called Googlebot Mobile, right?

Cindy Krum: That’s true. It’s interesting that they don’t, it’s hard to tell exactly what the point of Googlebot Mobile is anymore except that Googlebot Mobile does seem to focus on feature phones content or older text-based WAP sites. And so, for a while, the people in the SEO, the mobile SEO space figured that they would update Googlebot Mobile to address the Smartphones and do a better job indexing and organizing for Smartphones, and it doesn’t seem like that that’s happened.

It seems like Googlebot Mobile has really focused mostly on feature phone content, and that they are doing the mobile search queries that come from Smartphones from the traditional Google algorithm, and they are so doing slightly differently, but it seems like the indexing with Googlebot Mobile does not, as far as I can tell, have a super duper strong effect on your rankings in a mobile search from a Smartphone just from a feature phone.

Eric Enge: Yes, it’s interesting and that leads to the next big question that people face, do they implement the mobile sub-domain or do they implement something on the same URLs. Essentially so, if you are on the mobile device you are seeing the content and exactly the same URLs as you do on your desktop, but you see mobile phone version when for the content. What are your thoughts on that discussion?

Cindy Krum: So, there is a couple different ways you can go. For a very long time, I have said your best option is always to use the same URL on mobile as you do for your desktop searchers. The reason is because you get to leverage all of the links, and history, and good stuff, good SEO efforts that you have been doing on those URLs. You don’t have to start over. If you wanted to do an “m.” mobile subdomain, which is very common, that’s fine, but you have to acknowledge, in terms of SEO, you might be starting over to get those “m.” pages to rank.

Now, even if that’s the case, it’s not the end of the world because what we are seeing now is that if you have page-A on your desktop site and you have same page, but a mobile version of page-A on an “m.”, and they both have the same keyword saturation and you know, essentially the same SEO indicators. When you do a search from an iPhone or an Android phone, anything with a true web browser, what’s going to happen is that traditional page (the desktop page) is still going to outrank the mobile version of the page.

Possibly, because it has more history and links or possibly because the Google algorithm just isn’t that good or isn’t sophisticated enough to say hey, this is a mobile page and that is traditional page, these are the same thing and they are on the mobile device so we should rank the “m.” one better. So, you see lots of problems with that where people will build an “m.” site and they will say look we did everything you said we have these light, grey, awesome pages that work really well on mobile phones, but Google is still ranking the traditional versions better.

I don’t expect that that’s going to happen forever, but the best solution or the best way to share the values that you have created on your desktop site with your mobile pages is to setup what’s called user agent detection and redirection on all of your desktop pages and you can even do it on all of your mobile pages too. And, the idea is that it detects if you are on a mobile phone or not and redirects you to the right version of the page.

So even if your desktop site outranks your mobile site in a mobile search, when people click on the desktop version of the page, it will get them to the mobile version of the page. So, from the search results, even if the desktop one is there, it gets sent to the mobile version of the page. So, that’s a really good, strong, solid workaround; it’s not considered cloaking or anything like that. Google has come out and said that this is okay because it’s for the benefit of your users, as long as the pages, as long as you are not trying to do anything sneaky between the two pages.

Eric Enge: Right. And, I believe Google has also said that it isn’t cloaking provided that you are also bringing Googlebot Mobile to those mobile pages, right?

Cindy Krum: Yes.

Eric Enge: That’s one of their requirements. So, there is a version of Googlebot that is doing the exactly the same as what’s the users see.

Cindy Krum: And, you can even take it a step further and I always offer this recommendation with lots of caution, be careful when you do this, but you can canonicalize the mobile versions of your pages up to the traditional desktop ones, so that you consolidate the SEO value. As long as your user agent detection and redirection is really strong and reliable, you can use that rel=canonical tag.

Eric Enge: Right, yes because otherwise if you have a mobile sub domain the mobile sub domain might accumulate some links which aren’t really occurring to the benefit of the overall site.

Cindy Krum: Right.

Eric Enge: So, I guess one of the reasons why someone would do a mobile sub domain rather than the same URL strategy, if I understand this right, isn’t the complexity of the development much higher when you are dealing with feature phones support plus Smartphone support, especially giving that there are so many different form factors with feature phones. In that case it is easier to deal with an “m.” approach from a technical perspective?

Cindy Krum: It does, and what we find is if you do a mobile site that’s targeted at midlevel Smartphones, but not your, not necessarily the true web rendering Smartphones, then you do a good job for both worlds so the real older feature phones and it should still look fine and you know, work well on the true web rendering phones. And then, it will also force you to keep your page size and load time low, which is a good indicator for Googlebot Mobile and just better for your users in general.

Eric Enge: Right, so that way you are sending positive signals in terms of bounce rates and things like that.

Cindy Krum: And, mobility. Remember that Google is rearranging the rankings based on what’s going to work best on the phone and in general what’s going to work the best on the most mobile phones are clean fast loading pages.

Eric Enge: Yes. So I guess it would be fair for you summarize what we have talking about, it would say that if we are doing feature phone and other mobile device support then arguments for going with the mobile sub domain are stronger. If you are just in Smartphone on up, we would lean towards the same URL implementation, but mobile sub domain with good user agent detection and rel=canonical is okay too.

Cindy Krum: Yes, and there is trade offs all over the place. So if you go with the same exact URL, that’s fine, but that doesn’t mean that you should go with the same exact design. You can do things on the server or with the style sheets to rearrange how that page lays itself out on the mobile phone, because what you want to avoid in terms of in terms of user experience is you don’t want to send people to, people on an iPhone to a page that’s formatted for their desktop, because it’s like 1995 all over again.

I mean with the left to right scrolling, that’s miserable. You don’t want to give people that experience. That’s one of my easy to determine if this is a good mobile design is if there is left to right scrolling or if you have to pinch and zoom to interact with the page or to even understand the page, that’s also not a good user experience.

Eric Enge: Right, and the other thing you want to do even in the same URL is do things to make the page a little lighter weight without fundamentally changing content of course, but you don’t want to be loading a 150 K.

Cindy Krum: Yes, so what you have to do, one of important tests is that you want to make sure that user agent detection is detecting the mobile bot in the same way as it would detect the mobile phone, and that’s the separate test that you want to make sure that you have done to make sure that the mobile bot is triggering the mobile version of the page.

Eric Enge: Yes.

Cindy Krum: Even if it’s on the same URL, so that it can index that information rather than trying to index the desktop site.

Eric Enge: Thanks so much for joining us today!

Cindy Krum: Thank you!

Filed Under: mobile, SEO

How Bing Is Rethinking The Way We Search: Part 1 w/ Stefan Weitz

12/19/2011 by Eric Enge 3 Comments

photo of Stefan Weitz Stefan Weitz is a Director of Search at Microsoft and is charged with working with people and organizations across the industry to promote and improve Search technologies. While focused on Microsoft’s product line, he works across the industry to understand searcher behavior and in his role as an evangelist for Search, gathers and distills feedback to drive product improvements. Prior to Search, Stefan led the strategy to develop the next generation MSN portal platform and developed Microsoft’s muni WiFi strategy, leading the charge to blanket free WiFi access across metropolitan cities. A 13-year Microsoft veteran, he has worked in various groups including Windows Server, Security, and IT. Stefan is a huge gadget ‘junkie’ and can often be found in electronics shops across the world looking for the elusive perfect piece of tech. You can follow Stefan on Twitter

Key Points from Interview with Stefan Weitz

In this interview Stefan outlines the key areas where Bing sees itself diverging from Google. The discussion provides a clear and direct look at the way Bing plans to build its market share over time. There are three major components to Bing’s strategy:

Become a personal assistant for the searcher, one that knows enough about you to highly customize the search experience (see the discussion about Mission Impossible below)
Move away from a search box and make it totally transparent (for more on this read the discussion of the Xbox Kinect below)
Focus on partnerships instead of acquisitions, allowing Bing to leverage the creativity and accumulated data of others

Stefan also sees great algorithmic search as “table stakes”, but that the real value add in the future is come through additional layers built on top of the raw algorithmic piece. These layers will handle the personalization, the embedding in different platforms, managing partnerships, etc.

This summary covers the basics. Please read the full interview transcript to get the full impact of the discussion.

Full Interview Transcript

Eric Enge: During our last interview we talked Google being very algorithm-focused, and how Bing was going to take a different path. I suspect that this divergence is beginning to grow. Is that right?

There is this shift in how we view the web and what it can do and the way Google does …

Stefan Weitz: Yes. Let me take you a step back, and cover some new stuff we haven’t talked about before. This bifurcation you referred to is happening. There is this shift in how we view the web and what it can do and the way Google does, and neither of them by the way are bad and they both are necessary, but there is a difference. They have done great work focusing on index size, index freshness, speed optimizations, and user experience models. They offer a great keyword search experience, and that’s good. On our side we obviously have to do an amazing job with that core index, with retrieval of URLs based on two and a half keywords per query.

… in many cases we actually outperform Google for algorithmic search and in almost all cases we are at least as good or better.

All that stuff is table stakes, and we’ve always known that. With a couple of our more recent algorithmic updates, in many cases we actually outperform Google for algorithmic search and in almost all cases we are at least as good or better. But, there is this new thing, this notion that the web itself has changed and continues to change at an accelerating pace.

Bare Search Pages Search really is predicated on the structure of the web, and as it changes, search needs to change with it. Over time search looks less and less like a search box on a web site. It looks much more like a service, almost like a platform that spans across any input modality from a gesture on your Xbox, to voice on your phone, to touch on a tablet device, to even implicit queries that you don’t even know necessarily are happening on your behalf.

Of course, the service has to be personally useful. We want to create something that takes into account who you are, who you know, what you do, all those types of things, and then focuses on delivering experiences that help turn those ideas of things you want to actions. To do this well, we have to do an amazing job of brokering resources from all across the web to enable new levels of functionality.

In our view, search will start to be less about the traditional keyword to URL model, although we know we have to do that extremely well, because that’s what folks expect today. But, we see both the fun and innovation is around this notion of a horizontal service that can span across all of our properties, across other people’s properties, and across all devices and input methods that literally allow people to make the best use of the web that they can, given who they are, where they are, what they want to get done.

Eric Enge: Part of what you are talking about a distribution of search, so instead of going to a place to search, search is where you are already.

Today you are in the flow of doing something, and you have to stop and shift and go do a search.

Stefan Weitz: Exactly, it will be ubiquitous. Today you are in the flow of doing something, and you have to stop and shift and go do a search. This is similar to what it used to be like to connect to the Internet. You used to have to “dial up” the Internet. Today it is always available and you don’t have to think about it. We believe the same transition will happen with search. You will no longer have to think about taking some actions to do it because it will be embedded in the flow of what you are already doing, right where you are already.

In addition, search is much bigger than one single entity.

In addition, search is much bigger than one single entity. A lot of what people are looking for resides in other services and databases that are not readily available online. We are working hard to develop access to that information by setting up partnerships with other companies and acting as a universal broker to integrate that content directly into search.

Eric Enge: Can you help illustrate this concept with an example scenario?

Stefan Weitz: Sure. One I like a lot is a feature in the Bing iPad application called Lasso. When a user views a page, they can just take their finger and circle something. The act of you circling it gives us some intent cues. Let’s say they circled Mission Impossible. We then do a re-query, and use all of the power of Bing’s backend to analyze it. We know that Mission Impossible is a movie, and movies have attributes like show times, reviews, trailers, pictures, and casts. We return back in the iPad interface this really beautiful page that has all these components that describe the movie right there so you can take action.

You can click a button, buy a ticket, and see the restaurants around there. That’s a really simple example of where we are taking this notion of integration, where we recognize the context of your action, circling something on the device, and then build an experience on the fly that has all the information and services in the right place so you can actually do something.

As you begin to think through how that could evolve, it gets a little more interesting. There are actually four Mission Impossible movies. There is also the TV series, and video games, and all sorts of books. Traditionally, what search is going to do is going to look at those keywords, and attempt to use static rank and figure out what the most relevant Mission Impossible link would be for most people. It might assume that the most relevant result is one of the older ones because it has more static rank to it then the brand new movie.

In our example, perhaps the page is talking about the latest Mission Impossible (MI) film. The opportunity for us is to take the global context of that page (what the overall page is talking about), the engine should know that what I am really asking for is the latest Ghost Protocol film, not Mission Impossible 1, or the series from the sixties or Peter Graves or anything else.

That’s the first level of understanding more of the context behind the query, in this case me circling that MI link. Next is to start thinking about all the different things that I can actually do, and that’s where this notion of experiences and actions come into play. Perhaps the engine knows that I’ve rented Mission Impossible #1 and #2 from Netflix, and it can then makes the suggestion that I want to see #3 before I see #4. Next, it knows that I am in my office in Bellevue today, and my calendar says I am busy till four, and the most logical place to go see it might be the one closest to me at six.

The idea is to be far more intelligent about the things that I personally want to do based on the various signals available.

The idea is to be far more intelligent about the things that I personally want to do based on the various signals available. This gets away from this notion that all objects are the same to all people, and in this example it focuses on how I want to interact with this particular object given all the information the system knows about me.

Eric Enge: This is a permission-based system, where you share details about your intent, and you get a personal assistant in return.

Five years ago this would have sounded like science fiction, but now it’s fairly trivial to do …

Stefan Weitz: Yes. Five years ago this would have sounded like science fiction, but now it’s fairly trivial to do, but it requires a different view of the search experience.

Eric Enge: This is an extension beyond the original notion of becoming a decision engine because you are trying to add another layer of insight to it.

Stefan Weitz: One example that I used to use was the ability we developed to look at all the reviews on the web for a consumer device, such as a Smartphone. Say there were a thousand reviews of a device; we developed the ability to actually have machines read all those reviews, segment the reviews into what the reviews were about, and then apply sentiment analysis to see what they were saying. For example, we may determine that 48% of them had something positive to say about the battery life.

… the emotional component of decision-making is in many cases more important than the rational one.

Neuroscientisit Jonah Lehrer What we did is try to take all that data and make it into knowledge that people can use to decide to do something. That’s exactly where we began, but then just in the last year or so we began to look more at this notion of how people make decisions. I’ve done a lot of work with some neuroscience people out there, and then the research guys have done a lot of work as well to look at this. And, people like Jonah Lehrer, who is a great neuroscientist, talk a lot about the fact that the emotional component of decision-making is in many cases more important than the rational one.

The lizard brain inside the human cortex is the first place that gets invoked when you have some stressful decision to make. So, we began to look at this notion of how people make decisions. We had built rating tools, Bayesian model predictions for traffic, and all these really cool tools that we’ve built to help people rationally make decisions.

… we are actually literally bringing in that other critical piece of decision-making which is who you know, what they know, and how much you trust them.

I like to think of myself as a rationalist, but, it’s the other side, the right side of the brain that we were missing. That’s where a lot of the work with Facebook and Twitter and all the different social signals we are getting really comes in, to augment the ability for people to make decisions with Bing and to take action with Bing. It’s not just tools, but now we are actually literally bringing in that other critical piece of decision-making which is who you know, what they know, and how much we trust them.

Eric Enge: Can you see Bing collecting more information from people then what they give in Facebook and Twitter? Perhaps collecting profiles directly from the users?

Stefan Weitz: For years people have tried to collect profile data as you know. The number of people who actually opt to give that data is quite low, not because of privacy concerns, but just because they don’t see the value of it.

There is some more explicit stuff that we think we can do, but we are also very aware of the fact that people have been trying to do profiles on the web for fifteen years and have very limited success. It won’t work unless you can tie it back to some direct benefit that the user will see.

Eric Enge: Yes, because the person needs to take time to enter that information, and as we know in our wonderful world of the web, seconds are an eternity. Can you talk a little bit about the new arrangement with Twitter?

Going forward you’ll see more integration into the core search experience. We’ve been experimenting with what else you could do with Twitter.

Stefan Weitz: It really is an extension of what we had before. That being said, we did it because we value that data. We still use it to figure out when new things are happening, and that helps our news index find out what’s going on in that particular area. Going forward you’ll see more integration into the core search experience. We’ve been experimenting with what else you could do with Twitter.

Today we put it on a news page, for example, so when you are on a news article you might be reading about the latest Dell laptop, you can see relevant, de-duped, high quality tweets right in that page. So, you’ll see more of that type of thing from us rather than just paste it across all pages. We think there are cool things we can do by looking at what people are trying to do, and what decisions they are trying to make.

Eric Enge: Do you see yourself having an ability to extract information about the person to use in personalizing their search results from Twitter as well? For example, you may find out that they are currently looking at a trip to Rome because I asked three people on Twitter about that.

With adaptive search, we look at the queries you are issuing, understand the category of those queries, and then begin to re-rank future search results given the category.

Stefan Weitz: That’s a really good idea, I am not sure we’ve got that in our plan yet, but I can see how it might work though. One thing we released is this thing called adaptive search, and the reason I am bringing it up is because I can see how what you just said applies. With adaptive search, we look at the queries you are issuing, understand the category of those queries, and then begin to re-rank future search results given the category.

The tweets could work the same way where you have been tweeting a ton about your cocker spaniel, and then you search on “dog movies”, we might emphasize movies that have cocker spaniels in them. That might be the most bizarre example ever, but you get the idea.

Eric Enge: Now you’ve got Facebook and Twitter together, and that’s great. This smells like a shift away from having such a high level of algorithm focus and more towards people-oriented signals.

Stefan Weitz: That’s right. When you think about the initial algorithm it’s all static rank really does, and all PageRank really does, is use people to rank pages, because at the end of the day its people who contribute links, and people who use anchor text.

But now, links don’t have to be our only signal map. Now we can use the amazing social graph inside of Facebook and to an extent Twitter. And of course, even more importantly then just the connections of people to each other, it’s what I, Stefan am doing across the web on all of the different sites, not just on Facebook. It’s just marvelous.

Eric Enge: Tell me a bit about Xbox Kinect.

It gets back to the core principles of Bing: a) Bing can be integrated everywhere, and b) there can be many different kinds of interfaces.

Stefan Weitz: I’ve been looking at voice recognition since I was a kid. The software used to be horrendously expensive, and the quality of the recognition was frankly not that good. But today, speech recognition is fairly good across mobile devices and across desktops; in fact, it’s stunning how good it is. The Kinect work is literally a revolutionary way to interact with a computing device. In Kinect the device is controlled entirely by voice and your body movements. It gets back to the core principles of Bing: a) Bing can be integrated everywhere, and b) there can be many different kinds of interfaces.

The last thing I think that highlights really well is this notion of partnerships, and how we at Bing do a lot more partnerships than we do acquisitions.

The last thing I think that highlights really well is this notion of partnerships, and how we at Bing do a lot more partnerships than we do acquisitions. We look at partnerships as a way to accelerate the transition of search away from keywords to this more action-oriented model. Partnerships allow us to leverage the great content and tools being built by others. It also allows us a great deal of flexibility in combing the value of one third party service with that of another third party service to create something of much higher value.

Eric Enge: Can you talk a bit more about the mobile side of things?

Stefan Weitz: I was in New York a month ago. I walked into a little book store which had these amazing books, absolutely gorgeous, and there were many coffee table books, the big, huge, heavy $40 books that were far too big for me to carry on the plane although I am sure I would have tried if I had thought about it hard enough.

Windows Phone with Mango I had my phone with me and I actually just pointed my phone at the book I saw on twentieth century architecture. Sure enough my Windows Phone with the Mango OS found it, and showed me all the places I could buy it. I was curious if I could get it in Seattle because it was a huge heavy book. It showed me where I could do that, and it also offered me a way to buy it online from the shop I was standing in!

To do this the camera on the back of the phone had to understand what it was seeing. Then it kicked into the search system itself which understood it was a book, it understood the title of the book, and then it actually issued a query to Bing’s backend for shopping and found the places where I could buy it, and displayed them right there.

You have the same thing with the translation on the device, you can point at something, hit a button, and it translates it into twenty-three languages. You have the same with music in the device now. If you are sitting around and you hear something on, you can hit a button and boom, it tells you what the song is, how much it costs, and where you can buy it right away.

This new type of mobile functionality is awesome, because you always have it with you.

Eric Enge: Mobile often also comes with the context of being out of your office or house as well.

I think it also speaks to one of the big tenets of our vision which is that when you are using a phone or a tablet or a mobile device of some sort, the last thing you want to do is have to actually go search.

Stefan Weitz: A mobile device really is the most personal of personal computers because you often don’t share your device. All your contacts are in there, all your social networks are on there; you can get all your documents on there. Everything is on there, and it knows where you are, and you can let it know where you are going. I think it also speaks to one of the big tenets of our vision which is that when you are using a phone or a tablet or a mobile device of some sort, the last thing you want to do is have to actually go search. It just seems wrong, because likely you are doing something else at the time.

You want the device to be smart enough to be executing things on your behalf, and then doing things. If you are on Bing maps, for example, and you are trying to figure out where you are, you can tap on the little city icon on Mango, it’s called Local Scout. Local Scout fires up and shows nearby places to eat and drink, upcoming shows in the area, and attractions all around you. It’s like nineteen queries in one. It’s doing all these things, pulling them back based on where you are, what day it is, and all this different data at the search backend. This is search in context, and its way different than the traditional notions of search.

Eric Enge: It sounds like you have a core underlying algorithm, and then you have a whole layer on top of it which is the layer that adapts everything based on context. The context layer becomes the place where really the bulk of the added value lies. So, the layer at the bottom becomes a commodity.

Stefan Weitz: You are actually almost exactly right. There are a bunch of other services such as personalization, social services, and the actual broker service that understands what services exist on the web and how they can be used to help the person accomplish a task. There is the object service which understands that this thing is a book or this thing is a bottle of vitamin water, and what that means, and what you can do with it.

We actually just created a whole team under Brian MacDonald now that is focusing literally on that what you just said, not customization, but the dynamic experience given the intent, the context, and the access device.

We actually just created a whole team under Brian MacDonald now that is focusing literally on that what you just said, not customization, but the dynamic experience given the intent, the context, and the access device. For example, on a mobile device, you don’t want to display a bunch of links. It makes no sense because it’s a small screen.

We need to take all of those things into account, and then let the magic of Bing stitch all those services together into some experience that makes sense given all those different variables. That is where we are pushing the future of search at Bing.

Eric Enge: Thanks Stefan!

Stefan Weitz: Thank you Eric!

Filed Under: Uncategorized

News Revenue Optimization Podcast with Dennis Mortensen

12/1/2011 by Eric Enge Leave a Comment

photo of Dennis Mortensen Dennis Mortensen is the Founder and CEO of Visual Revenue. He’s a pioneer and expert in the analytics, optimization and online marketing space and has been since its inception – he is also a fully-fledged entrepreneur and successfully delivered a number of company exits. Dennis is obsessed with the marriage between News Media and Analytics and this passion triggered the formation of Visual Revenue. He’s an accredited Associate Web Analytics Instructor at the University of British Colombia, the Author of Data Driven Insights with YWA from Wiley and a frequent speaker on the subject of analytics and media.

Dennis Mortensen November 11, 2011>Your browser does not support inline frames or is currently configured not to display inline frames.

Optimizing Revenue for News Sites – Full Podcast Transcript

Eric Enge: Hello, everybody. This is Eric Enge, I am the CEO of Stone Temple Consulting an online marketing firm that does SEO, Pay per click, social media consutling. Here today, we have Dennis Mortensen the CEO and founder of Visual Revenue, who is known in the industry for his also being founder of a company called Index Tools, a leading analytics package that got acquired by Yahoo a few years back. Thanks for joining us today, Dennis.

Dennis Mortensen: Thank you very much.

Eric Enge: Alright. So, I think we are going to talk today about the media industry and the online environment for that industry. And, I was hoping we could start by talking a little bit about just how far along traditional media players are in their shift towards a focus on online?

Dennis Mortensen: So, I think you can look at this in two ways. You can look at it from a view of how far they are in processing their revenue from offline to online, or you can view at, look at this problem from a viewpoint where you think of how far they are in their thinking of the fact that their business should move online. And, I think from a monetary point of view that’s certainly not where they want to be. I think from them deciding that they must move from a printed channel to an online channel, I think they are pretty far ahead. And, most of what we read which is that they don’t get it, they will never get there, half will die in the next six months is in fact too aggressive.

Eric Enge: So, they’ve got the basic idea, but they’ve got work to do in terms of getting there. So, that would suggest that they are facing a number of different challenges along the way. Can you talk a little bit about what those are?

Dennis Mortensen: Certainly. So, my opinion is actually slightly aggressive in the viewpoint of a journalist. But, if we are just honest for a second here, what they do and forget about the idea of some institutions keeping democracy in place or them being the fourth estate, think of what they do as not very different from any other business. They produce a set of products. Those products just happen to be content. And, if you are in the business of producing content, you should make sure that you have a decent return on that investment.

If you are the telegraph, you might produce eight hundred pieces of content on a daily basis. You want to make sure that you have a fair return on that. And, I think if you look at it from that viewpoint then you do have multiple challenges. First of you should figure out which products, and with that I mean which specific articles should I produce.

Even before you write that first word, and forget about the specific responsibility which you have, think about this at least for a moment as a business. And, that’s the first challenge that most of the production is based on gut feel. There is not much science really that goes into figuring out the demand. The demand is really figured out in a way where the editor decides that you must know this because you are a citizen of Connecticut, and thus this must be on your agenda. Some of it I think is justified; I think a lot of it might not be.

The first challenge as an editor or an organization and as a publisher you must figure out that demand. Then I think the second challenge is around being able to produce the content in the most cost effective way. So, that means some content, you should probably not even produce that, it’s been produced already. All you should do if you believe this is important to your audience, you should link to it. You should see no shame, just like you are seeing folks like The Huffington Post, or any other blogger being willing to do, link to a story or you might choose that this is the story that we are not going to write an in-depth story about how we believe this should be approached over the next four days or five days, you might just write a short abstract and that’s that.

Or, it might be one of those stories where you believe that we should heavily invest. This is the one way I should stand my own photographer, or we should do a small video that goes along side the story itself. We should follow-up tomorrow or we should create a topic pace, we would go all in the story. So, there are certainly a set of decisions around how to produce in the most cost effective way. That of course means also selecting where to invest, that’s the second point.

Then comes the third point, and I think this is where most people start today. They almost forget the two prior points which says demand, yes somebody figured that out, produce yeah we’ll do what we did yesterday. But, the third point is the ability to promote this content as effective as possible into the channels that you want to participate in. When I speak of channels that can be anything from your own homepage or your own section front to social and every in connection whether that be Facebook or Twitter through your RSS feeds, through your emails, through what have you.

Then finally, I think the last element which we need to take into consideration, and you can’t forget is how people are going to consume it. So, the consumption itself could be selective. It could be that some content doesn’t go on my website, it only goes in the iPad version, that’s the willingness of somebody being okay with paying me four ninety-five, they get the luxury of having a certain set of content. I am not saying that’s right, I am saying that’s an opportunity. And, you certainly need to kind of figure that out.

It kind of happens already. So, if you signed up for the New York Times RSS feed, you don’t receive five hundred stories a day, you receive a selection. So, we need to take that into consideration. So, that’s my view of the world which is very firm on figure out the demand, produce it the most cost effective way, promote it to the maximum extent into every channel that you want to participate in, and make sure that the consumption is thought about before you just randomly and haphazardly just post stories out to everyone single channel.

Eric Enge: Right. So, if I were to try to summarize all of that, you go from a world where you always produce a comprehensive piece, and you have a captive audience inside a traditional media vehicle like a newspaper, to one where the production changes to scaling to the incremental need. In other words an abstract that links to somebody else’s piece or comments on someone else’s piece might be all you produce because that’s the incremental need in your piece. So, that’s one thing that’s different. The promotional channels are different. Instead of having subscribers and going out to newsstands you are promoting in methods where the quantity that might be consumed changes in a dynamic way from day-to-day depending on how effective you are producing things that meet the incremental content need.

Then, the third thing is recognizing that you can get audiences not even just through websites, but through maybe as a mobile version, an iPad version, and various things. I mean now you have to tailor to a lot of different environments, and when I add all those things up that’s a lot of changes in a pretty different structural, or a very different sounding company and a traditional media company.

Dennis Mortensen: I think you are right, but I think we have other industries which we can learn from. So, if you and I were to walk into the trading floor of Deutsche Bank on Wall Street this morning, I’ll be very surprised if we would not see two hundred something traders looking into a Bloomberg up terminal. And, they would get efficient support for whether to trade a given stock. They might even have some automated trade setup so that’s when a stock reaches a certain threshold, a sale or buy will happen automatically, or they might even go completely against the market, but knowing that they are doing exactly that.

If you and I walk into the newsroom of some organizations, would you see a hundred a fifty people looking out the window trying to decide what to do. If that was the setup of Deutsche Bank, that is not where you are going to put your money, it seems almost ridiculous that they are about to setup a new fund saying so what is your trading philosophy, you know what, we are just going to wing it. The fund is going to be a $100,000,000 and we are going to wing it; that seems silly.

Eric Enge: Yes.

Dennis Mortensen: It seems okay for media to go out with similar sized budgets. And, figuring out demand, figuring out production, figuring out promotion in a nonscientific way, I do understand that the difference between trading a stock and publishing a news piece in a local region is different. But, I want to, and I am saying this with the utmost respect. I do think that some of these applications are businesses like anything else. And, you really need to be unique to hide behind the idea that your responsibility goes above and beyond revenue, which is that you are supposed to uphold democracy, or you are the fourth estate, or you are anything which is different from you being supposed to generate revenue off of this content.

So, I am not sure really that what used to be very manual in trading stocks, and what is very manual today which is producing content and promoting content will not be a whole lot more scientific in the future. We’ve already seen some companies, and it doesn’t matter whether we like or dislike Associated Content, or Demand Media. They are trying to apply science to at least the tail end of the market. And, you know what, in current events you see the exact same thing happen.

Eric Enge: Right. And, all this suggests to me, it feels like a very dynamic environment. So, it seems to me like there has got to be a huge need for freshness.

Dennis Mortensen: We’ve done tons of studies at Visual Revenue as part of us building our model on what stories to promote where on the homepage, and so on and so forth. And, you are absolutely right. One of the conclusions that we came to was, if I take just one of our customers as an example, the average article lifespan for them is about eight hours.

Eric Enge: Yes.

Dennis Mortensen: Eight hours, that means from when you produce the content to the point where it ends up in the archive, that’s eight hours. That means your opportunity to create revenue off this piece of content that is eight hours. After eight hours it doesn’t matter. That is an aggressive environment to be in. I think that’s if anything even more stressful than working at the New York Stock Exchange. So, if you want to work in that environment, you certainly need tools, technology, decision support, and other elements to make sure that you can survive in that.

Eric Enge: Right, yes absolutely. And also, it seems to me that you need a lot of tracking assistance so that you can tell what’s going on, right measurement and tracking systems?

Dennis Mortensen: So, I am of the opinion that collecting data in itself is a commodity. Anybody who is willing to collect and store data can do so for little or no cost. Deriving some sort of insight from it comes cheap, and it’s certainly something which you can work at; making sure that you take action and create value out of it that is a lot more difficult. So, when you see people handing out credentials to Omniture or Google Analytics if you will to editors or journalists or people in media in general, I think that if anything, it is more dangerous than not giving them access at all.

I would like an analyst to be the analyst. I would like editors to be editors. So some, and I like the analogy of traders, some people are not supposed to have access to raw data, but the access to raw data and the ability to slice and dice it doesn’t really assure you that you get to right conclusion. That might just be a conclusion which you will kind of pack your stuff on the back and say I am data driven, but it might be a completely wrong conclusion.

Anyway, you would want analyst to figure out whether you should produce more political content, or less political content, or whether you should write your stories longer or shorter, whether you should publish them in the morning or in the afternoon, whether you should have editors in place from midnight till morning, long-term strategic decisions. But, what you want to have in place in front of your editors are decision support systems that collect all the data, model out suggestions and conclusions which they can act on. You can say no thank you, or yes please, but they shouldn’t turn into mini analysts. Well, that’s not what you learn at Columbia over four years, and you are not supposed to.

Eric Enge: Yes, understood. So, can you take a minute or two and tell us a little bit about what Visual Revenue is doing?

Dennis Mortensen: So, absolutely. I would like to give you the daylong seminar, given the fact that we don’t have time for that, what we do is decision support for editors. So, we’ve come up with this model where we can take any piece of content and we can predict exactly how well that will perform in any given position on the homepage or the section front. And, based on that ability we can come up with a set of very specific recommendations for what content to put where, and for how long to keep it there.

So, in essence if you will, we created a really smart editor. And, we are not trying to change the workflow of the process really. All we are really trying to do here is if you will to become that Bloomberg terminal of the newsroom. We want the editors to be empowered; we want them to be able to take better decisions faster that are more profitable. But, we don’t want to take their job; we just want to make sure that they do their job really well.

Eric Enge: Right. So, you actually play the role of being the analyst to help them make better decisions about what, like you said what content to put up for how long, but also as you told me before they have an option of where they can just get it as passive information which they act on or not, or you make a recommendation and they can say yes, let’s implement it, and you actually do the backend part of making the change on the site as well.

Dennis Mortensen: Exactly. So, if we go back to my four points from earlier, what I really do here is once they figure out the demand, and once they produce the content, they reach that point of not being able to turning back the clock or the point of no return if you will. That means now the time starts, and you need to promote, sorry for being blunt, the hell out of that story. And, you have eight hours to do so. That’s where I start. So, once that piece of content entered the article tentative pool, I help you make sure you make the most of it such that telling you that story on Libya; we should put that into the hero spot. That story on Egypt, we are going to put that in right rail #2. That story on economics, that is not working at all, let’s pull that from the front page, put it into the second position on the section front. And, they’ll take me up on some of those suggestions, and some of them they’ll leave. But, the value of doing this and the change in the workflow process that we’ve seen is that they end up doing many more updates during the day, much more correct, and at a much higher value.

Eric Enge: Yes. And, that sounds very exciting. And, you obviously have landed a number of significant brand name customers already, which indicates that you are getting a good acceptance for this notion among major publishers. I see CNNMoney, Wall Street Journal, Forbes people like that.

Dennis Mortensen: I think we’ve been able to pull this off, but we started out with the idea that it was not us against the editors. It was us and the editors against the rest of the world. And, that means that any of the modeling that we do, we do that with what we call editorial tone modeling on top of it, simply assuring that I don’t turn The Financial Times into the New York Post or vice versa because they are supposed to stay exactly who they are. So, we assure that we don’t make recommendations that are outside the tone of who they are, and the integrity which they have we keep that intact.

Eric Enge: Right, now that’s great. Hey, thanks for joining us today, Dennis.

Dennis Mortensen: You are most welcome. Thank you very much for having me.

Filed Under: Analytics, News Optimization

Cool New Quality Score Metrics from AdCenter

11/15/2011 by Eric Enge Leave a Comment

Ping Jen is a Product Manager on the Microsoft Advertiser and Publisher Solutions Team. He has a passion for driving improvements into adCenter which helps advertisers optimize their campaigns and increase their competitiveness in the marketplace. Prior to joining Microsoft in 2009, Ping was a Business Administrator at University of Cincinnati Department of Neurosurgery. Ping is a Microsoft Certified Solution Developer (MCSD) and holds a MBA degree from the University of Notre Dame.

Briefing with Ping Jen

Ping Jen and I connected for a call last week and reviewed some of the current developments with Microsoft adCenter. Today’s post will review the main items we talked about and what they mean for adCenter advertisers.

New Quality Score Data Provided

1. Historic Quality Score History: adCenter now allows you to monitor the Quality Score of a keyword over time. One reason this is important is that the most common question that the adCenter team gets is: “What does it mean if we see our Quality Score drop on keywords when we have not made any changes recently in that campaign?”

Great question! What it means is that your competition has been doing optimization work that is causing their click through rate to go up. As a result, your Quality Score is dropping because your CTR no longer compares as well to theirs as it did before.

Historical Quality Score (HQS) allows you to see the trends on a keyword by keyword basis over time. This can provide some great insights into marketplace dynamics. It can also help you understand what keywords the marketplace sees as the most important.

To see HQS you need to request a report. You will need to request a keyword report in daily mode as this is the only:

Creating a Historical Quality Score Report

Once this is done, click on the link to change the columns and layout and then select the four “historic” columns as shown at the bottom of the following screen shot:

Then, when ready, you can look at the report itself. This particular example shows a scenario where the competition for the keyword got a lot more intense on 11/3 and 11/4:

Once you see something like this you can begin to investigate what the market dynamics are that caused that to happen. For example, the 3rd and 4th of November of this year were a Thursday and a Friday. Perhaps your competitor has learned that the last two days of the work week are the highest converting days related to this keyword. If that is the case, you can adapt your strategy as well.

2. Aggregated Quality Score: adCenter is also now showing advertisers an Aggregated Quality Score (AQS) at the Ad Group level. This is more than a curiosity. AQS will be a very significant factor in setting the Quality Score for new keywords that you add to the same Ad Group. Other factors such as keyword and landing page relevance still apply, but AQS will provide you with a sense as to what to expect.

To see AQS you will need to request an Ad Group performance report in daily mode as shown here:

Creating an Aggegrated Quality Score Report

Then you will need to go in and add the historic quality score column to your report as shown here:

This will allow you to see the AQS for the Ad Group over time as shown here:

This is similar to what we did with HQS at the keyword level, but now let’s look at the AQS across a number of different Ad Groups at once:

Now comes the fun part. First of all, you see two Ad Groups with an AQS of 2, and one with a 3. However, the number of impressions is pretty low. The biggest opportunity for increasing overall performance may come from optimizing the Ad Group showing an AQS of 5, since it has the most impressions of all the Ad Groups shown. Great stuff!

3. How can I tell if my broad match keywords are well optimized?: This is not really a new feature, but it is the 2nd most popular question asked of the adCenter team. One of the basic ways to do evaluate your broad match keywords is to measure whether or not you are getting conversions for your broad match keywords, and good ROI. But, you can also compare the Quality Score of your broad match keywords with the Quality Score of the same keyword in exact match mode to help you with this evaluation.

For example, if your exact match form of the keyword has an 8 out of 10 score and your broad match variation is at 2, 3, or 4, you have an opportunity to greatly improve your results for that phrase. This is true even if the phrase passes the ROI test I just suggested. On the other hand if the exact match word have a Quality Score of 8 and your broad match variation scores a 6, it is probably already pretty well optimized.

Summary

adCenter Quality Score provides some great insights that advertisers can use to enhance the performance of their campaigns. The adCenter team is continuing to work on developing new tools to improve the ROI for adCenter customers, so watch for more developments from them in the near future.

Filed Under: Microsoft, PPC, Search Engine Marketing

How Google Does Personalization with Jack Menzel

11/1/2011 by Eric Enge 2 Comments

Jack Menzel is a Product Management Director for Google Search. Jack leads the teams developing new technologies used for personalization, question answering, web page summarization, and image search. Prior to joining Google Jack worked as a Program Manager at Microsoft. Jack holds a MS in Computer Science from the University of Washington as well as an BS in Computer Science and Mathematical Economics from Brown University.

Key Points

One of the hot areas in search is personalization. Google recognizes that personalization is a way to offer people better search results. How this works has a big impact on SEO, and I had the opportunity arise to speak with Jack Menzel and jumped at it. Here are some of the key points from the discussion:

People confuse context with personalization, and these are different things. Context includes factors such as language, location, and time of year.
(Jack:) “A lot of people assume personalization is amazingly pervasive”. In fact only small changes are made to a results page based on personalization. Google recognizes for diverse query results.
Past query history is used for personalization. If you search for “rome”, and then “hotels”, some of the results will be for hotels in Rome.
Past click through history is a factor. If you show a clear preference for one site by clicking on it in the results, then it may be moved up in the results for you.
The recommendations of friends are used in personalization.
Google will look at your friend’s profile to see what networks they have included there, and then see what they recommend on those sites.
(Jack): “When people are signed out, their search results are personalized based on past search information linked to their browser for up to 180 days using an anonymous cookie”.
Appending &pws=0 to the end of a URL does work, but it only removes personalization, it does not remove context (language, location, time of year).
There are ways to turn off all personalized results. Google’s position is that user’s own their data. However, context will still be taken into account.

Interview Transcript

Eric Enge: Sometimes people confuse the notion of context with personalization, right?

If I respond to your query in your language that is really about context, not personalization.

Jack Menzel: That’s right. Sometimes results that are really a result of context get misinterpreted by people as personalization. If I respond to your query in your language that is really about context, not personalization. Personalization is more about recognizing that I like Dominion the card game and you really like Dominion the power company, and someone else really likes a videogame called Dominion. Imagine you turned off personalization, and suddenly Google was responding to all of your queries in the wrong language, you would be like “oh come on”.

Eric Enge: Another example would be that you are in the US and Halloween is in the near future.

Jack Menzel: Correct, right before thanksgiving there are a lot of searches about turkeys, and it often means people want turkey recipes.

Eric Enge: What are some of the other kinds of things that fit into the definition of context?

Jack Menzel: Let’s use a conversation based example. If we are both in Mountain View and I am talking to you about catching a bus, I don’t have to remind you that I am talking about bus in Mountain View, as opposed to one in Austin, Texas.

We take into account geography, language, and seasonality to a certain extent.

We take into account geography, language, and seasonality to a certain extent. The context of the previous queries is kind of on the borderline of what is personal and what isn’t.

Eric Enge: For example, if a person’s previous query was “Rome”, and then they search on “hotel”, there is going to be a tendency to show hotels in Rome.

Search results 3 to 5 for “hotels” when the prior search was for “rome”

A lot of people assume that personalization is still amazingly pervasive.

Jack Menzel: Your example may work, but I would have to check to make sure. A lot of people assume that personalization is still amazingly pervasive. We believe we are able to do some really useful things with personalization, but we may not get all of these things exactly right.

Eric Enge: What are some good examples of personalization that you think are handled well at this point?

We refer to this as “pattern” analysis, and it is based on recognizing preferences.

Jack Menzel: My interest in the card game Dominion is an example of this. I really don’t care about the power company at all. We refer to this as “pattern” analysis, and it is based on recognizing preferences. That’s an example of understanding the kind of topics that I am more interested in. Also, I do a lot of web programming, so when I talk about vectors, it will mean something very different than when a doctor talks about vectors.

Search results for “dominion” for someone with no related search history

We recognize patterns very well. If I keep going to visit my favorite scrabble dictionary over and over again I will see that the site that I tend to prefer will end up being boosted in the ranking because it makes it easier and faster for me. Pattern recognition is important because there is so much ambiguity in language.

Eric Enge: Jaguar, is my favorite example because you have the guitar, the operating system, the animal, and the football team. I would probably get the football team a lot, because I am a football fan.

Jack Menzel: Right, exactly. If you tend to gravitate towards football sites as opposed to operating system sites then you would end up getting that.

Eric Enge: How about social data?

We leverage social data pretty well. If your friend likes a restaurant, they can indicate it in a way that we (Google) can see that (such as a +1).

Jack Menzel: We leverage social data pretty well. If your friend likes a restaurant, they can indicate it in a way that we (Google) can see that (such as a +1). When you’re searching for a restaurant and you’re signed in, we may well boost that restaurant’s site in the rankings for you as well. We will also annotate the results, so that you can clearly see that this is content from your friend.

Search results for “reconsideration requests” with personalization on and off

Eric Enge: How do you determine what social properties people are on?

Jack Menzel: We look at people’s profiles and see what social profiles they have included in there, and we can then see what they share on those sites, provided that the information is public.

Eric Enge: If it’s not connected through your profile and your friend’s profiles then you are not going to use it to personalize results.

Jack Menzel: That’s correct.

Eric Enge: Do you need to be logged in to get personalized results?

We do a certain amount of personalization for people who are not logged in.

Jack Menzel: Being logged in is the best way to get personalized results. We do a certain amount of personalization for people who are not logged in.

Eric Enge: Is that cookie based?

Jack Menzel: Yes, we do some cookie-based personalization, which applies to search sequences where the subsequent searches feel more like a conversation. If you take that away from people it tends to be kind of frustrating. And so, there are certain parts of personalization that we still do.

Eric Enge: What kinds of personalization do you still do when people are logged out?

Jack Menzel: When people are signed out, their search results are personalized based on past search information linked to their browser for up to 180 days using an anonymous cookie. But if you’re signed out, we have much less data to personalize your results with than if you’re signed in.

Eric Enge: For example, you wouldn’t be able to use the social information.

Jack Menzel: That’s right. We have no idea about any of the social information. We only have a very limited knowledge of what your previous actions may have been, but we try to save you from having to repeat every detail in every query. But, it’s not as personalized as a signed in version.

Eric Enge: There aren’t any issues in this approach with shared IP addresses, because you are dealing either with people who are logged in or have a cookie.

Jack Menzel: That’s right. However, you can still run into the problem of shared computers where things get a little muddled. For example, if you are at an internet café and you are just doing a couple of searches to find out where the newest movie can be found. In general though, we don’t tend to have problems at the IP level because the system is based on cookies or being logged in.

Eric Enge: In the case of my machine at home, my 16-year-old daughter can come in and do some searches on it. That’s pretty hard to disambiguate I suspect.

Jack Menzel: That is very hard to do.

Eric Enge: In that environment if she does log me out and log herself in, is there a cookie involved at that point or does it just immediately switch to personalizing for her?

Jack Menzel: Yes. It doesn’t have much do with your cookie. It’s completely associated with your sign in.

Eric Enge: Does appending &pws=0 to the end of a search result URL still turn off personalization as it used to?

Procedure for turning off personalization with &pws=0

It (&pws=0) turns off “personalization”. However it isn’t really useful because people assume that it then will show them what everyone else sees. That simply isn’t the case.

Jack Menzel: Yes it does work. It turns off “personalization”. However it isn’t really useful because people assume that it then will show them what everyone else sees. That simply isn’t the case. There are a whole lot of contextual factors that make everyone’s results most relevant to them. This takes most of the wind out the sails of these types of analysis.

If personalization is turned off, we will still take a lot of context into account, including things such as location, language, and time of year. Of course, you can also get rid of context most of the time by getting more specific about your query. For example, if you live in the US but want to learn about the UK tax code, you would search on something like “UK tax code” to make that clear. Or you can conduct the search at www.google.co.uk instead too.

Eric Enge: When you use search history I assume you need to accumulate a certain enough data to achieve significance involved?

Jack Menzel: Yes, of course. We don’t want you to have done one query out of curiosity, and then suddenly decide that you are really into macramé. We are looking for a meaningful pattern.

Eric Enge: The other area that I think people get concerned about is the potential the loss of serendipity, but it’s not like you remap the entire results page around this.

Jack Menzel: It does make me kind of sad that when we talk about serendipity and search engines that we don’t point out the fact that search engines are the most amazing tool when it comes to discovering new things.

We have lowered the barrier and made it possible to research anything you could possibly imagine in the time it takes for you to type a query and hit enter. A fraction of a second later you’ve got some of the best results in the world for you to dig through. It makes it so easy. If we personalize the results page to the extent that we were only showing results tailored for you, that would be a bug for us. We would never want to do that.

So when we personalize a page the changes are pretty small, and we want to leave the other results untouched by personalization.

We try to give people the most usable page, but on the other hand we also try to give people the most relevant page to them. So when we personalize a page the changes are pretty small, and we want to leave the other results untouched by personalization. Using your interest in football as an example, even though you love football, some of the time you may actually want information on the animal, the guitar, or the OS instead.

Eric Enge: There is this long standing notion that’s been out there called query deserves diversity.

Jack Menzel: That’s right.

Eric Enge: This is obviously something that Google has known for quite some time because it’s many years since I first heard about query deserves diversity. Since you are trying to get as close to satisfying a 100% of the people 100% of the time over-personalizing would fail to do that.

Jack Menzel: That’s right. We really do want to show people a good representation of what the most relevant results would be, and people like that.

Eric Enge: Can you also discuss your approach to transparency and control?

Our position is that this is your data and you have control over your data.

Jack Menzel: We think is really important in any conversation about personalization. Our position is that this is your data and you have control over your data. You do have control over your web history, and you have control over how your browser manages cookies. We take the privacy of people’s data, and how we manage data, and how people have control over that data really seriously.

At the feature-level we try to make it very transparent to people what it is we are doing. We really are trying our best to be the industry leaders in how people have control over the data.

Eric Enge: Are there any aspects of personalization that people can’t turn off?

Jack Menzel: There are ways to turn all aspects of personalization off. If you do want to really not have your queries tracked between, or if you don’t want to have your content tailored to you in any way, shape, or form, you can set your browser to not accept cookies, and then we think you are a brand new person every time. Also, bear in mind that we will still take into account context, such as the right language for your results, your location, and the time of year.

Eric Enge: Thanks Jack!

Other Recent Interviews

Google’s Peter Norvig, October 17, 2011
Google’s Mayuresh Saoji, October 10, 2011
Google’s Frederick Vallaeys, September 29, 2011
Bing’s Ping Jen, September 28, 2011
Bing’s Duane Forrester, September 6, 2011
Danny Sullivan, August 8, 2011
Bruce Clay, August 1, 2011
Google’s Tiffany Oberoi, July 27, 2011
Vanessa Fox, July 12, 2011
Jim Sterne, July 5, 2011
Stephan Spencer, June 20, 2011
SEOmoz’ Rand Fishkin, May 23, 2011
Bing’s Stefan Weitz, May 16, 2011
Bing’s Mikko Ollila, June 27, 2010
Yahoo’s Shashi Seth, June 20, 2010
Google’s Carter Maslan, May 6, 2010
Google’s Frederick Vallaeys, April 27, 2010
Matt Cutts, March 14, 2010

Filed Under: Google, interviews, Search Engine Algorithms, SEO

Search Algorithms with Google Director of Research Peter Norvig

10/17/2011 by Eric Enge Leave a Comment

Peter Norvig is a Fellow of the American Association for Artificial Intelligence and the Association for Computing Machinery. At Google Inc he was Director of Search Quality, responsible for the core web search algorithms from 2002-2005, and has been Director of Research from 2005 on.

Previously he was the head of the Computational Sciences Division at NASA Ames Research Center, making him NASA’s senior computer scientist. He received the NASA Exceptional Achievement Award in 2001. He has served as an assistant professor at the University of Southern California and a research faculty member at the University of California at Berkeley Computer Science Department, from which he received a Ph.D. in 1986 and the distinguished alumni award in 2006. He has over fifty publications in Computer Science, concentrating on Artificial Intelligence, Natural Language Processing and Software Engineering, including the books Artificial Intelligence: A Modern Approach (the leading textbook in the field), Paradigms of AI Programming: Case Studies in Common Lisp, Verbmobil: A Translation System for Face-to-Face Dialog, and Intelligent Help Systems for UNIX. He is also the author of the Gettysburg Powerpoint Presentation and the world’s longest palindromic sentence.

Introduction

As you will see in the transcript below, this discussion focused on the use of artificial intelligence algorithms in search. Peter outlines for us the approach used by Google on a number of interesting search problems, and how they view search problems in general. This is fascinating reading for those of you who want to get a deeper understanding of how search is evolving and the technological approaches that are driving it. The types of things that are detailed in this interview include:

The basic approach used to build Google Translate
The process Google uses to test and implement algorithm updates
How voice driven search works
The methodology being used for image recognition
How Google views speed in search
How Google views the goals of search overall

Some of the particularly interesting tidbits include:

Teaching automated translation systems vocabularly and grammar rules is not a viable approach. There are too many exceptions, and language changes and evolved rapidly. Google Translate uses a data driven approach of finding millions of real world translations on the web and learning from them.
Chrome will auto translate foreign language websites for you on the fly (if you want it to).
Google tests tens of thousands of algorithm changes per year, and make one to two actual changes every day
Test is layered, starting with a panel of users comparing current and proposed results, perhaps a spin through the usability lab at Google, and finally with a live test with a small subset of actual Google users.
Google Voice Search relies on 230 billion real world search queries to learn all the different ways that people articulate given words. So people no longer need to train their speech recognition for their own voice, as Google has enough real world examples to make that step unecessary.
Google Image search allows you to drag and drop images onto the search box, and it will try to figure out what it is for you. I show a screen shot of an example of this for you below. I LOVE that feature!
Google is obsessed with speed. As Peter says “you want the answer before you’re done thinking of the question”. Expressed from a productivity perspective, if you don’t have the answer that soon your flow of thought will be interrupted.

Interview Transcript

Eric Enge: Can you outline at a layman’s level the basic approach that was used to allow Google engineers a translation system that handles 58 languages?

Peter Norvig: Sure — Google Translate uses a data-driven, machine learning approach to do automatic translation between languages. We learn from human examples of translation.

To explain what I mean by “data driven,” first I should explain how older machine translation systems worked. Programmers of those systems tried to teach the system vocabulary and grammar rules, like “This is a noun, this is a verb, and here’s how they fit together or conjugate in these two languages.”

Language is so fluid that programmers can’t keep up with the millions of words in all these languages and the billions or trillions of possible combinations, and how they change over time.

But it turns out that approach didn’t really work well. There were two problems. First, the formalisms for writing rules were absolute: this sentence is grammatical, and this other sentence is ungrammatical. But language has shades of gray, not just absolutes. Second, it is true that languages have rules, but it turned out that the rules don’t cover enough — language is more complicated and full of exceptions than people assumed, and is changing all the time. New words like “LOL” or “pwn” or “iPad” appear. Old words combine in unique ways — you can’t know what a “debt ceiling” is just by knowing what “debt” and “ceiling” are. Even core grammatical rules are uncertain — is “they” okay to use as a gender-neutral pronoun? What is the grammatical structure of “the harder they come, the harder they fall,” and what else can you say with that structure? Language is so fluid that programmers can’t keep up with the millions of words in all these languages and the billions or trillions of possible combinations, and how they change over time. And there are too many languages to keep rewriting the rules for how each language translates into each of the other languages.

So the new approach is a data-driven approach. Recognizing that we’ll need lots of examples of how to handle exceptions, we make the leap of saying: what if we could learn everything — the exceptions and the rules — from examples? We program our computers to look on the web for millions of examples of real-world translations, and crunch all that data to find patterns for which phrases translate into which other phrases. We use machine learning to look for recurring patterns — “this phrase in French always seems to translate into this phrase in English, but only when it’s near this word.” It’s analogous to the way you can look over a Chinese menu with English translations — if you see the same character keeps recurring for chicken dishes, you can guess pretty confidently that that character translates to “chicken.”

The basic idea is simple, but the details are complicated. We do some deep work on statistics and machine learning algorithms to be able to make the best use of our examples, and we were able to turn this technology into a world-leading consumer product. Google Research is a great place to come work if you want to tackle these kinds of problems in artificial intelligence.

If you visit a website in Thai or French or Urdu, Chrome will detect it and ask if you want to translate it into your native language.

We’re really pushing to have Translate available as a layer across lots of other products. You can always just go to translate.google.com, but it’s also built into our browser, Chrome. If you visit a website in Thai or French or Urdu, Chrome will detect it and ask if you want to translate it into your native language. It’ll automatically translate the whole page, and keep translating as you click on links. So you’re basically browsing the web in this other language. It’s very Star Trek.

There’s also a cool mobile app you should try — Google Translate for mobile is on Android and iPhone, and it does speech-to-text so you can speak and get translations.

Eric Enge: How does Google manage the process of testing and qualifying algorithm updates?

Peter Norvig: Here’s how it works. Our engineers come up with some insight or technique and implement a change to the search ranking algorithm . They hope this will improve search results, but at this point it’s just a hypothesis. So how do we know if it’s a good change? First we have a panel of real users spread around the world try out the change, comparing it side by side against our unchanged algorithm. This is a blind test — they don’t know which is which. They rate the results, and from that we get a rough sense of whether the change is better than the original. If it isn’t, we go back to the drawing board. But if it looks good, we might next take it into our usability lab — a physical room where we can invite people in to try it out in person and give us more detailed feedback. Or we might run it live for a small percentage of actual Google users, and see whether the change is improving things for them. If all those experiments have positive results, we eventually roll out the change for everyone.

We test tens of thousands of hypotheses each year, and make maybe one or two actual changes to the search algorithm per day. That’s a lot of ideas, and a lot of changes. It means the Google you’re using this year is improved quite a bit from the Google of last year, and the Google you’re using now is radically different from anything you used ten years ago.

If you define A.I. as providing a course of action in the face of uncertainty and ambiguity, based on learning from examples, that’s what our search algorithm is all about.

I’d say the resulting technology — Google Search as a whole — is a form of A.I. If you define A.I. as providing a course of action in the face of uncertainty and ambiguity, based on learning from examples, that’s what our search algorithm is all about.

The search engine has to understand what’s out on the web in text and other forms like images, books, videos, and rapidly changing content like news, and how it all fits together. Then it has to try to infer what the user is looking for, sometimes from no more than a keystroke or two. And then it has to weigh hundreds of factors against each other — hundreds of signals, like the links between content, the correlations among phrases, the location of the search, and so on — and provide the user information that’s relevant to their query, with some degree of confidence for each piece. And finally it has to present that information in a coherent, useful way. And it has to be done potentially for each keystroke, since Google results update instantly now.

Every time people ask a question, you need this machine to automatically and instantly provide an answer that helps them out. It’s a deep A.I. problem, and I think we’re doing a good job at it today, but we’ve got lots of room to grow too. Search is far from solved, and we have plenty of room for experts in A.I., statistics, and other fields to jump on board and help us develop the next Google, and the Google after that.

Eric Enge: Voice driven search seems like a very interesting problem to me. Even if you are dealing with only one language you have a vast array of dialects, accents, pronunciations, and ways of phrasing things.

This used to be addressed by having the user “train the system” to their manner of speaking. Are we are the point where we are past that now? What are the basic methods (in layman’s terms) being used to make this possible? Will this expand to automatically transcribing videos?

Peter Norvig: Speech recognition is actually quite analogous to machine translation. In translation we learn from past examples of (English, Foreign) pairs how to translate a new sentence we haven’t seen before; in speech we learn from past examples of (Soundwave, Text) pairs how to find the text in a new soundwave.

So instead of relying on one person talking for a long time to train the system, we rely on lots of people saying lots of things to train the system. So in effect, our users are training the system en masse.

Like you say, in old systems you’d have to sit there and train the thing for an hour before it would recognize your words. We wanted to build something anyone could pick up and just immediately start talking to, and have it understand them right away. So instead of relying on one person talking for a long time to train the system, we rely on lots of people saying lots of things to train the system. So in effect, our users are training the system en masse.

I can explain a little more how it actually works. There are three parts to our speech model. First, there’s the acoustic model, which maps out all the possible ways soundwaves can form phonemes, like “ah” or “mm” or “buh.” It’s tricky because acoustics vary a lot by what kind of mic you’re using, what background noise there is, how you’re holding the device, the gender and age of the speaker, and even what sounds come before or after the one you’re making. And like you say, there are lots of versions because accents and dialects vary so much. But with enough examples of speech, we can model what are the most likely ways of forming phonemes.

Then phonemes come together in our lexical model, which is basically a dictionary of how all the words in a language are pronounced. That also takes care of a lot of differences in accents — the model knows that there are multiple ways to pronounce things, and knows which are more or less likely. “Feb-yoo-ary” and “Feb-roo-ary” will both give you “February,” because the model sees both spoken a lot.

Finally, the words are strung together into a language model, which tells you which words are most likely to come after another word. There might be a soundwave that sounds like either “city” or “silly”, but if it follows the words “New York…” then the language model would tell us that “city” is more likely. We have a lot of text to train the system on — for Voice Search, where you speak your search to Google, we train this model on around 230 billion words from real-world search queries.

It’s all anonymized, of course — we don’t keep any training examples that could be tied to an individual speaker; it is all combined into our big model. We do give you the choice to opt in to have us learn from your own voice over time. You can turn this on, and the model will start to learn how your voice varies from our baseline model — say, if you have a strong accent, or a really deep voice. The model works well even without having to train it yourself, but you have the option to make it even better.

You can try this out on an Android phone or on the Google Search app on iPhone or Blackberry. On Android you can search, of course, but you can also compose emails by voice, or for that matter speak into any app where you’d use the keyboard — we added it into the Android keyboard so you can speak pretty much anywhere you might type. It’s also on Google on the desktop if you use Chrome.

Eric Enge: How about the problem of image recognition? For example, can we train a computer to recognize an image of the Taj Majal?

Peter Norvig: Yes. We do this on mobile phones and now on desktop Google Search too. You can actually use it to see where old vacation photos were taken — ones you scanned from back before digital cameras geo-tagged photos. If you took a photo of some cool-looking bridge you don’t recognize, and you can drag and drop the image onto the Google Search box, and there’s a pretty good chance it’ll recognize the bridge, tell you what it is, and give you all kinds of relevant information on it.

As with speech and translation, image recognition is data-driven and relies on machine learning algorithms across lots of examples. Luckily for us, the web has lots and lots of images of things, and most of them have captions that identify them. The more popular, the more images, so the better a chance we have at our algorithms being able to recognize it.

Here’s how image recognition works in a nutshell. It starts with identifying points of interest in an image — the points, lines, and patterns that provide sharp contrasts or really stick out from a bland, featureless background. It’s similar in some ways to how the human eye picks out edges and points by keying off the places where there’s sharp contrast.

Then it looks at how these points are related to each other — the geometry of the whole set of points. You could picture it as looking like a constellation of stars, even though really it’s a more sophisticated mathematical model of these points of interest and how they relate.

Now it compares that model to all the other models in a huge database. Those other models come from images it has already analyzed from around the web. It looks for a matching model, but it doesn’t have to be a perfect match. In fact, it’s important that it be a bit flexible, so it doesn’t matter if it’s turned around, or shrunken, or twisted a bit. The Taj Mahal still has the basic geometry of the Taj Mahal even if you photograph it from a little bit of a different angle or photograph it lower in the frame. When Google recognizes that it matches that model best, it guesses it’s probably the Taj Mahal.

There’s something profound here about asking a “question” that’s actually just an image. We’ve moved beyond every query being a string of text. Now you can just present Google an image and expect relevant information.

There’s something profound here about asking a “question” that’s actually just an image. We’ve moved beyond every query being a string of text. Now you can just present Google an image and expect relevant information. So it puts even more burden on the search engine to know what that’s supposed to mean. What’s the best answer to a question when the question is an image? We present some information we think is relevant today, but what exactly the interaction should be here is still ripe for research.

Eric Enge: For some of these tasks we currently must rely on batch processing instead of real time processing (e.g. the way that the Panda algorithm currently operates). How long before the processing power increases to the point where the Panda algorithm can be done in real time?

Peter Norvig: I wouldn’t separate out that one update from the rest of the Search algorithm that way; it was really just one improvement among many that we’ve made in the past year or so. But the question is certainly relevant to our Search algorithm overall.

Broadly speaking, you can think of the growth of the web and the growth of the computing power needed to instantly index it as a kind of arms race.

Broadly speaking, you can think of the growth of the web and the growth of the computing power needed to instantly index it as a kind of arms race. The web keeps growing. There’s a misperception that the web has become established or matured, but in fact the growth curve is a nice smooth exponential that hasn’t shown signs of slowing down yet. We’re still in the middle of the information explosion.

So we keep up with it a few ways. It helps that processors and disks keep getting cheaper. Even new categories of technology, like solid-state disks, have helped. We’re also getting smarter about delineating which content needs to be updated instantly, and which can be updated more slowly — again, we learn how to do this from examples. A lot of the smarts you see in Google Instant, and the predictive input suggestions that keeps guessing what word you might type next, are about anticipating what information is most likely to be needed, and queuing that up so it’s ready to go.

Google Code Articles of Speeding Up the Web

We’re really obsessed with speed at Google. Speed is a crucial feature of any information-intensive product. You never want your tools to slow you down or interrupt your flow of thought. There’s a cool feature we launched a little while ago called Instant Pages which takes Google Instant a step further: instead of just predicting what words you might type, and pre-loading the search results, if Google is really confident that the first result is the right one, it’ll start loading it in the background. So often by the time you click that result, it’s already loaded up — so the website appears to load instantly. It’s like a magic trick when it works well.

Eric Enge: Can you expound a little bit on the types of problems that AI can work on solving in the area of search over the next 5 years?

So you want the answer almost before you’re done thinking of the question. We think we can offer that now most of the time.

Peter Norvig: We’ll work more on speed. It used to be that a few seconds was really fast to learn what the height of the Eiffel Tower was — that’s a heck of a lot faster than a trip to the library to look it up in a reference book in the back shelves. But now even a few seconds feels slow, because again, it interrupts your flow of thought. So you want the answer almost before you’re done thinking of the question. We think we can offer that now most of the time.

But eventually this will stop being such a back-and-forth question-and-answer routine, and start just being a steady flow of relevant information. It should be right there when you need it, presented so it’s useful without being overwhelming. It’s going to take a lot of engineering and a really fine artistic touch to make that work the way we envision it.

And of course this gets to a deeper A.I. problem: not just understanding information and queries, but really understanding what the user needs and will find useful at a given moment, and serving it up in a way that’s perfectly digestible. It’s not just about human-computer interaction or information retrieval. It’s about how people learn and attain knowledge. We’re trying to move beyond just presenting information, and really focus on increasing people’s knowledge of the world. So Google needs to be “smart” in the sense of really understanding the user’s needs in order to help them build up their knowledge of the world.

Eric Enge: Thanks Peter!

Other Recent Interviews

Google’s Mayuresh Saoji, October 10, 2011
Google’s Frederick Vallaeys, September 29, 2011
Bing’s Ping Jen, September 28, 2011
Bing’s Duane Forrester, September 6, 2011
Danny Sullivan, August 8, 2011
Bruce Clay, August 1, 2011
Google’s Tiffany Oberoi, July 27, 2011
Vanessa Fox, July 12, 2011
Jim Sterne, July 5, 2011
Stephan Spencer, June 20, 2011
SEOmoz’ Rand Fishkin, May 23, 2011
Bing’s Stefan Weitz, May 16, 2011
Bing’s Mikko Ollila, June 27, 2010
Yahoo’s Shashi Seth, June 20, 2010
Google’s Carter Maslan, May 6, 2010
Google’s Frederick Vallaeys, April 27, 2010
Matt Cutts, March 14, 2010

Filed Under: Google, interviews, Search Engine Algorithms, SEO

Starting Up with Google Product Search, with Google’s Mayuresh Saoji

10/11/2011 by Eric Enge Leave a Comment

Mayuresh Saoji is a Senior Product Manager on the Google Commerce team. In this role, Mayuresh is responsible for leading efforts on Merchant Center, Content API and broad Google Product Search Policy issues. Previously, Mayuresh was a Product Manager on the Google Chrome team, and also lead the Distribution efforts for products like ChromeOS, Google Toolbar, iGoogle and Chrome browser. Prior to Google, Mayuresh was a Product Manager at Microsoft where he worked on Go-to-Market for Sharepoint 2007. Mayuresh holds a Bachelor’s degree in engineering from the University of Bombay, India and an MBA from the Kellogg graduate school of Management.

Key Points

Google product search offers a rich array of opportunities for publishers to place their products in front of shoppers (there is a bulleted list of the opportunities right at the start of the interview). Mayuresh does a great job of spelling out the way to get started with Google Commerce in this interview. If you sell physical products this interview can act as a guide on how to get started and how to prioritize your efforts from an optimization perspective. Here are the key points:

You must sell physical products online to participate.
One opportunity is to place Google Commerce Search on your site. This provides visitors a way to search your product catalog using Google’s search technology. It is a paid product.
The first step is to create a Merchant Center account.
The second step is to verify that you are the owner of the website.
The next step is to provide a data feed of all your products.
Implement a test feed before going live, as this will allow you to find and remove errors upfront.
The most important optimization step is good quality data. This is worth a lot of effort, as Google will lose faith in feeds that show errors.
Make absolutely sure that the pricing data is accurate.
Plan on having a ISBN code, UPC code, or EAN code (Europe) for all your products.
Have images for all of your products. (Mayuresh): “it’s to your benfit to send uys good images for every product”.
Update your feed (Mayruesh): “at least as often as your website is updated”.
The Content API is useful for large feeds where it may be desirable to make partial updates (e.g. change only the price for 200 products). However, you need programming expertise to use it.
(Mayuresh): “Product reviews are important, and they provide a good signal to users about products”.

Interview transcript

Eric Enge: What are the benefits of participating in product search?

Mayuresh Saoji: Any merchant that sells physical products online is a good candidate for participating in Google product search. Participating in product search provides you with a forum for sending structured data on your products to Google. It allows merchants to show more rich data in many formats:

On Google.com
Google Shopping
Google Product Search
Product Ads and Product Extensions
Google Shopper in Mobile Search

If they are Google Commerce Search customers, which is a paid product, then that same data is leveraged to power the search and discovery experience on their e-commerce website or mobile application.

The end goal is to drive a lot of qualified traffic to publishers, and that’s the best reason for doing this.

Eric Enge: Basically, it is like a Custom Search Engine, but for products?

Google Commerce Search (GCS) is an e-commerce search solution designed specifically with online and multi-channel retailers in mind.

Mayuresh Saoji: It has some general similarities, but Google Commerce Search (GCS) is an e-commerce search solution designed specifically with online and multi-channel retailers in mind. GCS has several advanced features besides product recommendations to help retailers improve their conversion rates.

Eric Enge: Great, what’s the best way for someone to get started?

Mayuresh Saoji: First you create a Merchant Center account. This is where you tell us about your business, your store, and provide us with your URL. The second step is to verify that you are the owner of your website. This is still part of the signup flow, and once that’s done then now you have a valid Merchant Center account. That’s one part of the story.

The other part of the story is to start submitting your data to us. Google has published a product feed specification, and you need to adhere to that specification, and then you can submit data in one of a number of formats to us. You can submit it as a tab delimited (TSV) file, a flat file, XML file, or via the Content API. Many of our larger retailers use the Content API, and that allows them to easily submit hundreds of thousands of items (and much more), and also makes it easy to make very quick changes to specific attributes of those items.

I’d also recommend creating a Test Feed file first and submitting test data.

I’d also recommend creating a Test Feed file first and submitting test data. This functionality can be found under the “Data Feed” tab in the Merchant Center (click on “New Test Data Feed”). The test feed is not indexed and displayed on Product Search, so it’s a perfectly safe environment. We also have great error reporting for the test feed, which will allow merchants to understand errors, iterate and quickly get a functional feed up and running.

Once you are done testing you can actually submit the data and we will ingest it, index it, and then show it on Google Product Search and some of these other properties.

So to summarize: Create your account, verify your website, create and submit a Test Feed, work out all the kinks, and then submit the actual data feed to us.

The Google Merchant Center is the hub for these interactions: It’s where you provide us information about your business, it’s where you submit your product data feed. It is also the place where you can go to see the status of your data, to see if there are any errors with your submission. We also provide you with reporting on clicks, etc. so you can see how your product listings are performing.

Eric Enge: I assume you need programming expertise to use the Content API?

Mayuresh Saoji: Yes. You do need programming experience because you have to make HTTP calls with the right parameters. Most of our merchants submit data to us in an XML file or a flat file today. The content API is used by some of our largest merchants, who have that in-house IT expertise, It’s also used by merchants who need to change their data quickly, and frequently

Eric Enge: What determines the order in which you show products?

Make sure you adhere to the feed spec and make sure you fix problems as we report them in the Merchant Center.

Mayuresh Saoji: There are some things that you can control, and the biggest thing is to give us good quality data. Make sure you adhere to the feed spec and make sure you fix problems as we report them in the Merchant Center. In the Merchant Center there is a data quality tab. For instance, if you submit 10,000 items and 300 of them don’t have images the Merchant Center will tell you that.

It gives you very concrete and specific feedback on the types of errors, and in many instances also provides actionable feedback on what you can do to fix those errors. Note that submitting a feed is sometimes an iterative process. You may have some errors at first, but the Test Feed can make it easy to figure out problems and get it right quickly, so I highly recommend using that tool from the Merchant Center

Eric Enge: What are the best ways to optimize your feed?

Mayuresh Saoji: There are a few best practices to keep in mind. Whenever possible, each product should have a unique ID (there are rare exceptions for custom or one-off products). This is an important attribute that we look at, for matching products on the backend. This could be a UPC code, it could be an ISBN number for a book, or an EAN code if you are in Europe. Fundamentally it’s the unique fingerprint for each product.

Make sure that your price and availability information is accurate.

Make sure that your price and availability information is accurate. For price, you should separate out tax and shipping. If you tell us an item costs $12.99 make sure that it is actually $12.99 on your website and not $13.99. A mismatch in price is a bad experience for the user, moreover, the clicks you get are not going to convert to a sale on your site because you have a different price advertised. This generally leads to a bad taste in the mouth for everyone.

We provide you with a mechanism for giving us the base price, giving us the tax, and giving us the shipping separately, and we also show those separately on the search results page.

The other key thing would be images. In a nutshell, good quality images provide clear information to the consumer. So, it’s to your benefit to send us good images for every product that you sell. Note that each visually distinct variant does need its own image.

We also recently introduced some new attributes for better categorization of your items. Make sure that you send us that category code, and this is especially important for things like apparel and accessories like shoes, and jewelry.

I would summarize this by saying the top things merchants would care about would be unique IDs, price, availability, tax and shipping, and images. In addition, for Apparel and variants of products, there are some very specific requirements that are extremely important … you should read our feed spec for more details

Eric Enge: Would items with different colors still need a separate UPC code or EAN code?

Mayuresh Saoji: In many cases they do have a separate UPC code or an EAN code, and in some cases they don’t. It depends on the product actually.

Q: How often should the feed be updated? A: At least as often as your website is updated.

Eric Enge: How often should the feed be updated?

Mayuresh Saoji: At least as often as your website is updated. It’s important to keep the data fresh. Keeping your data fresh is very important. Many merchants set this up such that they have an automated process which will just go into the backend and send us a new feed every night. Some people send it to us multiple times a day because that’s how often their website varies. For some products, pricing can vary, and more importantly availability can vary from hour-to-hour.

Many people use the Content API for these kinds of scenarios because unlike the feed spec, the Content API allows you to make very, very quick changes and incremental changes to price and availability for specific products without reloading the whole feed.

Eric Enge: The Content API gives you a lot less latency in terms of turning that around.

Mayuresh Saoji: Absolutely. It also gives you a lot more control in being able to change certain specific attributes for certain specific products.

Eric Enge: What are the advantages of the Content API?

… with the content API you can submit only the parameters that are changing (in this instance, Price) for each of those fifty products.

Mayuresh Saoji: With a flat file feed you have to give us all the attributes for every item you send. If you submit a thousand items total in your feed, and subsequently you need to update the price for fifty of them, you can submit a flat file with only those fifty items. However, you’ll need to submit each and every attribute for each of those fifty items. And then it will overwrite the whole thing, but with the content API you can submit only the parameters that are changing (in this instance, Price) for each of those fifty products.

Eric Enge: What role do product reviews play?

Mayuresh Saoji: Product reviews are important, and they provide a good signal to users about products, so this is something that merchants should encourage their shoppers to do and should provide this information.

Eric Enge: Can you talk a bit about the changes you announced on September 2nd and the changes to the product search feed specification you announced in July?

Mayuresh Saoji: We want to get to a richer, more visual shopping experience, and we want to ensure that shoppers are getting the rich and detailed information they are looking for. I think this has benefits for everyone. It’s good for our merchants, because we can deliver more valuable, more qualified traffic to them.

In order to support this goal, we needed to get better (and higher quality) data from merchants. This was the impetus behind the new feed spec requirements we announced in July 2011.

For instance, we’ve required that merchants submit a high-quality image for all of their products. We’ve given merchants the ability to have alternate views of those products as well (although alternate views are not a required attribute).
We’ve gotten much stricter and more prescriptive about availability and how to define it.
We added the Google product category attribute, which allows us to better categorize and classify products using a standardized taxonomy. It also allows us to make sure we apply the right set of rules for certain products.

There were also a bunch of requirements around apparel that we had announced. There is this concept of variance in apparel. Typically the variant attributes are colors, size, material, and pattern. We’ve specified those things. One big change we’ve made is we’ve asked merchants to submit one distinct item per variant. So, if you have a shirt sold in three colors and two sizes, you would need to send us six separate items. That’s a high-level of the changes that we’ve made to the spec.

Regarding the updates to the Google product search page we announced in September, the goal here was to help shoppers find new stuff. Again, it’s all about that richer experience. Our merchants have a better showcase for their products. It’s like walking into the mall and touching and feeling something. You want to get as close to that as possible

We wanted to make it easier for our users to browse and discover new products, be aware of trends, etc. If you look at the new product search homepage you will see many interesting changes. We’ve got more of a curated feel to the page now. We show popular products, we showcase new trends, we may show relevant Google offers. It’s very fashion-focused and apparel-focused at this point.

It provides a more visual way to shop for dresses. We simplified the UI and removed much of the text around the images. We’ve increased the size of each image; we’ve emphasized the visual aspects of apparel shopping. People often shop by color or genre or size or silhouette of a dress, and we’ve taken those things into account.

In addition, from each product page you can see visually similar products. Shoppers will have the ability to view similar items, and there is the serendipity that takes over and allows them to very quickly browse, and meander, and discover. The goal for us was to help shoppers browse and discover new products and new trends in a fun and visually appealing environment.

Eric Enge: Thanks Mayuresh!

Other Recent Interviews

Google’s Frederick Vallaeys, September 29, 2011
Bing’s Ping Jen, September 28, 2011
Bing’s Duane Forrester, September 6, 2011
Danny Sullivan, August 8, 2011
Bruce Clay, August 1, 2011
Google’s Tiffany Oberoi, July 27, 2011
Vanessa Fox, July 12, 2011
Jim Sterne, July 5, 2011
Stephan Spencer, June 20, 2011
SEOmoz’ Rand Fishkin, May 23, 2011
Bing’s Stefan Weitz, May 16, 2011
Bing’s Mikko Ollila, June 27, 2010
Yahoo’s Shashi Seth, June 20, 2010
Google’s Carter Maslan, May 6, 2010
Google’s Frederick Vallaeys, April 27, 2010
Matt Cutts, March 14, 2010

Filed Under: Google, interviews, Search Engine Algorithms, SEO

Real Time Quality Score Defined, with Google’s Frederick Vallaeys

9/29/2011 by Eric Enge 2 Comments

photo of Frederick Vallaeys Frederick Vallaeys is a Product Evangelist for Google AdWords. In this role, he helps advertisers learn which Google products can best solve their marketing needs. He also represents the needs of advertisers with the engineering and product management teams. His main product focus is on ads quality and bulk tools like the AdWords Editor and the AdWords API.

Key Points

Hoo boy! I went through this interview to try and extract the most important points made, and I will do the best I can here. However, if you are a serious AdWords professional, I’d suggest you read the entire interview from end to end.

The main thing you will get from this interview is that the Quality Score you see in your Google AdWords account differs significantly from the Real Time Quality Score that Google uses to determine how your ad ranks. There is definitely a strong correlation, so Quality Score is a useful metric, but an understanding of Real Time Quality Score can give you an extra edge in understanding what it is you need to do to make your optimization efforts as successful as possible.

Quality Score is the number you see in your Google AdWords account. It is a number between 1 and 10, where 1 is a horrible score, and 10 is an awesome score. Some key points about Quality Score are:

It is mostly based on historical clickthrough rates of the keyword and ad text.
Additional factors include landing page quality and load time of the page, but these are secondary factors.
Quality Score (QS) is based on data from exact match only. Even if you bid on a broad match keyword, such as “cruises”, only exact matches with the keyword are used to determine the QS.
The published number is the aggregate for all instances of that keyword in your account.
When you first add keywords into an new account, Google will show the system wide average for that keyword as your Quality Score.
If you have an existing account, and you add a new keyword, than the account history is a factor in the default Quality Score.

Real Time Quality Score is the number used by Google to help determine your ad rank. It has a lot in common with QS, but is calculated in real time and takes into account many additional factors. Some key points about Real Time Quality Score (RTQS) include:

Specific query performance is taking in to account. For example, if you bid on “tennis shoes” and someone searches on “discount tennis shoes”, but you sell only expensive tennis shoes, chances are that the resulting user interactions will end up in a low RTQS for this particular query.
RTQS is personalized to the user based on query history. For example, a recent search on “Rome” followed by a search on “hotels” is more likely to show adds for hotels in Rome.
RTQS personalization is session based. Once the session cookie is deleted the query history used for personalization is lost.
Other personalization factors include location and time of day.
The +1 button does not factor into RTQS … yet. However, it can impact QS and RTQS by increasing Clickthrough rate.
+1 is associated with the URL, regardless of whether or not it is clicked on in the ad, organic results, or on the web page.
Site links drive CTR increases ranking from 17% to 30% and can also result in more qualified customers (higher conversion).
CTR expectations are normalized by position. So if the number 1 position usually gets a 30% CTR and you are getting 20% that is a negative.
RTQS is determined at the keyword-ad level. There are no ad group or campaign components to RTQS.

That’s it for the summary points. However, in the body of the interview there is much more, including Frederick’s recommended process for optimizing your QS and RTQS, lots of examples, and why bidding your keywords high when you first launch them is a smart thing to do.

Full Interview Transcript

Eric Enge: Can you tell me how Quality Score is used?

Frederick Vallaeys: The Quality Score is Google’s way of ensuring that we show the most relevant ads to our users, and we deliver high quality leads to advertisers buying the clicks from us. The Quality Score obviously factors into the ad rank together with the advertiser’s bid.

It helps determine which advertiser has the highest position on that page. The Quality Score that you see in the account is determined by a number of factors and is mostly based on the historical click through rates of the keyword and the ad text.

The Quality Score is only based on data from results on exact match.

The Quality Score is only based on data from results on exact match. That means the keyword the user types in has to be exactly the same as the keyword chosen by the advertiser. There has to be an exact match between those two regardless of which match type the advertiser selected. Also, we only use data from google.com, not display network traffic or traffic from our search partners.

That’s the data that builds up the Quality Score. We also have additional factors such as landing page quality and load time of the page, but those are secondary factors. The biggest thing we look at is the historical click through rates of the ad text with the keywords inside of the account.

Eric Enge: That’s specific to what we see published in AdWords, is that correct?

Frederick Vallaeys: Exactly. What you see published in AdWords is going to be a number between one and ten. A Quality Score of one out of ten is a terrible Quality Score, and a score of ten is a fantastic Quality Score. What you have to keep in mind is that the number we publish is the aggregate for that specific keyword. It reflects all the data we have on that keyword for your account.

The key point here is that this is an average, and an average is never great which is why we also calculate a Real Time Quality Score internally.

The key point here is that this is an average, and an average is never great which is why we also calculate a Real Time Quality Score internally. The average you see in the accounts is good for figuring out where you have an issue.

As an advertiser, if I have to prioritize which keywords to optimize, this is a good indication. Any Quality Score below a seven is a place where you might want to start looking. The lower that number the bigger an issue you have.

Eric Enge: When you open up a new account, and there isn’t any click through rate history, I’ve seen situations where the Quality Score is quite low but the numbers come up as the account ages.

Frederick Vallaeys: Right. What typically happens when you start up a new account, or you put a new keyword for the first time into an existing account, is we take a system-wide average based on advertisers who have run on that keyword in the past. What often happens is that the keyword may be fairly broad and may not be the best performing keyword system-wide.

As your account ages and you start getting impressions and clicks on that keyword, we can build a specific picture of how you, an advertiser with those specific ad texts, will do on that keyword. If you are a good advertiser that knows how to write a compelling ad text for all the keywords, your Quality Score will certainly increase at that point and become much better. It also becomes your own Quality Score as opposed to that starting point system-wide average.

Eric Enge: Can keywords with a bad history have a negative impact on another keyword’s quality score?

If an account has a set of keywords that in aggregate have a low QS, this can have a negative impact. Zero impression keywords do NOT matter because those contribute no CTR data.

Frederick Vallaeys: In the absence of specific data about how a keyword performs with a specific ad, we rely on system wide data and account-level data. If an account has a set of keywords that in aggregate have a low QS, this can have a negative impact. Zero impression keywords do NOT matter because those contribute no CTR data.

Keywords with few impressions and few clicks could in aggregate have a large number of impressions with a low CTR and this could hurt the account. Keep in mind though that even if there is a negative impact on the account, this won’t matter as soon as we have enough data about how a keyword performs with a specific ad because we’d use that specific data for QS rather than the less specific account level data.

Real Time Quality Score

Eric Enge: Let’s say we have a keyword such as “tennis shoes.” How is Real Time Quality Score, both displayed and calculated?

Tennis Shoes

Frederick Vallaeys: Many people will type in “tennis shoes” but others may type in variations of that keyword such as “discount tennis shoes” or “Nike tennis shoes.” If you had that keyword in the broad match then your ad would have been eligible to show on these different variations.

For the Real Time Quality Score we calculate at the exact moment a user did the search and take into account what these variations are. If you sell expensive tennis shoes, and someone did a query for discount tennis shoes, we would show your ad and maybe that ad had an eight out of ten Quality Score. It’s a mismatch to what that specific user was looking for because they weren’t looking for expensive tennis shoes. In that case it would not be the best ad to show.

The real time system allows us, based on the additional data for this specific situation, to know this ad is not the best ad for that case, and to give preference to some of the other ads.

We think it’s a real positive for advertisers, because in the past we would aggregate and you would get clicks that maybe weren’t from the most qualified potential customers because we were looking at averages. Now we can look at how they formulate the query and how that impacts their likeliness of being interested in this advertiser’s ads.

Instead of a eight out of ten, the Real Time Quality Score might be a five out of ten telling us this ad is not a great ad for this query. This will affect the ad rank and, in some cases, the ad doesn’t show.

In the “tennis shoes” situation, when someone types in “discount tennis shoes” we are looking beyond exact match and you have a separate Real Time Quality Score calculated for the performance of the query “discount tennis shoes” against that keyword, that ad and that landing page. We could look at some interesting cases that would match really ambiguous keywords which are difficult to bid on.

If you as an advertiser pick that relatively generic keyword, we can find a subset of queries that do well for what it is you are selling.

For another example, consider the keyword “jobs.” You could be looking for Steve Jobs or you could be looking for jobs in San Francisco. How do we know? If you as an advertiser pick that relatively generic keyword, we can find a subset of queries that do well for what it is you are selling, whether it’s a blog about Steve Jobs’ company or whether it’s a blog or website for finding a job in San Francisco. Many years back, the AdWords system wasn’t quite as specific with its Quality Score. What it would do in these ambiguous cases is not run the advertiser’s ads because we would say, “okay, on average this is a pretty bad keyword, it doesn’t perform that well.” We would lose sight of the specific queries in which it actually did do well.

With the more sophisticated system we have today, if there is a small subset of queries that work well for you, we can find those and often show you in quite a high position even though all the other queries for that same keyword might not have done well for you.

Cruise Ship Another example I like to use is “discount cruises.” If someone looks for discount cruises, it’s not ambiguous in terms of what they are looking for, but it could be ambiguous in terms of the destination they are looking for.

There are companies that offer Alaskan cruises and companies that offer Caribbean cruises. Generally, people are more interested in the Caribbean or warm weather cruises. With that generic keyword “discount cruises” you might do well on most queries because most people want to buy your Caribbean cruise.

In those few instances where someone is looking for an Alaskan cruise, it would be a poor decision to show your ad because you don’t sell that cruise. If we had gone on the average, we would have shown the ad because most people look for Caribbean cruises.

This provides a better user experience because users aren’t seeing an ad for Caribbean cruises just because it happens to have a high overall Quality Score.

With the real time system we see that the user typed in the word “Alaskan” in addition to “discount cruises.” This is probably not the best time to show the ad, and it prevents the advertiser from showing an ad that’s unlikely to lead to a sale. This provides a better user experience because users aren’t seeing an ad for Caribbean cruises just because it happens to have a high overall Quality Score.

Personalization and selection of Ads

Eric Enge: In the scenario above, where the user provides more information based on adding a qualifying word to the query. For discount Alaskan cruises you don’t show the Florida or Caribbean cruises ad. Could you look at the user’s past query history and see that they recently read blogs about Alaska or things of that kind? Is there anything like that in play at this point?

There is a personalization factor in place. This works by looking at previous queries the user has done … when we talk about personalization it’s actually on an anonymous basis.

Frederick Vallaeys: Yes. There is a personalization factor in place. This works by looking at previous queries the user has done. A good example of this is a user came to Google, did a search for Rome, and the next search they did was for hotels. What Google knows is that they probably were thinking about hotels in Rome as opposed to hotels anywhere. Rather than show generic ads for hotels, we can look back at that session data and show more relevant ads based on that. That’s the extent of what we can do at this point.

I would like to note that when we talk about personalization it’s actually on a anonymous basis. It means we know what a certain cookie is doing, but we don’t know what a certain person is doing. We know that cookie ID 1234 searched for Rome before they searched for hotels, but we don’t know that the cookie is Frederick Vallaeys.

Eric Enge: You obviously have to avoid the privacy concerns. Does the cookie that allowed you to do this survive across its sessions?

Frederick Vallaeys: No. We found that’s usually not a great thing to do because the correlations you start seeing actually go down quite a bit. Also, we don’t always combine the previous searches to the current searches because if there is a clear shift in topic that the user is searching for then it doesn’t make sense to look at that previous data.

Eric Enge: This personalization that we spoke about is a factor in Real Time Quality Score?

Frederick Vallaeys: The other mechanics we look at are the location of the searcher and the time and day. There are a number of other factors we don’t disclose, but we do evaluate many factors that could potentially have some impact. We look at CTR, and if there a strong correlation between this factor and CTR, that’s a factor we could continue to use. Location and time are two good examples that do matter.

Eric Enge: If it’s November and somebody in Massachusetts typed in “discount cruises” are you more likely to show a Florida cruises ad than an Alaska cruises ad?

Frederick Vallaeys: Exactly. We might give preference to an ad on Florida and Caribbean cruises for people from a cold location.

Eric Enge: Correspondingly, if you have someone in California typing that, you might actually show a Hawaii cruise ad rather than a Caribbean cruise ad.

Frederick Vallaeys: Exactly.

Eric Enge: What are some of the correlations for time of day?

there have been a number of studies in the travel industry that show in the morning people tend to research hotels they may stay at. At lunch they talk to their spouses to get approval to book a certain hotel. In the afternoon they may be more likely to book that hotel.

Frederick Vallaeys: You can think about differences in behavior even if they were searching for the same thing at different times of day. For example, there have been a number of studies in the travel industry that show in the morning people tend to research hotels they may stay at. At lunch they talk to their spouses to get approval to book a certain hotel. In the afternoon they may be more likely to book that hotel.

So, if we find a query in the morning for a certain type of item, we might give preference to more research-oriented ads, and in the afternoon we may focus on more transaction oriented ads. That’s difficult so the system depends on having enough statistically significant data to make those decisions.

Eric Enge: Right, because you don’t know if they went and talked to their spouse, but you do know they tended to click on review-oriented ads as opposed to book-it-now oriented ads.

Frederick Vallaeys: Exactly.

The role that the +1 button plays into the Quality Score

Eric Enge: What about the +1 button that you now see appearing on ads. Is that something you factor into a Quality Score at this point?

Frederick Vallaeys: It doesn’t factor into the ranking yet. However, what we typically see whenever a new ad format or a new feature of an ad is introduced, such as the +1 button, is that it sometimes increases click through rates. If the click through rate increases, that leads to a better Quality Score so there is definitely an indirect factor by having strong +1 recommendations and endorsements that more people could click on your ad.

+1 is essentially bringing social to the moment of relevance.

+1 is essentially bringing social to the moment of relevance. If a user sees that five of his buddies have booked the same vacation or done business with the same cruise line that’s a pretty strong endorsement and that user is more likely to also click on the ad, check it out, and buy from them. If you as an advertiser can build that following of +1 clicks and get people to endorse you that should be positive for you. If that seems to be a useful thing to use in terms of Quality Score, we absolutely could start thinking about integrating that.

Eric Enge: Are people clicking on those +1 buttons in the ads in any volume? I could see +1′ing a great article, but I’m not sure what the proclivity would be of people to +1 an ad.

Frederick Vallaeys: That raises another good point which is if you are using +1 as a publisher, an advertiser, or a business the +1 actually is associated to a certain URL. So, even if you don’t have a +1 next to your ad, but you get people to +1 your website, that all feeds into the same pool of data.

Later on when somebody searches and sees your ad, those recommendations will show up even if those +1′s were done from your website or the organic results. It’s a whole ecosystem that persists across all the different touch points you might have with that customer, whether it be through Google or through your own website.

As far as the volume of how many people have done this, I can’t talk about that. It’s still early stages for this, but we are pleased with the way people are using it at this point.

The power of using the new ad extensions

Eric Enge: One of our clients is using the seller rating ad extensions. That’s kind of a corollary, this whole business of including reviews and ratings into the whole process.

Frederick Vallaeys: Exactly. I think it fits into the bigger picture of new ad formats you see on Google, and it stems from the fact that we realize that sometimes the picture is worth a thousand words, and the ad doesn’t have to be purely text.

You can also answer with a map. If it’s a local search you can enhance with product prices and images if it was a product search. If it was a search for a new movie then it might make sense to show the trailer right there. Positive seller ratings and reviews are a good thing to surface because it helps build trust and brings in those clicks that an advertiser was looking for.

We’ve seen site links drive increases in CTR anywhere between 17% to 30%.

A specific example to look at is site links, which is probably the easiest of the new ad formats to implement because it’s literally going into your campaign and putting in up to ten links associated to each of your campaigns. We’ve seen these drive increases in CTR anywhere between 17% (search) to 30% (mobile). These are fantastic increases in CTR simply by showing more information that’s useful to users.

Eric Enge: Similar to the +1 button, it’s something the eye notices and attracts a little bit of mind space.

Frederick Vallaeys: Exactly. We want to be careful because people are drawn to new things, but we need to make sure that those new things are not just drawing clicks because they are different, but because they are actually useful. We are careful in terms of launching these new features and testing them and making sure there is actual user benefit in them.

On the flipside, when the user sees more it typically also means they are better qualified by the time they make the click and come to you as an advertiser, so you are more likely to convert that customer. A great example of this is again in the travel space.

Let’s say someone is looking for a destination and you have a travel site with car rentals, hotels, flights, and vacation packages. In the past you would have taken that user to your generic page where they could have done all four of those things. But, if you now show four site links to each of those different areas of your site, you’ve done two things.

You’ve told that user “hey, by the way you might not have realized it, but we also do car rentals.” The second thing is the user goes directly to that page for the thing they were looking for. Now you can take them to a page where, instead of cluttering it with the things they weren’t looking for, you actually put special offers and pitch the product they were looking for.

In the case of car rentals you show them what discounts are available in the space that you might have otherwise had to use to say “hey, you can also book flights here” which they weren’t looking to do at the time. It’s a positive thing for both the user and the advertiser.

Eric Enge: I saw what Apple did with site links. They show their current hot offers. It’s a very, very smart way to use that feature.

Quality Score and Position Normalization

Clickthrough Rate Eric Enge: Coming back to Quality Score and the click through rates. I assume you have some way of adjusting expectations based on positions, because obviously one would expect the first ad to get the most clicks. To put a strawman concept out there, if we thought the first ad was going to get 30% of the page search clicks, and the second was going to get 15% and so forth then if the first ad gets 25% and the second ad gets 20% then that starts to be a sign that the second ad is the better ad. Am I interpreting that correctly?

Position normalization says that we have different expectations for CTR for the different ad positions.

Frederick Vallaeys: Yes, you are spot on with that. We call it Position Normalization, and it’s exactly as you described. Having a certain CTR, say 25%, could be a really good thing if we were expecting you to get 15% in the position that you were in. Your Quality Score could go up. Many advertisers look at the CTR in their accounts and try to judge everything on that. However, it’s important to look at both the CTR number as well as the Quality Score number in your account.

Eric Enge: You want to look at them together as it’s a relative thing.

Frederick Vallaeys: Exactly. You look at them in combination, and the more important thing to look at in your account is the return on investments you’ve received from those ads. The Quality Score is a number we put in there to help you figure out where it is you could perform better and possibly decrease your cost and increase your position by having more relevance. If that is driving ROI, then that’s the only thing that matters to advertisers.

Eric Enge: You don’t want to lose sight of the end goal. The Quality Score is basically a tool to help you better get to that goal. The point you just made about the Position Normalization, is that you get to look at all the things together. I need to look at it in a holistic fashion so it can tell me where the opportunities are.

Frederick Vallaeys: Exactly, and a simple technique is to look at which of your keywords have a sub-bar Quality Score; and that could be any number. That could be the lowest ones in your accounts or it could be literally at a one level or a two level. Then you can look at your search query report.

From that you start seeing these different variations, and now you can start figuring out why is it that it wasn’t performing well at the aggregate level, and then how can I make my account more specific by building out new ad groups for these different search queries that we are also triggering.

Typically, when you do that, you increase your relevance because you are now taking more specific keywords and building ad text specifically for those which help you boost up your click through rate.

Tips for optimizing your AdWords account

Eric Enge: If you are a publisher that wants to do optimization on your account, what are the steps you recommend publishers should go through?

Frederick Vallaeys: I recommend that you look holistically at your accounts. Sort it on a keyword basis from lowest to highest Quality Score and apply some filter so you are not looking at anything that doesn’t have a lot of impressions yet.

Look at which ones have the highest volume and not a great Quality Score.

I would say a thousand impressions and up. That’s the baseline where you would start looking at it, and then do a secondary sort on that. Look at which ones have the highest volume and not a great Quality Score. Go after the high volume first even if it’s not necessarily the absolute lowest Quality Score, but it’s still in that bucket where the Quality Score is not quite where you want it to be, and start optimizing on those.

Then try to figure out if you could write better ad text for that keyword as it stands now or do you need to break that keyword into more specific variations, build new ad groups around that to create ad text that’s more compelling and maybe lead it to a landing page that’s also more specific.

Eric Enge: Is there an ad group or campaign level component to Quality Score?

Frederick Vallaeys: The QS is at a keyword-ad level. So the way you structure ad groups plays a large role in determining QS. However there is no ad group or campaign QS component. I.e. if you took the same keyword and ad and moved it to a different ad group or campaign, the quality score would remain the same.

Eric Enge: We did talk about Position Normalization earlier, but is there an argument in some situations for bidding higher? To drive history faster, or do things to try to help the Quality Score go up?

I think you hit the nail on the head with the statement that it (bidding higher) helps you build history faster in some cases.

Frederick Vallaeys: I think you hit the nail on the head with the statement that it helps you build history faster in some cases. Keep in mind when you bid higher it usually means you are going to get a higher position on the page.

In those higher positions if you go from being on page two to page one, that’s going to have a huge impact on how quickly you accrue impressions. It’s those impressions that will give Google the confidence to make a Quality Score judgment that’s specific to your account as opposed to the system-wide averages.

If you, as an advertiser, are doing much better than the system-wide average then it would benefit you to prove to us as quickly as you can because that will then decrease your costs in the long run.

It’s about building that volume, but not about anything else because there is Position Normalization. Bidding up to a higher position and getting that higher CTR isn’t a guarantee of getting a better Quality Score in the long run.

Eric Enge: Right, because presumably the Position Normalization is adjusted on a keyword basis. Position normalization for market expectations on one keyword might be different than the expectations on another keyword.

Frederick Vallaeys: Right.

Eric Enge: That eliminates any possibility that you could fool the Position Normalization algorithm with the bids. The only thing you gain is that you can accelerate the development of your own history.

Frederick Vallaeys: Exactly.

How the real time math helps advertisers

Eric Enge: In summary, the Quality Score we see in AdWords is actually a very valuable proxy basically for the real numbers because you can’t possibly handle the data for the real numbers as a human.

Frederick Vallaeys: Exactly. That brings up another good point. One thing I like to harp on is that Google has a lot of data, and we are very good at using that data to give the best results to advertisers. Conversion optimizer is actually a good example of this.

To the point that you just made, we at Google collect data on a query-by-query basis, can have an expectation of how that’s going to perform. The problem is that even if you had that as an advertiser, there would be no way for you to bid in real time based on those factors.

That’s where Google can actually do a good job for those advertisers, and that’s where conversion optimizer comes into play. That’s using all of Google’s power of crunching numbers to make sure that you are meeting your ROI targets, and let us handle all the heavy lifting of determining the right CPC.

Eric Enge: Thanks Fred!

Other Recent Interviews

Bing’s Ping Jen, September 28, 2011
Bing’s Duane Forrester, September 6, 2011
Danny Sullivan, August 8, 2011
Bruce Clay, August 1, 2011
Google’s Tiffany Oberoi, July 27, 2011
Mona Elesseily, July 18, 2011
Vanessa Fox, July 12, 2011
Jim Sterne, July 5, 2011
Stephan Spencer, June 20, 2011
SEO by the Sea’s Bill Slawski, June 7, 2011
Elastic Path’s Linda Bustos, June 1, 2011
SEOmoz’ Rand Fishkin, May 23, 2011
Bing’s Stefan Weitz, May 16, 2011
Bing’s Mikko Ollila, June 27, 2010
Yahoo’s Shashi Seth, June 20, 2010
Google’s Carter Maslan, May 6, 2010
Google’s Frederick Vallaeys, April 27, 2010
Matt Cutts, March 14, 2010

Filed Under: Google, interviews, PPC

adCenter Quality Score Defined, with Bing’s Ping Jen

9/19/2011 by Eric Enge 1 Comment

photo of Ping Jen Ping Jen is a Product Manager on the Microsoft Advertiser and Publisher Solutions Team. He has a passion for driving improvements into adCenter which helps advertisers optimize their campaigns and increase their competitiveness in the marketplace. Prior to joining Microsoft in 2009, Ping was a Business Administrator at University of Cincinnati Department of Neurosurgery. Ping is a Microsoft Certified Solution Developer (MCSD) and holds a MBA degree from the University of Notre Dame.

Key Interview Points

One of the things I learned when I was out at SMX Advanced in June was that the Quality Score that you see in the search engine PPC services (both adCenter and AdWords) was that the Quality Score you see displayed in your account is not the same as what is used by the engines in ranking your ad. For that reason I asked Ping Jen of the adCenter team to join me for an interview. Below we talk through exactly how Quality Score works in adCenter. Here are the key points from the interview:

Original content, content relevance to the ad, location, and layout are all factors in landing page relevance.
Advertisers whose pages are deemed to be harmful will get banned from the adCenter marketplace.
(Ping Jen) “Our philosophy is that we want our advertisers to have high ROI, and one of the ways we do that is by requiring them to have higher quality landing page user experiences and relevance. To help advertisers, we provide feedback through the Landing Page User Experience subscore and Landing Page Relevance subscore.”
(Ping Jen) “If you have some outliers within an ad group or campaign, determine why. Should I use this keyword? Does it belong to this ad group or campaign because usually KWs that share an ad group are tied to the same landing page? In some cases, it may be time to move those keywords to another ad group because they don’t fit into this landing page.”
(Ping Jen) “Before clicking your ads, search users will look at the content of your ads. Immediately, they can see if they are relevant to what they are looking for. We follow the same logic to validate your ad copy relevance.”
(Ping Jen) “badly spelled ad copy immediately reduces the confidence a user will have with the ad and they will shy away from it.”
(Ping Jen) “Placement is still determined by relevance, the landing page experience, the historical CTR and the advertiser’s price.”
(Ping Jen) “We have always been upfront that adCenter Quality Score is not directly tied into a rank score.”
(Ping Jen) “How do you know your KW performance against others bidding on the same term? We tell you with the keyword relevance sub score.”
Rank score, which is the term adCenter uses for the actual method used to determine ranking, is calculated on a marketplace by marketplace basis. This is done because the needs of each marketplace are different.
(Ping Jen) “Ads must comply with the adCenter Relevance and Quality guidelines. Then the ad’s competitiveness in KW relevance, landing page relevance and their bids will decide their ranks.”

Landing Page Quality and Relevance with adCenter

Eric Enge: Can you give us an overview of how Quality Score operates in adCenter?

Ping Jen: Our Quality Score is a strong signal of campaign quality and performance. The reason we introduced the adCenter Quality Score was to help our advertisers enhance campaign performance and raise the visibility of improvement opportunities.

We consider campaign quality an important factor and we want to showcase the best experiences in the marketplace and continue to grow the traffic volume and increase market share.

Eric Enge: How do you measure that landing page user experience? What factors are involved?

Ping Jen: If you Bing “adCenter relevance and quality guidelines”, you will find that adCenter has published very specific requirements for landing page and user experience. We measure the Landing Page User Experience and then validate whether advertisers have followed the guidelines and show the results through the Landing Page User Experience subscore.

Eric Enge: In the landing page guidelines you recommend analyzing the content and the user’s interaction with the content to make sure there is value.

Ping Jen: Yes, besides original content, content relevance location and layout is also very important.

Eric Enge: Your guidelines warn against too much advertising on the page, SEO manipulation, and doorway pages. You can break these guidelines into two categories.

One category is pages that are overtly harmful; for example, pop ups, parked sites and automatic software downloads. The second category includes guidelines that point towards an evaluation of the quality of the user experience when they click on an ad and arrive at the page.

Ping Jen: For the first category, we will prevent them from our marketplace if they create a harmful experience. For the second category, if everything else is equal we want to promote the content that provides the best user experience.

Eric Enge: At the SMX Advanced panel you and I attended recently, either Frederick Vallaeys of Google or Craig Danuloff indicated that landing page evaluation was not much of a factor in the Google Quality Score algorithm. I am not asking you to comment on Google’s approaches, I simply want to bring up the fact that adCenter’s approach appears to be different than the Google approach.

Ping Jen: Our philosophy is that we want our advertisers to have high ROI, and one of the ways we do that is by requiring them to have higher quality landing page user experiences and relevance. To help advertisers, we provide feedback through the Landing Page User Experience subscore and Landing Page Relevance subscore.

Eric Enge: It’s the key differentiating point in terms of driving a higher ROI.

Ping Jen: Correct, through Quality Score, adCenter aims to help advertisers improve their landing pages relevance and show ads accordingly. Initially the traffic volume may not be at its full potential but the traffic that comes to our advertisers is good traffic. In time advertisers will be able to achieve both quality and quantity goals by leveraging Quality Score.

Eric Enge: Can you tell us more about landing page relevance?

Ping Jen: Yes, adCenter has devoted a lot of resources to analyze landing page relevance and shares the findings through landing page relevance subscore. We want to help advertisers align their landing pages content with user intent. When a search query such as “golf” comes in, adCenter analyzes what golf means, which may be different context for different people.

For example, golf could be an event, sporting goods, a clothing line, or a golfer like Tiger Woods. The searcher might be looking for a golf vacation or information on the Volkswagen car. We can analyze these possible user intents from the search query side and validate the landing page to see whether you have enough information on your landing site to engage user intent for all possibilities.

Eric Enge: If an advertiser has an ad for golf and the keywords aren’t clear, you can look at their page and understand whether they are right for a golf clothing line or golf course query and perhaps target when their ads show. If it’s too vague, you can lower their Quality Score because a generic word like golf would be used by people for many purposes and, if they only match one of those, they will only fit a percentage of those users.

Ping Jen: We will list all the possible intents behind the term golf and then assess your landing page to validate whether it has enough related information to engage any of these possible intents. That’s the basics behind our landing page relevance check. adCenter has invested a lot of resources enhancing this feature so I strongly encourage people to take a deeper look at their Landing Page Relevance subscore. They can use this sub score as a strong signal to say “hey, do I have good information on my landing page to convert a search user for this term?”

Eric Enge: If someone has a poor landing page relevance score they have work to do, right? However, there are many reasons why their score could be poor. Are there tools you can suggest that would help them break this down and figure out what aspect is causing their landing page sub score to be low?

Ping Jen: First, look at the whole picture. At the campaign and ad group levels, do you have a good landing page relevance or not? If you can say this ad group or campaign is pretty solid, that’s the first step. If the answer is no, you need to reconsider your KW selections and how they are tied to landing pages.

Second step, look closely at the details. If you have some outliers within an ad group or campaign, determine why. Should I use this keyword? Does it belong to this ad group or campaign because usually KWs that share an ad group are tied to the same landing page? In some cases, it may be time to move those keywords to another ad group because they don’t fit into this landing page.

Role of Ad Copy in adCenter Quality Score

Eric Enge: What about the role of an ad copy on your overall score?

Ping Jen: We check the relevance of your ad copy as well.

Eric Enge: What are some of the aspects you look at inside the ad?

Ping Jen: There are simple ways to identify whether your ad copy can engage with user intent. Before clicking your ads, search users will look at the content of your ads. Immediately, they can see if they are relevant to what they are looking for. We follow the same logic to validate your ad copy relevance.

Eric Enge: I see one of the things you focus on includes correct grammar.

Ping Jen: Absolutely, badly spelled ad copy immediately reduces the confidence a user will have with the ad copy and they will shy away from it.

Eric Enge: Many people type in search phrases which are misspelled. One popular technique is to show the user’s keywords back to them exactly as they typed it. Is that something you recommend?

Ping Jen: No, we identify possible misspellings and what could be the correct term first. We then ask the search users if this is what they are looking for, and then show ads based upon search user’s choice.

Eric Enge: I’d like to talk more about the golf scenario. Let’s say you have someone who promotes golf vacations. A user types in the generic keyword “golf” and the golf vacation company wants to bid on it. A popular technique in your ad tells them they entered golf but your ad is about golf vacations so you screen out the people who were looking to buy golf clubs. Do you encourage this approach?

Ping Jen: Absolutely. Ad copy is advertisers’ first line of defense to low-value traffic. Having the search user help filter out unwanted traffic is a good technique that we recommend.

jaguar Eric Enge: A more dramatic example is when someone enters the search phrase “jaguar.” As you know, this is an animal, a football team, an operating system and a guitar. No one is going to cover all these situations so how do you handle this scenario?

Ping Jen: First we identify what the possible user intents are behind it. We assess advertiser’s landing pages to see whether those pages match up with any of possible intents.

Eric Enge: So would you show the most popular meaning when people type in jaguar, which might be the animal?

Ping Jen: No, we don’t try to alter the ad placement on this basis. Placement is still determined by relevance, the landing page experience, the historical CTR and the advertiser’s price.

Quality Score and Ad Position

Eric Enge: In terms of how the Quality Score is used, is it used to help determine the position of an ad or only whoever the ad shows or not? At SMX Advanced Craig Danuloff told us that Google Quality Score is not the Quality Score they use to generate rank score.

We have always been upfront that adCenter Quality Score is not directly tied into a rank score.

Ping Jen: That’s correct. That was news to a lot of people. However, there are good reasons why it’s almost impossible to have the displayed Quality Score determine the ranking of the ad. We have always been upfront that adCenter Quality Score is not directly tied into a rank score.

Eric Enge: So are you saying the Quality Score you display is a hint.

Ping Jen: It’s an indicator of how competitive your keyword is in our marketplace.

Eric Enge: If you do things to improve the Quality Score you see inside of adCenter, then you will probably be moving in the right direction?

Ping Jen: It’s not probably; you are definitely going in the right direction to make your KW/ad copy more competitive and enhance your landing page relevance and user experience.

Eric Enge: But it’s not a one-to-one with what is actually used in the algorithm.

Ping Jen: That’s correct, but it is a very strong indicator.

Click Through Rates and the Keyword Sub Score

Eric Enge: Do the click through rates play a large role in the algorithm?

Ping Jen: When we decide adCenter Quality Score, we also consider the advertiser’s expectation. They look at their CTR as a strong indicator of how their ad performed so we respect that notion and make sure adCenter Quality Score is moving consistently with their CTR performance.

Eric Enge: So the rank score is derived from a combination of factors including keyword relevance, ad copy, landing page, click through rate and, finally, the bid price.

Ping Jen: Correct. Hopefully you can help me and adCenter to drive this message home to the heart and mind of every advertiser. The message is very straightforward. Landing page user experience and landing page relevance is the cornerstone of the search alliance marketplace. We want to make sure you meet our requirements and grow together with our marketplace.

After you enter our marketplace, you have to compete with others looking for the same traffic. How do you know your KW performance against others bidding on the same term? We tell you with the keyword relevance sub score.

You have three different results, poor, no problem, and good. With the search term “jaguar”, the CTR average is 10% so if you are about 10.2 or 9.8 you are average. If you are below the marketplace average for this term you are poor. If you are better than the market average you are good.

Sub Score It is a useful tool to gauge your performance. Occasionally, people come to us and say I have a CTR of 5% so why are my ads not showing, or why isn’t it in the number 1 position? The answer is the CTR average for your keyword is 10%. Some people come to us and say I only have 0.8% so why do I have a high quality score? I tell them it is because they are doing better than the 0.2% average and have “No Problem” on landing page relevance and landing page user experience.

Eric Enge: Right, because you calculate the average performance based on each marketplace?

Ping Jen: Each specific term. There is a lot of nuance behind our Quality Score when compared with Google. This is a major difference.

How positioning works in adCenter

Eric Enge: If we were to write down on a piece of paper which ads show first, which ads show second, and how that is calculated, what would that look like?

Ping Jen: Ads must comply with the adCenter Relevance and Quality guidelines. Then the ad’s competitiveness in KW relevance, landing page relevance and their bids will decide their ranks.

Eric Enge: Is there anything else you would like to add?

The adCenter Quality Score is a channel we use to communicate to the advertisers they can monitor for improvement opportunities.

Ping Jen: I want to make sure people understand that the adCenter Quality Score is a channel we use to proactively communicate to the advertiser. For example, your ads are losing strength compared to your competitors or due to a new policy you suddenly see your user experience drop down to poor.

I encourage all the adCenter users to closely monitor their quality score and provide feedback to us on how we can fine tune the signals.

Eric Enge: Thanks Ping this was very helpful and we appreciate you taking the time to chat with us today.

Other Recent Interviews

Duane Forrester, September 7, 2011
Danny Sullivan, August 8, 2011
Bruce Clay, August 1, 2011
Google’s Tiffany Oberoi, July 27, 2011
Mona Elesseily, July 18, 2011
Vanessa Fox, July 12, 2011
Jim Sterne, July 5, 2011
Stephan Spencer, June 20, 2011
SEO by the Sea’s Bill Slawski, June 7, 2011
Elastic Path’s Linda Bustos, June 1, 2011
SEOmoz’ Rand Fishkin, May 23, 2011
Bing’s Stefan Weitz, May 16, 2011
Bing’s Mikko Ollila, June 27, 2010
Yahoo’s Shashi Seth, June 20, 2010
Google’s Carter Maslan, May 6, 2010
Google’s Frederick Vallaeys, April 27, 2010
Matt Cutts, March 14, 2010

Filed Under: interviews, Microsoft, PPC

Key Points from Interview with Frederick Vallaeys

Full Interview Transcript

Mobile Search and Mobile SEO – Full Transcript

Key Points from Interview with Stefan Weitz

Full Interview Transcript

Optimizing Revenue for News Sites – Full Podcast Transcript

Briefing with Ping Jen

New Quality Score Data Provided

Summary

Key Points

Interview Transcript

Other Recent Interviews

Introduction

Interview Transcript

Other Recent Interviews

Key Points

Interview transcript

Other Recent Interviews

Key Points

Full Interview Transcript

Real Time Quality Score

Personalization and selection of Ads

The role that the +1 button plays into the Quality Score

The power of using the new ad extensions

Quality Score and Position Normalization

Tips for optimizing your AdWords account

How the real time math helps advertisers

Other Recent Interviews

Key Interview Points

Landing Page Quality and Relevance with adCenter

Role of Ad Copy in adCenter Quality Score

Quality Score and Ad Position

Click Through Rates and the Keyword Sub Score

How positioning works in adCenter

Other Recent Interviews

Sign-up for Email Updates

Latest Blog Posts

Blog Categories

Invesp #27

27th Most Influential Marketer!

Art of SEO Book

The Art of SEO

Matt Cutts Discusses STC Case Study

Latest Posts

Blog Categories

27th Most Influential Marketer!

2006-2008 Interviews and Podcasts

Latest Tweets