Transcript of Podcast with Jim Sterne
Podcast Date: February 22, 2007
The following is a written transcript of the February 22, 2007 podcast between Eric Enge and Jim Sterne:
Eric Enge: Excellent. So, today we are here with Jim Sterne. Jim is the President of Target Marketing one of the most well-known people in the Web Analytics space, and is also President of the Web Analytics Association. This is Eric Enge speaking. I am the President of Stone Temple Consulting and you can see our website at www.stonetemple.com. So with that Jim, we would like to get started.
Jim Sterne: Thanks very much. I am delighted to participate.
Eric Enge: Excellent! So, what advice would you give to companies just getting started in analytics?
Jim Sterne: Boy, the main number one single most important thing is to know what your goals are. And, it is true for every company, every size company in every country. When I started doing consulting in this area it was how do we make our website better? And, the answer is better at what? What are you trying to accomplish? Today, it's what should we measure? And, the answer is well that depends what are you trying to accomplish? And if you have specific business goals, from which you can derive specific web goals, then that will point the way towards which metrics will tell whether or not you are reaching your goals.
It sounds very simplistic and you want to dive into technology, and is page-tagging better then block file enlisting. The most important thing is that you understand not just the total purpose of the website. But, then much more specifically we can get down to each individual page. Why is that page there? Well if I know what its goal is, then I could determine whether it is working, and measure whether a change is an improvement or not. So, that's the number one thing to get started with. Number two would be, take a look at whatever you have on hand. If you are a small company, and your website is hosted elsewhere, chances are excellent that there is some sort of report available to you giving you visits and sessions and bytes; just really basic, basic stuff, which is not a bad place to begin.
And, then it is a matter of slowly moving up that food chain of complexity, one step at a time, so that you are assured that you are using the available data before you try to bite off more than you can chew. There is a good beginning. Oh and the other advice of course, I have to, because this is the portion of the program where we plug the speakers' business, the Emetrics Summit, is coming up in the beginning of May in San Francisco, and for those of you listening from overseas, even sooner towards the end of March in London, and in the middle of April in Germany. It's a great place to get the big broad picture of what it takes to make your website better; optimizing online marketing value, and a good place to learn from others. So, there end of plug.
Eric Enge: My apologies for forgetting to mention that, because I was aware of that as well.
Jim Sterne: Oh I was going to make sure it got in there some time.
Eric Enge: Oh yes, I could count on that. So, if we are looking at the goals piece in a little more depth it's, a lot of people are going to say well my goal is to sell a product.
Jim Sterne: Good.
Eric Enge: But, it gets a little more complicated than that once you get into the details of it, doesn't it?
Jim Sterne: Sure. You know that; an ecommerce website is the simplest one from the perspective of web analytics. Because, you have an absolute goal that has an absolute value, boy what if; thanks you just took care of fifty percent of the problem of making all the stuff work. You clearly know what your goals are.
Eric Enge: Right.
Jim Sterne: But, what we want of them look at is, well what does it take to sell something? What is the persuasion process? How good at are you at attracting attention? So, how good are you at making noise, getting people to look your way, and recognize your logo, and click on your banner ad or choose your search engine keyword to click through on? Once they show up, what do they need to know in order to make a buying decision? And, what is their persuasion path that they have to go on, in order to say yeah that's a good idea, I will buy that.
For some companies it's; well gee I mean I go to Amazon usually because I have something in mind. I put in a search, I hit the button, it comes right up, I click on the link, and then there is a buy now button. So, in three clicks I am done. Other companies, if I am selling a refrigeration system for a warehouse; well there is going to be multiple types of people who are going to be interested in different kinds of information, and are going to take months to decide. And, they are going to have to visit the website repeatedly in addition to visiting a showroom or having a salesperson come out and make a call, or calling the call centre several times.
So, we've got multiple people who have multiple needs at different points in time. Oh! Well, now it gets very complex very quickly. If it's pure ecommerce buy now great, easy; well, hope it's not easy but easier right on, if it's a considerate purchase with an eighteen month sell cycle, and a committee is involved in the purchase. So, there is a huge amount of it depends that is going on there as far as; but what specifically should we do today? If I have a clear idea of not the sales process, but the buying process; if I have a clear idea of what my prospects are going through while they are making their decision. I can build a website to accommodate them, then I can measure with the success of that website against those criteria.
Eric Enge: Right. It's what our mutual friend Bryan Eisenberg, would call persuasion architecture.
Jim Sterne: Exactly.
Eric Enge: So, what are the major sources of errors in web analytics?
Jim Sterne: Well, first of all believing the reports that come out of the box, that's a problem. It is the desire to have all of the answers; oh look, I bought this product or I've signed up for this service, I tagged all my pages, and here come a bunch of reports. Oh good, there my problem is solved. Well, actually the effort has only just begun; probably the biggest mistake is not investing in the interpretation of those reports. So, we've got to talk about errors in a couple of different ways; errors in using the tools, and the errors that the tools produce.
Eric Enge: Right.
Jim Sterne: Web analytics data; they are not precise. So, if I had a yardstick that's three inches short, and I measured myself. And, you had a yardstick that was the right length, and you measured yourself, I am going always be taller then you, and I am going to feel great about that. I am going to be wrong; the way the reason that this is not a problem for web analytics is that I am going to use the same yardstick to measure everybody.
What's important is not that it's exactly three feet per stick; what's important is how much did I grow between yesterday and today? How much different was the response to my website when I changed the background from blue to green? How much more traffic did I get when I changed the call for action in my landing page or my banner ad between version A, and version B? How much better was the conversion rate when I shortened the persuasion path?
I have very frequently found that we just don't know what the result is going to be, and you just have to test stuff. So, if today I have got 2,783 people visiting my one particular page, and I make a little change to the preceding page, the next day I get 5,627 people. The fact that it was exactly 5,627 is immaterial. What matters is I almost doubled my traffic. So, that's actually better to double my traffic. So, that's what's important. It's the difference between measurement A, and measurement B.
When people get upset about web analytics not being precise, they are worrying about the wrong thing. Now, there is one area where it matters, and that is I am trying to compare apples-to-apples. If I am buying a million impressions on MSN, and a million impressions on Yahoo, and a million impressions on Wall Street Journal's website. How do they measure impressions, and how are they calculating, and how do those numbers compare? That's a tough problem to solve; because I do need apples-to-apples comparison. But, classic web analytics, click through page views, revenues on my own website, precision is not as important as understanding that I need to measure the difference between version A and version B.
Eric Enge: Right. So, in an ideal world gee it would be great, if you could have a tool that was a hundred percent precise. But, since there are inherent reasons in the whole structure of web technology and where analytics fits into it, to prevent that from being the case, the lesson is to learn that the tool is still really useful and powerful if you just look at it as a tool that allows you to measure progress and differences, and things that you try. Is that a fair summation?
Jim Sterne: Very, very fair, and probably the thing that is most disconcerting to me is that, any other marketing tool, or method that we have, communication device, you know whether it's television, or magazines, or newspapers, or hanging flyers on door knobs, the internet is so massively more measurable than any of those.
Eric Enge: Right.
Jim Sterne: That to expect perfection is a total waste of time.
Eric Enge: That way, we are all geeks you know.
Jim Sterne: Yeah, well.
Eric Enge: Entirely.
Jim Sterne: Yeah.
Eric Enge: Yeah, we expect perfection, but yes that's an excellent point.
Jim Sterne: I would like to quote Einstein on this one that perfection is the enemy of good enough.
Eric Enge: Oh yes. Well, and as you say compared to the kind of measurement you get of other marketing campaigns, it is perfection in comparison.
Jim Sterne: Absolutely.
Eric Enge: Alright. So, let's talk a little bit about the key performance indicators, and what they are and the process; and I am getting back to the goal conversation earlier.
Jim Sterne: Yeah.
Eric Enge: You know the process of figuring them out.
Jim Sterne: Well, a key performance indicator is that which tells you whether you are headed in the right direction. So, for some website it is purely how many people showed up. We have got a website, and its only purpose is to try to change people's minds. Hey, we want you to recycle more, we want you to drive less, we want you to give blood, and the more people who see the message the better. So, the key performance indicator for us is number of visits. Well, gee we actually have take that a step further; what's more important is number of unique visitors.
If we have a million visits and its just one guy who comes back million times, we are not reaching our business goals. If it's a million people who all visit once and stay for five seconds, we are not reaching our business goals. So, the key performance indicator is going to be a little trickier there. It's going to be a combination of the number of people who came, the links that they stayed, the amount of content that they consumed, and perhaps whether they came back, perhaps whether they subscribed to the news letter, or registered for the webinar.
So, it's a matter of identifying what our goals are and then figuring out what the matrix are behind them. The methodology for identifying key performance indicators is, its one of those things that if you troll around in the good blogs, you know if you start with the web analytics demystified.com Eric Peterson's blog, and you go over to Avinash Kaushik's Occam's Razor blog, and then you know just follow all the blog roles from there. You are going to find some really good ideas about how to create KPIs.
They have to be specific; they have to be well-defined. They have to have a goal attached to them, I mean it's not you know I am six feet tall, so what. Am I trying to grow taller, oh well, if that's the case then I am going to measure a little bit more carefully? Do I want to measure this to compare myself to others? So, that oh now we are doing talking about analysis. So, I want to look at specific goals for specific metrics over a specific period of time. So, if my website goal is to increase sales, that's useless; to increase sales by fifteen percent, okay, well that's at least better. But, to increase sales by fifteen percent over the next three months, now we have got something we can work with, and from that we can start to write the KPIs.
Eric Enge: Right. So, that's measuring now the things that are helping you accomplish that fifteen percent growth in three months.
Jim Sterne: Exactly.
Eric Enge: And, you can begin to draw a picture. Well, if I am going to do that, and I need this to happen to you know visits to this page, visitors to this page, and these kinds of transactions to occur, and you can start to measure material things.
Jim Sterne: And, I want to know where I should invest my efforts. If I have got a great website, and nobody shows up, obviously I want to spend some money promoting it, and putting people in the top of the persuasion funnel. If I have got a gazillion people coming to my website, but they take one look and run away, I don't need to spend anymore money on advertising, I need to spend some money on the right up front grab you, explain the offer, explain the value and take you to participate further. Thus, people get half way through that process and bail out. Oh, well then I need to spend time figuring out what's wrong with my website, is it too slow, does it not have enough information, does it have too much information, am I taking advantage of too many hi-tech flash interactive things that people just get put off and they run away.
So, I need to investigate not just by click through page views, but by doing surveys, and getting attitudinal response to my website. This is a piece that it's not classic web analytics, but it is a critical factor, how do people feel about it. I can measure what they do, but I also need to ask them how they feel. So, that will tell me where people are dropping out of the persuasion path. Finally, I get down to the call the action, now that might be purchased, it might be registered, down load, participate, respond to a survey, could be all kinds of things. It's an event; the visitor does something that is the conversion event.
What is happening right at the tail end there that is causing people to abandon the shopping cart, or refuse to register. If I have got a giant funnel that comes to a little tiny pin hole at the bottom, I need to figure out where to open up the hole at the bottom to get more people to follow through. So, keeping track of how people go through the persuasion process helps me identify where I need to pay attention.
Eric Enge: Right. So, in terms of the Key Performance Indicators, is it fair to say that if you see the numbers and some variation in the numbers, it is not likely to cause you to take some action, that it's not a key performance indicator?
Jim Sterne: Well, I mean it is a; so a key performance indicator in my automobile is my oil temperature, as long as it's stable I don't care.
Eric Enge: That's right. But, if it moves to some point where it's not stable that's what I meant.
Jim Sterne: Oil pressure goes down, and temperature goes up. Oh, now I have got, yeah now I have to take action. So, a Key Performance Indicator can be valuable, and that it tells me no action is necessary.
Eric Enge: Yes.
Jim Sterne: But, none of this has value unless I have specific numbers that I am looking at that clearly tell me, oh I need to do something about this, or I have an opportunity to take advantage of that, and that's where you know ad hoc reporting become so valuable. When people in the company wake up to the fact that you know what, I am not just, oh look we got three percent more people today than yesterday, okay everything is fine, I will go back to the rest of my work.
But, instead they say, well gee, I wonder how I could move; what levers do I have to change that number, what if I did this, or what if I did that, can you show me how many people who came from this source, accomplish that task as compared to came from that source? And, the response from the analytics person, from the analyst should be, well, yeah I can, but what are you going to use it for. What action are you going to take, based on numbers? If the person comes back and says, oh, well if I see that twice as many people from source A are converting then from source B, I am going to increase my spend on getting people from source A by twenty five percent, oh great, here is the number, here is the report, help yourself out.
Eric Enge: So, let's talk a little bit about the daily reports that come out, you know the automated reports that we put together here, and what role those play, and how you might tailor those to different audiences in your company?
Jim Sterne: Well, the biggest role they play is to put people to sleep. The standard reports that come to out of the box are fascinating. So, we can go through a process, you get it you; the first time you implement a web analytics tool, and you crank out the standard reports, and you spend two hours following through them to see what the heck is this? You are going to find two or three really weird things.
There are going to be some weird traffic patterns, and some strange places where there is leakage from the persuasion path. You go to the page that you are looking at to see what the heck, how come everybody is running away from this page, and the buy now button is broken, or the important information is below the scroll line, or something that's going on strange there. You go, oh, how do we miss that, just fix that; and suddenly your conversion rate goes to the ceiling.
Everybody is happy, and the return on investment is instant and very gratifying. The second time you print out those reports, oh well, we have already take care of the abnormalities, now what we do. So, now it's a matter of knowing why we are looking at the reports not just; when I did my, the White Paper back in 2000, we interviewed 25 companies. How are you using web analytics? Oh! We are overwhelmed with data; we don't know what to do? So, the White Paper was five pages of survey results and sixty pages of what they should be doing. In 2002, when I turned that into a book, I interviewed another 25 companies, how are you, what are you doing for web analytics? Oh! We have got a web analytics tool; we have got a huge amount of data, but now it's being managed, it's being captured, it's being massaged, it's being cleansed, and we are reporting out on it every week. And, then what happens. Well, then we do it again next week.
And, who is looking at the reports? If nobody is actually consuming it, they are a total waste of time. So, what we want to do is get the cultural change management thing happening, to get people to comprehend the value of web data. So, that they will use it to make business decisions at that, so on; the KPIs that are just telling that I don't need to do anything, great, there is a dashboard that says everything is green, or yellow, and that's fine, oh, something's red, I got to go pay attention.
The real value web data is when people say, okay, the dashboard is fine, but now let's go experiment, let's try this different offer, let's try this different page design, let's try this different photograph, and see what impact it has, and when you get into a mindset of doing that, it's a process of continuous improvement. We have always got this instant feedback of; we made a change, we got a result, oops let's pull that change out, or let's try a different change, oh that was an improvement; let's try some of those changes elsewhere in the website, well, everything improved. How do we improve it again next week? And that's where the stuff all comes together.
Eric Enge: Right. So, the standard daily reports that I referred to, I mean you can include in those dials and gages, that you know tell you when the oil pressure has blown, or something like that, which tell you that you need to do something. And, if you have a campaign underway, and you have got a set of key performance indicators to help you get this fifteen percent growth in three month. You could put that in your daily report so that people can see how their campaign is progressing.
Jim Sterne: Yeah.
Eric Enge: But, the goal is to keep all the information that something that the various people who are receiving it actually use.
Jim Sterne: Yes.
Eric Enge: Right, and then you know on top of that, there is also an aspect, isn't there that different people in the company might need different things?
Jim Sterne: Everybody needs different things, and for a number of reasons people need different things because they have different job responsibilities. So, they need to have different insights into what's happening on a website. Different people respond to data representations in different ways. I am a visual person. I want to see a pie chart, because I can instantly grasp the ratios.
But, other people are numbers people, and they look at a spreadsheet, and they can instantly see the relationship between the numbers, because they are looking at the numbers. Other people need to hear a story. They need to hear it in terms of well, this kind of individual shows up on our website, and this is the kind of experience they have, and if we change this it might make that better; and majority of people are doing this. So, it's not just where am I in the organization, but what kind of cognitive assimilation works best for me.
Eric Enge: Right.
Jim Sterne: The higher up I am in the organization, the less data I want to see. If I am a CEO, I want a dashboard that includes one dial that says website. And, it's red, yellow, or green, and if it's green, great, I got a gazillion of other things I've got to worry about. If it's red, then I will click on it and drill-bound, and see the five things that make up my one dial. So, is it traffic, is it maybe conversion, maybe its amount of content consumed. Now, if I see one of those as red, I am going to pick up the phone and call Sally, say hey Sally, you are; the area you are responsible for showing up red what's going on.
Sally needs a whole different set of data to look at, because Sally knows a lot more about what's going on under the hood then the CEO does, and, needs to see the ten or eleven things that cause CEO's dial turn red, and then so on so on down to the guy doing the AB split test, who needs to see absolute results, absolutely right now.
Eric Enge: Right.
Jim Sterne: The CEO could checklist.
Eric Enge: So, the other aspect of this that you mentioned earlier is, you know the ad hoc type analysis. And, this is now where; it's not the automatic, the generative reports, not the dials and gages. It's digging in and really figuring out what makes you know some aspect of the website tick, so maybe a big conversion problem of some sort. This is an incredibly important part of analytics as well.
Jim Sterne: Yeah, and it's; I contend that we have all the answers already. The technology is better, and more complete, and more capable then we are able to take advantage of it. We have all the answers. We are not good at asking questions. So, the person who figures out how to ask good questions is going to be the one with a competitive edge. It's you know the question asker of the question it's not; so here is an example from years ago macys.com, IT people there said, you know what this is really a pain maintaining Netscape for --, for PC and Netscape for Mac.
So, what percentage of people come to our website, and it was, it's like eighty percent PC or twenty percent Mac. Okay, well twenty percent that's okay, but when it got down to fifteen percent they went to marketing department, and said it's only fifteen percent. We think we should stop doing this. Somebody in the marketing department said, well, wait a minute. What's the comparative conversion rate? IBM people, the PC people that showed up, it was two percent conversion; perfectly reasoned. Macintosh people who showed up thirty percent conversion; nobody knew why. But, when we look at not the percentage of visitors, but the percentage of revenue, suddenly we had a different answer.
So, the important thing is asking that question behind the question, and that's what the ad hoc reporting is all about. It's this engaging your mind with the process, instead of just looking at the number and saying, okay, it's within normal ranges, we don't need to worry.
Eric Enge: Right. So this notion of understanding how to ask the question, I think is really a, it's a great insight. Because, as you suggested there is so much web analytics data available, you can get completely lost in it, and you know find yourself wasting hours of time accomplishing little.
Jim Sterne: Yeah, the real drawback here is that the web analytics data is really interesting, but until it's useful it's a waste of time.
Eric Enge: Right. So, you got to ask the right question.
Jim Sterne: Yeah.
Eric Enge: Or else you will be investing that time in the wrong way. And, that leads to next question really which is, the role of the analyst and how important it is to have a good one?
Jim Sterne: It's critical. The analyst is the arbitrator between raw data and meaning. So, what you know I've got a ten percent increase in the number of people who went from page A to page B. Why is that and how are we measuring that, and is it statistically significant to look at it that way. I'll go back to one of my favorite quotes from a guy from macys.com, and he was from the IT department. And, he said you know with all this data sometimes you just got to tell the marketing department what the numbers are, what they mean, and what they should do about it? And, the challenge here is as a marketing you know if I have fifteen years of experience in marketing, I don't have fifteen years of experience in building websites.
So, what do these numbers actually mean, how are they derived; what is this conversion rate or ratio off? And, that's a big time consuming effort. The analyst sits in between the raw data and the reports that land on the desk of the business people. And, the analyst is the one to say based on these numbers, we are seeing this kind of behavior happening on the website, and perhaps this kind of a test would be an order. The analyst is the one there when the marketer says well, what about this? The analyst can say we don't have enough data to give you an answer on that, or do you really mean this question instead. And, these people are not easy to find; they are mathematicians who understand business really well. And, there watch their way to go.
Eric Enge: Absolutely. Yeah, just as an example of the; a really simple example of what do these numbers mean; people get confused with that, what the term visitors means?
Jim Sterne: Oh, yeah.
Eric Enge: On their site, and they may not be aware of the fact that you know many users, humans use multiple computers.
Jim Sterne: Oh, yeah.
Eric Enge: And, each computer they go to, they show up as a different visitor; or you can have a family computer that has multiple users. So, it could be four humans using a computer, but it will show up as only a single visitor. So, it's really interesting; I know that's a very simple example compared to what you are learning to; I think it sort of illustrates just the tip of the iceberg in a puzzle.
Jim Sterne: Yeah, it is; I mean there is two things going on; there is the puzzle of the data, and then there is the mystery of the meaning.
Eric Enge: Right.
Jim Sterne: And, it's different kinds of people who are good at solving puzzles then people who are good at solving mysteries. So, you need a team to help you out with this.
Eric Enge: Right, great. So, for our last question, how would you advice website owners to allocate their budget between software tools, and people resources?
Jim Sterne: We all point to Avinash Kaushik, who famously said that for every ten dollars you spend on a tool, you need to spend ninety dollars on the people. The tools are great, the tools are anywhere from free to a couple of thousand dollars a month, or maybe I mean a big website, maybe as twenty thousand dollars a month. And, they will crank out data right, left, centre, so then you got a big pile of data. What you really need is somebody who understands statistics, who can draw meaning from the data, and then a business person who clearly understands the business goals?
And, that again is the initial challenge; what are your goals? But, also understands enough of the underlying message, the underlying machinery of a website to say based on these numbers, let's try this; let's try that. The people who have the biggest advantage of this, have an experimentation mentality, and I point to just Jeff Bezos at Amazon. Everybody I've talked with Amazon says the same thing; this guy walks into a meeting, and he's got four hundred ideas in about thirty seconds. And, he is not married to any of them, he is; you know it is throw the spaghetti at the wall, if it sticks go with it, if it doesn't, get rid of it. And, Amazon has enough traffic that they can get statistical significance in fifteen minutes.
Eric Enge: Right.
Jim Sterne: They could try infinite number of experiments, and there you have started to automate those experiments. So, that they are constantly, constantly try and do things. That's the way to win.
Eric Enge: Right. But, doing that obviously does take a significant amount of people resources to execute effectively.
Jim Sterne: Absolutely.
Eric Enge: Now, imagine for a smaller business that their ratio might be a little bit different then Avinash's famous ninety ten ---.
Jim Sterne: Well, I don't know; for a smaller business I can use google analytics for free, or I can buy; I can use ClickTracks for very inexpensive.
Eric Enge: Well, then it's a hundred, zero isn't it?
Jim Sterne: Sure, exactly. You know, the point is that somebody says Oh! Look we can get this web analytics tool for free, it won't cost us anything. Oh, not true. It's free, but there will be a cost in implementation, and there will definitely be a cost in interpretation.
Eric Enge: Right.
Jim Sterne: So, the people are absolutely critical, and so we've got a growing industry of outside consultants. You can go to any of the web analytics tool of vendors' websites, and click on company partners. And, those partners are small, you know anywhere from two guys to a hundred people companies who will come in, and workshop with you to figure out your goals, identify what matrix you should use, help you interpret the reports, tell you what's going on, and then tick the box and they are off to the next client.
And, you've got six months worth the work to do, and when you've accomplished that bring them back in. And, that's you know that's a way to kind of outsource the intelligence if you will. And, these people are out there helping all kinds of companies, all time of day and night. So, they recognize patterns, and can identify the practices, and you know their worth; they too are worth their way.
Eric Enge: Alright, great. Well, thanks for taking the time to speak with us today, Jim. I enjoyed it, and I certainly hope our listeners did as well.
Jim Sterne: I enjoyed it, and I have same hopes that you do, and if they want to know more emetrics.org. Thank you for including me, and thanks for the great questions.
About the Author
Eric Enge is the Founder and President of Stone Temple Consulting (STC). STC offers Internet marketing optimization services, including SEO, Social Media and PPC optimization, and its web site can be found at: http://www.stonetemple.com.