Comparison of 7 Different Analytics Packages

Stone Temple Consulting is pleased to announce the publication of the 2007 Web Analytics Shootout – Interim Report. A product of many months of effort (we started on this on October or 2006), and can now show data from 7 different web analytics packages running side by side across 4 sites.

This allows us to do some very interesting comparisons of all the packages. We also provide detailed analysis of the results, and offer commentary as to what we think it tells us.

One special test we conducted was designed to measure how the deletion of third party cookies differes from the deletion of first party cookies. Our analysis shows that this differs by 13%. Check it out.

Comments

  1. Please take a look at http://www.nextstat.com. It is a great alternative to the ultra-high priced alternatives.

    Doug

  2. Wow! Awesome job Eric. Very interesting report. We have been waiting for something like this to hit the web. You shed some very interesting light on several areas of web analytics. What I found very useful was the comparison to TPC and FPC. I gained a lot of insight into some specifics on how analytics packages actually functions. I never knew that analytics companies created a standard on the definition of a session and that after 30 minutes of inactivity the session ends. Some very useful insight here Eric!!

    Scott

  3. This is very useful and I am sure will help vendors and consultants such as ourselves reduce the amount of [free] time we spend in engagements trying to convince the client that there is a good reason why our numbers are different to the previous vendors. (whether better or worse, although there is less argument if they are better!) Even being ABCe accredited doesn’t convince them so maybe this will.
    Thanks
    Paul

  4. Very interesting – I’m looking forward to the full report! I spend a lot of time looking into various data collection methodologies, and how these compare across different systems, and so this interim report has made interesting reading.
    A key challenge at the moment is the transition of the analytics audience away from technical teams and into the boardroom. As a consequence terminology, planning, and constant communication are absolutely essential to ensure optimal interpretation of web analytics.
    Without good planning and implementation, as these results show, the metrics can be confusing and the accuracy questionable.
    Finally, I’d be interested to know the order in which the ‘tag’ include files were placed on the pages of the sites. We have found that ‘tags’ placed beneath other ‘tags’ can be affected detrimentally, i.e. lower reported traffic figures, as they will be processed after the initial tag.
    I look forward to the next installment!
    Regards,
    Sean.

  5. hi, Eric, how are you?

    I just wanted to post to thank you for this. There’s some fantastic info in there & the charts are really useful. For an ‘interim’ report, this is just incredible. Looking forward to the full version!

    daniel

  6. This is really great information that site owners and webmasters need to see before making decisions on analytics packages.

  7. Hi,

    I have been scouring the web for information like this, I can’t to see all of your findings. This is a huge experiment that will be highly commendable. My only thougts are you worried at all that having different dates in such a controled experiment will skew your information. (though the cookie interim is great so far!) Sites like brick and mortar businesses have seasons of low and high visitors, so comparing these tools and how true they are in measuring visitors and unique might throw some people off. I was wondering if in the full report you were going to disclose your equations that gave you your findings.

    Thank you!

  8. Eric Enge says:

    Hi Chacon,

    It’s certainly possible that the data may skew a bit for highly seasonal businesses, but I suspect that the differences in results are driven primarily by the way that the implementation of the software by the analytics vendor matches up with the implementation of the site, rather than the specific visitor flow.

    I will seek to remember to include the equations used within the final report. If I forget, don’t be afraid to remind me!

  9. Does the final report indicate the sequential order of vendor javascript on the page? Slow loading pages(heavy with 3rd party ads, Flash objects, the standard garbage, etc.) may not load completely before a visitor clicks to leave the page. One vendor’s code placed at the bottom of the page with another vendor at the top of the page could result in a page view|visit not being recorded for the vendor falling last in sequence. Additionally, depending on the weight of the “customized” code(I’m looking at you WebTrends), javascript code coming after a 30-40kb+ block of code or external reference may not have a chance to execute.

    Would be interesting to know if this was factored into your measurement.

  10. Dave – The Final Report does an analysis of Javascript placement on the page, and how it affects results. This includes a test related to Javascript order on the page. It should provide some insight on this issue.

  11. Eric,

    Great work in the publication of this test. I, like many others, learned more about how these analytics systems gather and report information. Who knew a session times out after 30 minutes?! As I am currently researching analytics systems for my own company this comparative information came at just the right time.

    I’m eagerly awaiting the delivery of the final report!

    Thank You!

  12. Letícia says:

    Despite all the information provided on the report really helped me and opened my eyes to issues I have never though about, I am still feeling unsure about which is the best web analytical tool to use.
    I have been analyzing data from WebTrends and Google Analytics, and thinking about using the WebSideStory. But as far as I could understand this one delivers the lowest numbers from all tools?! So, how can I believe it is the best option???

  13. Hi Leticia,

    The final report will be released on August 27th, 2007, which is 1 week from this coming Monday. In it, you will see that Visual Sciences (formerly WebSideStory) reporting the lowest numbers is not necessarily a negative for them.

    I think the bigger question is whether or not the features and functions of the product meet your needs.

    That aside, there are issues that you need to manage with any analytics package to ensure the best possible accuracy. Watch for the report. It’s coming in 12 days!

  14. Letícia says:

    Hi Eric Enge,

    Thanks for your update.
    I agree with you about the bigger question being “whether or not the features and functions of the product meet my needs”. But even if the toll provides me all the functionalities I need, it seems still essential to have the most real results to analyze and understand the best direction to take on the website, don’t you think?! My most concern is analyzing the results as they are bad, when the truth is the opposite, and take wrong conclusions due to the incorrect numbers…
    Don’t you think this can happen if the tool doesn’t provide me the most accurate numbers?!

    Many thanks
    Letícia

  15. Leticia – This is really a very common concern for people – whether or not their analytics software is accurate.

    Here is the way I personally think of it: Even if the package only accurately tracks 80% of the visitors, that provides an enormous wealth of data.

    I know people who work in the printed magazine industry, and they regularly run their businesses on surveys of their subscriber base, and they are thrilled when they get 20% of the people to respond.

    It is well known that a sampling of 20% of this kind is statistically significant, and they have run their businesses this way for many decades. Of course, these people would die to get data on 80% of their audience.

    While you need to make your own decisions, we use analytics as a tool in our business.

  16. Letícia says:

    Eric,

    many thanks for your response. It is very good to share opinions about this delicate decion!

    letícia

  17. Anirban Biswas says:

    Hi Eric
    This is mindblowing and eye opening, we in India need to be educated on this stuff desperately, how about coming down for a talk some time, ill foot the bill on the drinks bit, seriously :)
    Anirban Biswas
    SEO Black Hat

  18. Eric

    awesome job.

    Was there a final report, were there any significant differences from the interim report?

    Is the report being re-run in 2008?

Speak Your Mind

*

*