Stats Weenie - December 24th, 2004

Eric Meyer wrote a piece last week about browser stats. To quote the lines that summarise the entire post:

Don't Care About Market Share

[...]

The only stats that matter are the ones that come from the site you’re designing.

Given that stats_weenie exists entirely for the purpose of discussing other peoples' stats, it doesn't take psychic powers to conclude that I disagree with this. Eric does make several cogent points about the problems inherent in using such stats, and his piece is well worth a read.

However, my version of the above summary is: "the most important stats are those from the site you're working on". Unfortunately, there are some problems and limitations to those stats (several of which are picked up by Eric and those commenting on his post).

As Eric points out, stats for a single site can be skewed by compatilibility issues with the existing site - if it's completely unreadable in Browser X, then visitor numbers for that Browser are going to be suppressed. However, in these days of improved standards support (compared to the IE4/NN4 days) if you're publishing a website which only works in a couple of browsers, then you're a dolt (disregarding demonstration pieces - single-browser intranets have an excuse, too, though they run the risk of lock-in or a nasty upgrade surprise).

Another issue picked up by the commenters is: what if you can't rely on stats for the site in question? There are several reasons why this could happen:

• The site doesn't exist yet. Professionally, this is a big issue for me. Many of the sites I work on are completely new, from-the-ground-up.
• The existing site doesn't have any stats tracking set up. Laughable as it may seem, this is distressingly common. Many people buying websites are completely clueless. They often don't even realise that browser compatibility may be an issue (until they get complaints, then It Must Be Fixed Immediately, If Not Sooner, natch). In some cases, they are actively hostile - we've actually had clients say "damn accessibility/minority browsers" in meetings.
• The owner of the existing site won't let you see the stats. Yes, it's unprofessional, but this is also depressingly common. Sometimes it's because of a "turf war" with the Client's own IT people, who are sore at the development being outsourced. Sometimes it's because the Pointy-Haired Bosses are loathe to let such commercially vital(!) information outside their immediate sphere of influence. In any case, it happens.
• The site stats are compromised in some way. For example, the existing site is so low-traffic that anything below 30% is statistically meaningless. Or, to pick a real-world example from my experience: 50% of the hits on the site are automated bots, sending IE5/6 user-agents. To those bots, most CSS/HTML changes were completely irrelevant, but if we moved a form field from one page to another, they went down in flames. Another 20% of the visits were from automated monitor software, which wouldn't care if we recoded the whole thing in Klingon. Remove the bots from the browser stats, and suddenly "minority" browser shares are more than tripled, making them far more worthy of attention.

Stats for your site record the people who have visited recently. This does not necessarily correspond with your future visitors - and when you think about it, it's the future visitors that you should be designing for. While the recent visitors may be a good guide to the future, it's far from guaranteed, especially if there are outside factors involved. I would also point out that there may be significant differences between your present audience and your intended target audience.

Another real world example: It's fairly common for our systems to be an "embedded" module within a larger site. One of these systems is buried deep within a large corporate site, which was (until recently) almost pathologically hostile to all browsers other than Internet Explorer (and Netscape 4.x, for historical reasons, though it looked terrible in that browser - see my rant about IE/NN4.x Boilerplate)

In the last few months, we've been re-engineering our bit. They've been reworking the wider site to be less browser-hostile. In the near future, they're going to make our module far more visible to the casual visitor (linking direct from the front page, rather than half-a-dozen clicks deep, that sort of thing). I'm also told that there's going to be a non-trival national advertising campaign pointing people at this thing. My best guess is that all these changes are probably going to have a noticable effect on the profile of users; any existing stats from the site probably aren't going to be much help in guessing what happens next.

So, if the stats for your site don't give you the information you want, where do you get it? I say: this is where Other Peoples' Stats come in. They provide clues about what's happening in the wider world. Mass market stats can give you some idea of what's out there. If your stats differ from the mass-market stuff, work out whether you care, and if so, why they differ (this is where single-site niche stats might help). It could be something simple and unimportant, or it could be an indicator of a problem. For example, a site carrying news and tips for users of MacOS X is probably going to have far more Safari users than is "normal", and the people running such a site shouldn't be too upset about this. However, if a general-interest news site shows no hits at all from IE5, then the owners should start asking why there's no evidence of one the most popular browsers hitting their site.

Of course, Other Peoples' Stats have their problems. Lots of problems, some of which are huge. Demographic bias, poor or inconsisent browser detection or methodology, and so on. I hope I draw sufficient attention to such issues here (example: my rants about OneStat's recent press release).

In many ways, the most interesting part of looking at stats is comparing and contrasting different sets of figures, and trying to work out the implications. To some degree, cross-comparison of multiple sources can help work out what's "real" and what's just a blip. For example, I'm pretty sure that the market share for mozilla-based browsers is increasing - because a trend is visible in virtually every stats source. So what is mozilla's share? That's much harder to answer - some sources say 3%, some say 6%, some say 30%. My gut instinct is 3-6%. However, as Eric points out, why should we care what the global share is?

I think that such stats do provide useful information, however. For instance, the sites I work on professionally are aimed purely at UK users, and are likely to stay that way for the forseeable future. To pick countries at random, US or Indian users are irrelevant. However, the sites are squarely aimed at a general UK audience, and if Mozilla's market share is increasing globally, it seems likely that it will increase in the UK, too. So we note this, and design sites accordingly.

I suspect that the main target of Eric's post is people who read one press release from Joe Q Internet-Consultancy, and totally re-jig their browser support strategy as a consequence. On this, I can agree with him wholeheartedly - that's just plain dumb. The stats for the site you're working on should be your primary influence. Other Peoples' Stats can provide useful additional information, but come with many caveats and should be treated with caution.