At first glance, it sounded like a great idea.
A client who shall remain nameless, a tech startup, was trying to make a go of a somewhat questionable - but certainly workable - Web-based service concept. They'd have to be very careful about how they went about marketing themselves, because with the market risk involved there would be little room for error.. but, as long as they kept things on track, we all clearly saw that it could work. The problem was deciding what message to run with on the website landing page.
I suggested two very short headlines and focused directions. However, their web designer - who resented having to negotiate these issues with a copywriter - argued for long headlines and lots of them, and sold them on the idea of testing: put these long headlines all on different landing page versions, rotating randomly with each hit. They'd then pull the server logs, figure out which headline generated the greatest clickthrough, and that'd be their new marketing message. (I suspected that her real reason for suggesting this had less to do with getting solid results, and more to do with getting paid to design multiple webpages instead of one.)
I strongly disagreed. I said it wouldn't work. They went ahead anyway.
And, indeed, it didn't work. The result was a normal distribution, a bell curve that represents a randomized result data set. (English translation: any trend that could be inferred from the data was little better than horoscope-reading.)
So why did their testing fail? Because their methodology was scientifically and mathematically flawed - right from their starting premises.
Each different headline was accompanied by a completely different graphic image. Each image featured different models, was done in a different style, and offered an entirely different sense of context to the website landing copy. Was it the headline that caught the visitor's eye? The image? One of the models? By allowing so many different variables into the testing, my client guaranteed that they'd never be able to determine which ones were actually important.
They were appealing to a collection of highly specific and varied demographics - but the images rotated perfectly randomly. This ensured that my client would have no way of knowing what exactly was effective for whom. That graphic with the cute smiling girl: was it appealing more to men or women? Old or young? To entrepreneurs or to corporate employees? Since their audience wasn't homogenous, they had no earthly way of determining exactly how effective any of these marketing messages were - every prospect interaction was a random matching of demographic to message. So, of course, their resulting data set was randomized.
They attempted to test their messages based on server logs. Every web server generates a text file on an ongoing basis, listing every action it performs and when it performs it. This provides a running record of every hit and every page request, recording the visitor's IP address, a time and date, the exact action taken, and a handful of other useful bits of information.
The server log is a very, very useful tool for optimizing a business website. If you read it closely, it will provide a map of average visitor activity - how long they tend to stick around, what order they tend to read your pages, etc. Over the long term, analyzing server logs will tell you how much overall traffic you're receiving and how it changes from month to month and year to year.
What the server log won't do is tell you anything at all about the visitor themselves, other than a set of four numbers (the IP address), the type of browser they used, and what pages they viewed. It's nearly impossible to reliably relate that data to an actual human being, much less to a demographic or intention to buy. So while the server log is very useful for many things, accurately testing a marketing message based purely on logged clickthrough is not one of them.
So now you know what doesn't work. How does the small or midsized business justify an ongoing marketing expense in a way that directly relates real data to bottom line dollars?
By gathering data the right way.
Next Time, in Part 2: Gathering Marketing Metrics - The Right Way.
Rob