Hence, I'm well placed to write this article. So here goes, the most common errors and/or challenges I've spotted in my now rather long marketing career:
I could write this entire blog post on one topic alone - Probability. It is constantly misunderstood. Even well-known publications like The London Times, or CNN, often make errors. The most common one is confusing correlation with causation; Just because two things happen regularly at the same time, it does not prove that there is a connection. Here are some funny examples to demonstrate my point.
The second most common mistake I've seen in my career regards A/B testing, and Statistical Significance. It's amazing how often this type of evidence goes unquestioned, even in big rooms of senior executives.
Let's say we are testing out two images in an email we send out, which we send to 20,000 contacts.
- Image A: 71 clicks
- Image B: 87 clicks
Clearly, image B is 'the winner', right? But hold up just one minute. Have you tested this experiment for statistical significance? To what degree are you certain that this test is conclusive?
This is where the concept of Confidence Intervals is invaluable. I can go into the equations on how you calculate this number, as I had to do in my first year MBA Statistics basics class (thank you Raj, my stats tutor, who helped me after hours to get through this tough class with a good grade!).
But fortunately, understanding the math is not critical to running a good A/B test. In fact, Hubspot has a handy AB testing kit, that contains the formulas you need, so you can simply plug in your numbers, and get your answer!
Here you can see that I've done this myself. And look at the results!
As you can see, it has failed both the standard, 90% confidence interval, and more rigorous, 95% confidence interval tests.
In fact, we are only 22.5% confident that this data is conclusive. In other words, Clicks went up, but not by enough to confidently say it wasn’t just randomness.
Here's what those confidence intervals look like on a probability bell curve
The 'Moneyball' problem
So if you're a bit of a statistics fan like me, you may well have read a few books by Nassim Nicholas Taleb, like 'Fooled by randomness' and 'The Black Swan' - in my case, after learning all about statistics at business school, he brilliantly debunked some common statistical errors for me.
And I'm sure you will have either read the book 'Moneyball' or seen the excellent movie.
The Moneyball problem is when everyone in your company, and perhaps even everyone in your industry, is measuring the wrong things. Moneyball is often summarised as 'use data to win'. But the deeper lesson is harsher: you can measure brilliantly and still lose if you’re measuring the wrong thing.
In the movie, Baseball didn’t suffer from a lack of stats, it suffered from mispriced stats. Batting average and RBIs looked like performance, but they weren’t the most reliable predictors of wins. Oakland’s edge came from shifting attention to what compounded: getting on base, over and over, from undervalued players.
You could perhaps compare this to marketing that measures 'last click' attribution, when marketing mix modelling is the real driver of success - the combination of all channels that produces the win. You are focusing on the metrics, not because they are the best, but because a) either those are the only ones you have, or b) because your company, or industry are all measuring these things - 'everyone is doing it!' It is the industry standard.
“The fact that a great many people believe something is no guarantee of its truth.”
― W. Somerset Maugham, The Razor’s Edge
I would just like to add that I hope everyone reading this takes it in the right spirit. Just because I've mastered some of these subjects, does not mean I consider myself particularly smart, or even an 'expert' - it's simply that I've been lucky to have had the opportunity to study these concepts, to devote a lot of time to them, and that I'm genuinely interested in them.
I don't believe understanding these concepts makes me more of a professional, or even a better marketer. Others have many insights into this topic, or other directions, which could be equally, or indeed, far more cogent and useful.
"To know, is to know that you know nothing. That is the meaning of true knowledge."
- Plato
