MeritDirect presents the third and final installment of our three part blog series.
In the two previous posts, MeritDirect’s Kevin Hope and Maria Jaime discussed direct response rates and PPC campaigns and how using the law of averages in determining campaign strategy can often cause marketers to lose sight of the overall potential of less than average variables. Part three focuses on how marketers can miss hidden gems in their testing results.
We’ve tested mailing our customers 6x vs. 3x across three months and the control treatment(3x) won. How many times have you heard this result before? It is not uncommon to look at the average results of a test and stamp it as the only outcome. However, the results can differ as you slice the data by different dimensions. Let’s take a look at a specific example:
A company which typically mails its customers once a month decided to test increasing the frequency to 2x/month across three months. The goal of the test was to see if they can drive incremental revenue in their best customer segments. At the end of the test, they saw that on average the 3x panel drove the same revenue per customer as the 6x panel. However, when they sliced the results by industry, they saw that a specific industry of customers preferred the 6x vs. 3x treatment.
It was a small segment of their audience but the revenue per customer of the 6x increased drastically vs. 3x. This was a hidden gem in the results that was masked by the average performance of the entire overall audience.
Test results can be sliced by various other segmentation systems such as model scores, recency, acquisition channel, etc. The goal is to not get deceived by the average results. It is always best to view the results by several dimensions to ensure the best strategy for each segment.
Senior Marketing Analyst
Catch up on parts one and two of this series:
Part One: The Deception of Averages – In Direct Response Rates
Part Two: The Deception of Averages – In PPC Campaigns
For comments, questions and more information on the ideas presented in this blog series, please contact our BI + Analytics team.