Experimentation and Testing Exposed - Optimization Approach #1

Last week's post on Optimization was my most popular post ever.  In it, I outlined four main types of experience optimization: Experimentation, Targeting, Predictive, and Social.  All have very high value. All can improve relevance and results. And they are not out of reach.

In this post, we will explore experimentation in more detail. In case you are curious, experimentation also goes by the names of A/B testing, multivariate testing, experimental design, and a number of known aliases. It is the "wise old man" of optimization, because its methods and math are well-established and are the basis of many newer automated approaches.

Testing is a "white box" approach that offers confidence and transparency.  When done thoughtfully, it gives feedback you can trust about what your consumer actually responds to.

And frankly, knowing what the customer responds to might be one of the most strategic parts of marketing.

(One note to consider when reading: if any of this sounds complicated or slightly intimidating, remember that the benefits of testing far outweigh the slight challenges involved in getting testing up and running. Plus, there are tools that allow marketers to rapidly create and deploy different tests - tools that handle all the elements I outline below - so that the marketer can concentrate on discovering what is most effective in helping them reach a stated goal, rather than on constructing a useful test.)

What is it?

Experimentation has a long and storied tradition in Western scientific thought.  You pose a hypothesis, design an experiment that isolates the effect you are investigating, and measure the results.  If the data supports your hypothesis, then you go with it.  If not, you re-think and try again.  The power of testing is in its simplicity - we intuitively learn through observation and experimentation.

In online testing, the objective is to evaluate the effectiveness of various elements of an online offering based on response.  Whether you are testing ad creative, navigation, or promotion, the basic approach is the same.  We explicitly vary certain elements while controlling for others so we can clearly identify what has influenced a consumer. 

In the example above, we show four different form pages from a multivariate test we ran a while back. We varied layout, imagery, and messaging.  (Can you guess which one had the highest conversion?)

Talking to the Monkey-Brain

Testing/experimentation are valuable for one simple reason: you are not your consumer (even in B2B). As a result, you cannot intuit what they want, even if you ask them. That's because focus groups (and other approaches where the consumer is aware of the experiment) lead to misleading results. Testing gives the ability to learn who your customer really is.

When you are doing simultaneous testing, you are measuring unconscious response to different marketing approaches: "Do you like more copy or more images", "Should we do more interactive merchandising?", or "Is a two page or one-page form easier to execute?". 

I like to call it "Talking to the monkey brain". When the consumer is unaware, and giving token attention, you are likely marketing to the older, reactive part of the brain -- the monkey-brain -- not our high-consideration conscious self. The fact is that the monkey brain decides faster, and we use it intuitively when we consume information. So having a dialog with your consumer's unconscious is likely a great way to get usable answers.

Since you are dealing with the monkey-brain, you don't ask for a considered opinion, you just show different versions to different visitors *at the same time* and measure which version leads to a response that is favorable to the outcome you want.

As one scientist put it "The reptilian brain has managed to FOOL the subconscious into believing that IT is in control... when all along it is the reptilian brain that is doing the basic controlling." I don't know that I totally agree, but start testing and you will see that what we consciously predict is infrequently true.

Succeeding with Testing

The real advantage of testing is that most of us have been trained to understand the basics of experimentation, and almost every company already does it to some extent. Think of any page of your site — the home page, a category page, a page that requests more information from the user. Now think of all the times you have tweaked copy, changed images, tried something new. Every time you have done that, you have been experimenting with optimization.

Unfortunately, too many folks are constrained by resources and do it in a relatively sloppy way. They run one version for a week and another for the next week. They change a page and they measure the results against some other day. Or they continually vary elements of a site, ad, or email and do not isolate the variants.

The keys to testing successfully are fourfold:

1. Be able to Set Up Content Quickly - You cannot test with just a reporting tool. You must have the ability to change the web page/ad/mobile phone image that different people see to create the test data. The heart of any testing service must be a content serving and routing system that you can use.
2. Traffic - Testing is the most rigorous approach, and seeks precision.  As such, it requires traffic to achieve results.  Traffic does not have to be massive, but if the population of consumers who experience a test is too small or too varied, then it may be hard to find signal.  Running tests to larger groups or running to more well-defined segments can help.
3. Things that Matter, Matter - Signal is a word for a measurable effect. If it doesn't matter to your consumer what color the button is, there is no signal. If using a video on a landing page cuts conversion 30%, there is a lot of signal. You are searching for signal when you test, so you need to think about alternatives that are sufficiently different.
4. Confidence - The heart of testing is statistics. Regardless of your approach, you must measure statistical confidence or you cannot trust results. High statistical confidence (95% or above) means you can trust that if you ran the same test again you would have the same lift. Low confidence means that results will vary or are entirely random. If you are not measuring confidence, test results are really useless.
5. Frequency - The amount of experimentation you do is just as important as (if not more important than) the things you test. Finding "signal" can be elusive, and reducing the cost of running a test makes it more likely that you will try newer and riskier ideas - the very ideas that are transformative to a marketing organization.

This screen cap shows some of the basics - what are you measuring, traffic, which version offers the best outcome, and what confidence. Note that only the third recipe  has high confidence when compared to control (four bars). This means the changes in the other recipes really didn't have signal. Luckily one branch really nailed it.

With Offermatica, our approach is to simplify the presentation of content across different channels like ads and the site. Our Mbox allows us to change content elements on a page, and the AdBox allows us to change ads out in the network. This simplifies the creation of simple tests (A/B) or complex multivariate and non-orthogonal approaches.

From our perspective, it is critical that marketers be able to rapidly create and deploy different experiences and that the tool manage segments, confidence, and population. 

The Good, the Bad and the Ugly

Testing can provide significant feedback to a marketer through its transparency, and can be applied almost universally to any type of offer, layout, content or messaging. Simple testing can provide instant feedback on decisions and should be an element in *every* decision a marketer makes online.  There are a broad range of approaches (A/B, A..n, multivariate, and others), but they all share the discipline of controlled experimentation and can provide invaluable insight to guide future marketing.

However, testing is rigid by design and is less suited for environments with high variability or where environmental "signal" cannot be controlled. Running a/b tests to figure out what product to offer a given visitor or which article would be tedious, ineffective, and likely impossible.

Testing also requires participation by a marketer or agency - if version A of a test wins and there is no one there to listen, it may as well not have happened.

Testing can be an integral part of the ongoing marketing experience, not simply a project you do periodically. Testing can provide instant feedback on decisions and that it can (and should) be an element in every decision you make online.

Next Post - Targeting