Tips and Ideas for Landing Pages and Sites

How Experimentation Boosted Restoration Hardware's ROI 267%

We're in the third month of what we have been calling "The Year of Optimization." (We're calling it that because it has become apparent to us at Offermatica that the companies that don't focus on optimizing their sites, in order to give their customers exactly what they are looking for online, will die... In the same way, the companies that do consider optimization an ongoing part of their marketing mix will thrive and speed past their competitors.)

As I wrote recently, the first and easiest phase of optimization is simple experimentation, and almost every company already does it to some extent. Any marketer who has tweaked copy, changed images, or tried something new has experimented with optimization.

But such simple tweaking is merely a baby step. As marketers become skilled at testing elements on a page, watching to see what works, and implementing winning elements, they become able to test and optimize for more intricate needs.

For example, a couple of years ago, marketing executives at Restoration Hardware wanted to improve usability on their website. They started with simple tests, experimenting with where they placed certain details, such as product copy, on the product detail page.

Then they tested the placement of key gift items on a page, during the holiday season.

Finally, they tested different cross-sell products on the checkout page.

Overall, by experimenting with various elements, the company was able to increase ROI by 267% over the duration of the test.

But perhaps more important, they learned how to test quickly and often, and they learned that experimentation can lead to valuable knowledge about their consumers.

For an article on the lessons Restoration Hardware learned from testing, click here.

Google - Testing using Offermatica won't harm SEO

Updated: Check out Is It Cloaking on searchengineland.com.

One question we get asked at Offermatica is whether conducting tests (A/B, multivariate, and other) will affect SEO ranking.  Our response has always been:

1. We do not alter the underlying HTML that is spidered, and Google has assured us that this is not forbidden provided that the underlying HTML is a legitimate page viewed by a portion of the visitors.
2. Google used Offermatica to test its own AdWords interface in September of 2005, and supported our mbox implementation on their own properties.

I was very comforted today to see this response from the GWA group:

---------- Forwarded message ----------
From: Website Optimizer Beta Advisor <WOBetaAdvisor@google.com>
Date: Mar 21, 2007 10:43 AM
Subject: [Website Optimizer] Does Website Optimizer affect my organic search rankings?
To: Google Website Optimizer Beta <websiteoptimizer@googlegroups.com >

We've recently noticed some confusion about whether Website Optimizer
affects organic search rankings, so we wanted to take a moment to
clear things up.

Website Optimizer is designed to keep your original content visible in
the HTML source code of your page at all times. As a result, your
original content is visible to crawlers, which means there should be
no major impact on search engine ranking.  However, if you implement
changes to your content after using Website Optimizer, they'll have
the same effects as any content changes that you would typically make
to your website.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Google Website Optimizer Beta" group.
To unsubscribe from this group, send email to websiteoptimizer-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.com/group/websiteoptimizer?hl=en

I trust this will settle the issue.  The GWA mechanism and the Offermatica mbox operate the same way.

note: I am not subscribed to the Google Website Optimizer beta.

Experimentation and Testing Exposed - Optimization Approach #1

Last week's post on Optimization was my most popular post ever.  In it, I outlined four main types of experience optimization: Experimentation, Targeting, Predictive, and Social.  All have very high value. All can improve relevance and results. And they are not out of reach.

In this post, we will explore experimentation in more detail. In case you are curious, experimentation also goes by the names of A/B testing, multivariate testing, experimental design, and a number of known aliases. It is the "wise old man" of optimization, because its methods and math are well-established and are the basis of many newer automated approaches.

Testing is a "white box" approach that offers confidence and transparency.  When done thoughtfully, it gives feedback you can trust about what your consumer actually responds to.

And frankly, knowing what the customer responds to might be one of the most strategic parts of marketing.

(One note to consider when reading: if any of this sounds complicated or slightly intimidating, remember that the benefits of testing far outweigh the slight challenges involved in getting testing up and running. Plus, there are tools that allow marketers to rapidly create and deploy different tests - tools that handle all the elements I outline below - so that the marketer can concentrate on discovering what is most effective in helping them reach a stated goal, rather than on constructing a useful test.)

What is it?

Experimentation has a long and storied tradition in Western scientific thought.  You pose a hypothesis, design an experiment that isolates the effect you are investigating, and measure the results.  If the data supports your hypothesis, then you go with it.  If not, you re-think and try again.  The power of testing is in its simplicity - we intuitively learn through observation and experimentation.

In online testing, the objective is to evaluate the effectiveness of various elements of an online offering based on response.  Whether you are testing ad creative, navigation, or promotion, the basic approach is the same.  We explicitly vary certain elements while controlling for others so we can clearly identify what has influenced a consumer. 

In the example above, we show four different form pages from a multivariate test we ran a while back. We varied layout, imagery, and messaging.  (Can you guess which one had the highest conversion?)

Talking to the Monkey-Brain

Testing/experimentation are valuable for one simple reason: you are not your consumer (even in B2B). As a result, you cannot intuit what they want, even if you ask them. That's because focus groups (and other approaches where the consumer is aware of the experiment) lead to misleading results. Testing gives the ability to learn who your customer really is.

When you are doing simultaneous testing, you are measuring unconscious response to different marketing approaches: "Do you like more copy or more images", "Should we do more interactive merchandising?", or "Is a two page or one-page form easier to execute?". 

I like to call it "Talking to the monkey brain". When the consumer is unaware, and giving token attention, you are likely marketing to the older, reactive part of the brain -- the monkey-brain -- not our high-consideration conscious self. The fact is that the monkey brain decides faster, and we use it intuitively when we consume information. So having a dialog with your consumer's unconscious is likely a great way to get usable answers.

Since you are dealing with the monkey-brain, you don't ask for a considered opinion, you just show different versions to different visitors *at the same time* and measure which version leads to a response that is favorable to the outcome you want.

As one scientist put it "The reptilian brain has managed to FOOL the subconscious into believing that IT is in control... when all along it is the reptilian brain that is doing the basic controlling." I don't know that I totally agree, but start testing and you will see that what we consciously predict is infrequently true.

Succeeding with Testing

The real advantage of testing is that most of us have been trained to understand the basics of experimentation, and almost every company already does it to some extent. Think of any page of your site — the home page, a category page, a page that requests more information from the user. Now think of all the times you have tweaked copy, changed images, tried something new. Every time you have done that, you have been experimenting with optimization.

Unfortunately, too many folks are constrained by resources and do it in a relatively sloppy way. They run one version for a week and another for the next week. They change a page and they measure the results against some other day. Or they continually vary elements of a site, ad, or email and do not isolate the variants.

The keys to testing successfully are fourfold:

1. Be able to Set Up Content Quickly - You cannot test with just a reporting tool. You must have the ability to change the web page/ad/mobile phone image that different people see to create the test data. The heart of any testing service must be a content serving and routing system that you can use.
2. Traffic - Testing is the most rigorous approach, and seeks precision.  As such, it requires traffic to achieve results.  Traffic does not have to be massive, but if the population of consumers who experience a test is too small or too varied, then it may be hard to find signal.  Running tests to larger groups or running to more well-defined segments can help.
3. Things that Matter, Matter - Signal is a word for a measurable effect. If it doesn't matter to your consumer what color the button is, there is no signal. If using a video on a landing page cuts conversion 30%, there is a lot of signal. You are searching for signal when you test, so you need to think about alternatives that are sufficiently different.
4. Confidence - The heart of testing is statistics. Regardless of your approach, you must measure statistical confidence or you cannot trust results. High statistical confidence (95% or above) means you can trust that if you ran the same test again you would have the same lift. Low confidence means that results will vary or are entirely random. If you are not measuring confidence, test results are really useless.
5. Frequency - The amount of experimentation you do is just as important as (if not more important than) the things you test. Finding "signal" can be elusive, and reducing the cost of running a test makes it more likely that you will try newer and riskier ideas - the very ideas that are transformative to a marketing organization.

This screen cap shows some of the basics - what are you measuring, traffic, which version offers the best outcome, and what confidence. Note that only the third recipe  has high confidence when compared to control (four bars). This means the changes in the other recipes really didn't have signal. Luckily one branch really nailed it.

With Offermatica, our approach is to simplify the presentation of content across different channels like ads and the site. Our Mbox allows us to change content elements on a page, and the AdBox allows us to change ads out in the network. This simplifies the creation of simple tests (A/B) or complex multivariate and non-orthogonal approaches.

From our perspective, it is critical that marketers be able to rapidly create and deploy different experiences and that the tool manage segments, confidence, and population. 

The Good, the Bad and the Ugly

Testing can provide significant feedback to a marketer through its transparency, and can be applied almost universally to any type of offer, layout, content or messaging. Simple testing can provide instant feedback on decisions and should be an element in *every* decision a marketer makes online.  There are a broad range of approaches (A/B, A..n, multivariate, and others), but they all share the discipline of controlled experimentation and can provide invaluable insight to guide future marketing.

However, testing is rigid by design and is less suited for environments with high variability or where environmental "signal" cannot be controlled. Running a/b tests to figure out what product to offer a given visitor or which article would be tedious, ineffective, and likely impossible.

Testing also requires participation by a marketer or agency - if version A of a test wins and there is no one there to listen, it may as well not have happened.

Testing can be an integral part of the ongoing marketing experience, not simply a project you do periodically. Testing can provide instant feedback on decisions and that it can (and should) be an element in every decision you make online.

Next Post - Targeting

What is Marketing Optimization? Testing, Targeting, and Behavior

It is the Year of Optimization.  The recent acquisition of TouchClarity by Omniture is yet another confirmation of an intense surge in interest in technologies that make computers sell better to people. 

But what in the world *is* optimization?

As a matter of disclosure I am not a PhD.  My ADD is a strong inoculation against advanced scholarly pursuits.

However, I have the unique viewpoint of experience.  I co-ran a company, Fort Point Partners, that was responsible for deploying a dazzling range of technology for companies like Nike, Best Buy, and about 50 other firms.  We launched rules-based systems (ATG Scenario Server, e.Piphany), search systems (EasyAsk and Endeca) and more advanced segmentation and modeling software (LikeMinds, Personify, netPerceptions to name a few).  We also ran a lot of tests.

Our goal was simple - make the computer capable as a salesperson. For us, optimization is a fancy word for making a selling process more relevant and engaging for your customer so that they make you more money. And the best optimization tool was one where a marketer could adapt and learn, but the machine did the work.

I see four major approaches to optimization that each have critical value for the marketer (I will use this space over the next week or so to go into more detail on each approach):

1. Experimentation - testing approaches including A/B, multivariate, Taguchi, optimal design and others. Showing different experiences to different control groups to determine a "winner" or "best recipe" based on conversion rate, revenue, or other outcome. Read more here.
2. Targeting - also referred to as "rules-based optimization".  Defining explicit segments and rules for delivering content experience. These can be simple definitions like "show the iPod when our customer searches for "iPod" on Yahoo or very sophisticated behavioral segments.
3. Behavioral - applying AI or linear regression to prior data to determine predictive factors from data to drive the display of content.
4. Social - offloading the work of relevance to the community through ratings, reviews, tagging, or other forms of participation.

Take Google, for example.  They are algorithm guys, right? They use a predictive model that is finely tuned to determine the elusive grail of "relevance" and their results are unbelievable. Yet they also use targeting and testing. True, they outsource the work of specifying the rules to us through keyword selection, bidding, and match type, but this is targeting at its finest.  And they test regularly - evaluating different treatments of the SERPs.

So what is the best optimization approach? Optimization is just marketing with math. If your user base ratings improve the relevance of your search results, then do it!  If testing helps to eliminate your CEO's bias towards acres of copy, do it! The marketing "mix" for optimization is going to take time to get right, but will yield tasty morsels of revenue improvement every step of the way.

We started Offermatica not because we discovered the magic algorithm that turned a computer into a selling machine, but because we found out that the keys to selling online were speed and control. Speed - because marketers had no time, so the machine was going to have to do the work.  And Control, because the marketer still needed to be "in the loop", either driving new ideas or removing crazy outcomes.

And remember this: Marketing is done by marketers. Machines just help us listen and aim better.

When Listening Hurts - Yahoo TV Gets an Earful

So Yahoo TV launches its new look, and gets an earful.  What is the net?  It is better to get negative feedback fast and out in the open than it is to hide problems.  And it didn't have to go the way it did.

According to the blognoscenti at TechCrunch and others, Yahoo! is getting negative ratings for the redesign and credit for being both open to feedback (through their own blog). Kudos to Yahoo for having the cajones to open a public forum for feedback and taking their lumps.  And nice work from Sal Taylor Kidd, the Director of Product Management, for engaging in the conversation.

What caught my eye in the discussion was a simple post on the Yahoo blog in defense of Sal and the team:

"I have to say I’m amazed and dismayed by the tone of these posts. “This sucks!”, “I hate it!”, “Cheap, worthless stunt.” I’m sorry, but would you people talk to the people at Yahoo like this if you met them in person? Of course not."
                            Comment by Charlie Wood - Dec 1st, 2006 at 1:26 pm

A perfect gem of wisdom that captures exactly what we have been saying here at The Site is Dead.  Your customers will not tell you what you most like to hear if you just ask their opinion.  Unless you are willing to put it out there, you will be living in a focus-group fantasy world that is not a mirror image of our own.

One of the unerring truths of digital products is that you always know exactly what you should have built 10 seconds after you launch (or run out of budget...).  From where I stand, this means that you have two choice: Launch what you have and gird for criticism or Never make changes.

Given that the second option is untenable, does it mean that we all have to go through what Sal and Yahoo TV did? Will public scorn be added to the list of indignities that plague the position of Product Manager? Probably.  But it doesn't have to be this bad.

Charlie Wood is right.  People will not be as honest, and you won't learn what you need.  But how did Yahoo get in a position where they were not doing testing?  Why did the new interface require a damn the torpedoes launch?  Why couldn't they introduce some of the interactivity to evaluate how it affected the user experience gradually?

Whether the new Yahoo TV site is good or bad is not my bailiwick.  From my standpoint, if more people engage with the site, if more can find what they want, then it is a success, and if not, it isn't.

But other companies who are contemplating introducing more interactivity to their sites should take heed - you will likely get grief, and Yahoo is a good model for openness.  But you can avoid some of it by selectively testing your ideas, either through Beta or live testing, to understand how it will affect your visitors.

Consumers Don't Know Why They Buy -- How Can You?

Consumers do not necessarily purchase items based on the ads that they really like. Consumers who remember funny ads don't always purchase the products. And ads which consumers think are ineffective sometimes work to move the needle in terms of purchases no matter what consumers think about them.

I was reading a blog post from MarketingSherpa which pointed out the fact that in a USA Today Ad Track report, consumers thought that Hyundai's Sonata ads are 55 percent less effective than other ads on TV. However, Sonatas are being sold at a rate of 44 percent more this year than last.

H'm. Could it be that consumers do not always know, themselves, just what moves them to make a purchase? And if the individual who is making the purchase can't be trusted to know what causes them to buy, how can we, as marketers, pretend to know? Just because our marketing team likes an ad, how can we be sure it will push a consumer toward a purchase?

We can't. So, as Anne Holland so succinctly pointed out, only through testing can we tell which ads really work.

Of course, during my work with Offermatica, that has been my mantra for years: gut instincts, while sometimes correct, can be gut-wrenchingly wrong, too. When marketers make changes based solely on gut instinct, the results can be disastrous. On the other hand, when they test those assumptions, they can move the needle significantly.

See, for example, this case study from Intuit's QuickBooks division. The team assumed that people who searched for QuickBooks came from one of two different "groups," so they created two separate landing pages depending on which group the potential customer fell into. Turns out both groups behaved exactly the same, and could be targeted as a single group, saving the QuickBooks team the time they were taking creating two different versions of their messaging.

Let the Games Begin

We at Offermatica have been evangelizing online testing for 3 years. Statisticians have been practicing multivariate testing (MVT) for decades.

But landing page optimization and multivariate testing leapt in awareness this week with the announcement of Google Website Optimizer, their new offering.

I am delighted by Google's announcement because knowing that MVT tools work is one thing, having Google say Do it is a marching order for many.  There is virtually no amount of marketing budget that I could spend that would be as validating.

I am thrilled because, with any luck, independent marketers, developers, and bloggers will play with Google's free tool and develop skills in continuous optimization. Simply put, the more you test, the better marketer you can become.

You could not give Offermatica a better gift than this.  The marketers are Offermatica's future.

So am I concerned about Google's optimizer?  When a company with $10B in cash enters your market, things are going to change, and we over here at Offermatica may be delusional, but we are not stupid.

But I trust that the market is (mostly) rational, and so I am very optimistic that this will be a tremendous catalyst for Offermatica.

Why? Because Google works for Google, and Offermatica works for advertisers:

1. To use their new tool, you must use Google Analytics.  Forget about Omniture, Coremetrics, or your other package.

2. Google's freebie tool is for Google Adwords.  Offermatica is for Google, Yahoo, MSN, Display Ads, email, affiliates and any other source of traffic.

3. As Bryan Eisenburg pointed out during the eMetrics show, "The Page is Dead".  If you want better advertising, you have to connect the experience, and this requires more that just single page content-switching.

It is instructive to consider what Google is *not* offering to their advertisers - like more price transparency, lower click fraud, or better control of where ads are displayed in the AdSense network. Many advertisers are interested in MVT, every advertiser is interested in lower fraud.

When Google offers consumer services for free (like YouTube or Gmail), they have a clear incentive to create real estate that can be monetized through some form of advertising. The deal is simple and clear.

When Google offers advertiser services for free, it is simply to increase their control of the advertiser value chain. Every advertiser must ask what is in their best interest.

I want to be clear.  Hats off to Brett (and thanks for the beer on Monday) and his team. We are passionate about better online marketing, we appreciate Google's participation, and we toast their success.

There is no looking back now, marketing is now officially beyond the site.

Case Study: How Monster Scored Millions through Multivariate Testing

About a year ago, we ran some exciting tests with Monster.com to see if we could help the company improve revenue per visitor. The tests played with a number of variations:

--promotional copy (default or existing copy vs. stronger offer copy)
--copy for the job posting button
--"learn more" link (link present vs. link not present)
--"Search and Buy Resumes" button (existing copy vs. "Search Resumes" new copy)
--"Search and Buy Resumes" test drive ("Take a free test drive" present vs. not present)
--main button design (depth added to buttons vs. no depth)
--layout of top 3 boxes (button at the top of the boxes vs. button at the bottom).

By testing different combinations of all those elements, we were able to help Monster.com improve revenue per visitor by 8.31 percent, on that page alone.

Another test, on the jobs page, improved revenue per visitor by 11.6 percent.

For a website as large as Monster.com, such a revenue increase means millions of dollars of additional revenue per year -- for very little effort.

What's more, Monster learned important behavioral insights about their customers that they could use in future marketing tests.

To read the entire case study, as well as to view screen shots of various "recipes" we tested, check out the Offermatica newsletter, here.

Just Testing Is Good, Too

Still thinking about language. It occurs to me that marketers aren't always word people. More often, we're idea people. And yet here we are, trying to convey these new ideas -- testing and optimization, Web 2.0, consumer generated content -- in words that we're hoping other marketers can understand.

Sometimes I'm overly ambitious with my words. I see great potential for these tools we're building and I want to push marketers to use them in the way that I see fit. I believe that, by taking online testing and optimization to the next level, marketers can begin to target content on a one-to-one basis, targeting and personalizing so closely that every single consumer will be served the content that is most relevant to him or her at the exact moment they are seeking it.

I really believe that. I know we can do it, and I know we're getting there. But sometimes I forget that, in the actual marketing trenches, we're not quite there yet. I talk and write about the potential, rather than the actuality.

The actuality is, we're still convincing our CMOs that personalization has validity, that if we take an email offer one step further -- bringing the offer onto the landing page and reinforcing it throughout the site, for example -- conversions will improve measurably. We're still convincing our brand marketers that testing can set them free, rather than set them back.

So while I know that we can target content down to the most minute level -- based, for example, not only on the fact that a visitor has been to our site in the past but also on what he bought, what he searched for, where he came from, what search term brought him there, and whether he's in a shopping or browsing mode -- that's not necessarily our most immediate need.

Right now, what most marketers need is to increase revenue and ROI, and to do it quickly. Sure, we need to innovate, and I'd argue that we need to begin exploring these ideas sooner rather than later. But in the meantime, we need to become comfortable with testing and optimization, in order to learn to use them as an ongoing marketing tool.

What Does Testing Mean, Really?

I'm struggling with language this week. It occurs to me that with testing and optimization, we're creating a whole new vocabulary. What does testing really mean? When I say online testing and optimization, do marketers really get it? And if marketers do understand, are they all thinking the same thing?

So often in recent years, I have asked a marketer if they currently conduct testing, and they say, "Oh, sure, we tested that email campaign on 10,000 before we ran out the campaign to the entire 100,000." Then, I say, "And what did you test?"

"Well, we tested the email."

"But what about the email did you test?" I ask.

"We tested the whole email. We got a good response, so we launched the campaign."

I like that testing matters. Because testing is a major way of listening to customers.  But just pre-sending one version to a subset to see whether anyone buys is not really testing, and it is not really listening.

When we talk about testing and optimization, it means testing something against something else. In email, it can mean 10,000 emails sent out with one subject line vs. another 10,000 sent with another subject line.

It could mean that the emails had different offers. It could mean that the same email was sent to different groups of customers. In each case, though, it would be a true test of one something against another something, to see which something generated the best results.

But what it doesn't mean, as far as I'm concerned, is a test against... nothing. And that's what a lot of marketers believe.

Testing is really a very basic term, a very basic concept. Perhaps I should be careful to always say "AB testing," or "multivariate testing" or use some other qualifying word.

But maybe, in order to move the online testing and optimization field forward, we need a glossary of sorts, a dictionary that describes the most often-used terms, so we're on the same page when it comes to these tactics.

An article, here, talks about the basic differences between A/B testing and multivariate testing, and explains what is meant by each.