Testing Engagement Rather than Preferences

I've recently been thinking about what testing really means, at its core. I think it means listening, really listening, to consumers.

Mostly, we tend to test things online: colors of buttons, wording of language, copy in a headline, position of the navigation bar, number of fields in a form, size of images, or placement of merchandise.

When we test elements, we often learn interesting factors - a hard sell in the call to action may perform better, indicating that consumers of a given company may need a good, hard push toward the purchase.  That in itself is important knowledge. But it leaves out significant information about the way the consumer engages with the company.

What if, instead, we begin to test the underlying assumptions about how we engage with our customers, and how our customers engage with us? Should you do more suggesting, or more effective responding to searches? Should you give a lot of results or just a few?  Should you create a shipping club or just offer reasonable shipping rates?

This sounds simple, but it marks a fundamental difference in the way we have been thinking about testing and optimization.

Think of it this way: You want to buy a sweater for someone you don't know very well. You might ask, "What's your favorite color?" She says red, and you know a tiny bit about her. Her answer might help you make a decision - but there are an awful lot of red sweaters available.

Now imagine that, instead, you ask, "What's your favorite thing to do on a cold weekend day in January?"

If she answers that she loves ice fishing in 20-below weather, that gives you a very different feel for how she engages in the world (and thus what type of sweater to buy) than if she answers that she likes to sit in front of a fireplace with a good book.

Both answers are equally revealing. And both, I would argue, give a better idea of what kind of sweater to purchase than simply knowing her favorite color is red.

So keep seeking elements that change results, but seek to be bolder.