Below, you'll find five pertinent, useful tips that were pulled from an article by Alan Rimm-Kaufman which first appeared in the September 2005 issue of Catalog Success. (I'll post the next five in the next few days.)

Almost any question can be answered, cheaply, quickly and finally, by a test campaign. And that's the way to answer them—not by arguments around a table. Go to the court of last resort—the buyers of your product.

–Claude Hopkins, Scientific Advertising, 1923

Tip #1: Move Bigger Levers First: List, Offer, Creative

In descending importance, the three essential elements of a direct marketing campaign are "list" (who, and how many, people receive the offer), "offer" (what merchandise you offer those people, at what price, and with what service), and "creative" (how is the merchandise presented, described, and displayed.)

If you goal is to double online sales, your best bet is doubling qualified traffic to your site. This is generally easier than doubling average order value or doubling site conversion. (Worthy goals, too, but harder to achieve.)

If you've not yet tested paid search, paid inclusion, local search, affiliates, Ebay, Amazon, and so on, do so. Such "list" tests offer you the greatest chance of bumping sales.

Then focus on "offer." Is your site presenting the right merchandise at the right prices? Would the conversion lift from a free free shipping offset the cost? (Suggestion: when setting a minimum order size for an offer, place it above your average order size.)

Finally, focus on how your site looks and works. Does your homepage highlight the breadth of your merchandise? Are your product detail pages clear, with relevant information above the fold (visible without scrolling)? Is your checkout processes smooth, fast, and intuitive?

Tip #2: Test Shouts, Not Whispers

Testing takes effort, attention, and sometimes money.

Don't test tiny tweaks.

Favor bold tests that have the potential to really change your business. Suggestion: a sure sign of a bold test is that it may make some insiders slightly uncomfortable.

Subtle tests will, at best, yield subtle results, often too small to detect.

Tip #3: Keep Test Notebooks

For each test, document what you tested, why, and what happened. Short pre-test and post-test summaries keep you from repeating mistakes or wasting time on questions already answered.

Before the test, write down a clear hypothesis of what you're trying to prove or disprove. Here's an example:

Test #6, October 2005: Our hypothesis is that bringing visitors into our site from paid search to the new simplified product page template will increase conversions relative to the current grid-style product page template.

Before the test, also record your decision metrics and the roll-out plans.

If the new pages increase closing by a significant amount, that is 50 more orders than the control for the week, we'll discard the grid template in favor of the simpler template.

After the test, record your numeric results, your interpretation, and suggestions for next steps.

Results: Despite one large order, the simple treatment stunk, actually reducing conversion a bit. Next steps: keep the grid, test another challenger later this month.

Given the value of test notebooks – indeed, they become your marketing department's shared institutional memory – it's worth maintaining two copies.

Tip #4: Test One Thing At A Time

To isolate the effect of a variable, traditional testing mandates changing a single factor at a time.

Testing online is usually cheaper and faster than a traditional in-the-mail test. Because of this, there's less need to cram everything into one massive test. Start with a series of fast, simple one-factor tests – you'll quickly learn what matters.

More advanced marketing teams should look into multivariate testing (also known as MVT, scientific testing design of experiments, or Taguchi testing). MVT offers marketers the chance to vary many factors at once in a statistically valid way.

Tip #5: Separate Signal From Noise

All tests have some element of random statistical noise. Suppose you took a mailing list of 10,000 people, randomly split it into two cells of 5,000 people each, and on the same day mailed each cell the same catalog. Even with exactly the same treatment to both groups, one cell by chance alone will have a few more orders, and thus a higher response rate.

Be sure you can distinguish marketing signal from marketplace noise. Familiarize yourself with basic statistical significance calculations. As a very rough rule of thumb, if you plot conversion rates over time, a test needs to increase conversion by more than 1.5 times the normal range of variability to be significant.

If your team isn't using these statistical significance formulas yet, you can get training from the DMA (http://www.the-dma.org/seminars/statistics/), or we've put a Excel spreadsheet with these basic formulas on our website (http://www.rimmkaufman.com/statistics).

To read the entire Rimm-Kaufman article, click here.