An article that appeared recently in a national direct marketing publication recounted the case study of a database solutions company that used focus groups to evaluate two different creative concepts for an upcoming lead generation mailer.

One concept took a humorous approach; the other was more straightforward and to-the-point. Every one of six different focus groups said that the humorous piece was the best. Wisely, the company decided to test both versions in the mail anyway. The results: The more straightforward mailer outpulled the humorous version by more than 2 to 1.

The story reminded me of a similar experience we had with a client a few years ago for a campaign that targeted software developers. The client wanted brash, colorful creative to "break through the clutter." We convinced them to let us develop both a colorful self-mailer and a more sedate letter package in a plain (no teaser copy) #10 envelope.

At final layout, the client took the comps round to all the developers in their organization and asked them which they'd be most likely to respond to. To a man (this was an all-male sample), they chose the self-mailer. In the mail, the letter package outpulled the self-mailer by a factor of 3 to 1.

Focus groups have tremendous value for many things - notably to help evaluate brand advertising and basic positioning. But for direct mail, they're notoriously unreliable. One problem is sample size. I'm no statistician, but I'd wager that as a sample, twelve people probably isn't statistically significant. (If you have one hundred people in a focus group, and only five people say they'd respond to your mailer, you've got a very successful campaign on your hands.)

Secondly, focus groups are ideal for subjective evaluation of a positioning statement or an image ad, but they're hopeless as an objective measure of whether a direct marketing campaign will work. The only way to get a true, objective evaluation of a given program is to test it in the field. After all, one of the things that makes direct marketing so valuable - whether print or online - is that it's measurable.

If you simply must get some indication of a direct mail campaign's potential merits before committing to even a limited test, then consider using online media - broadcast e-mail or banner ads, say - to test message, audience, offer or creative. Then use those results to alter your direct mail design if necessary.