A Quick Guide to Testing Messages
Testing messages and materials before using them is probably the single best thing you can do to improve your outreach and avoid expensive mistakes. Yet not many people conduct message tests. Why? Perhaps many of you are unsure about how to proceed or fear that you have to spend a lot of time and money to get useful information. Worry no more. This guide outlines easy ways to conduct a message test.
Why bother with a message test?
- A message test allows you to validate key assumptions about your target audience. It helps guard against assuming that we know what our audience wants to hear.
- By asking a sample of your target audience to review your materials, you’ll find out if your original materials were off-target or not focused enough and often get some information to improve them. Even for experienced communications professionals who know their audience well, message testing is a common and valued practice because it helps to refine messages and generates confidence in the final product.
- Finally, if you’ve generated a few ideas, message testing helps to identify the message (or combination of messages) that is likely to be most effective with your target audience.
While message tests are very useful, they are not a substitute for good thinking and judgement. The results of your test are only as good as the messages that you are testing. Your audience can tell you how well the tested messages work for them, and they can give you some clues about what works and what doesn’t. But you can’t expect them to come up with good message and design ideas. So, do your homework to develop the best possible messages before you put them to the test.
How to conduct a message test
What to test
It’s always a balancing act to figure out how “finished” your materials should be before you test them. On the one hand, you want to know how your audience will response to your final materials. On the other hand, you don’t want to spend too much time and money on finalizing materials if you’re not on the right track. And while it’s great to test different options, you might not have the resources to develop and test 2-3 different materials.
One way to address this is to test one finished message but have some alternatives for key elements of the message. Let’s say you are planning to use a postcard for your outreach. You would first test your best version of the postcard and then focus on testing some options for key elements such as headlines, logos, and visuals.
Also make sure the materials you test are brief enough for the landowner to digest in real time, so they can give you a meaningful response. If you’re testing something more detailed (like a website or a book), then either give landowners time to read the materials (perhaps at home), or direct their attention to specific elements on which you want feedback (such as the cover page, title, format, etc.).
How to test
The right format to test your messages depends on what you’re testing. Generally speaking, it is better to conduct message tests in person or online, especially if you’re testing images. Phone conversations can work for testing short, pithy headlines that can be easily read out.
You’ll also have to decide whether to test messages with individual landowners or in a group setting. This decision depends on many factors, but generally speaking, it is easier to get people’s honest reactions when talking to them individually. Group conversations can be interesting but are also subject to peer influences.
Who takes the test?
Luckily, you don’t have to test materials with a lot of people -- 8-12 are usually enough. In statistical terms, this is not a large number; but based on experience, we know that responses from 8-12 landowners will likely give you good information to validate your approach and refine your materials.
To have confidence in your findings, you must test your messages with landowners beyond your immediate professional and personal circles. You could, for example:
- Ask your colleagues or landowners you know to put you in touch with landowners they know. Just make sure that these aren’t “model” landowners who are already converted to your point of view.
- Look at institutional records to identify landowners in the area and call them.
- Go to community events where you’re likely to find members of your target audience and ask people if they’d be willing to talk with you for 10 minutes. Maybe you could offer them a small gift, like a pen or hat, to thank them for their help.
Finally, it helps to have a sense of whether or not the “test” landowners fit your target audience segment. You might want to include a few questions in the test that help you determine whether these landowners are indeed the type of landowner that you’re hoping to reach and influence.
What to ask
A good questionnaire is precise and efficient. Resist the urge to cram in questions concerning everything you want to know about the landowners. Stay focused on testing the effectiveness of your message and the key elements that you think will drive landowner action. Here are some sample questions that you might use. We’ve worded these for testing a postcard, but you can adapt them to different kinds of materials.
Give respondents a copy of your postcard. Give them some time to look it over and then ask:
- On a scale of 1 to 5, where 1 means you “don’t like it at all” and 5 means you “love it” how would you rate this postcard? (Follow up question: Why do you say that?)
- Please read the text carefully. Circle the parts that you like and strike out those that you don’t like or find confusing. (Follow up questions: Why did you circle “X”? Why did you strike out “Y”?)
- If you got this postcard in the mail, would you read it? Why/why not? (Possible follow-up question: What could we change that would get your attention so you would read it?)
- After reading this postcard, would you do anything? (Possible follow-up questions: What would you do? Why would you/why would you not [insert call to action]?)
- On a scale of 1 to 5, where 1 means you “don’t like it at all” and 5 means you “love it” how would you rate this picture? A follow up question could be: Why do you say that?
- What do you feel when you see this? (Ask needed follow up questions to understand their response.)
- If it had this [other headline/other image], would that be better or worse? Why?
- If we added X logo, would that change your response?
We have worked through one example here, but you can apply these techniques and questions to test all sorts of messages. Feel free to build on what you have here. You won’t go wrong if you keep the following things in mind:
- The most important thing in a message test is to learn why landowners said or think something. “Why/why not?” is a great follow-up question!
- Don’t ask leading questions, i.e., questions that put words in the respondents’ mouth. An example would be: “Does this make you feel confident?” Instead ask: “How does this make you feel?” Make sure your questions are phrased neutrally.
- Don’t explain, argue or correct respondents, or encourage “good” feedback over “bad” feedback. Remember, it is their perceptions that you’re looking for. Always be professional and neutral.
- Don’t ask them if they think other landowners will like/dislike something. Respondents should be asked to respond on their own behalf. It’s up to you to make generalizations and draw conclusions.
- Avoid double-barreled questions. Double-barreled questions are basically two questions crammed into one statement. An example would be: “Would this be of interest to you and other landowners?”
- Have backup options ready to test in case your preferred option does not test well.
Finally, please remember that you can always reach out to the TELE team if you’d like additional guidance in designing your message test. We are here to help!