TELE - Tools for Engaging Landowners Effectively

Step 6: Evaluate Your Work

Evaluation is an integral part of audience-focused communication. Audience responses to your outreach provide valuable insights for improving current and future efforts.

Generally speaking, evaluation activities fall into three categories--

Process measures tell you how well your outreach was conducted
Outcome measures tell you how your target audiences responded to your outreach (i.e., whether they took the actions you suggested)
Impact measures that assess whether the communication helped accomplish your overall program objectives

Because behavior change takes time, and it takes even longer for human actions to translate into measurable environmental impacts, the complete impact of your program may not be apparent for several years. That is why it is important to measure how well your program was implemented and how audiences received your message. These process measures are early signals that the program is working as planned. They also give you the ability to refine and adjust your program based on what's working. For example, if you find that most people who call your hotline say they heard about you on the radio, you might want to increase your radio advertising budget. Conversely, if some channels are not attracting a good response, you should consider altering those activities or stopping them in favor of those that are working well.

Evaluation does not have to be expensive. It can be qualitative or quantitative, formal or informal. Often, a lot of useful data can be collected while you are implementing the program, e.g., by having sign-in sheets at presentations, asking landowners where they heard about your program, or monitoring how people are getting to your Web site. As a general rule, communication campaigns set aside 5% of their total budget to conduct process and/or impact evaluation. Think of this money as a way to maximize success for this campaign and an investment in learning for the future.

1. Outcome indicators. To develop appropriate outcome indicators, work backwards from your communication and program objectives. Ask yourself:

Is there any indication that people are doing as we asked? Can we measure their behavior or get them to report it to us?

For example, if the objective of your communication is to get more people to request management plans, monitor the number of plans requested during the campaign. You can also compare this to a similar time frame in previous years.
Is there any indication that our audiences' actions are advancing our conservation objective?

For example, depending upon your program objectives, you might want to monitor the number of sustainably managed forest acres, the number of forested acres, or the number of acres in easements in your state.

2. Process indicators. To develop good process indicators, focus on who you want to reach, how you're doing that, and what you want them to think/believe about the target behavior. Ask yourself:

Are we reaching enough members of our target audience?

For example, you can assess your reach by counting meeting attendees or getting them to sign in; asking media outlets for readership and viewership numbers, etc.
Are our messages and materials being received well? Do people like our messages and materials and think they are relevant and important?

For example, ask meeting attendees to fill out a short survey; include a feedback card with your collateral material; or simply count the number of people who request additional information.

3. Proxy measures. When you cannot get direct answers to your questions, use proxy measures. For example, you may not have an exact idea of how many people saw your billboard advertisement, but you can estimate that number based on traffic volume figures compiled by the Department of Transportation.

4. Time frame for results. Consider the time frame over which you expect to see changes in specific indicators. For actions like calling for additional information, you can expect immediate results. For others, like consulting a forester before harvesting timber, audience members may not have the opportunity to act on your recommendations for several years. In such situations, changes in audience members' beliefs and intentions are often good proxy measures.

To clarify how you will evaluate this effort, answer the following questions:

What are your primary outcome indicators and how will you track them? (In other words, how will you know whether your target audience is taking the actions you intended?)
What process or intermediate indicators will you monitor to get early feedback on how well the campaign is working?
How will you measure the long-term impact of this communication on your overall program objectives?

Also consider: The time frame over which you expect change to occur.

Here is how the CBYC campaign addressed these questions:

Primary Outcome Indicators: The primary outcome indicator for the campaign is the number of landowners who are using foresters to help plan and/or manage their timber harvests. Depending upon availability of funds, the campaign plans to conduct a one-year follow-up survey with a random sample of the people who requested the informational packet to see whether they used a forester or did anything differently as a result.

Process/Intermediary Indicators: Each state will assess the cumulative impact of its outreach by monitoring the number of people who call the campaign hot-line to request additional information about timber harvesting. Additionally, they will also monitor:

How the number of calls compares with similar requests before the campaign?

What proportion of these calls are from people who are actually thinking about harvesting their trees in the near future?

How many packets of information about timber harvesting were mailed out to callers?

In addition, each of the CBYC implementing states will monitor process indicators that are tailored to their specific outreach activities, e.g., attendance at meetings, media impressions, etc.

Impact on Program Objectives: The CBYC currently has no plans to monitor this, but good indicators include: a reduction in the number of "bad" cuts; increase in use of consulting forester services; and a reduction in the number of people who call state foresters to complain about poor harvesting practices.

Time Frame: Since most people cut trees only once or twice in their lifetime, the opportunities to act on this information are rare. The conservation impact of the campaign will be seen over several years. However, the process indicators and the annual survey will give an indication of how well the campaign is working.