Earlier this year, Greenlights published research on the successes and challenges Central Texas nonprofits face in evaluating their programs and measuring results and impact. A key finding from our study is that only about half of organizations surveyed regularly collect data on program outcomes (specific changes which occur in program participants as a result of the program or service). So why do so many organizations struggle with collecting outcome data? While the organizations in our study reported a number of challenges (including lack of dedicated staff and funding for evaluation), I suspect that part of the reason also has to do with the fact that some outcomes are frankly, difficult to develop and measure.
In my role as a consultant and trainer, I often field questions about how to develop meaningful outcomes. While there aren’t always easy or simple solutions, below are some tips for some of the most commonly mentioned challenges.
What do we do if the outcomes of our program can’t be expected for many years?
The trick here is to not only focus on long term outcomes, but also to figure out the intermediate results you want to achieve. For example, if you run a high school tutoring and mentoring program, your long term outcome might be student graduation. However, shorter term outcomes might focus on improving attendance and grades. These intermediate outcomes will (hopefully) lead to the longer term outcomes for the students in your program.
My organization only provides short-term assistance. Does it still make sense to collect outcome data?
Disaster relief and emergency food and shelter are just a couple of examples of programs that focus on providing temporary help to clients. These services may result in almost immediate improvement in an individual’s status or condition. Therefore, the actual provision of the service can be considered an outcome. In addition, it may be possible to follow-up with the client in order to determine if and how the assistance impacted their well-being over time.
How can we measure results of prevention-only programs?
With prevention programs, you may want to focus on the number of participants that did or did not experience the negative event that the program was attempting to prevent. For example, a teen pregnancy prevention program could follow up with participants 6 months or a year after program completion to determine how many participants did in fact become pregnant. Prevention programs can also measure short-term changes in participants’ knowledge or attitude. In the example above, staff could survey teens during the program to see if their knowledge regarding how to prevent unwanted pregnancies has increased.
Our outcomes are too “intangible” to truly measure. What do we do?
It may seem that way on the surface, but almost anything can be measured. For example, I recently read about an international program focused on the empowerment of women. Program staff determined that “increasing women’s decision making power in the home” would serve as an important outcome related to empowerment. To measure the outcome, staff surveyed participants before and after program completion and asked about their role in various household decisions. On the surface, “empowerment” may not seem like a concept that can be measured, but with creativity and effort it can be done.
My organization is engaged in advocacy efforts. How can short-term success be measured?
The key here is to focus on intermediate outcomes. Your ultimate goal may be to pass legislation or create policies. However, consider what intermediate outcomes will take you one step closer to reaching your ultimate outcomes. One example could be the number of times an issue is mentioned in the media. (For more information on resources related to advocacy outcomes, check out the Aspen Institute’s Advocacy Planning and Evaluation Program.)
What other types of “hard-to-measure” outcomes do you struggle with? Any tips or ideas you would add to this list? If you’re looking for additional support in developing outcomes and setting up an evaluation system for your organization, consider signing up for Greenlights’ June workshop, Building Evaluation Capacity: Tools and Best Practices for Measuring Results.