PPN Home > Service Delivery > Evaluation

Sign up for PPN updates by email

Suggestions?

PPN welcomes your ideas and suggestions for new information to be considered for the Service Delivery pages.

Please send comments to promisingpractices@rand.org

As a rule, we will not link to sites that do not have objective research or credible information that pertains to the topic areas listed on this site, nor will we link to commercial sites.

To stay up to date on new additions to the site, sign up for the PPN email newsletter.

Service Delivery


Evaluating the Process and Monitoring Outcomes


How should one decide which data to collect and what specific outcome to monitor?


The Center for Substance Abuse Prevention defines an outcome as "changes observed on targeted measures."[1] "Targeted measures" are those that will provide feedback about the effectiveness of a specific program. Other terms for targeted measure include outcome measure, benchmark, or indicator. We can observe changes in these measures by collecting data and analyzing them as part of an outcome evaluation. These outcome measures are tied directly to the goals the program is intended to achieve.

Perhaps the biggest challenge in arriving at these outcome measures is translating program goals into a measurable form. For example, the goal of a prevention program may be to "reduce domestic violence in our community." In order to develop a specific strategy to measure progress toward this goal, some very specific questions must be answered. For example:

  • What is meant by the term "domestic violence"? What types of relationships does this term refer to (such as spouse, cohabitants, dating partners, all family relationships) and what type of abuse is it intended to capture (such as physical assaults, sexual abuse, emotional abuse, or isolating and controlling behavior)?

  • What exactly is "our community?" Is this a geographic description (such as Los Angeles County or a particular school district in Boston) or a population subgroup (such as Spanish-speaking Americans living in El Paso)?

  • How can we get information about the level of "domestic violence" in "our community?" This question is specifically about the source(s) of data being sought and depends heavily on the way one chooses to define key terms. If one is interested in physical assaults by spouses in Los Angeles County, he or she may seek existing countywide data from police records or hospital emergency rooms. If one is interested in emotional abuse by Spanish-speaking cohabitating partners in El Paso, he or she may need to develop a way to identify and survey those residents because it is unlikely one would find an existing source for this information.

No matter which data sources are used, there will be limitations in the measurement. For example, police records contain only those incidents that are officially reported and surveys (aside from their considerable expense) can also miss important segments of the target group. Because of data limitations, it is best to use more than one measure to capture the outcome(s) of interest and limitations must be taken into account when drawing conclusions about program effectiveness [2].

  • How can you tell if domestic violence has been reduced? The definition of outcome, "changes observed on targeted measures,"1 requires that one have some information about the level of domestic violence prior to beginning the program activities. If data is collected only for the period after the program has begun or fully implemented, there is no "baseline" against which to compare the outcomes. Thus, the best strategy is to identify appropriate outcome measures when the program is in the planning phase and to collect "baseline" data on those measures before launching program activities [2].

See Important Issues for Conducting Evaluations.

More Information

Here are some sources of information on how to identify important data to collect and important outcomes to monitor.

  • The National Science Foundation (NSF) prepared two handbooks that discuss a number of relevant issues, including an overview of the evaluation process, methodological approaches, data collection and analysis, and preparing a written presentation of results. www.ehr.nsf.gov and www.nsf.gov. Also sponsored by NSF, along with the Directorate for Education and Human Resources, the Online Evaluation Resource Library provides an additional resource with details about collecting data and conducting evaluations www.oerl.sri.com.

  • A Web page sponsored by the International Consortium for the Advancement of Academic Publication provides a gateway to a broad array of free and largely online resources on planning and conducting evaluations. The page has links to online textbooks, classroom instructional materials, and technical assistance reports providing how-to information on survey development, data collection, outcome measure selection, and development of appropriate research questions gsociology.icaap.org.

  • The National Highway Traffic Safety Administration has produced a booklet entitled "Community How To Guides On, " which covers several topics, including evaluation. This guide includes a description of myths and facts about evaluation, stages of evaluations, methods of evaluations, and selecting an evaluator www.nhtsa.dot.gov.

  • The Community Toolbox provides a discussion of the basics of evaluation and how to identify meaningful process and outcome indicators http://ctb.ku.edu/tools/tk/en/tools_tk_12.jsp.

Footnotes

[1] Rossi, P. H., and H. E. Freeman, Evaluation: A Systematic Approach, Thousand Oaks, Calif.: Sage Publications, 1993. To order this textbook on evaluation, visit: www.sagepub.com.

[2] MacKenzie, D. L. and L. J. Hickman, What Works In Corrections? Olympia, Wash.: Washington State Legislature Joint Audit and Review Committee, 1998. Available at: www.bsos.umd.edu, see Chapter 1.