Considerations When Planning a Program Evaluation Study
A program evaluation is an empirical examination of the goals, processes, inputs, outputs, and impacts of a program. Unlike purely descriptive research, a program evaluation generally results in recommendations, which if implemented, will result in improvement of a program.
When embarking on a program evaluation study—whether actually performing the research or enlisting the aid of a consultant—it is useful to have a clear understanding of study objectives. Considerations when planning a program evaluation study include the following.
1. For what purpose is the evaluation being conducted?
2. What decisions do you hope to be able to make, based on research findings?
Actionable research is that which provides an empirical basis for improving a program or policy. While actionable research often describes a program or policy, descriptive statistics by themselves rarely provide a solid basis for action.
An example of a descriptive research finding is, “This research showed 40% of university faculty with children ages 5 and under are dissatisfied with current childcare support.” While informative, merely knowing the extent to which employees, clients, or customers are dissatisfied does not tell us why they are dissatisfied or what can be done to increase their satisfaction.
An actionable program evaluation study will usually provide much greater detail, will use measures more objective than "satisfaction,” and will conclude with recommendations. An example of an actionable research finding is, “This research suggests at least one additional childcare facility is needed to accommodate an estimated 300 faculty with one or more children between 3 and 5 years of age. Study participants indicated a strong need for childcare that is located on campus. Respondents indicated a need for childcare evenings and weekends, and for sick children. Competitive pricing was deemed essential.”
3. Who is the audience for research findings?
The audience can shape the goals of a study and influences the format in which results will be presented. For example, if the researcher knows beforehand their audience has particular concerns, the researcher can strive to address these issues directly in the study.
4. What kinds of information are needed to make the decisions you need to make and/or to enlighten your intended audience?
Do you want to…
- Understand the process of the program—its activities, inputs, and outputs?
- Understand the users of the program—their characteristics, preferences, and needs?
- Estimate demand for, and preferred characteristics of a new program?
- Determine the extent to which a program is bringing about specific short- or long-term outcomes?
5. How can program objectives be operationalized?
In identifying “observable” and “trackable” measures of program performance, it can be useful to:
- Review program policy and mission statements
- Review program materials and employee training manuals
- Discuss program objectives with program managers (who often conceptualize program goals in a concrete, practical way)
- Examine existing data on program characteristics, activities, and performance
- Examine past research performed in house
- Review the academic literature to explore measures used by others who have assessed the performance of similar programs
Defining valid measures of performance is a key component of designing an effective program evaluation study.
6. From what sources should data be gathered?
- Users of the program?
- Non-users of the program? (e.g., to determine why they don’t use the resource)
- Potential program participants? (e.g., to identify their needs and preferences)
- Program administrators?
- Organizational databases?
- Paper files or written records?
7. How can the needed information be collected?
Common data collection methods in program evaluation include surveys, and individual and focus group interviews. Most organizations maintain databases or written records documenting program activities and the clients they serve. Sometimes existing data can be of value in a program evaluation study.
- Surveys provide information regarding characteristics of a population such as the percentage of program participants who became employed, or the percentage of students who rate the quality of their education as “excellent.”
- Individual Interviews are appropriate when discussing a sensitive topic or gathering qualitative information individuals may not feel comfortable sharing in a group. Individual interviews do not allow for generalizations to be made about the frequency opinions are held or the prevalence of certain characteristics within a population.
- Focus Groups can be a great starting point for exploring how people think and talk about an issue. Focus groups are useful for understanding the nature of people’s experiences. Focus groups do not allow for generalizations to be made about the frequency opinions are held or the prevalence of certain characteristics within a population.
- Secondary Data Collection and Analysis entails using existing data to describe or evaluate a program. While data gathered to document program operations can sometimes be useful in quantifying the inputs and outputs of a program, such data often lack utility in measuring a program’s impact.
- Observation is not used particularly often in program evaluation. A researcher might employ this method to evaluate the extent to which spatial arrangements are conducive to social interaction (e.g., examine how program participants engage with one another in a new living skills facility).
- Mixed Methods approaches entail employing multiple methodologies that complement and enhance the value of one another. For example, a researcher may conduct focus groups to identify key research issues and to lay the groundwork for the development of a survey instrument.
8. When must the study be completed?
9. What resources are available to gather and analyze data, and to prepare a report of findings?
10. How will findings be presented?
A slide presentation is generally less time-consuming to produce than a written report of findings, but is usually inadequate for fully documenting methods and results. If there is no detailed record of prior research, future research cannot build on past research efforts.
Formal written reports usually contain a detailed introduction, methods, results, and discussion section. Depending on the length of the report, an executive summary may be provided. It is ideal to include a copy of all data collection tools (e.g., survey instruments, interview protocols, coding programs, etc.) in an appendix for future reference.