Evaluator Exclusion in the Program Planning and Design Phase

Program leadership’s decision to include an evaluator during the program planning and design phase is the critical first step necessary for evaluators to provide the programmatic benefits associated with the evaluation profession. Several recent developments have promoted evaluator inclusion in program planning and design activities, including federal legislation that mandates evaluator inclusion and advocacy efforts from evaluation academics. However, the evaluation literature presents a collective frustration within the evaluation field due to ongoing exclusion from program planning and design activities. Utilizing the defensive attribution hypothesis, this quantitative study gathered responses from 260 American Evaluation Association members and 61 Project Management Institute members to determine an evaluator exclusion rate, develop a taxonomy of exclusion factors, and explore the extent to which program leaders and program evaluators demonstrate defensive attributions when rating these factors’ influence on evaluator exclusion in program planning and design activities. Results indicated an approximately 70% evaluator exclusion rate in respondents’ most recent program experiences. Furthermore, the defensive attribution hypothesis was not supported in the study, as program evaluators more strongly attributed their lack of inclusion to deficiencies outside of the evaluation practice, but program leaders also more strongly attributed evaluator exclusion to deficiencies outside of the evaluation practice. Program evaluators most strongly attributed their exclusion to program leaders’ insufficient training and knowledge on the role of evaluation during the program planning and design phase. Program leaders most strongly attributed evaluator exclusion to their own staffing decisions, indicating a preference to not include evaluators in program planning and design activities due to achieving previous program success without them, assigning evaluation activities to non-evaluation staff, and a funding process that allows the practice to occur. As the first study to explore evaluator exclusion in the program planning and design phase, it sets a foundation for future research studies to corroborate and build upon its findings, identify policies that encourage evaluator inclusion, and continue efforts to establish mutually beneficial relationships in the program planning and design phase.