ECUR 809 Assignment 2: Selecting an Evaluation Method

For our second assignment, we were asked to select an appropriate method to evaluate a  program designed to prevent diabetes in high risk groups, by providing free access to exercise programs to pregnant aboriginal women. Participation in the program would hopefully prevent cases of gestational diabetes mellitus, and type 2 diabetes in pregnant women, therefore hopefully preventing occurrences of type 2 diabetes in the children of these women (Klomp, Dyck, Sheppard, 2003).

I feel that the Outcomes-Based Evaluation method fits with this program, but I realize that it also probably speaks to my own personal philosophies about what makes a program successful. The Outcomes Based Evaluation method, as described on managementhelp.org, focuses on the impact that the program has on the users.

To make my decision,  I envisioned the program using a Program Logic Model, as explained on the University of Missouri's website, reflecting on who is involved in the program, and the desired short, medium and long-term outcomes. This lead me to the Outcomes-Based Evaluation method, which is especially appropriate for the non-profit sector (Plantz, 2006).  A review of the guiding questions on Managementhelp.org also directed me to this method.  Although there are many things that could be assessed, including how the classes were delivered, the qualifications of the instructors, and several other factors, the only thing that truly matters is whether or not the program helps to reduce occurrence of diabetes.

Managementhelp.org provides a brief overview of the steps involved in this type of evaluation, and based on the information provided in the case study, I feel that it would be possible to work through these steps. Beginning with identifying the outcome to be measured (I would consider repeat participation over multiple weeks, as well as instances of GSM), moving on to identifying observable measures (attendance lists and instances of GSM). Information about the target group is clearly available, as described in the case study, and the outcomes could be assessed using a telephone survey.  Participant contact information is available, as participants were contacted each week with a reminder to attend, therefore a follow up telephone survey or interview should be easy to obtain. The biggest challenge with using this method would be tracking the long-term outcomes desired. Short term outcomes - for example, participation in the program, could be easily tracked. Medium-term outcomes - the decreased occurrence of GSM, could be tracked with telephone surveys. The decreased occurrence of type 2 diabetes in the children of the participants would take years to track. However, success in short and medium term goals would likely be enough to evaluate the program in the meantime, as studies have proven that these outcomes do impact the occurrence of type 2 diabetes in children. This program evaluation would likely drive further participation, by program participants, and program leaders or employees, as it is the method that most clearly focuses on the good that comes from this initiative.  

References

Candidate Outcome Indicators: Health Risk Reduction Program. (n.d.). The Urban Insitute. Retrieved February 6, 2014, from http://www.urban.org/center/met/projects/upload/Health_Risk_Reduc.pdf

Klomp, H., Dyck, R., and Sheppard, S. (2003). Description and evaluation of a prenatal exercise program for urban Aboriginal women. Canadian Journal of Diabetes, 27: 231–238.

McNamara, C. (n.d.). Basic Guide to Program Evaluation (Including Outcomes Evaluation). Basic Guide to Program Evaluation (Including Many Additional Resources). Retrieved February 06, 2014, from http://managementhelp.org/evaluation/program-evaluation-guide.htm#anchor184773

Plantz, M. C., Greenway, M. T., & Hendricks, M. (2006, March 3). Outcome management: Showing results in the nonprofit secotr. Outcome Measurement Resource Network. Retrieved February 6, 2014, from https://www.nationalserviceresources.gov/files/legacy/filemanager/download/ProgramMgmt/Outcome_Measurement_Showing_Results_Nonprofit_Sector.pdf

Program Development & Planning: ProgramLogic Model. (n.d.). Program Development & Planning: ProgramLogic Model. Retrieved February 06, 2014, from http://extension.missouri.edu/staff/programdev/plm/

ECUR 809 Assignment 1: Evaluating a Program Evaluation

As an an introduction to the field of Program Evaluation, the first assignment in my current course, ETAD 809, is to review a completed evaluation.

The Calgary AfterSchool program is a project put in place by the City of Calgary and UpStart (The United Way). This city-wide version of the program was launched in September 2009, and the evaluation period ended in June 2013. The AfterSchool program is the implementation of afterschool programming for children and youth across the city, free of charge, in order to promote positive child and youth development. The program claims to be the only of its kind in Canada.

The final program evaluation was published in October of 2013, although there were several interim evaluations along the way. I reviewed the interim evaluation from 2011, mid-way through the project.

The evaluation is based on the programs two goals, identified in the final evaluation as “(i) to increase the participation of Calgary’s children and youth in high-quality after-school programming by increasing the quantity and accessibility of high-quality recreational and developmental programming throughout the city, and (ii) to improve participants’ social and emotional development and school engagement.” Both the interim and final evaluations concluded that the program met both of these goals, and was a success.

Due to the large size of the program, data collection was challenging throughout the study. Pre-and post questionnaires were administered to participants. The results were collected, and for statistical reasons that I don’t fully understand, the evaluators then selected only locations which were able to collect data from all participants to use when analyzing the results.

Two things stand out for me in these documents. First, the results, which are available in graphical format, are separated into categories for children and youth, and then further into “overall”, and those “with poor pre-test scores”. These divisions allow the reader to see how the program impacts each group differently, and highlights the results that are most important, the children and and youth deemed to be at-risk, with poor pre-test scores. I agree with this separation in results, because for many youth, they likely just transferred into the AfterSchool program from another paid program. To know if the programs have a real impact, the customer would need to see results for children that are not receiving this services elsewhere. This does raise, two follow-up questions which I would investigate if running this type of program: What percentage of the participants are included in the at-risk results? Are City’s funds going to the children and youth who really need it?

The second thing that jumped out at me was that the surveys were not administered to any students in grades one to three because they required reading and writing skills. The results in the report do not reflect how well the program serves these children. Although I understand the complications in administering surveys to such young participants, another method, perhaps interviews, anecdotal evidence from program instructors, or parent surveys, could have been used in order to provide a more complete and accurate picture of the program’s effectiveness. Without this information, how can those who manage the program truly know if it is worthwhile to deliver programs for the youngest participants?

____________________________________________

Note: I wrote my original post in Google Docs. Using the Research Tool, I included proper APA footnotes, although I have no idea, yet, if and how that needs to be done in a blog. In any case, they did not copy over to Blogger, nor can I find a way to add them in. Here's a link to my Google Doc that includes the footnotes.