Results and Evaluation of HOME WORKS!

Children who participate in home visiting programs have improved language, literacy and social skills and are more likely to graduate high school. And research finds that parent participants are more likely to retain jobs and stay off public assistance.
Diana Mendley Rauner, Ounce of Prevention Fund

HOME WORKS! values evaluation to promote program accountability, knowledge development, and continuous quality improvement. HOME WORKS! partners with an external evaluator who conducts a process evaluation of its funded programs each year to assess overall implementation quality and program efficacy. The process evaluation proposed for the 2016-17 school year will incorporate data from several sources, including training participant feedback surveys, teacher logs, and parent surveys administered at school-sponsored family dinners. Training participant feedback surveys will be completed by teachers, school administrators, site coordinators, and other school staff at two administration time points: at the conclusion of each training event and at 3-month follow-up time point. Feedback surveys will be used to assess the overall quality of training delivery, the completeness and relevance of training content, and the applicability of training concepts to the home visit experience.HOME WORKS! also maintains an internal tracking system to monitor information logged by home visitors at the completion of each teacher home visit. Information maintained in the data tracking system is used to ensure program accountability and assess fidelity to the program model, and to generate appropriate reimbursements. The information logged by teachers into the system includes: the number of students and families targeted for participation, the number and percentage of visit attempts that resulted in completed home visits, the date and duration of completed visits, the individuals present at each visit, the demographic characteristicis and needs of participating students and families, and the perceived outcome of the intervention. Information is exported for analysis on a monthly basis and is analyzed more in-depth as part of the annual evaluation reporting process.

The process evaluation also includes the use of parent feedback forms administered to parents attending school-sponored family dinner events on two scheduled dates each school year. The parent feedback form asks parents to share their perceptions of the family dinner event, and their perceptions regarding the impact of the teacher home visit on parent-teacher relationship quality, parent-teacher communication, parent engagement in their child’s education, and improvements in their child’s school performance. HOME WORKS! establishes formal Memorandums of Understanding with each district, following federal data access and confidentiality guidelines, to collect school data, and pays Site Coordinators in each school to facilitate the data collection process.

HOME WORKS! also partners with an external research firm who is conducting an exploratory-phase analysis, as defined by the National Science Foundation, that will provide a preliminary evaluation of student outcomes. The analysis uses a quasi-experimental design to compare differences in student attendance, classroom behavior, and test performance between students whose families received a teacher home visits and students whose families did not both within participating schools and between participating and non-participating schools. The results of the analysis will be used to help refine the program model and pave the way for a more rigorous summative evaluation planned for the program. HOME WORKS! has convened a national evaluation team who has requested federal funding through the Institute of Education Sciences (IES) to implement a randomized control trial (RCT). The RCT will measure the impacts of the HOME WORKS! program model on classroom behavior, school attendance, and school achievement in early elementary school settings. The RCT study involves a partnership with the St. Louis Public Schools and is set to commence in the 2016-17 school year.

An important priority for the HOME WORKS! evaluation is to produce relevant, actionable findings that will inform program planning, and promote successful program expansion and replication in other school environments. The process evaluation measures fidelity to the program model and captures feedback from teachers, school administrators, and parents to identify program strengths, challenges, and areas in need of refinement. The evaluation team provides regularly scheduled updates on evaluation findings to the HOME WORKS! team and facilitates dialogue on how these findings can contribute to project improvements.The more rigorous RCT-based impact evaluation will be used to determine which school factors or program adaptations facilitate successful implementation, and will help distill lessons learned from the teacher home visit experience that can shape future program refinements. Ultimately, the RCT-based impact evaluation will produce the hard evidence answering the question we most care about – Are we moving the needle in regard to student behavior and academics?

Ultimately, we want to produce the quality of evidence that will establish programs like HOME WORKS! as cost-effective game-changers for educating our children. This would justify incorporating this type of rich, structured parent outreach as a regular school function in the minds of the education community and the public.