The Reading Mentors Program: A Language and Literacy Partnership 
 
Rosaline Tio 
 
2016-2017 Mid-Year Evaluation Summary 
 
February 2017 
 
 2016-2017 Reading Mentors Program Mid-Year Evaluation Summary 
Mid-Year Evaluation Summary 
Overview 
The Reading Mentors Program: A Language and Literacy Partnership (RMP) aims to improve the efficacy of kindergarten through third grade (K-3) teachers in literacy instruction through comprehensive coaching support for teachers and leaders at participating schools. Language and Literacy Specialists (LLSs) provide research-based professional learning, personalized one-on-one coaching based on individual teacher needs, and support for data-driven instruction to teachers and administrators in participating Georgia public schools. The RMP aims to build capacity and facilitate the change needed in each participating school to strengthen instructional practices and help more students read at grade level by the end of third grade. 
Governor Deal first began the RMP in 2012. The 2016-2017 school year marks the second year of the newly revamped three-year RMP that has expanded its focus to include not only conventional reading skills, but also foundational reading skills such as speaking, listening, and writing. The RMP currently serves approximately 26,600 students in 77 schools and 24 districts across the state. Participating schools submitted applications to be a part of the program. The Governor's Office of Student Achievement (GOSA) reviews school applications, hires the RMP staff, and provides overall guidance to the program. The RMP team consists of 2 Program Managers and 20 LLSs. The LLSs work directly with teachers and leaders in approximately three to five schools each, and the Program Managers oversee and support LLSs. 
Program Goals 
The 2016-2017 one-year goals for the RMP, as identified in the strategic plan, include: 
 90% of principals and teachers indicate that they are well supported by their LLS. 
 85% of teachers will indicate that they have learned a variety of instructional strategies from their respective LLS. 
 100% of the LLS team will actively coach 10 or more teachers in a virtual coaching platform. 
 100% of district central office staff indicate that they received timely and professional communication regarding the status of the program from their respective Program Manager.1 
 Generate statistically significant increases in the percentage of students meeting grade-level benchmarks in Oral Reading Fluency on the Dynamic 
1 The RMP has two Program Managers who oversee and support the activities of the LLSs and frequently interact with all district-level program participants. 
1 
 
 2016-2017 Reading Mentors Program Mid-Year Evaluation Summary 
Indicators of Basic Early Literacy Skills (DIBELS) Next assessment from beginning-of-year administration to end-of-year administration. 
The three-year goals for the RMP (to be completed by the end of the 2017-2018 school year), as identified in the strategic plan, include: 
 90% of students will be reading at or above grade level by the end of third grade. 
 95% of stakeholders (teachers, coaches, administrators, and district staff) will indicate that participation in the RMP is valuable in meeting individual teacher professional learning goals, as well as school- and/or system-level literacy goals. 
Evaluation Methodology 
GOSA's Evaluation team will produce annual mid-year updates and end-of-year reports for each year of the three-year program, as well as a summative report on the RMP as a whole at the end of the third year. GOSA's Evaluation team developed several evaluation instruments to collect information to inform developmental and summative analyses of the RMP. The evaluation focuses on three areas: program implementation, stakeholder satisfaction and impact, and student outcomes. This summary presents preliminary findings from multiple evaluation instruments for the 2016-2017 school year and presents the baseline data collected. Other instruments are either still in progress or have yet to be administered. Evaluation instruments include weekly logs submitted by the LLSs, mid-year surveys, the School Literacy Needs Assessments, the Teacher Progress Monitoring Forms, and baseline student reading performance data. The 20162017 mid-year summary provides an overall status update of the RMP's implementation so far, and the 2016-2017 end-of-year report will provide more evaluative conclusions and recommendations. 
Preliminary Findings 
EVALUATION FOCUS AREA I: PROGRAM IMPLEMENTATION 
GOSA's Evaluation team analyzed data from weekly logs completed by LLSs to evaluate program implementation. Preliminary findings include2: 
 During the first half of the school year, 45% of LLSs' coaching time was spent conducting observations and conferencing with teachers. 
2 The preliminary findings do not include all categories of LLS coaching activities, so percentages do not add up to 100%. Additionally, LLSs report hours for coaching activities and content area simultaneously, so these percentages also do not add up to 100%. 
2 
 
 2016-2017 Reading Mentors Program Mid-Year Evaluation Summary 
 LLSs spent 21% of their coaching time conducting professional learning sessions in schools. 
 LLSs spent 9% of their coaching time providing virtual coaching to teachers. 
 In terms of content area, LLSs spent 47% of their time supporting the implementation of research-based instructional strategies, 26% on data review, and 20% on assessment strategies. 
EVALUATION FOCUS AREA II: STAKEHOLDER SATISFACTION AND IMPACT 
Data collected from mid-year surveys, the School Literacy Needs Assessment, and Teacher Progress Monitoring Forms provide baseline data and preliminary findings on stakeholder impact and satisfaction. Preliminary findings include: 
 The RMP is on track to meet its goal for 90% of administrators and teachers to indicate that they are well supported by the LLS with 89% of administrator respondents feeling very or extremely supported. However, only 79% of teachers who responded to the survey feel very or extremely supported. 
 Although the RMP is on track to meet its three-year goal for 95% of administrators and district staff to indicate that participation in the RMP is valuable to meeting literacy goals, only 72% of teacher respondents feel the RMP is valuable. 
 100% of district staff who responded to the survey indicated they received good or excellent professional and timely communication from the Program Managers. 
 Of the teacher survey respondents participating in video coaching, 38% feel comfortable with video coaching, and 61% feel video coaching will be valuable to their professional development. 
 Teachers who began participating in the RMP before 2015-2016 feel more positively about the value of program participation and LLS supports. 
 All stakeholder respondents recommended increasing the amount of faceto-face time with the LLSs beyond one day a week. 
 According to the School Literacy Needs Assessment, over half of participating schools were at least operational in implementing engaged leadership, ongoing assessments, best practices in literacy instruction, and tiered interventions for students.3 However, schools could use more support in establishing continuity of literacy instruction schoolwide and improving instruction through effective professional learning. 
3 Tiered interventions are part of the Response-to-Intervention (RTI) model in which teachers provide individualized supports in addition to regular classroom instruction to students who are performing below grade level according to the student's specific needs. 
3 
 
 2016-2017 Reading Mentors Program Mid-Year Evaluation Summary 
 Larger percentages of schools who began participating in the RMP in 2015-2016 are at least operational in all building blocks when compared to schools who began participating this school year. 
 LLSs identified 34% of teachers as proficient or exemplary in employing research-based instructional strategies to engage students in active learning and 31% of teachers as proficient or exemplary in using assessment data to drive instruction. 
 LLSs indicated that teachers still need support in using flexible groups to support targeted small group instruction, developing higher-order thinking among students, using data to inform instruction, and encouraging student articulation of data and goals. 
EVALUATION FOCUS AREA III: STUDENT OUTCOMES 
GOSA's Evaluation team analyzed student performance on the DIBELS Next assessment at the beginning of the year (BOY) to establish baseline data for 20162017. GOSA's Evaluation team will also analyze the College and Career Ready Performance Index (CCRPI) Third Grade Lexile indicator to evaluate any changes in school performance over time.4 Preliminary findings include: 
 53% of all students were meeting DIBELS Next benchmark goals at the beginning of the school year, which is 37 percentage points below the RMP's three-year goal for 90% of students to be reading at or above grade level by the end of third grade. 
 The percentage of students meeting benchmark goals at the beginning of the year was highest in second grade (58%) and lowest in first grade (49%). 
 52% of schools had BOY percentages of students meeting benchmarks that were greater than the overall program percentage (53%). 
 Of the students in schools who began the RMP in 2015-2016, 58% of students met benchmark goals at the beginning of the year compared to only 41% of students in schools who began the RMP this school year. 
 51% of students tested in oral reading fluency met benchmark goals at the beginning of the year.5 
Next Steps 
The preliminary findings establish strong baseline data for more in-depth analysis of the RMP's impact at the end of the year and in the future. The program 
4 Since only 2016 CCRPI data is available, GOSA will release this analysis as an addendum to the 2015-2016 RMP end-of-year report. 5 Only second and third graders are assessed on oral reading fluency. Oral reading fluency is measured by taking the median number of words read correctly by a student on three one-minute passages. 
4 
 
 2016-2017 Reading Mentors Program Mid-Year Evaluation Summary 
implementation findings indicate that LLSs may need to increase the time spent video coaching to provide more coaching support to teachers. The stakeholder satisfaction and impact findings reveal general satisfaction with the program, but a desire for more time with LLSs. The student outcomes data demonstrate a need for continued reading support in all grade levels to improve student achievement, especially in schools that are new to the RMP. GOSA's Evaluation team will continue to use consistent evaluation instruments to collect data on program implementation, stakeholder satisfaction and impact, and student outcomes for each year of the three-year program. The annual mid-year and end-of-year reports will monitor any growth and inform ongoing program developments. The summative report at the end of the RMP will evaluate any three-year trends and the program's overall impact. 
5