Reading Mentors program 2015-2016 mid-year evaluation report

The Reading Mentors Program: A Language and Literacy Partnership

Rosaline Tio

2015-2016 Mid-Year Evaluation Report

January 2016

This page is intentionally left blank. ii

Executive Summary
Overview The Reading Mentors Program: A Language and Literacy Partnership (RMP) aims to improve the efficacy of K-3 teachers in literacy instruction through comprehensive coaching support for teachers and leaders at participating schools. Language and Literacy Specialists (LLSs) provide research-based professional learning, personalized one-on-one coaching based on individual teacher needs, and support for data-driven instruction to teachers and administrators in participating Georgia public schools. The RMP aims to build capacity and facilitate the change needed in each participating school to strengthen instructional practices and help more students read at grade level by the end of third grade.
Governor Deal first began the RMP in 2012. The 2015-2016 school year marks the beginning of the newly revamped three-year RMP that has expanded its focus to include not only conventional reading skills, but also foundational reading skills such as speaking, listening, and writing. The RMP currently serves approximately 21,000 students in 60 schools and 22 districts across the state. Participating schools submitted applications to be a part of the program. The Governor's Office of Student Achievement (GOSA) reviews school applications, hires the RMP staff, and provides overall guidance to the program. The RMP team consists of 2 Program Managers and 17 LLSs. The LLSs work directly with teachers and leaders in approximately three to five schools each, and the Program Managers oversee and support LLSs.
Evaluation Methodology GOSA will produce annual mid-year and end-of-year reports for each year of the three-year program, as well as a summative report on the RMP as a whole at the end of the third year. GOSA has developed several evaluation instruments to collect information to inform developmental and summative analyses of the RMP. The evaluation will focus on three areas: program implementation, stakeholder impact and satisfaction, and student outcomes. This report presents preliminary findings using all available data from the evaluation instruments with baseline data collected. Other instruments are either still in progress or have yet to be administered. Thus, this report provides an overall status update of the RMP's implementation so far, and the 2015-2016 RMP End-of-Year Report will provide more evaluative conclusions and recommendations.
Preliminary Findings The preliminary findings below include baseline data from weekly logs submitted by the LLSs, School Literacy Needs Assessments, Teacher Progress Monitoring Forms, and the Mid-Year Satisfaction Survey.
iii

PROGRAM IMPLEMENTATION
Data collected from the weekly logs completed by LLSs provide insight on typical LLS activities:
LLSs spend 42% of their time on supplemental LLS operations, which include conducting professional learning sessions for school faculty, gathering resources, and other administrative work.
LLSs spend 21% of their time providing one-on-one coaching to teachers. LLSs spend the remainder of their time providing data analysis and
assessment support to schools, attending their own professional development, and completing additional administrative work. In terms of content area, LLSs spend 41% of their time on instructional strategies, 28% on assessment strategies, and 27% on data review.
STAKEHOLDER IMPACT AND SATISFACTION
Data collected from the School Literacy Needs Assessment, Teacher Progress Monitoring Forms, and Mid-Year Satisfaction Survey provide baseline data and preliminary findings on stakeholder impact and satisfaction:
As reported by LLSs and school administrators on the School Literacy Needs Assessments, over half of participating schools had certain elements of a supportive environment for literacy instruction present at the beginning of the school year, such as engaged school leadership, ongoing use of assessments, and systems of tiered intervention to support students. However, the majority of schools still need to establish stronger supports for professional learning and continuous instruction throughout the school. LLSs may need to provide more support to schools in gaining community support for literacy instruction and developing more effective professional learning opportunities for teachers on literacy instruction across all disciplines.
According to LLS ratings on the Teacher Progress Monitoring Forms, less than 30% of observed teachers are proficiently employing effective instructional strategies and using assessments to guide instruction. LLSs need to provide extensive support for teachers on how to implement research-based literacy instructional practices and use assessments to make decisions about instruction.
The Mid-Year Satisfaction Survey results reveal that, in general, participating administrators and district personnel feel well supported by LLSs, receive timely and professional communication from Program Managers, and feel participation in the RMP has been valuable.
iv

Some administrators and district personnel are concerned that LLSs are not able to spend enough time at each school because they are working with multiple schools at once. Additionally, some RMP participants feel overwhelmed by certain program expectations, such as adhering to RMP expectations for progress monitoring students in addition to existing school or district requirements.
STUDENT OUTCOMES
All participating schools are using the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Next assessment to measure student performance. GOSA will use DIBELS Next benchmark scores throughout the school year to evaluate any student growth. GOSA also plans to analyze the College and Career Ready Performance Index (CCRPI) Third Grade Lexile indicator to evaluate any changes in school performance over time. DIBELS Next baseline data and CCRPI Third Grade Lexile indicators are not yet available and thus not included in this report. Next Steps The preliminary findings establish strong baseline data for more in-depth analysis of the RMP's impact in the future. GOSA plans to use consistent evaluation instruments to collect data on program implementation, stakeholder impact and satisfaction, and student outcomes for each year of the three-year program. The annual mid-year and end-of-year reports will monitor any growth and inform program developments. The summative report at the end of the RMP will evaluate any three-year trends and the program's overall impact.
v

Table of Contents
EXECUTIVE SUMMARY .......................................................................................................... III OVERVIEW ................................................................................................................................... III EVALUATION METHODOLOGY ..................................................................................................... III PRELIMINARY FINDINGS .............................................................................................................. III Program Implementation.........................................................................................................iv Stakeholder Impact and Satisfaction .......................................................................................iv Student Outcomes .....................................................................................................................v NEXT STEPS .................................................................................................................................. V
TABLE OF FIGURES AND TABLES......................................................................................VII TABLE OF ABBREVIATIONS ................................................................................................VII INTRODUCTION ...........................................................................................................................1 RMP MISSION AND GOALS .......................................................................................................2 PROFILE OF PARTICIPATING SCHOOLS .............................................................................3 EVALUATION METHODOLOGY ..............................................................................................5 PRELIMINARY FINDINGS .........................................................................................................7
PROGRAM IMPLEMENTATION.........................................................................................................7 LLS Weekly Logs ......................................................................................................................7
STAKEHOLDER IMPACT AND SATISFACTION ................................................................................10 School Literacy Needs Assessment .........................................................................................10 Teacher Progress Monitoring Forms .....................................................................................12 Mid-Year Satisfaction Survey .................................................................................................15
STUDENT OUTCOMES...................................................................................................................16 CONCLUSION .............................................................................................................................. 17 APPENDIX A: LIST OF PARTICIPATING SCHOOLS AND DISTRICTS IN THE RMP 19 APPENDIX B: DEMOGRAPHIC PROFILES FOR ALL RMP PARTICIPATING SCHOOLS .....................................................................................................................................21 APPENDIX C: LLS ACTIVITIES BY CATEGORY................................................................25 APPENDIX D: SCHOOL LITERACY NEEDS ASSESSMENT SURVEY ITEMS ..............27 APPENDIX E: AVERAGE RATINGS AND PERCENT OF SCHOOLS AT OPERATIONAL OR FULLY OPERATIONAL FOR ALL INDICATORS OF SCHOOL LITERACY NEEDS ASSESSMENT ..........................................................................................35 APPENDIX F: TEACHER PROGRESS MONITORING FORM SURVEY ITEMS............37 APPENDIX G: PERCENTAGE OF TEACHERS PROFICIENT OR EXEMPLARY FOR ALL INDICATORS AND GRADE LEVELS ON THE TEACHER PROGRESS MONITORING FORM ................................................................................................................39
vi

Table of Figures and Tables Table 1: Demographic Profile Comparison of RMP and the State......................... 4 Table 2: Summary of GOSA's RMP Evaluation Plan............................................ 5 Table 3: Grade Distribution of Teachers Monitored............................................. 13 Table 4: Overall Percent of Teachers Proficient or Exemplary ............................ 14 Table 5: Mid-Year Satisfaction Survey Results.................................................... 15
Figure 1: Map of Participating RMP Schools......................................................... 3 Figure 2: Distribution of LLS Activities by Category through October 30............ 8 Figure 3: Distribution of LLS Activities by Focus Area through October 30 ........ 9 Figure 4: Average Percent of Schools Performing at the Operational or Fully Operational Level by Building Block ................................................................... 11

Table of Abbreviations
CCRPI College and Career Ready Performance Index
FRL Free/reduced-price lunch

GOSA RMP
TAPS

Governor's Office of Student Achievement Reading Mentors Program: A Language and Literacy Partnership Teacher Assessment on Performance Standards

DIBELS
GaDOE LLS RTI
TKES

Dynamic Indicators of Basic Early Literacy Skills
Georgia Department of Education Language and Literacy Specialist Response to Intervention
Teacher Keys Effectiveness System

vii

2015-2016 RMP Mid-Year Evaluation Report
Introduction
The Reading Mentors Program: A Language and Literacy Partnership (RMP) aims to improve the efficacy of K-3 teachers in literacy instruction through comprehensive coaching support for teachers and leaders at participating schools. The program's Language and Literacy Specialists (LLSs) provide research-based professional learning, personalized one-on-one coaching based on individual teacher needs, and support for data-driven instruction to K-3 teachers and administrators in participating schools. LLSs aim to build capacity and facilitate the change needed in each participating school to strengthen instructional practices and help more students read at grade level by the end of third grade.
Governor Deal first introduced the RMP in 2012 as part of an initiative to improve the percentage of children reading at grade level by the end of third grade. Beginning with the 2015-2016 school year, the Governor's Office of Student Achievement (GOSA) revamped the RMP. The focus of the program has now expanded to encompass not only conventional reading skills--including decoding or sounding out words, oral reading fluency, reading comprehension, and spelling, but also foundational early literacy skills including speaking, listening, and writing.1 GOSA believes that by providing more research-based and comprehensive coaching support to teachers and staff in participating schools, the RMP can foster lasting changes in literacy instruction that will improve student reading performance across the state. The RMP currently services 60 participating schools in 22 districts. Schools applied to be in the RMP, and GOSA reviewed and selected schools from the applicant pool.
The 2015-2016 RMP Mid-Year Evaluation Report is a status update of the RMP's activities since the beginning of the 2015-2016 school year. GOSA's Research and Evaluation unit conducted this evaluation. The evaluation team collaborated with RMP staff to establish the evaluation plan and collect and analyze baseline data. The 2015-2016 RMP Mid-Year Evaluation Report includes:
A summary of the RMP's mission and goals, A profile of participating schools, A description of the evaluation methodology, and Preliminary findings from collected baseline data.
1 Decoding is the ability to apply letter-sound relationships to sound out written words.
1

2015-2016 RMP Mid-Year Evaluation Report
RMP Mission and Goals
The mission of the RMP is to build teacher and leader capacity to deliver highquality literacy instruction that increases student achievement. LLSs help Georgia's teachers and leaders establish supportive and long-lasting environments that enable effective literacy instruction to ensure students are reading at or above grade level at the end of each grade.
The one-year goals for the RMP, as identified in the strategic plan, include:
LLSs will show marked improvement in their knowledge of early literacy skills related to oral language and phonological awareness.2.
90% of principals and teachers indicate that they are well supported by their LLS.
100% of district central office staff indicate that they received timely and professional communication regarding the status of the program from their respective Program Manager.3
Generate statistically significant increases in the percentage of students meeting grade-level benchmarks in Oral Reading Fluency on the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Next assessment from beginning-of-year administration to end-of-year administration.4
The three-year goals for the RMP, as identified in the strategic plan, include:
90% of students will be reading at or above grade level by the end of third grade.
95% of stakeholders will indicate that participation in the RMP is valuable in meeting individual teacher professional learning goals, as well as school- and/or system-level literacy goals.
2 Phonological awareness is the ability to understand that words are made up of different sound units. 3 The RMP has two Program Managers who oversee and support the activities of the LLSs and frequently interact with all district-level program participants. 4 Specific goals will be set once all baseline DIBELS Next data are received and analyzed.
2

2015-2016 RMP Mid-Year Evaluation Report
Profile of Participating Schools The RMP currently serves 60 schools in 22 districts across the state. All participating schools had to submit an application to be a part of the program. Since program participation is application-based, LLSs serve districts and schools with varied socioeconomic and academic backgrounds. Some LLSs work with high-capacity urban districts and schools, but other LLSs serve districts and schools who have never had an individual dedicated to language and literacy support placed in their building. Figure 1is a map showing the geographic distribution of the 60 participating schools. A full table of participating schools and districts is available in Appendix A.
Figure 1: Map of Participating RMP Schools 3

2015-2016 RMP Mid-Year Evaluation Report

LLSs serve all K-3 teachers, staff, and students at participating schools. GOSA used the Georgia Department of Education's (GaDOE) recent release of school enrollment by ethnicity/race, gender, and grade level to produce demographic profiles of RMP participating schools.5 Currently, the RMP serves 21,343 K-3 students across the state, or approximately four percent of the state's total K-3 student population.

Although there is great diversity among the student population of RMP schools, the demographic profile of the RMP as a whole reveals overall racial/ethnic differences when compared to the state's K-3 student population. Table 1 shows the racial/ethnic distribution of K-3 students in the RMP and the state. 63% of RMP students are black, which is 26 percentage points higher than the overall state percentage. Additionally, the RMP has a lower percentage of white students (23%) compared to the overall state percentage (39%). The RMP student population also consists of a smaller share of Hispanics (9%) than the state's K-3 student population (16%). However, it is important to remember that the demographic profile presented in Table 1 is merely an overall summary of the racial/ethnic demographics for all students in the RMP and does not capture school-level differences. A full breakdown of racial/ethnic demographics for each participating school is available in Appendix B.
Table 1: Demographic Profile Comparison of RMP and the State

RMP Total

K-3 Students in Georgia

American Indian <1%

<1%

Asian

2%

4%

Black

63%

37%

Hispanic

9%

16%

Pacific Islander

<1%

<1%

Multi-Racial

3%

4%

White

23%

39%

Source: GaDOE October 6, 2015 FTE Enrollment by Ethnicity/Race, Gender,

and Grade Level

5 The demographic analysis does not include data on Students with Disabilities and Free/Reduced Price Lunch status because GaDOE has not yet released these data at the school-level.
4

2015-2016 RMP Mid-Year Evaluation Report

Evaluation Methodology

Given that the RMP is a three-year program, GOSA's Research and Evaluation unit plans to produce annual mid-year and end-of-year reports, as well as a summative report on the RMP as a whole at the end of the third year. GOSA's evaluation of the RMP will analyze developmental and summative information in three focus areas: program implementation, stakeholder impact and satisfaction, and student outcomes. Table 2 lists each evaluation focus area with its respective evaluation questions and instruments. The following sections will present preliminary findings in each of the three focus areas from the evaluation instruments with available data, including the LLS Weekly Logs, Mid-Year Satisfaction Survey, Teacher Progress Monitoring Forms, and School Literacy Needs Assessments.
Table 2: Summary of GOSA's RMP Evaluation Plan

Evaluation Focus Area

Evaluation Question(s)

Instruments

Program Implementation

Did LLSs present research-based strategies that provide instructional support for teachers and school staff to better serve students?
Was the grant program implemented with fidelity?

LLS Evaluation Tool
LLS Weekly Logs of coaching activities*

Stakeholder (Teacher, Administrator, District Personnel) Impact and Satisfaction

Do stakeholders (teachers, administrators, district personnel) feel satisfied with and believe there was a value add from the instructional support provided by LLSs?
How impactful are the LLSs in developing highly effective teachers and strong literacy instructional environments in participating schools?

Mid-Year Satisfaction Survey of administrators and district personnel*
Ongoing Teacher Progress Monitoring Forms*
Survey of teachers, administrators, district personnel, and LLSs
Program Manager Site Visit Forms

School Literacy Needs Assessments*

5

2015-2016 RMP Mid-Year Evaluation Report

Evaluation Focus Area

Evaluation Question(s)

Instruments

DIBELS Next

benchmarking scores

(beginning of year,

Are students benefitting from greater middle of year, and end

Student Outcomes

teacher preparation from coaching of year), disaggregated and professional learning provided by subgroup

by LLSs?

CCRPI Third Grade

Lexile Indicator from

Georgia Milestones

*Asterisks denote instruments with baseline data available that are discussed in

this report.

6

2015-2016 RMP Mid-Year Evaluation Report
Preliminary Findings
Since the beginning of the 2015-2016 school year, GOSA has collected baseline data using several of the evaluation measures in Table 2. This report includes preliminary findings from the LLS Weekly Logs, School Literacy Needs Assessments, Mid-Year Satisfaction Survey, and Teacher Progress Monitoring Forms. Other evaluation measures are either still in progress or have yet to be administered. Therefore, the 2015-2016 RMP Mid-Year Evaluation Report focuses on providing an overall status update of the RMP's implementation so far, and the 2015-2016 RMP End-of-Year Report will provide more evaluative conclusions and recommendations. The preliminary findings below are organized according to the evaluation focus areas in Table 2.
Program Implementation
GOSA plans to use two evaluation measures to assess implementation of the RMP. The LLS Evaluation Tool is a performance rubric Program Managers will use to assess LLS performance throughout the year. The Program Managers are conducting LLS evaluations currently, so no data from the LLS Evaluation Tool are available for this report. The LLS Weekly Log is a cumulative tracking sheet documenting all LLS activities in a day that LLSs submit weekly to their Program Manager. This report includes data collected from all LLS Weekly Logs between August 10, 2015 and October 30, 2015.
LLS WEEKLY LOGS
The RMP team consists of 2 Program Managers and 17 LLSs. The LLSs work directly with teachers and leaders in participating schools, and the Program Managers oversee and support LLSs. On average, each LLS serves between three to five schools, so LLSs must divide their time among each of her schools.6 LLSs complete and submit Weekly Logs to document the amount of time spent on the myriad of coaching activities that comprise their work. The Program Managers monitor Weekly Logs to track LLSs' work and ensure coaching activities are appropriate and justifiable. GOSA's Research and Evaluation unit analyzes hour totals from the Weekly Logs to provide an overall understanding of LLS activities.
GOSA analyzed all submitted LLS Weekly Logs from the beginning of the school year through October 30, 2015. Typical LLS activities are organized into seven different categories: one-on-one coaching, data analysis and assessment support, LLS professional learning, supplementary LLS operations, school
6 All of the LLSs and Program Managers are female.
7

2015-2016 RMP Mid-Year Evaluation Report
closures/personal leave, LLS committee work, and additional work.7 A complete list of the LLS activities included in each category is available in Appendix C. Figure 2 shows the percentage distribution of LLS activities based on LLSreported hours in the Weekly Logs through October 30. The majority of the LLSs' time (42%) was spent on supplementary LLS operations, which includes developing and delivering group professional learning sessions, gathering resources for schools and individuals, providing technical support, preparing materials for school staff, collaborating with other LLSs, and completing administrative work for the program. As this is the first year of the second iteration of the RMP, LLSs spent a significant amount of time in the beginning of the school year conducting school-wide or grade-level professional learning sessions to establish relationships and build a strong foundation for the three-year partnership.
Figure 2: Distribution of LLS Activities by Category through October 30
7 "LLS Professional Learning" refers to trainings the LLSs personally attend to develop their skills. GOSA is committed to ensuring LLSs have adequate personal professional learning to build a strong foundation of knowledge so they can best serve their schools for the remainder of the RMP. Trainings conducted by LLSs are housed under "Supplemental LLS Operations." "Additional Work" includes administrative work for GOSA outside of RMP requirements, such as reviewing grant applications.
8

2015-2016 RMP Mid-Year Evaluation Report
Furthermore, LLSs have spent 21% of their time so far providing one-on-one coaching support for teachers. One-on-one coaching support has mostly consisted of classroom observations, educational discussions with teachers, and gathering additional resources to support teachers. Given the emphasis on data-driven instruction during this second iteration of the RMP, LLSs also spent 16% of their time providing support for data analysis and assessment strategies. All participating schools are using the DIBELS Next assessment to support literacy instruction. Since many schools are new to DIBELS Next, LLSs provided extensive training for teachers and administrators on how to administer and analyze DIBELS Next. LLSs have also facilitated frequent data conferences with teachers and administrators to model progress monitoring practices and ensure instructional decisions are truly data-driven. As part of the Weekly Logs, LLSs also categorize their daily activities according to four focus areas: instructional strategies, data review, assessment strategies, and classroom management. Figure 3 shows the percentage distribution of LLS activities by focus area based on the Weekly Logs.
Figure 3: Distribution of LLS Activities by Focus Area through October 30 Whereas the analysis of LLS activities by category shows what types of activities LLSs are engaged in on a daily basis, the analysis of LLS activities by focus area reveals the content of LLS coaching support thus far. 41% of the LLSs' time is spent providing coaching support to teachers and staff on how to implement
9

2015-2016 RMP Mid-Year Evaluation Report
research-based instructional strategies. 55% of the LLSs' time is spent modeling successful assessment strategies and facilitating data review at schools. LLSs' have only spent 4% of their time so far providing classroom management support. As the school year progresses and schools begin to implement the strategies learned so far, GOSA expects the distribution of LLS activities by category and focus area to fluctuate according to the developing needs of participating schools. The LLS Weekly Logs are meant to inform GOSA on the RMP's activities and where LLSs are focusing their time, rather than serve as an evaluation of the LLSs. Thus, the preliminary findings from the LLS Weekly Logs summarize the distribution of activities from August through October of 2015.
Stakeholder Impact and Satisfaction
GOSA plans to use several tools to evaluate stakeholder impact and satisfaction for the RMP. No data have been collected yet for the survey and Program Manager Site Visit Forms. Thus, this report only discusses preliminary findings from the School Literacy Needs Assessment, Teacher Progress Monitoring Forms, and Mid-Year Satisfaction Survey.
SCHOOL LITERACY NEEDS ASSESSMENT
The School Literacy Needs Assessment is a survey that LLSs complete with the administrators at each school to assess the literacy instructional environment at the beginning and end of the school year. The School Literacy Needs Assessment evaluates schools according to six building blocks identified by GOSA and the RMP Program Managers as important foundations for effective literacy instruction:
1) Engaged leadership, 2) Continuity of instruction, 3) Ongoing formative and summative assessments 4) Best practices in literacy instruction, 5) A system of tiered intervention (Response to Intervention, or RTI) for all
students, and 6) Improved instruction through professional learning.
The School Literacy Needs Assessment consists of 25 total questions that use a 4point scale measuring the functionality of a school's literacy environment.8 Each building block has its own list of indicators that the LLS and administrator use to rate the school. All 60 schools completed a School Literacy Needs Assessment at the beginning of the school year to establish a baseline. Schools will complete the
8 The response options are Not Addressed (1), Emergent (2), Operational (3), or Fully Operational (4). Definitions of the response options were provided to LLSs, who then shared the information with school administrators.
10

2015-2016 RMP Mid-Year Evaluation Report
assessment again at the end of the school year to determine any growth. A full list of the School Literacy Needs Assessment survey items is available in Appendix D.
Figure 4: Average Percent of Schools Performing at the Operational or Fully Operational Level by Building Block
Figure 4 shows the average percentage of schools performing at the Operational or Fully Operational level for each building block of the School Literacy Needs Assessment. Operational performance means that the building block or indicator is visibly present in the school; Fully Operational performance means that the building block or indicator is visibly present and successfully implemented in the school. Roughly 70% of all schools are performing at the Operational or Fully Operational level in the building blocks for ongoing formative and summative assessments and established systems of tiered intervention (RTI).9 Approximately 60% of schools are also performing at the Operational or Fully Operational level in the engaged leadership building block and best practices in literacy instruction. In fact, the indicator with the highest average percentage of schools receiving Operational or Fully Operational ratings was in the engaged leadership building block; on average, 90% of schools have administrators who are committed to
9 The purpose of formative assessments is to monitor student learning to provide ongoing feedback that teachers and students can use to improve instruction and learning, respectively. The purpose of summative assessments is to evaluate student learning at a particular point in time by comparing it to a standard or benchmark. For more information on the difference between formative and summative assessments, please see the GOSA RMP web page.
11

2015-2016 RMP Mid-Year Evaluation Report
learning about evidence-based literacy instruction and 88% of administrators use efficient scheduling and collaborative planning to ensure the best use of personnel's time.
A smaller percentage of schools are Operational or Fully Operational in the building blocks for consistent literacy instruction across the curriculum (45%) and improved instruction through professional learning (49%). The indicators with the lowest percentages of schools designated Operational or Fully Operational were assessing out-of-school community support for literacy instruction and preservice teacher preparation to teach literacy in all content areas. These findings indicate that LLSs may need to work with RMP schools to garner more community support for literacy instruction and develop stronger professional learning opportunities that better prepare teachers for effective literacy instruction across all disciplines. A full table of the average ratings for all indicators as well as the percentage of Operational and Fully Operational schools in each building block is available in Appendix E.
TEACHER PROGRESS MONITORING FORMS
The Teacher Progress Monitoring Form is an observation rubric that LLSs use to identify strengths and areas of improvement for teachers. Although LLSs monitor and work with teachers frequently as a major element of their work, GOSA plans to formally collect and analyze Teacher Progress Monitoring Forms three times a year to track instructional changes at the beginning, middle, and end of the year. The Teacher Progress Monitoring Form assesses teacher performance according to two Teacher Assessment on Performance Standards (TAPS) standards from the Teacher Keys Effectiveness System (TKES) that the RMP Program Managers identified as priority coaching areas for this school year.10 Though the Teacher Progress Monitoring Form is aligned to TAPS to better serve RMP teachers, the tool is not meant to be evaluative and will not be used as part of a teacher's formal TKES evaluation; GOSA developed this tool solely to aid LLSs in their work and to collect internal data on teacher instructional changes as a result of participation in the RMP.
The first priority area assessed in the Teacher Progress Monitoring Form is TAPS Standard 3 on Instructional Strategies--the teacher promotes student learning by using research-based instructional strategies relevant to the content to engage students in active learning and to facilitate the students' acquisition of key knowledge and skills. The other priority area is TAPS Standard 6 on Assessment Uses--the teacher systematically gathers, analyzes, and uses relevant data to measure student progress, to inform instructional content and delivery methods,
10 For more information on the TAPS standards, please see the TAPS Standards and Rubrics Reference Sheet. For more information on TKES, please see GaDoe's TKES website. Please note, the TAPS standards refer to general instruction and are not literacy specific.
12

2015-2016 RMP Mid-Year Evaluation Report

and to provide timely and constructive feedback to both students and parents. The RMP Program Managers established indicators for each TAPS standard for LLSs to use as a guide when monitoring and coaching teachers. LLSs use a four-point scale to evaluate a teacher's performance.11 A copy of the Teacher Progress Monitoring Form survey items is available in Appendix F.

LLSs submitted a sample of Teacher Progress Monitoring Forms for teachers from various grade levels in each of their schools.12 For the beginning-of-year collection, GOSA received 131 Teacher Progress Monitoring Forms out of the approximately 1,000 teachers LLSs are working with in the RMP. Table 3 shows the grade distribution of the Teacher Progress Monitoring Forms received. Though the LLSs work with all teachers in grades K-3, there were slightly more forms submitted to GOSA for kindergarten and 1st grade teachers than 2nd and 3rd grade teachers.
Table 3: Grade Distribution of Teachers Monitored

Grade

Total Number
of Teachers

Total Percentage

Kindergarten

40

1st

38

2nd

29

3rd

23

Other13

1

Total

131

31% 29% 22% 18% 1% 100%

Table 4 shows the percent of teachers scoring Proficient or Exemplary overall in Standards 3 and 6. LLSs found that only 29% of teachers were employing research-based instructional strategies to engage students in active learning at the Proficient or Exemplary level. Similarly, only 26% of teachers are using assessment data to drive instruction and provide feedback at the Proficient or Exemplary level. Among the individual indicators, the highest percentage of teachers are Proficient or Exemplary at building upon students' existing knowledge and skills (47%). Very few teachers (12%) are Proficient or Exemplary at teaching students how to self-assess and use metacognitive strategies in support of lifelong learning. A complete table of the percentages of

11 The response options are Ineffective (1), Needs Improvement (2), Proficient (3), or Exemplary (4). 12 Some LLSs submitted Teacher Progress Monitoring Forms for all of their teachers in each school, while others submitted a minimum of three Teacher Progress Monitoring Forms from each school. 13 One LLS is working with an instructional coach who has been placed in the classroom temporarily. This person's data was not included in the grade-level calculations for Appendix G.

13

2015-2016 RMP Mid-Year Evaluation Report

teachers receiving Proficient or Exemplary ratings for each indicator and grade level is available in Appendix G.
Table 4: Overall Percent of Teachers Proficient or Exemplary

Standard 3: Instructional Strategies Standard 6: Assessment Uses

Percent of Teachers Proficient or Exemplary
29% 26%

These findings serve as a baseline of teacher performance and indicate that the majority of RMP teachers need coaching support in implementing research-based literacy instructional strategies and using assessments appropriately to make datadriven decisions. As LLSs continue to work with teachers one-on-one throughout the school year, GOSA will analyze future collections of Teacher Progress Monitoring Forms to see if teacher performance improves.

LLSs also provide additional comments and suggested strategies for teachers as part of the Teacher Progress Monitoring Form. For instructional strategies, common suggestions included:

Utilizing more small group literacy instruction as opposed to only whole group instruction,
Improving differentiation and active engagement strategies, Establishing clearer classroom management policies, Communicating expectations to students, and Planning more thoughtful and frequent checks for understanding during
instruction.

For assessment uses, common recommendations from LLSs included:

Providing more self-assessment opportunities for students, Using DIBELS Next benchmark and progress monitoring data to group
students and differentiate instruction, Developing better questioning techniques for quick checks for
understanding, Giving constructive feedback to students, and Incorporating goal-setting into student conferences.

These findings provide valuable insight into the strategies LLSs plan to focus on in their work with teachers. GOSA expects the suggested strategies reported in the Teacher Progress Monitoring Forms to evolve over time.

14

2015-2016 RMP Mid-Year Evaluation Report

MID-YEAR SATISFACTION SURVEY

GOSA plans to administer a comprehensive pre/post retrospective survey at the end of the school year to all teachers, administrators, and district personnel to collect data on the RMP's perceived impact and stakeholder satisfaction. However, to inform ongoing programmatic improvements, GOSA also administered a brief mid-year satisfaction survey to administrators and district personnel to receive feedback on whether stakeholders' needs are currently being met by the LLSs and Program Managers. The survey was sent via email to all administrators and district personnel participating in the RMP. Respondents were given two weeks in November to complete the survey online and had the option to respond anonymously.

The survey was sent to 80 participating school administrators and district personnel. GOSA received 51 responses, which is a response rate of 64%. The survey only consisted of three questions, as listed below. Respondents used a four-point Likert scale to determine their agreement with three given statements.14

Table 5: Mid-Year Satisfaction Survey Results

Survey Item

Percent who Agree or
Strongly Agree

1. I feel well supported by the Language and Literacy

Specialist.

80%

2. I have received timely and professional

communication from the Reading Mentors Program

Program Manager (Jamie Ray or Kimberly Turner).

66%

3. My participation in the Reading Mentors Program

has been valuable to me, my school, and/or my

district.

73%

Table 5 shows the percent of respondents who agreed or strongly agreed with each statement in the survey. In general, survey respondents agreed with the survey statements. Eighty percent of respondents feel they are well supported by LLSs, which is close to meeting the RMP's goal for 90% of participants to feel well supported by LLSs after the first year. 73% of respondents feel that participation in the RMP has been valuable. However, given the 64% response rate and the fact that 37% of submitted responses were anonymous, it is difficult to infer that the survey results are representative of all RMP administrators and district staff participants. Nevertheless, the survey results reveal general satisfaction with the RMP halfway through the school year.

14 The response options were Strongly Disagree (1), Disagree (2), Agree (3), Strongly Agree (4).
15

2015-2016 RMP Mid-Year Evaluation Report

The statement regarding timely and professional communication from the Program Managers had the lowest percentage of respondents who agreed or strongly agreed (66%). Program Managers may want to prioritize communication in the future, but GOSA also recognizes that Program Managers interact more with district personnel than school administrators, so school administrators may not be as familiar with the Program Managers as the LLSs. The RMP staff may want to consider developing stronger ties between Program Managers and school administrators if this is a priority for the program.

The Mid-Year Satisfaction Survey also included an optional section for respondents to provide any additional comments on the RMP. 41% of respondents provided additional comments

"The support from this program cannot be put into words. It's simply

voluntarily. The majority of the comments were awesome."

positive. Many respondents expressed extreme

gratitude for the support they have received from their LLS and enthusiasm for

being in the RMP. LLSs were frequently described as "an asset" or "excellent

resource" for the school.

"I am new to the Reading [Mentors]

Program but it is the best thing that However, some respondents also

has happened to my school."

expressed some initial concerns with the RMP. A couple of

respondents felt that the LLSs are great assets to the school, but wish that LLSs

could spend more time with their schools. As mentioned earlier in the report, most

LLSs have to divide their work among three to five schools. A number of

participants expressed a desire for their LLS to be in their school every day rather

than only once or twice a week. Other respondents reported feeling overwhelmed

by all of the RMP's requirements, such as the DIBELS Next progress monitoring

expectations. There are concerns that participating in the RMP is simply adding to

already demanding district requirements. The RMP staff may want to reexamine

timelines and expectations so that they better accommodate schools' needs, or

facilitate more discussions with program participants to set mutual expectations

and goals that satisfy the missions of both schools and the RMP.

GOSA will share the Mid-Year Satisfaction Survey results with the RMP staff so they can make any necessary programmatic changes to address the reported concerns. GOSA will also include the same survey items in the end-of-year survey to determine any changes in the attitudes of program participants over time.

Student Outcomes

GOSA will use two academic indicators to analyze the impact of the RMP on student achievement. All participating schools are using DIBELS Next as the universal assessment to drive literacy instruction. DIBELS Next measures

16

2015-2016 RMP Mid-Year Evaluation Report
phonemic awareness, phonics, accurate and fluent reading of connected text, reading comprehension, and vocabulary and language skills.15 DIBELS Next provides universal screening benchmarks and progress monitoring resources and allows teachers to differentiate literacy instruction through its performance tiers. As such, GOSA can use DIBELS Next benchmark scores at the beginning, middle, and end of the year to analyze student achievement for the entire RMP. Though all students have been benchmarked for the beginning-of-year period, GOSA has not yet received the benchmark scores for students from all schools so this report does not include any baseline data.
GOSA will also use the state's College and Career Ready Performance Index (CCRPI) Third Grade Lexile indicator score to analyze school performance while participating in the RMP. Since the CCRPI Third Grade Lexile is derived from Georgia Milestones scores, GOSA expects a lag in when these data will be available to analyze and will report on the CCRPI Third Grade Lexile indicator when available.
Conclusion
The 2015-2016 RMP Mid-Year Evaluation Report provides a status update of the RMP's activities as of the middle of the school year. As a mid-year update, many of the evaluation measures GOSA plans to analyze do not have any available data, including student outcome data. Thus, this report presents preliminary findings for two of the three evaluation focus areas: program implementation and stakeholder impact and satisfaction.
Program Implementation Using data collected from the LLS Weekly Logs, LLSs spend the majority of their time on supplemental LLS operations, which include professional learning sessions, gathering resources, and other administrative work. LLSs also spend a significant amount of time providing one-on-one coaching to teachers. The primary content focus of LLSs work so far has been research-based instructional strategies, assessment strategies, and data review. LLSs have spent limited time providing classroom management support.
Stakeholder Impact and Satisfaction According to data collected from the School Literacy Needs Assessment, schools are performing just below the Operational level in the identified building blocks for an effective literacy instructional environment. The preliminary findings indicate that LLSs may need to provide more support to schools in gathering community support for literacy instruction and producing more effective professional learning programs for teachers on teaching literacy across all
15 For more information on DIBELS Next, please visit the Dynamic Measurement Group's website here: https://dibels.org/dibelsnext.html.
17

2015-2016 RMP Mid-Year Evaluation Report
disciplines. Data collected from the Teacher Progress Monitoring Forms reveal that the majority of teachers are at the Needs Improvement level for instructional strategies and assessment uses. LLSs need to provide extensive support for teachers on how to implement research-based literacy instructional practices and use assessments like DIBELS Next to drive instruction and student achievement. Finally, the Mid-Year Satisfaction Survey results demonstrate that program participants in general feel well supported by LLSs, receive timely and professional communication from Program Managers, and feel participation in the RMP has been valuable. The RMP staff may want to address participant concerns about LLSs not spending enough time at each school and overwhelming program expectations, such as frequent progress monitoring. Overall, the preliminary findings establish strong baseline data for future analysis of the RMP's impact. GOSA will produce annual mid-year and end-of-year reports to monitor any growth and inform program developments.
18

2015-2016 RMP Mid-Year Evaluation Report

Appendix A: List of Participating Schools and Districts in the RMP

District Atlanta Public Schools Atlanta Public Schools Atlanta Public Schools Atlanta Public Schools Atlanta Public Schools Atlanta Public Schools Barrow County Barrow County Barrow County Barrow County Barrow County Bibb County Bulloch County Bulloch County Bulloch County Calhoun County Chattahoochee County Clay County Cobb County Dekalb County Dekalb County Dooly County Effingham County Effingham County Fulton County Fulton County Fulton County Fulton County Fulton County Fulton County Fulton County Fulton County Fulton County Ivy Preparatory Kirkwood Campus (Dekalb County)

School Bolton Academy Fain Elementary FL Stanton Elementary Gideons Elementary Usher-Collier Elementary Woodson Primary School Auburn Elementary Bramlett Elementary County Line Elementary Kennedy Elementary Statham Elementary Hartley Elementary Sallie Zetterower Elementary Mill Creek Elementary Stilson Elementary Calhoun County Elementary Chattahoochee County Education Center Clay County Elementary Powder Springs Elementary Brockett Elementary Oak Grove Elementary Dooly County Elementary Marlow Elementary South Effingham Elementary Asa Hilliard Elementary Bethune Elementary College Park Elementary Feldwood Elementary Hamilton E. Holmes Elementary Heritage Elementary Nolan Elementary Parklane Elementary Seaborn Lee Elementary
Ivy Preparatory Academy at Kirkwood for Girls

19

2015-2016 RMP Mid-Year Evaluation Report

District Ivy Preparatory Kirkwood Campus (Dekalb County) Macon County Meriwether County Meriwether County Meriwether County Murray County
State Commission Charter School (serving Baker, Calhoun, Clay, Early, and Randolph counties)
Pelham City Richmond County Richmond County Richmond County Richmond County Richmond County Richmond County Richmond County Richmond County Richmond County Richmond County Richmond County Richmond County Savannah-Chatham County Savannah-Chatham County Savannah-Chatham County Vidalia City Vidalia City Wayne County

School Ivy Preparatory Young Men's Leadership Academy Macon County Elementary George E. Washington Elementary Mountain View Elementary Unity Elementary Spring Place Elementary
Pataula Charter Academy
Pelham Elementary Barton Chapel Elementary Bayvale Elementary Copeland Elementary Craig Houghton Elementary Diamond Lakes Elementary Glenn Hills Elementary Hains Elementary Jenkins-White Elementary Charter School Lamar-Milledge Elementary Meadowbrook Elementary Wheeless Road Elementary Wilkinson Gardens Elementary Haven Elementary Hodge Elementary Spencer Elementary J. D. Dickerson Primary Sally Dailey Meadows Elementary Martha Rawls Smith Elementary

20

2015-2016 RMP Mid-Year Evaluation Report

Appendix B: Demographic Profiles for All RMP Participating Schools

Demographic Profiles for All RMP Participating Schools

School

American Indian Asian Black Hispanic Pacific Islander Multi-Racial

State of Georgia

0.2

3.8 36.9

16.1

0.1

3.8

RMP Total

0.1

1.7 62.6

9.3

0.1

2.8

Asa Hilliard Elementary School

0.0

0.8 87.4

11.1

0.0

0.6

Auburn Elementary School

0.2

9.0 5.1

16.2

0.0

8.3

Barton Chapel Elementary School

0.0

0.0 86.9

3.7

0.0

2.8

Bayvale Elementary School

0.3

0.3 62.2

25.6

0.3

1.9

Bethune Elementary School

0.2

0.7 94.6

4.1

0.0

0.2

Bolton Academy

0.8

1.4 31.1

40.3

0.0

2.8

Bramlett Elementary School

0.2

8.0 6.7

13.8

0.2

4.0

Brockett Elementary School

0.3

20.5 29.0

30.7

0.0

4.0

Calhoun County Elementary School

0.0

0.0 95.4

3.1

0.0

0.0

Chattahoochee County Education Center

0.6

0.0 34.5

3.4

0.6

4.0

Clay County Elementary

1.7

0.0 95.0

0.0

0.0

0.0

College Park Elementary

0.0

0.9 88.9

9.0

0.0

1.1

Copeland Elementary School

0.0

1.5 83.4

4.0

2.5

3.4

County Line Elementary School

0.2

4.7 10.8

21.4

0.2

6.8

Craig-Houghton Elementary School

0.0

0.0 97.9

1.1

0.0

0.4

Diamond Lakes Elementary School

0.6

0.3 80.6

4.7

0.3

4.7

Dooly County Elementary School

0.0

0.2 66.1

21.1

0.0

2.4

Dorothy Hains Elementary School

0.0

0.6 78.6

1.7

0.0

1.4

Values highlighted in yellow represent school percentages that are greater than the state percentage for that racial/ethnic category.

White 39.1 23.4 0.0 61.1 6.7 9.4 0.2 23.5 67.1 15.5 1.5 56.9 3.3 0.2 5.2 55.9 0.7 8.8 10.1 17.7

21

2015-2016 RMP Mid-Year Evaluation Report

Demographic Profiles for All RMP Participating Schools

School

American Indian Asian Black Hispanic Pacific Islander

State of Georgia

0.2

3.8 36.9

16.1

0.1

RMP Total F. L. Stanton Elementary School Fain Elementary School Feldwood Elementary School

0.1

1.7 62.6

9.3

0.1

0.0

0.0 99.4

0.0

0.0

0.0

0.0 91.0

8.7

0.3

0.2

0.0 94.5

3.4

0.0

George E. Washington Elementary School

0.0

0.0 70.4

3.1

0.0

Gideons Elementary School Glenn Hills Elementary School Hamilton E. Holmes Elementary

0.0

0.4 99.2

0.4

0.0

0.0

0.0 88.9

4.8

0.8

0.0

0.4 82.5

13.5

0.0

Hartley Elementary School

0.0

0.7 97.9

0.0

0.0

Haven Elementary School Heritage Elementary School Hodge Elementary School

0.4

0.0 95.2

0.0

0.0

0.2

0.4 95.3

2.7

0.0

0.0

0.0 96.7

0.3

0.0

Multi-Racial 3.8 2.8 0.6 0.0 0.9 5.8 0.0 2.0 1.4 1.4 2.2 0.9 2.3

White 39.1 23.4 0.0 0.0 0.9 20.8 0.0 3.6 2.2 0.0 2.2 0.5 0.7

Ivy Preparatory Academy at Kirkwood for Girls

0.0

0.6 98.9

0.6

0.0

0.0

0.0

Ivy Preparatory Young Men's Leadership Academy, Inc.

0.0

0.0 97.9

1.4

0.0

0.7

0.0

J. D. Dickerson Primary School

0.0

0.0 55.4

9.1

0.0

3.9

31.6

Jenkins-White Elementary Charter School

0.0

0.6 95.3

1.6

0.0

2.2

0.3

Kennedy Elementary School

0.0

3.4 16.1

21.7

0.0

7.4

51.5

Lamar-Milledge Elementary School

0.0

0.4 88.1

0.4

0.0

1.9

9.3

Values highlighted in yellow represent school percentages that are greater than the state percentage for that racial/ethnic category.

22

2015-2016 RMP Mid-Year Evaluation Report

Demographic Profiles for All RMP Participating Schools

School

American Indian Asian Black Hispanic Pacific Islander

State of Georgia

0.2

3.8 36.9

16.1

0.1

RMP Total Lee Elementary School Macon County Elementary School Marlow Elementary School

0.1

1.7 62.6

9.3

0.1

0.3

0.0 94.6

3.4

0.0

0.2

1.1 81.1

8.8

0.0

0.0

0.6 5.0

5.8

0.0

Martha Rawls Smith Elementary School

0.0

1.7 34.1

10.1

0.0

Meadowbrook Elementary School Mill Creek Elementary School Mountain View Elementary School

0.0

0.3 92.1

2.1

0.0

0.2

2.2 57.8

4.5

0.2

0.0

0.6 48.5

2.3

0.2

Nolan Elementary School

0.0

0.0 97.6

1.9

0.0

Oak Grove Elementary School Parklane Elementary School

0.3

13.0 6.7

7.0

0.3

0.0

0.6 62.8

32.8

0.0

Multi-Racial 3.8 2.8 1.0 1.8 4.3 7.0 2.6 4.5 6.1 0.2 7.0 1.2

White 39.1 23.4 0.7 7.0 84.2 47.0 2.9 30.5 42.2 0.2 65.8 2.6

Pataula Charter Academy

0.0

0.6 17.3

5.8

0.0

1.7

74.6

Pelham Elementary School

0.0

0.2 55.1

7.9

0.0

3.5

33.3

Powder Springs Elementary School

0.6

1.7 65.5

16.0

0.0

3.7

12.5

Sallie Zetterower Elementary School

0.0

2.9 47.3

6.4

0.0

3.1

40.2

Sally Dailey Meadows Elementary School

0.3

1.0 52.6

6.4

0.0

2.8

36.9

South Effingham Elementary School

0.0

1.9 10.2

8.7

0.0

3.6

75.6

Spencer Elementary School

0.0

0.0 95.2

0.4

0.0

3.7

0.7

Spring Place Elementary School

0.0

0.0 0.8

39.6

0.0

0.5

59.1

Statham Elementary School

0.2

4.7 14.1

15.8

0.0

3.4

61.9

Values highlighted in yellow represent school percentages that are greater than the state percentage for that racial/ethnic category.

23

2015-2016 RMP Mid-Year Evaluation Report

Demographic Profiles for All RMP Participating Schools

School

American Indian Asian Black Hispanic Pacific Islander Multi-Racial

State of Georgia

0.2

3.8 36.9

16.1

0.1

3.8

RMP Total Stilson Elementary School Unity Elementary School Usher-Collier Elementary School

0.1

1.7 62.6

9.3

0.1

2.8

0.0

0.5 6.8

1.9

0.0

3.9

0.0

0.0 43.2

5.9

0.0

5.0

0.0

0.8 96.8

2.0

0.0

0.4

Wheeless Road Elementary School

0.6

0.0 88.9

2.2

0.0

2.8

Wilkinson Gardens Elementary School Woodson Primary School

0.0

0.0 93.1

3.1

0.0

0.9

0.0

0.0 94.9

3.9

0.0

0.9

Values highlighted in yellow represent school percentages that are greater than the state percentage for that racial/ethnic category.

White 39.1 23.4 86.9 45.9 0.0 5.5 2.8 0.3

24

2015-2016 RMP Mid-Year Evaluation Report
Appendix C: LLS Activities by Category
One-on-One Coaching: CE=Classroom Environment LLS is helping a teacher plan for a positive learning environment, focusing specifically on the room setup (Is it conducive to learning?) and atmosphere (Is it a positive learning environment where students feel safe?) CL=Collaborative Lesson - LLS is working with a teacher to plan a lesson to meet student needs. The LLS may even be in the classroom when the lesson is taught. CO=Coaching Observation - LLS goes into the classroom (or observes remotely) to observe instruction provided as a follow up to a coaching conference or feedback. Or, LLS is visiting the classroom to gather information for future conversations with the teacher. ED=Educational Discussion - LLS is having an educational discussion with administrators or literacy coaches to make plans for the school as a whole. GR=Gathering Resources - LLS is working to gather resources for teachers to use in their classrooms. Many times schools have resources they do not even know exist hidden in an old classroom or storage closet. M=Modeling - LLS is modeling a lesson or portion of a lesson for a teacher in his/her classroom. PO=Post-Conference - LLS is conducting a conference with a teacher following an observation. PR=Pre-Conference - LLS is having a conference with a teacher before observing in his/her classroom.
Data Analysis & Assessment Support: ADC=Administrative Data Conference - LLS is sharing data with the school administrator. DC=Data Conference with Teacher or Grade Level - LLS is analyzing data with at grade level or an individual teacher. AC=Assessment Check - LLS is making sure assessments are being given with fidelity using the Assessment Accuracy Checklist in the Data Resources Notebook.
LLS Professional Learning: T=Training - LLS is participating in a training, not conducting training. SG=Study Group - LLS is participating in a study group for her professional knowledge.
25

2015-2016 RMP Mid-Year Evaluation Report
Supplemental LLS Operations: PL=Professional Learning - LLS is conducting professional learning for teachers/administrators. PLP=Professional Learning Preparation - LLS is preparing to present professional learning for teachers/administrators. GR=Gather Resources - LLS is gathering resources to present during professional learning. CP=Coaching Partner Work - LLS is working with assigned partner from their team. FO=Field Office - LLS is working on travel, time sheets, and other administrative tasks. TS=Tech Support - LLS is working on tech support for a school, teacher, or themselves.
School Closures/Personal Leave: SC=School Closed PC=Partial School Closing AL=Annual Leave SL=Sick Leave ESL=Education Support Leave
LLS Committee Work: LLS is working (independently or collaboratively) on tasks assigned by their Program Manager to distribute to the entire team. Additional Work: LLS is completing other administrative tasks for GOSA.
26

2015-2016 RMP Mid-Year Evaluation Report

Appendix D: School Literacy Needs Assessment Survey Items

For the following survey, administrators and LLSs were asked to rate their school (grades K-3) using a four-point scale where 1=Not Addressed, 2=Emergent, 3=Operational, and 4=Fully Operational. The definitions of each rating for each indicator is listed below.

Building Block 1. Engaged Leadership

A. Administrator demonstrates commitment to learn about and support evidence-based literacy instruction in his/her school.

Fully Operational

Operational

Emergent

Not Addressed

Administrator seeks out and participates in
professional learning in literacy with his/her faculty.

Administrator researches and secures professional learning in
literacy for his/her faculty, but does not
participate in it.

Administrator researches professional learning in
literacy.

Administrator has not yet demonstrated a
commitment to learning about literacy
instruction.

B. A school literacy leadership team organized by the administrator is active.

Fully Operational

Operational

Emergent

A literacy leadership team, led by the
administrator, meets regularly and provides substantive direction for
the school and community.

A school literacy leadership team has been formed, meets regularly, but has not yet begun effecting change in the course of literacy instruction.

A school literacy leadership team is
envisioned and stakeholders have been
identified.

Not Addressed
No action has yet been taken in the formation of a literacy leadership
team.

C. The effective use of time and personnel is leveraged through scheduling and collaborative planning (K-3).

Fully Operational

Operational

Emergent

Not Addressed

Daily schedules include a 90-120- minute literacy
block, a set time for intervention, instruction in disciplinary literacy in
content areas, and collaborative planning.

Daily schedules include a 90-120minute literacy block but do not include scheduled time for all of the following:
intervention, disciplinary literacy in
content areas, and collaborative planning.

Daily schedules include 90-120 minute literacy block but do not include scheduled time for any of the following: intervention, disciplinary literacy in
content areas, and collaborative planning.

Daily schedules are elastic and do not specify a 90-120 minute literacy block. Teachers may or may not devote that much time to literacy.

Fully Operational

Operational

Emergent

Not Addressed

27

2015-2016 RMP Mid-Year Evaluation Report

Building Block 1. Engaged Leadership

D. A school culture exists in which teachers across the content areas accept responsibility for literacy instruction as articulated in the Common core Georgia Performance Standards (CCGPS).

Fully Operational

Operational

Emergent

Not Addressed

Faculty and staff know and consistently use effective instructional
practices for disciplinary literacy across the content areas.

Faculty and staff have received professional learning in disciplinary
literacy across the content areas, but implementation is not
consistent.

Faculty and staff have received professional learning in disciplinary literacy in some content
areas but not all.

Professional learning in disciplinary literacy
has not formally begun.

E. Literacy instruction is optimized in all content areas.

Fully Operational

Operational

Emergent

Not Addressed

Content area teachers consistently incorporate the teaching of academic vocabulary; narrative,
informational, and argumentative writing; and the use of disciplinespecific text structures.

Content area teachers consistently
incorporate the teaching of two but not all of the following: 1. academic vocabulary;
2. narrative, informational, and argumentative writing; and 3. the use of discipline-specific text
structures

Content area teachers consistently incorporate the
teaching of one of the following: academic vocabulary; narrative, informational, and argumentative writing; and the use of disciplinespecific text structures.

Content area teachers do not consistently incorporate the teaching of the following: academic
vocabulary; narrative, informational, and
argumentative writing; and the use of
discipline-specific text structures.

F. The community at large supports schools and teachers in the development of students who are collegeand-career-ready as articulated in the Common core Georgia Performance Standards (CCGPS).

Fully Operational

Operational

Emergent

Not Addressed

A community literacy council participates in achieving literacy goals through support of and/or participation in a network of learning supports (e.g., tutoring, mentoring,
afterschool programming).

A community literacy council participates in
developing literacy goals, but a system of learning supports has
not yet developed.

A community literacy council is being planned. Stakeholders have been identified and meetings are
being planned.

A community literacy council has not yet begun to take shape.

28

2015-2016 RMP Mid-Year Evaluation Report

Building Block 2. Continuity of Instruction
A. Active collaborative school teams ensure a consistent literacy focus across the curriculum (See Engaged Leadership, 1. D, E).

Fully Operational

Operational

Emergent

Not Addressed

Cross-disciplinary teams meet regularly to
examine student work and to collaborate on the achievement of literacy
goals shared by all teachers.

Cross disciplinary teams have allocated
various aspects of literacy instruction across all content areas.

Cross-disciplinary teams meet regularly to examine
student work, but all teachers have not fully assumed responsibility for achieving literacy goals.

Cross disciplinary teams are not currently
meeting.

B. Teachers provide literacy instruction across the curriculum (See Engaged Leadership, 1. D, E).

Fully Operational

Operational

Emergent

Not Addressed

Literacy instruction, supported by a systematic,
comprehensive core language arts core program, occurs in all
content areas.

Literacy instruction, supported by a systematic,
comprehensive core language arts core program, occurs in only one or two content
areas.

Literacy instruction, supported by a systematic,
comprehensive core reading core program occurs only in language
arts classrooms.

Literacy instruction is not guided by a systematic,
comprehensive core program. C. Out-

C. Out-of-school agencies and organizations collaborate to support literacy within the community.

Fully Operational

Operational

Emergent

Not Addressed

A comprehensive system of learning supports within the community complement literacy instruction within the classroom.

A few community organizations provide learning supports to complement literacy instruction within the
classroom.

Out-of-school organizations and agencies
are making plans to develop learning supports
to complement literacy instruction.

As of yet, there is no system of learning supports available in
the community.

29

2015-2016 RMP Mid-Year Evaluation Report

Building Block 3. Ongoing Formative and Summative Assessments

A. An infrastructure for ongoing formative and summative assessments is in place to determine the need for and the intensity of interventions and to evaluate the effectiveness of instruction.

Fully Operational

Operational

Emergent

Not Addressed

Effective screening, progress monitoring and
diagnostic tools have been selected to be used
along with a complementary system of mid-course assessments that are common across
classrooms.

A system of midcourse assessments that
are common across classrooms is in place, but as of yet screening, progress monitoring, and diagnostic tools have not been selected.

Teachers have agreed that a system of common midcourse assessments across classrooms is needed but
those assessments have not been developed or located
yet.

The district is complying with state
requirements for summative
assessments, but formative assessments have not been selected
or developed.

B. A system of ongoing formative and summative assessment (universal screening and progress monitoring) is used to determine the need for and the intensity of interventions and to evaluate the effectiveness of instruction.

Fully Operational

Operational

Emergent

Not Addressed

A full range of formative and summative assessments are
administered regularly and are used to guide
classroom and intervention instruction.

A full range of formative and summative assessments are administered regularly, but review of assessments is not
consistent.

Some formative and summative assessments are
administered.

The district is complying with state
requirements for summative
assessments, but no true formative
assessments have been selected or developed.

C. Problems found in literacy screenings are further analyzed with diagnostic assessment.

Fully Operational

Operational

Emergent

Not Addressed

Problems found in literacy screenings are routinely followed up by diagnostic assessments that are used to guide placement and/or inform
instruction in intervention programs.

Problems found in literacy screenings in
some cases are followed up by diagnostic assessments that are used to guide placement and/or inform instruction in intervention programs.

Problems found in literacy screenings are sometimes followed up by diagnostic assessments, but are rarely used to guide placement and/or to inform instruction in intervention programs.

Problems found in literacy screenings are
not followed by diagnostic assessments.

D. Summative data is used to make programming decisions as well as to monitor individual student progress.

Fully Operational

Operational

Emergent

Not Addressed

30

2015-2016 RMP Mid-Year Evaluation Report

Time is devoted in teacher team meetings to
review and analyze assessment results to
identify needed programmatic and instructional adjustments.

Teacher team meetings to analyze summative assessment results of individual students are
used to make adjustments to instruction, but rarely impact programmatic
decisions.

Teachers meet with administrator to discuss progress of individual students on summative
assessments.

Teachers rarely have time to review
summative data for their former or future
students.

E. A clearly articulated strategy for using data to improve teaching and learning is followed.

Fully Operational

Operational

Emergent

Not Addressed

All appropriate staff members have access to
data and follow the established protocol for
making decisions to identify the instructional
needs of students.

All appropriate staff members follow the established protocol for making decisions when they have access to
necessary data to identify the
instructional needs of students.

Some staff members have access to data and follow the established protocol for
making decisions to identify the instructional
needs of students.

Staff members have difficulty obtaining data necessary for making informed
decisions about instruction.

Building Block 4. Best Practices in Literacy Instruction

A. All students receive direct, explicit instruction in reading (K-3).

Fully Operational

Operational

Emergent

Not Addressed

The faculty is thoroughly trained to use the core program which provides continuity based on a carefully articulated scope and sequence of skills that is integrated into a rich curriculum of literary and informational texts.

The core program provides continuity based on a carefully articulated scope and sequence of skills that is integrated into a rich curriculum of literary and informational texts, but the faculty is not yet fully trained in its use.

A core program is in use, but it does not provide a strong basis for instruction in all aspects of literacy.

A core program is available but is not used to guide sequential skill instruction (or is not available at all).

B. Extended time is provided for literacy instruction.

Fully Operational

Operational

Emergent

Not Addressed

31

2015-2016 RMP Mid-Year Evaluation Report

Building Block 4. Best Practices in Literacy Instruction

Daily schedules include a 90-120- minute literacy block, a set time for intervention, instruction in disciplinary literacy in content areas, and time for collaborative planning.

Daily schedules include a 90-120minute literacy block but does not include scheduled time for all of the following: intervention, disciplinary literacy in content areas, and time for collaborative planning.

Daily schedules include 90-120 minute literacy block but does not include scheduled time for any of the following: intervention, disciplinary literacy in content areas, and time for collaborative planning.

Daily schedules are elastic and do not specify a 90-120 minute literacy block. Teachers may or may not devote that much time to literacy.

C. All students receive effective writing instruction across the curriculum.

Fully Operational

Operational

Emergent

Not Addressed

A coordinated plan has been implemented for writing instruction across all subject areas that includes explicit instruction, guided practice, independent practice.

A coordinated plan has been developed for writing instruction across all subject areas that includes explicit instruction, guided practice, independent practice.

Teachers are beginning to develop a plan for writing instruction across all subject areas.

Writing is only taught by English language arts teachers.

D. Teachers are intentional in efforts to develop and maintain interest and engagement as student progress through school.

Fully Operational

Operational

Emergent

Not Addressed

Teachers regularly implement strategies for
developing and maintaining interest and engagement appropriate
to their grade levels.

Teachers have received professional
development in strategies for developing and maintaining interest and engagement appropriate to their grade levels, but implementation is not consistent.

Professional development is planned and teachers have been encouraged to seek out strategies for developing and maintain interest and engagement appropriate to their grade levels.

Teachers have not yet formally begun learning about strategies for developing and
maintaining interest and engagement in
their students.

32

2015-2016 RMP Mid-Year Evaluation Report

Building Block 5. System of Tiered Intervention (RTI) for All Students

A. Information developed from the school-based data teams is used to inform RTI process.

Fully Operational

Operational

Emergent

Not Addressed

Data from formative assessments is gathered and analyzed regularly to ensure that all students are receiving instruction in appropriate tiers and that instruction in each
tier is effective.

Formative assessments are administered
regularly to students in each tier of instruction.

Intervention is monitored regularly to ensure that it occurs regularly and is implemented with fidelity.

RTI is not currently being formally implemented.

B. Tier I Instruction based upon the CCGPS in grades K-3 is provided to all students in all classrooms.

Fully Operational

Operational

Emergent

Not Addressed

Student and classroom data have been analyzed
to determine the instructional areas and classrooms in greatest
need of support.

Current practice in literacy instruction has been assessed using the
Literacy Instruction Checklist, GA, or its
equivalent.

Student data is examined regularly to determine if fewer than 80% of students are successful in any area.

Student data is examined regularly to determine if fewer than 80% of students are successful in any area.

C. Tier 2 needs-based interventions are provided for targeted students.

Fully Operational

Operational

Emergent

Not Addressed

Interventionists, ELA, and content area teachers
meet regularly for collaboration and planning to ensure that the goals of interventions are being achieved.

Interventionists participate in ongoing professional learning in
program use and in how to diagnose and
correct reading difficulties.

Interventions are provided by competent instructors, in spaces that are adequate, and with sufficient blocks of time in the schedule.

Tier 2 instruction is not provided by
interventionists, but is expected to be done by the classroom teacher
at some time during the day.

D. In Tier 3, Student Support Team (SST) and Data Team monitor progress jointly.

Fully Operational

Operational

Emergent

Not Addressed

Data team/SST team meet regularly to ensure that a student's lack of progress is not due to a preventable cause (e.g., too large a group, lack of regularity or fidelity of
instruction).

Interventions in Tier 3 are provided by a
trained interventionist with fidelity, but are not yet on a 1:1-1:3
basis.

SST team meets to ensure that interventions are at appropriate teacher-student ratio and is delivered with
fidelity.

Requirements for Tier 3 have not yet been implemented.

33

2015-2016 RMP Mid-Year Evaluation Report

E. Tier 4-specially-designed learning is implemented through specialized programs, methodologies, or strategies based upon students' inability to access the CCGPS any other way.

Fully Operational

Operational

Emergent

Not Addressed

To ensure that the most highly qualified teachers
provide Tier 4 instruction, SpEd, ESOL,
and gifted teachers participate in
professional learning communities to maintain
strict alignment with CCGPS.

School schedules are developed to ensure that students receive instruction in the least
restrictive environment.

Building and district administrators are familiar
with funding formulas affecting students in special programming.

Special education functions separately within the school with little communication with regular education or with little input from the administrator.

Building Block 6. Improved Instruction through Professional Learning

A. Preservice education prepares new teachers for all aspects of literacy instruction including disciplinary literacy in the content areas.

Fully Operational

Operational

Emergent

Not Addressed

Representatives from the community and/or
administration have met with representatives from
the Professional Standards Commission
(PSC) to ensure that preservice teachers receive coursework in disciplinary literacy in
the content area.

School administrators have begun to include
questions about whether potential hires
have received coursework in disciplinary literacy in the content area in their preservice training.

Representatives from the community and/or
administration are aware of the need for preservice teachers to receive
coursework in disciplinary literacy in the content area.

Preservice education does not include coursework in all aspects of literacy
instruction including disciplinary literacy in
the content areas.

B. In-service personnel participate in ongoing professional learning in all aspects of literacy instruction including disciplinary literacy in the content areas.

Fully Operational

Operational

Emergent

Not Addressed

All administrative and instructional personnel
participate in professional learning on
all aspects of literacy instruction including disciplinary literacy in
the content areas.

Administrators and ELA instructors (certified and noncertified) participate in
professional learning on all aspects of
literacy instruction including training on
use of the core program.

ELA instructors participate in professional learning on
the use of the core program.

Professional learning in literacy has not begun formally.

34

2015-2016 RMP Mid-Year Evaluation Report

Appendix E: Average Ratings and Percent of Schools at Operational or Fully Operational for All Indicators of School Literacy Needs Assessment

Indicator

Average Rating

Percent of Schools at Operational or Fully Operational

Building Block 1: Engaged Leadership

Administrator demonstrates commitment to learn about and

3.4

90%

support evidence-based literacy instruction in his/her school.

A school literacy leadership team organized by the administrator is active.

2.6

50%

The effective use of time and personnel is leveraged through

3.3

88%

scheduling and collaborative planning (K-3).

A school culture exists in which teachers across the content

areas accept responsibility for literacy instruction as articulated in the Common core Georgia Performance

2.6

51%

Standards (CCGPS).

Literacy instruction is optimized in all content areas.

2.5

44%

The community at large supports schools and teachers in the

development of students who are college-and-career-ready as articulated in the Common core Georgia Performance

2.3

40%

Standards (CCGPS).

Building Block 2: Continuity of Instruction

Active collaborative school teams ensure a consistent literacy

2.5

51%

focus across the curriculum.

Teachers provide literacy instruction across the curriculum.

2.6

49%

Out-of-school agencies and organizations collaborate to support literacy within the community.

2.3

36%

Building Block 3: Ongoing Formative and Summative Assessments

An infrastructure for ongoing formative and summative

assessments is in place to determine the need for and the intensity of interventions and to evaluate the effectiveness of

3.1

80%

instruction.

A system of ongoing formative and summative assessment

(universal screening and progress monitoring) is used to

3.1

85%

determine the need for and the intensity of interventions and

to evaluate the effectiveness of instruction.

Problems found in literacy screenings are further analyzed

2.7

53%

with diagnostic assessment.

35

2015-2016 RMP Mid-Year Evaluation Report

Indicator

Average Rating

Percent of Schools at Operational or Fully Operational

Summative data is used to make programming decisions as

2.9

67%

well as to monitor individual student progress.

A clearly articulated strategy for using data to improve teaching and learning is followed.

2.8

60%

Building Block 4: Best Practices in Literacy Instruction

All students receive direct, explicit instruction in reading (K3).

3.0

70%

All students receive effective writing instruction across the

2.4

37%

curriculum.

Extended time is provided for literacy instruction.

3.0

75%

Teachers are intentional in efforts to develop and maintain

2.7

58%

interest and engagement as students progress through school.

Building Block 5: System of Tiered Intervention (RTI) for All Students

Information developed from the school-based data teams is

3.0

68%

used to inform RTI process.

Tier I Instruction based upon the CCGPS in grades K-3 is provided to all students in all classrooms.

3.2

82%

Tier 2 needs-based interventions are provided for targeted students.

2.9

65%

In Tier 3, Student Support Team (SST) and Data Team monitor progress jointly.

2.8

70%

Tier 4-specially-designed learning is implemented through

specialized programs, methodologies, or strategies based

2.8

65%

upon students' inability to access the CCGPS any other way.

Building Block 6: Improved Instruction through Professional Learning

Preservice education prepares new teachers for all aspects of

literacy instruction including disciplinary literacy in the

2.3

35%

content areas.

In-service personnel participate in ongoing professional learning in all aspects of literacy instruction including disciplinary literacy in the content areas.

2.7

63%

36

2015-2016 RMP Mid-Year Evaluation Report
Appendix F: Teacher Progress Monitoring Form Survey Items
For the following survey, LLSs were asked to rate teachers using a four-point scale where 1=Ineffective, 2=Needs Improvement, 3=Proficient, and 4=Exemplary.
Standard 3: Instructional Strategies
1. Engages students in active learning and maintains interest. 2. Builds upon students' existing knowledge and skills. 3. Reinforces learning goals consistently throughout the lesson. 4. Uses a variety of research-based instructional strategies and resources. 5. Effectively uses appropriate instructional technology to enhance student
learning. 6. Communicates and presents material clearly, and checks for
understanding. 7. Develops higher-order thinking through questioning and problem-solving. 8. Engages students in authentic learning by providing real-life examples and
interdisciplinary connections. 9. Overall: The teacher promotes student learning by using research-based
instructional strategies relevant to the content to engage students in active learning and to facilitate the students' acquisition of key knowledge and skills.
10. Additional notes on Instructional Strategies:
11. Suggested Strategies for Instructional Strategies:
Standard 6: Assessment Uses
1. Uses diagnostic assessment data to develop learning goals for students, to differentiate instruction, and to document learning.
2. Plans a variety of formal and informal assessments aligned with instructional results to measure student mastery of learning objectives.
3. Uses assessment tools for both formative and summative purposes to inform, guide, and adjust instruction.
4. Systematically analyzes and uses data to measure student progress, to design appropriate interventions, and to inform long-term and short-term instructional decisions.
5. Shares accurate results of student progress with students, parents, and key school personnel.
6. Provides constructive and frequent feedback to students on their progress toward their learning goals.
37

2015-2016 RMP Mid-Year Evaluation Report
7. Teachers students how to self-assess and to use metacognitive strategies in support of lifelong learning.
8. Overall: The teacher systematically gathers, analyzes, and uses relevant data to measure student progress, to inform instructional content and delivery methods, and to provide timely and constructive feedback to both students and parents.
9. Additional notes on Assessment Uses: 10. Suggested Strategies for Assessment Uses:
38

2015-2016 RMP Mid-Year Evaluation Report

Appendix G: Percentage of Teachers Proficient or Exemplary for All Indicators and Grade Levels on the Teacher Progress Monitoring Form

Percent of All
Teachers Proficient/ Exemplary

Percent of Kindergarten
Teachers Proficient/ Exemplary

Percent of 1st Grade Teachers Proficient/ Exemplary

Percent of 2nd Grade Teachers Proficient/ Exemplary

Percent of 3rd Grade Teachers Proficient/ Exemplary

Standard 3: Instructional Strategies

Engages students in active learning and maintains

interest.

43%

49%

37%

38%

48%

Builds upon students' existing knowledge and skills.

47%

49%

45%

45%

48%

Reinforces learning goals consistently throughout

the lesson.

38%

41%

29%

45%

35%

Uses a variety of research based instructional strategies and resources.

32%

36%

29%

28%

32%

Effectively uses appropriate instructional technology

to enhance student learning.

31%

22%

39%

31%

30%

Communicates and presents material clearly, and

checks for understanding.

32%

36%

30%

28%

30%

39

2015-2016 RMP Mid-Year Evaluation Report

Percent of All
Teachers Proficient/ Exemplary

Percent of Kindergarten
Teachers Proficient/ Exemplary

Percent of 1st Grade Teachers Proficient/ Exemplary

Percent of 2nd Grade Teachers Proficient/ Exemplary

Percent of 3rd Grade Teachers Proficient/ Exemplary

Develops higher-order thinking through questioning

and problem-solving activities.

19%

24%

19%

11%

17%

Engages students in authentic learning by providing

real-life examples and interdisciplinary connections.

35%

45%

30%

25%

35%

OVERALL: The teacher promotes student learning

by using research-based instructional strategies

relevant to the content to engage students in active

learning and to facilitate the students' acquisition of

key knowledge and skills.

29%

34%

27%

25%

27%

Standard 6: Assessment Uses

Uses diagnostic assessment data to develop learning

goals for students, to differentiate instruction, and to

document learning.

30%

33%

22%

33%

29%

Plans a variety of formal and informal assessments

aligned with instructional results to measure student

mastery of learning objectives.

36%

44%

25%

37%

33%

40

2015-2016 RMP Mid-Year Evaluation Report

Percent of All
Teachers Proficient/ Exemplary

Percent of Kindergarten
Teachers Proficient/ Exemplary

Percent of 1st Grade Teachers Proficient/ Exemplary

Percent of 2nd Grade Teachers Proficient/ Exemplary

Percent of 3rd Grade Teachers Proficient/ Exemplary

Uses assessment tools for both formative and

summative purposes to inform, guide, and adjust

instruction.

31%

36%

22%

37%

25%

Systematically analyzes and uses data to measure

student progress, to design appropriate interventions,

and to inform long-term and short-term instructional

decisions.

27%

28%

21%

33%

21%

Shares accurate results of student progress with

students, parents, and key school personnel.

32%

45%

22%

28%

28%

Provides constructive and frequent feedback to

students on their progress toward their learning

goals.

22%

21%

22%

19%

24%

Teaches students how to self-assess and to use metacognitive strategies in support of lifelong learning.

12%

17%

13%

8%

10%

41

2015-2016 RMP Mid-Year Evaluation Report

Percent of All
Teachers Proficient/ Exemplary

Percent of Kindergarten
Teachers Proficient/ Exemplary

Percent of 1st Grade Teachers Proficient/ Exemplary

Percent of 2nd Grade Teachers Proficient/ Exemplary

Percent of 3rd Grade Teachers Proficient/ Exemplary

OVERALL: The teacher systematically gathers,

analyzes, and uses relevant data to measure student

progress, to inform instructional content and deliver

methods, and to provide timely and constructive

feedback to both students and parents.

26%

29%

22%

23%

25%

Note: The total number of teachers receiving a rating for each indicator varied slightly because some indicators were not observable by the LLS during the time of observation. Thus, the sample size used to calculate the percentages for each indicator by grade level is different for each indicator.

42

2015-2016 RMP Mid-Year Evaluation Report
43