Annual report [Dec. 1, 2002]

December 1, 2002

TO:

Honorable Richard Marable

State Senator

Chair, Education Committee

Honorable Jeannette Jamieson State Representative Chair, Education Committee

FROM: RE:

Office of Education Accountability 2002 Annual Report

The Office of Education Accountability (OEA) is pleased to present its second annual report of activities from December 2001 through November 2002. During OEA's second year of operation, the following major activities were accomplished and/or initiated:

I.
II.
III. IV. V. VI.
VII. VIII.

Student Level Standard Setting - Certification of cut scores for CriterionReferenced Competency Tests in grades 1-8 Development and Approval of K-12 Definitions for Below Grade Level, Mobility, Dropout Indicator Rating Criteria for K-12 Schools Development of Placement and Exit Criteria for the Early Intervention Program Production of a 2001-2002 Comprehensive State Education Report (P-16) Identification of Indicator Performance elements for: University System of Georgia, Department of Technical and Adult Education, the Office of School Readiness and the Georgia Professional Standards Commission Development of a single federal/state education accountability system Initial Development of an Audit Manual and Initiation of Audit Responsibilities

K-12 Timeline OEA continues to pursue a careful, deliberate process to develop report cards, benchmarks, and standards. As the Georgia Department of Education (GDOE) implements new tests, data will be captured and included on report cards - for informational purposes only. It is anticipated OEA will not issue its first "accountable" report card that will carry awards/interventions for grades K-8 until the 2003-04 school year and for grades 9-12 during the 2004-05 school year. OEA published a 5-year timeline for local school systems outlining the phase-in of baseline years, standards setting years, hold harmless "pilot" report card

years, and "accountable" report card years. The timeframe is posted on OEA's website [www.ga-oea.org] so everyone will have a clear understanding of the report card and development process. The three remaining years in the development timeline are as follows:
K-12 Report Card Development Timeline
2002-2003 School Year
Standards Set for K-8 Schools All High School End-of-Course Tests first administered Baseline Year for 9-12 Standard Setting for Schools Report Cards Issued in December 2003
o K-8 Hold Harmless "Pilot" Report Card - No awards o 912 Report Card (No Grades)
2003-2004 School Year
Standards Set for 9-12 Schools Report Cards Issued in December 2004
o K-8 - First "Accountable" Report Cards issued with awards and interventions
o 9-12 Hold Harmless "Pilot" Report Card No awards
2004-2005 School Year
K-8 - Report Cards issued in December with awards and interventions 9-12 - First "Accountable" Report Cards issued December 2005 with awards
and interventions
To construct a fair and reasonable accountability system, many steps must be taken. The remainder of this report outlines each of the major accomplishments during this past year.
I. Student Level Standard Setting O.C.G.A. 20-14-31 requires OEA to establish the level of performance considered to be satisfactory on each assessment instrument administered under

O.C.G.A. 20-2-281 by establishing the standard that should be achieved by students in each subject area at each grade level. Working in conjunction with GDOE, OEA approved student performance standards for grades 4, 6, and 8 in English/Language Arts, Mathematics, and Reading after the first Criterion-Referenced Competency Tests (CRCT) operational administrations in spring 2000. OEA also approved the equating process for CRCT for grades 4, 6, and 8 as administered Spring 2001. This process allowed standards to remain consistent from year to year. In the spring of 2002, additional CRCT were operationally administered in grades 1, 2, 3, 5, and 7 in English/Language Arts, Reading, Mathematics, and in grades 3-8 in Science and Social Studies. Once again using the same methodology in the standard setting process, OEA approved student performance standards for CRCT in Grades 1, 2, 3, 5, and 7 in English/Language Arts, Mathematics and Reading. Student performance standards were also approved for Grades 3-8 in Science and Social Studies. Student level cut scores were established for Levels I (Did not meet), II (Meets), and III (Exceeds). II. Development and Approval of K-12 Definitions for Below Grade Level, Mobility, and Dropout Part of the process of developing an accountability system is to operationalize key elements. OEA working with various committees defined the following key elements. These definitions were approved by the Education Coordinating Council and the State Board of Education. Below Grade Level Student performance will be considered "below grade level" if a student's performance level is that as described by one or more of the following:
1. Students entering kindergarten who score at the "needs extra instructional assistance" level on the Fall Georgia Kindergarten Assessment Program-Revised (GKAP-R) or whose documented performance is that as described in the Performance Descriptions at the Beginning of Kindergarten as "needs extra instructional assistance";
2. Students entering first-grade who score at the "not ready for first grade" or "ready with extra instructional assistance" level on the Spring GKAP-R or whose documented performance is that as described in the Performance Descriptors at

the end of Kindergarten as "not ready for first-grade" or as "needs extra instructional assistance"; 3. Students who score at Performance Level 1 (Does Not Meet) on the CRCT in reading or mathematics; 4. Students who do not have GKAP-R or CRCT scores but who score below the 35th percentile on a standardized norm-referenced test; 5. Students who do not have GKAP-R or CRCT scores but whose documented performance indicates "below grade level" performance on local assessments, portfolios, other checklists, or student work as described by GKAP-R or CRCT Performance Descriptors for the Beginning of Kindergarten or for Performance Level 1 of the CRCT; 6. Students who meet standards on the GKAP-R or CRCT but whose documented performance indicates "below grade level" performance on local assessments, portfolios, other checklists, or student work as described by GKAP-R or CRCT Performance Descriptors for the Beginning of Kindergarten or for Performance Level 1 of the CRCT.
Mobility
Mobile students will be included for grading purposes using the following:
For Grades K-8--Absolute Measures:
Count students who were enrolled in the same school system in the October FTE
Count and who take the Spring GKAP-R or CRCT.
For Grades K-8--Progress/Gain Measures:
Count students who were enrolled in the same school system in the October FTE
count and who take the Spring GKAP-R or CRCT, and who also have a GKAP-R
or CRCT score from anywhere in the state.
For Grades 9-12--Absolute and Progress/Gain Measures:
Count students who have been enrolled in a course within the same school
system for at least one day more than half the days in the course prior to the
End-of-Course Test (EOCT) date.
Dropouts for Grades 7-12
Students are reported as dropouts if they leave school for one of the following
reasons: Marriage, Expelled, Financial Hardship/Job, Incarcerated/Under
Jurisdiction of Juvenile or Criminal Justice Authority, Low Grades/School Failure,
Military, Adult Education/Postsecondary, Pregnant/Parent, Removed for Lack of
Attendance, Serious Illness/Accident, and Unknown.

III. Indicator Rating Criteria for K-12 Schools Ten indicators for measuring K-12 performance are listed in H.B. 1187. The bill requires that each indicator be rated as exemplary, acceptable, or unacceptable. Working in conjunction with the Standards/Grading Committee, the Implementation Task Force, the Education Coordinating Council, and the State Board of Education, criteria for the following legislatively mandated indicators were adopted:

1. Dropout rates for each school

The criteria for rating schools as exemplary, acceptable, or unacceptable based

on dropout rates is as follows:

Exemplary

School dropout rate - 2% or more below the state average and no increase over the previous year

Acceptable

School dropout rate - 1% or more below the state average

Unacceptable

School dropout rate - less than 1% below the state average or greater than the state average

2. Attendance rates for each school

The criteria for rating schools as exemplary, acceptable, or unacceptable based

on student attendance is as follows:

Exemplary

Students absent more than 15 days - 5% or less

Acceptable

Students absent more than 15 days - Greater than 5% but less than 15%

Unacceptable Students absent more than 15 days - 15% or more

3. Percent of students taking GAA - (Report only)
Report, not rate, the number and percent of students participating in the GAA, including the number and percent of students participating by each category of exceptionality.
4. Completion rates for each school - in progress 5. Percent of graduating students passing the GHSGT - in progress 6. Percent of graduating students meeting course requirements - in
progress 7. Percent of students taking EOCT - in progress

8. Percent of high school students passing EOCT in core subjects - in progress
9. Results of the SAT and ACT - in progress 10. Average time that a student remains in EIP before attaining grade
level status - in progress
IV. Placement and Exit Criteria for Early Intervention Program The development of Placement and Exit Criteria for the Early Intervention Program (EIP) was the result of collaboration between OEA staff, GDOE staff, and various committees and panels of educators. The placement and exit procedures use multiple indices of student performance for each grade level. While results from State curriculum-based tests are the preferred method for identifying students for EIP, teachers are allowed flexibility and may use such documented evidence as local assessments, checklists, portfolios, etc. More specific information can be found on the OEA website.
V. 2001-2002 Comprehensive State Education Report OEA releases its first comprehensive education report on K-12 schools, the University System of Georgia (USG), the Department of Technical and Adult Education (DTAE), the Georgia Professional Standards Commission (GAPSC), and the Office of School Readiness (OSR) in December. Previously only a K-12 Report Card had been issued.
The 2001-2002 Report Card contains test results as well as other information relevant to schools and their performance toward the goals of student achievement and school completion. The report has four major sections:
1. Student performance results from Georgia tests 2. School performance indicators 3. School demographic information 4. National tests results
In addition to information contained in OEA's 2000-2001 K-12 Report Card,

assessment results from Criterion-Referenced Competency Tests (CRCT) for grades 1, 2, 3, 5, and 7 in Reading, English/Language Arts, and Mathematics. CRCT results for grades 3-8 in Science and Social Studies have been added to the report. No grades or ratings are included in the 2001-02 Report Card.
Multi-year comparisons at the school, system, and state levels are now available. In addition, while viewing a school the user can opt to compare the school data to system and state data.
The report card includes assessment and other data disaggregated by different student groupings. These groupings are based on race/ethnicity, gender, disability, and English proficiency status as required by the A Plus Education Reform Act of 2000 and the Federal No Child Left Behind Act of 2001.
Part of the uniqueness of Georgia's accountability system is that it encompasses additional elements beyond the K-12 public school sector to provide a seamless, unified educational system. For the first time, indicators and data elements are being reported for OSR, DTAE, USG, and GAPSC. The following section describes the data elements for each of these education entities: VI. Identification of Indicator Performance Elements
Office of School Readiness (OSR) Indicators
Accessibility Indicator - The first indicator on accessibility of prekindergarten services to four-year-olds focuses on Pre-K enrollment and the proportion of economically disadvantaged or at-risk students served during the 2001-2002

academic year. Information is available for each of the 1,683 OSR directed Pre-K programs, by each of the 159 counties, and at the state level. For each program, OEA reports:
1. Percent of four-year-old children participating in Pre-K 2. Number of four-year-old children identified as at-risk participating in Pre-K 3. Percent of Pre-K enrollment comprised of at-risk students
For each county, OEA Reports:
1. Number of four-year-olds enrolled in a Pre-K program 2. Percent of county's estimated four-year-old population served by lottery-funded
Pre-K programs in the county 3. Number of four-year-olds identified as at-risk enrolled in Pre-K programs within
the county 4. Percent of total Pre-K enrollment comprised of at-risk children
At the state level, OEA reports:
1. Number of four-year-old children enrolled in a Pre-K program in the State 2. Percent of state's estimated four-year-old population served by OSR directed Pre-
K programs 3. Percent of total Pre-K enrollment comprised of at-risk children
Quality of OSR Directed Pre-K Services - The second indicator focuses on the quality of services provided by Georgia's Pre-K programs. Beginning with the 2001-2002 fiscal year, OSR implemented a formal data collection process using its newly developed Program Quality Assessment (PQA). Annually, a representative from OSR visits each Pre-K facility and verifies the certifications of instructors as well as evaluates staffing levels, the physical facilities of the location, and instruction and curriculum. OSR provided the PQA database to OEA. OEA's report on OSR includes the ratings for each program and at the state level, the percent of programs receiving each rating. Some of the key elements defining a quality Pre-K program reported in the 2001-2002 Report Card include:
Lead teacher certification Approved curricula

Clearly defined learning areas enhancing children's growth and development Facility meets child/staff ratios Facility has an appropriate daily routine Facility provides an environment and instruction that promotes language
development
(DTAE) Indicators
Three indicators are reported for DTAE. Information is presented by institution and is also disaggregated by race/ethnicity, socioeconomic status, and gender. DTAE provided OEA with data from the 2000-2001 academic year.
1. Retention rates of first-time, full-time award-seeking students (Retention rates reflect the percentage of fall quarter first-time, full-time entering students who continue enrollment the following fall quarter. The report presents both institution-specific retention rates and system retention rates.)
2. Graduation rates (DTAE provided information on graduation rates for associate degree programs and rates for students who complete a program or earn a professional certificate.)
3. Pass rates on licensure/certification exams (The 34 technical institutions offer different combinations of degrees and programs. The list of licensure exams for which OEA reports a passing rate varies for each institution. As with all other data, OEA reports for the DTAE system as a whole as well as by each institution.)
University System of Georgia (USG) Indicators
OEA's 2001-2002 Report Card on USG focuses on the following four indicators:

1. Retention rates of first-time, full-time award-seeking students (Retention was based on continuous student enrollment from their first year to their second year, and rates reflect the percentage of fall semester first-time, full-time freshmen who continued enrollment the following semester)
2. Graduation Rates (OEA's Report Card presents information on graduation rates for baccalaureate degree programs; associate degrees and certificate programs combined with rates of transfer to four-year institutions)
3. Pass rates on the Regents' exams (Information provided by USG indicates the percentage of students who pass the test before earning 45 credit hours)
4. Pass rates on licensure/certification exams (Pass rate for USG students on licensure or certification exams)
The data provided OEA from USG reflects information from the 2000-01 academic year. The above indicators are reported for the university system as a whole, for each sector, and for each of the 34 institutions. When data are available, information is disaggregated by race/ethnicity, gender, and socioeconomic status. Socioeconomic status is based on whether the student applied for and received a Pell Grant (federal student aid).
Georgia Professional Standards
(GAPSC) Indicators
OEA's Report Card on GAPSC 2001-2002 Report Card focuses on two indicators:
1. Pass Rate on Praxis I Exam (OEA reports the 2001-2002 pass rates on the Praxis I for public and private post-secondary institutions)
2. Pass Rate on Praxis II Exam (For each public and private postsecondary institution offering a teacher preparation program, OEA reports an overall Praxis II pass rate. OEA also includes individual pass rates on specific content exams from each institution and offers the ability to compare institutional pass rates on a specific content exam.)
GAPSC provided OEA with the data. Results are disaggregated by race/ethnicity and gender.

The GAPSC has the full responsibility for the preparation, certification, and conduct of certified, licensed, or permitted personnel employed in the public schools of the State of Georgia. Its mission is "to provide a qualified teacher in every classroom by setting and applying high standards for the preparation, certification, and continued licensing of Georgia public educators." GAPSC uses the Praxis I tests of mathematics, reading, and writing to assess the basic skills of individuals seeking certification. Praxis I is considered a preprofessional skills test. GAPSC sets the passing scores on the Praxis I exam. GAPSC uses the Praxis II exams to ensure that Georgia educators are well versed in their teaching field(s). To be recommended for licensure, a person must earn satisfactory scores on the Praxis II Subject Assessments in the appropriate subject area(s) for the certification sought. VII. Development of a Single Federal/State Education Accountability System With passage of the No Child Left Behind (NCLB) Act that reauthorized federal education funding, states are now required to have a single education accountability system for all public schools.

Georgia's Single
Accountability System

1. Status of federal adequate Yearly Progress Goals (AYP)
2. Grades on state assessments
3. Ratings on state indicators

VIII. Initial Development of an Audit Manual and Initiation of Audit Responsibilities Until grades are given to K-12 schools, the audit function will be used as a vehicle for:
addressing ideas that address anomalies in data; improving test administration; and improving data integrity and reporting.

After analyses are conducted, if unexpected student participation trends or data integrity issues emerge, or if information received from external sources suggests irregularities, then the audit process will be initiated. OEA will also work closely on a proactive basis with local schools and school systems using a three strikes policy as follows:
First, Inquiry Level 1 (Desk Review) - A letter will be forwarded and a telephone call will be placed to the principal of the affected school asking the school to research/report back to OEA on the identified audit issue(s).
Second, Inquiry Level 2 (Desk Review) - If the results of the letter/telephone inquiry with the principal are insufficient to indicate a clear and valid reason for closing the audit issue(s), a letter to the school system superintendent will be forwarded requesting a written explanation of the irregularity or irregularities that have been observed.
Third, Inquiry Level 3 (Site Visit) - When telephone calls and two letters of inquiry are not sufficient to resolve the questions pertaining to the audit item(s), an on-site audit will be conducted. On-site audits will be made in accordance with O.C.G.A. 20-14-35.
Looking Ahead The major focus of OEA's activities for calendar year 2003 will be centered on collaborative efforts to establish the framework for a single statewide education accountability system-integrating state and federal requirements.