Statewide analysis : spring 2009 CRCT grades 1-8 : presented to the State Board of Education on February 10, 2010

GOSA CRCT Analysis Report

February 2010

Statewide Analysis: Spring 2009 CRCT Grades 1-8 Presented to the State Board of Education on February 10, 2010

Rationale for this Investigation
The Governor's Office of Student Achievement is charged with auditing and inspecting schools and Local Education Agencies (O.C.G.A. 2012-24). A comprehensive analysis of the 2009 Spring CRCT data conducted by the state's vendor, CTB McGraw-Hill, showed an unusually high number of answers changed from wrong to right (WTR) in some classrooms. Based on a conservative criterion for identifying unusual results, OSA makes the recommendations in this report to help eliminate test tampering and to help students adversely affected where applicable.
Because important decisions for individual students and for schools are based on CRCT data, it is vital that results indicate an accurate representation of students' knowledge.
Purpose of the Criterion Referenced Competency Test (CRCT)
The CRCT is a standardized assessment administered to students in grades 1-8 in Georgia. It is designed to measure how well students at each grade level have acquired the knowledge and skills within the state's curriculum, the Georgia Performance Standards.
CRCT results are used to determine whether schools have made Adequate Yearly Progress (AYP) as required by the No Child Left Behind (NCLB) Act.

Executive Summary
Erasure Analysis The state's test vendor for the CRCT, CTB-McGraw Hill, conducted an analysis of erasures on the answer sheets used in the spring 2009 CRCT. The analysis was conducted for grades 1-8, in Reading, Language Arts and Math and was designed to identify whether answers were changed from wrong to right more frequently in any classrooms compared to the rest of the state test population in each grade and subject.
Using a professional grade scanner, CTB scanned the answer documents to determine the total number of erasures and the total number of wrong-to-right (WTR) changes on each answer document. CTB then aggregated those results at the classroom level. Any classroom in which the number of WTR changes was 3 standard deviations (SD) or more (adjusted for class size) above the state average for that particular grade and subject was "flagged" as having an unusually high number of WTR changes. CTB aggregated these results at the school level.
Erasure Analysis Results The analysis indicates that some classrooms show an unusually high number of wrong answers changed to right answers on the grades 1-8 Spring 2009 CRCT in Reading, Language Arts and Math. At the school level, the percentage of classrooms flagged using the conservative criterion of 3 SD above the state average ranged from 0% to 89.5%, with a statewide average of 4%.
GOSA placed schools into four categories based on the percentage of classrooms within each school flagged according to CTB's results: Clear of concern; Minimal concern; Moderate concern; and Severe concern.
80% of Georgia's elementary and middle schools fell into the "Clear" category, 10% fell into "Minimal concern," 6% fell into "Moderate concern," and 4% fell into the "Severe concern" category.
~ 1 ~

Erasure Analysis
Submitted by CTB-McGraw Hill
January 22, 2010
With the high-stakes nature of large-scale assessments such as the CRCT, there are times when student's responses, and hence their scores, may not be a true representation of their own abilities. Various activities may take place, such as a student copying from another student's paper, students receiving inappropriate assistance before or during testing, or students' responses altered after testing. To maintain the integrity of the CRCT and the validity of the results, it is important that any such instances be discovered.
The present study investigated student responses on the Reading, English Language Arts, and Mathematics Spring 2009 CRCT that a) were erased and b) changed from wrong to right answers
Reviewers should note that results should only be used to facilitate identification of systematic problems within individual schools. That is, these types of analyses must be supported by additional, collateral information.
Method
The basis for the erasure analysis is to count erasures in items where an answer choice was erased and replaced with another answer choice. Often the data captured is useful for identifying cases of cheating. During erasure analysis, two sets of erasures were analyzed: all erasures and wrong-toright erasures where an incorrect answer choice was erased and replaced with the correct answer choice. Please note that, for the erasure analyses, all items (either the operational or field-test) were included, as field test items were all embedded in CRCT.
The basic idea underlying the procedure is a statistical test of the null hypothesis (H0) that the mean number of erasures for a class constitutes a random sample from the state distribution of erasures. The hypothesis is tested against the (right-sided) alternative (H1) that the mean number is too high to be explained by random sampling. Classes for which H0 has to be rejected are flagged for further scrutiny. A well-known central limit theorem in statistics tells us that the sampling distribution of mean number of erasures for class i (mi) is asymptotically normal with mean and standard deviation

Mean(mi ) =

(1)

SD(mi ) =

ni

(2)

~ 2 ~

where ni and mi denote the size and mean number of erasures for class i, respectively. In addition, and denote the mean and the standard deviation of the distribution of the number of erasures of the population of individual students in the state of Georgia.
It is evident in the formula for the state standard deviation that the class flagging criterion for each class is adjusted for the number of test takers in a classroom. For example, if the state mean and SD of erasure count are 1.73 and 2.11, respectively, the flagging criterion for a class size of 20 is
adjusted to 3.15 (1.73 + 3 2.11 = 3.15 ). 20
This adjustment ensures that the flagging criterion is equally stringent for classes with considerably different numbers of test takers. In addition, minimizing the probability of false positive (Type I) errors in this statistical test is crucial in this analysis.
The classes were flagged if their mi was larger than + 3 . Statistically, the flagging criterion ni
set at or above 3 is conservative. The standard normal table shows that under random sampling the (asymptotic) probability of a sample mean being more than three standard deviations above the population mean is around 0.001. However, rejection of H0 only tells us that the observed mean number of erasures is unlikely to be the result of random sampling.
Results
Table A.1 reports the state summary for erasure counts. The mean number of erasures across grades and subjects ranged from 1.40 to 3.07 for the 2009 spring CRCT. The erasure count at the 95th percentile point was between 5 and 9. Table A.2 reports the state summary for wrong-to-right erasure counts. As can be expected, the mean wrong-to-right erasure count and the count at the 95th percentile point were lower than those obtained from all erasure counts. The mean number of wrong-to-right erasures ranged from 0.81 to 1.91 for the 2009 spring CRCT. The wrong-to-right erasure count at the 95th percentile point was between 3 and 6.
Table 1 summarizes all erasure analyses and wrong-to-right erasure analyses. Table 1 presents the number of schools flagged across three content areas Reading, English/Language Arts, and Mathematics - within each analysis of spring CRCT. For each analysis, the number of schools was computed in two ways: flagged for at least one content area or flagged for all three content areas.
Separate reports were produced displaying the results of all erasure analyses and wrong-to-right erasure analyses. The number/percentage of schools that had zero flags for all erasures and wrongto-right erasures in Reading, English/Language Arts, and Mathematics and is provided in Table 2.
The number/percentage of schools that had less than 1% of classes flagged for all erasures and wrong-to-right erasures in Reading, English/Language Arts, and Mathematics and across grades is provided in Table 3.
~ 3 ~

Table 1. The number of schools flagged across three content areas

Grade
1 2 3 4 5 6 7 8

Total Number of
Schools
1228 1230 1232 1233 1235 591 567 566

All Erasure Analyses

Number of

Number of

Schools Flagged Schools Flagged

for at Least One for All Content

Content Area

Areas

478

191

424

174

368

94

298

68

316

90

220

45

204

55

220

65

Wrong-to-Right Erasure Analyses

Number of

Number of

Schools Flagged Schools Flagged

for at Least One for All Content

Content Area

Areas

455

180

392

161

316

80

250

50

252

69

154

23

148

23

156

39

Table 2. The number and percentage of schools that had zero flags for all erasures and wrong-toright erasures (Spring CRCT)

Reading,

Reading

English/Language Arts

Mathematics

English/Language Arts,

and Mathematics

Grad

No. of % of

No. of % of

No. of % of

No. of % of

e No. of School School No. of School School No. of School School No. of School School

School s with s with School s with s with School s with s with School s with s with

s

zero zero

s

zero zero

s

zero zero

s

zero zero

flags flags

flags flags

flags flags

flags flags

1 1226 852 69% 1226 880 72% 1226 845 69% 1226 700 57%

2 1229 909 74% 1229 914 74% 1229 903 73% 1229 771 63%

3 1231 969 79% 1231 994 81% 1231 1014 82% 1231 816 66%

4 1231 1036 84% 1231 1019 83% 1230 1063 86% 1231 884 72%

5 1233 1019 83% 1232 1007 82% 1233 1057 86% 1233 879 71%

6 590 436 74% 589 438 74% 589 484 82% 590 350 59%

7 566 414 73% 566 434 77% 566 449 79% 566 341 60%

8 564 401 71% 564 409 73% 564 450 80% 564 330 59%

Table 3. The number and percentage of schools that had less than 1% of classes flagged for all erasures and wrong-to-right erasures across grades

No. of Schools
1857

Reading
No. of School s with
<1% flags across grades 1191

No. of School s with
<1% flags across grades
64%

English/Language Arts

No. of Schools
1857

No. of School s with
<1% flags across grades
1220

No. of School s with
<1% flags across grades
66%

Mathematics

No. of Schools
1857

No. of School s with
<1% flags across grades
1260

No. of School s with
<1% flags across grades
68%

Reading, English/Language

Arts, and Mathematics

No. of No. of

School School

No. of Schools

s with <1% flags

s with <1% flags

across across

grades grades

1857 1594 86%

~ 4 ~

Discussion With respect to the erasure analyses, the following caveats are always applicable:
1. The normal distribution holds only for large classes; for smaller classes the result is approximate.
2. Rejection of H0 does not necessarily imply cheating. Alternative explanations are possible. 3. The flagging criterion should thus be taken as a stimulus to look for additional evidence and
find out what happened in the school. This erasure analysis is considered a check for unusual numbers of erasures to student responses. Without additional layers added to the analysis, this kind of check only addresses the possibility, not the certainty, of teachers or administrators altering the responses of students.
~ 5 ~

Appendix

Scanning Operations Documentation
CTB-McGraw Hill
With the high-stakes nature of large-scale assessments such as the CRCT, there are times when student's responses, and hence their scores, may not be a true representation of their own abilities. Various activities may take place, such as a student copying from another student's paper, students receiving inappropriate assistance before or during testing, or students' responses altered after testing. To maintain the integrity of the CRCT and the validity of the results, it is important that any such instances be discovered.
This memorandum is designed to provide some additional documentation in support of a research study that investigated student responses on the Reading, English Language Arts, and Mathematics Spring 2009 CRCT that a) were erased and b) changed from wrong to right. Specifically, it provides information regarding the scanning operations as they pertain to erasure analyses and associated quality control checks.
It should be emphasized that results from erasure analyses performed in 2009 should only be used to identify potential problems within individual classrooms. That is, these types of analyses must be supported by additional investigations and collateral information before conclusions regarding any improprieties are reached.
Scanning Operations
The GA CRCT answer documents were processed using high speed 5000i optical scanners which reliably capture document images and optical mark read data. The sophisticated proprietary scoring software system WinScore reviews the integrity of each batch of documents scanned according to pre-defined guidelines and services. This flexible system reduces scanning/scoring time and provides a high degree of quality control.
The WinScore system provides a mechanism for identifying multiple-marks and identification of erasures for scanned data. The basis of the erasure analysis is to count erasures for multiple choice items where two or more responses have been made with specified intensity. Erasure analyses provide a mechanism to differentiate between three kinds of answer changes: a) wrong-to-wrong, b) right-to-wrong and c) wrong-to-right. Capturing the frequency of answer changes from wrong to right can be useful for identifying potential instances of cheating at the student level. Erasure analyses results can be grouped to tentatively identify problems at the classroom and school levels.
An erasure analysis involves an analysis of scanner data where additional marks have been made on student answer documents. As part of our quality control check for the 2009 CRCT erasure analyses, we pulled 19487 student papers from 17 schools and checked whether the wrong-to-right erasures could be confirmed. Over 96% of answer changes were confirmed by raters. The remaining 4% of answer changes were likely due to unintended marks on answer documents made and were distributed throughout the 20K papers examined. This extensive review of the CRCT answer changes indicates that the scanning software employed in the erasure analysis accurately identified students changing wrong to right answers.

Table A.1. State summar y for all types of er asur e counts

Content RD LA MA

Grade 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8

N_state 128,257 129,365 128,837 125,267 124,798 120,336 120,692 120,859 128,226 129,328 128,767 125,239 124,744 120,283 120,629 120,724 128,434 129,540 129,046 125,470 125,019 120,478 120,777 120,953

ERA_state 221,495 181,429 251,810 212,860 251,418 174,097 168,982 188,964 306,476 259,145 301,406 276,556 285,415 191,496 173,625 203,763 321,641 328,724 367,994 368,661 368,460 308,007 249,930 342,054

Mean_state 1.73 1.40 1.95 1.70 2.01 1.45 1.40 1.56 2.39 2.00 2.34 2.21 2.29 1.59 1.44 1.69 2.50 2.54 2.85 2.94 2.95 2.56 2.07 2.83

Std_state 2.11 1.87 2.70 2.44 2.71 2.04 1.98 2.12 2.65 2.37 2.85 2.83 2.85 2.29 2.10 2.35 2.70 2.72 3.25 3.39 3.48 2.96 2.62 3.33

Min_state 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Max_state 31 40 44 50 43 49 41 41 34 50 55 60 53 56 49 52 40 61 57 68 60 67 54 66

P10_state 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

P25_state 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 1 1 1 1 0 1

P50_state 1 1 1 1 1 1 1 1 2 1 2 1 1 1 1 1 2 2 2 2 2 2 1 2

P75_state 3 2 3 2 3 2 2 2 3 3 3 3 3 2 2 2 4 4 4 4 4 4 3 4

P90_state 4 4 5 4 5 4 4 4 6 5 6 5 5 4 4 4 6 6 7 7 7 6 5 7

P95_state 6 5 7 6 7 5 5 5 7 6 8 7 7 6 5 6 8 8 9 9 9 8 7 9

Table A.2. State summar y for wr ong-to-r ight (WTR) er asur e count

Content RD LA MA

Grade 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8

N_state 128,257 129,365 128,837 125,267 124,798 120,336 120,692 120,859 128,226 129,328 128,767 125,239 124,744 120,283 120,629 120,724 128,434 129,540 129,046 125,470 125,019 120,478 120,777 120,953

WTR_state 138,593 118,288 151,060 124,590 141,789 98,587 97,579 111,596 205,551 170,610 194,150 175,974 178,967 115,767 110,213 125,520 224,507 223,945 241,822 235,186 228,146 171,897 143,487 186,083

Mean_state 1.08 0.91 1.17 0.99 1.14 0.82 0.81 0.92 1.60 1.32 1.51 1.41 1.43 0.96 0.91 1.04 1.75 1.73 1.87 1.87 1.82 1.43 1.19 1.54

Std_state 1.49 1.38 1.87 1.64 1.83 1.30 1.27 1.39 1.95 1.73 2.05 2.03 2.04 1.55 1.48 1.63 2.05 2.05 2.43 2.44 2.54 1.96 1.77 2.14

Min_state 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Max_state 23 28 35 36 37 29 33 35 30 43 37 41 34 36 41 41 30 44 41 47 45 43 41 45

P10_state 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

P25_state 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

P50_state 1 0 1 0 1 0 0 1 1 1 1 1 1 0 0 1 1 1 1 1 1 1 1 1

P75_state 2 1 2 1 2 1 1 1 2 2 2 2 2 1 1 1 3 2 3 3 2 2 2 2

P90_state 3 2 3 3 3 2 2 2 4 3 4 4 4 3 3 3 4 4 5 5 4 4 3 4

P95_state 4 3 4 4 4 3 3 3 5 4 5 5 5 4 4 4 6 5 6 6 6 5 4 5

For mal Recommendations that Requir e Action by the State Boar d of Education

Recommendations are intended to eliminate test tampering and help students who were adversely affected by intentional wrongdoing.

Severe Concern (25% or more of classrooms flagged)

State Action State Monitors during Spring 2010 CRCT

LEA Corrective Action 1) LEA to conduct an investigation 2) Submit results of investigation to OSA 3) Rotate teachers during CRCT.

LEA Student Support 1) Notify parents 2) Offer student support services as appropriate based on outcomes of LEA investigation

Moderate Concern (11% - 24% of classrooms flagged)

Random Spot Checks by State Monitors

1) LEA to conduct an investigation 2) Submit results of investigation to OSA 3) Rotate teachers during CRCT. 4) Monitor test environment.

1) Notify parents 2) Offer student support services as appropriate based on outcomes of LEA investigation

Minimal Concern (6%-10% of classrooms flagged)

1) LEA to monitor test environment of flagged schools; OR 2) LEA to rotate teachers in flagged schools.

1) Offer student support services as appropriate based on any concerning irregularities found.