Research notes [June 2005]

Evaluating the Effectiveness of Learning Support Placement: Actual versus Expected Performance in College-Level Courses
The purpose of most pre-college remedial programs, including the Learning Support (LS) program of the University System of Georgia, is to prepare students for success in college-level courses. A way to examine the success of Learning Support students and to evaluate the placement system is to compare the grades of students who completed LS with the grades they would have been expected to receive if they had not taken LS. Although we have not conducted experimental studies comparing the grades students with low scores would receive with and without remediation, we can use the relationship between placement test scores and grades in college-level courses to predict how students in LS would have performed without remediation.
An effective remedial placement system has two essential components: a method of identifying students who would fail without remediation and the provision of effective remediation to increase the chances of success for the students who are predicted to fail. If a placement test is to be useful for identifying students who would fail without remediation, there must be a relationship between scores and grades. Higher test scores must predict higher grades in the college-level course, as illustrated in figure 1:

4

Grade

3

in

Core

Curriculum 2

Course

1

0

55

65

75

85

95

Placement Test Score

Figure 1

Page 1 of 15 June 2005

If placement test scores are used to identity students who would fail without remediation, and if effective remediation is provided to those students predicted to fail, the failing grades predicted will not occur. In other words, effective remediation will serve to eliminate or reduce the relationship between placement test scores and actual grades. The relationship between scores and grades for students above the placement cutoff score can be used to predict success for students below the cutoff score. Figure 2 shows the relationship between placement test scores and grades in Math 1101 (Introduction to Mathematical Modeling) for students who received passing scores (75 or higher) on a mathematics placement test (CPE) and took Math 1101 in FY2000.
CPE Math 4
3
Grade 2
1
0 50 55 60 65 70 75 80 85 90 95 CPE Score
Math 1101 No LS
Figure 2 Based on the relationship between placement test scores and grades for students who were not placed in remediation, the grades for students with failing scores can be predicted as shown by the dotted line in Figure 3.
Page 2 of 15 June 2005

CPE Math 4
3
Grade 2
1
0 50 55 60 65 70 75 80 85 90 95 CPE Score
Math 1101 No LS Math 1101 Hypothetical No LS
Figure 3
Students below the placement cutoff score are predicted to earn failing grades (below 2.0 or "C") if they do not receive remediation. The actual relationship between placement test scores and grades is shown in Figure 4 for students who scored less than 75 and were placed in LS and those who scored 75 or above and were not placed in LS.
CPE Math
4
3
Grade 2
1
0 50 55 60 65 70 75 80 85 90 95 CPE Score
Math 1101 No LS Math 1101 LS Math 1101 Hypothetical No LS
Figure 4
Page 3 of 15 June 2005

In this example, students placed in LS did not earn the failing grades they would have been expected to earn if they had not been placed in LS. Instead, they earned a predicted grade of approximately 2.0 ("C") regardless of placement test score. Thus, it appears that remediation was effective in increasing the probability of success in the college-level mathematics course for students who failed the placement test. Linear Regression Analyses Analyses were conducted for students enrolled in English 1101 (Composition I), Mathematics 1101 (Introduction to Mathematical Modeling), Mathematics 1111 (College Algebra) and Political Science 1101 (American Government) at University System of Georgia institutions. Linear regression analyses were conducted separately for students who took the Collegiate Placement Examination (CPE), the paper and pencil placement test developed by ACT, Inc. for use in the University System of Georgia, and COMPASS, the computerized placement test available from ACT, Inc. Results are based on students who took one of the college-level courses in FY2000 and had one of the placement test scores available. Students who passed the placement test at the System minimum cutoff score were included if they were not placed in LS; students who did not pass the placement test at the System minimum cutoff score were included only if they were placed in LS. Descriptive statistics for the samples are provided in Table 1. The results are shown on pages 5 through 8. In each graph, the line below the placement test cutoff score is based on students who completed Learning Support in the area; the line above the cutoff score is based on students who did not enroll in Learning Support in the area. For Political Science 1101, reading is the LS area used in the analysis.
Page 4 of 15 June 2005

RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND GRADES IN CORE CURRICULUM COURSES BY LS PLACEMENT:
LINEAR REGRESSION FY2000

Math 1101/CPE

4

3
Grade
2

1

0

55

65

75

85

95

CPE Score

Math 1101/COMPASS
4 3
Grade
2 1 0
15 25 35 45 55 65 75 85 95
COMPASS Score
Page 5 of 15 June 2005

RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND GRADES IN CORE CURRICULUM COURSES BY LS PLACEMENT:
LINEAR REGRESSION FY2000 (CONTINUED)

Math 1111/CPE

4

3
Grade
2

1

0

55

65

75

85

95

CPE Score

Math 1111/COMPASS
4 3
Grade
2 1 0
15 25 35 45 55 65 75 85 95
COMPASS Score

Page 6 of 15 June 2005

RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND GRADES IN CORE CURRICULUM COURSES BY LS PLACEMENT:
LINEAR REGRESSION FY2000 (CONTINUED)

English 1101/CPE

4

3
Grade
2

1

0

45

55

65

75

85

CPE Score

English 1101/COMPASS
4 3
Grade
2 1 0
0 10 20 30 40 50 60 70 80 90 100 COMPASS Score
Page 7 of 15 June 2005

RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND GRADES IN CORE CURRICULUM COURSES BY LS PLACEMENT:
LINEAR REGRESSION FY2000 (CONTINUED)

Political Science 1101/CPE

4

3
Grade
2

1

0

55

65

75

85

95

CPE Score

Political Science 1101/COMPASS
4 3
Grade
2 1 0
15 25 35 45 55 65 75 85 95
COMPASS Score

Page 8 of 15 June 2005

Table 1
Sample Sizes, Means and Standard Deviations of Grades, Correlations between Placement Scores and Grades, and
Standard Errors of Estimates for Linear Regressions

English 1101 No LS LS
Math 1101 No LS LS

CPE ___________________________

Grades _

N Mean SD R

SEE

2474 2.37 1.18 .15

1.16

1393 2.03 1.07 .11

1.06

860 2.52 1.36 .37

1.26

574 1.85 1.28 -.02* 1.28

COMPASS ___________________________

Grades _

N Mean SD R

SEE

2380 2.41 1.19 .21

1.16

1317 2.04 1.09 .10

1.08

734 2.29 1.38 .20

1.35

612 2.19 1.29 00* 1.29

Math 1111 No LS LS

735 1.76 1.35 .23

1.32

1072 1.60 1.25 .02* 1.26

Pol Science 1101

No LS

1729 2.26 1.2 .19

1.17

LS

779 1.86 1.1 .08

1.09

402 1.72 1.35 .32

1.28

503 1.56 1.28. .09* 1.28

1436 2.19 510 1.77

1.26 .23

1.22

1.13 -.06* 1.13

* Not significant at p< .05. Other correlations are significantly greater than 0. Note: Correlations are reduced because of restriction in range. Students with grades of A, B, C, D, F, or WF are included.

Page 9 of 15 June 2005

Logistic Regression Analyses
Another method used to examine the performance that would be expected of students in remediation compared with the performance that would be expected of the same students had they not been placed in remediation was logistic regression. In these analyses, the relationship between placement test scores and passing rates in core curriculum courses was examined. Results are provided on pages 10 through 13. Similar to the linear regression, the performance of students above the cutoff scores who were not placed in LS is shown. A separate regression is shown for students who scored below the placement cutoff score and who were placed in LS. As with the previous analyses, the relationship for those above the cutoff can be extrapolated to predict the performance of LS students had they not been placed in remediation.
Table 2 indicates the sample sizes and passing rates for the samples used in the logistic regression analyses. The sample sizes for the logistic regressions are larger than those for the linear regressions because the logistic regressions included students who received grades of W (as not passing) in addition to those with grades of A, B, C, D, F, and WF.

Table 2 Sample Sizes and Passing Rates for Logistic Regressions

English 1101 No LS LS

CPE

______________________

N

% C or Better

2749

73.0

1574

65.1

COMPASS

_______________________

N

% C or Better

2706

71.8

1460

67.3

Math 1101

No LS

992

67.6

LS

734

48.4

854

63.6

780

55.6

Math 1111

No LS

918

46.3

LS

1391

41.1

510

43.9

688

39.2

Political Science 1101

No LS

1996

65.9

LS

911

57.1

1657

63.1

615

52.4

Page 10 of 15 June 2005

RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND PASSING RATES IN CORE CURRICULUM COURSES BY LS PLACEMENT:
LOGISTIC REGRESSION FY2000

Math 1101/CPE

1 0.9 0.8

0.7

Passing 0.6

Rate

0.5

(>D)

0.4

0.3

0.2

0.1

0

55

65

75

85

95

CPE Score

Math 1101/COMPASS
1 0.9 0.8 0.7
Passing 0.6 Rate 0.5 (>D) 0.4
0.3 0.2 0.1
0 15 25 35 45 55 65 75 85 95
COMPASS Score

Page 11 of 15 June 2005

RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND PASSING RATES IN CORE CURRICULUM COURSES BY LS PLACEMENT:
LOGISTIC REGRESSION FY2000 (CONTINUED)

Math 1111/CPE

1

0.9 0.8

0.7

Passing 0.6

Rate

0.5

(>D)

0.4

0.3

0.2 0.1

0

55

65

75

85

95

CPE Score

Math 1111/COMPASS
1 0.9 0.8 0.7
Passing 0.6 Rate 0.5 (>D) 0.4
0.3 0.2 0.1
0 15 25 35 45 55 65 75 85 95
COMPASS Score
Page 12 of 15 June 2005

RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND PASSING RATES IN CORE CURRICULUM COURSES BY LS PLACEMENT:
LOGISTIC REGRESSION FY2000 (CONTINUED)
English 1101/CPE

1

0.9

0.8

0.7

Passing 0.6

Rate

0.5

(>D)

0.4

0.3

0.2

0.1

0

45

55

65

75

85

CPE Score

English 1101/COMPASS

Passing Rate (>D)

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1
0 0

10 20 30 40 50 60 70 80 90 100
COMPASS Score

Page 13 of 15 June 2005

RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND PASSING RATES IN CORE CURRICULUM COURSES BY LS PLACEMENT:
LOGISTIC REGRESSION FY2000 (CONTINUED)

Political Science 1101/CPE

Passing Rate (>D)

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1
0
55

65

75

85

CPE Score

Political Science 1101/COMPASS
1 0.9 0.8 0.7
Passing 0.6 Rate 0.5 (>D) 0.4
0.3 0.2 0.1
0 15 25 35 45 55 65 75 85 95
COMPASS Score
Page 14 of 15 June 2005

Conclusion
For each test and subject area, students who were placed in remediation as a result of the placement test had higher grades than they would have been predicted to earn without remediation. The logistic regressions indicated higher than expected passing rates for LS students in all subjects except English 1101 for students placed with the CPE.
In many cases, students who were placed in remediation had higher performance than students near the cutoff score who were not placed in remediation. Especially in mathematics, the results provided evidence that students slightly above the cutoff score would have benefited from placement in LS. Based on this as well as other information indicating that the cutoff scores for mathematics were too low, the System minimum placement scores in mathematics were raised in 2002 from 75 to 79 on the CPE and from 30 to 37 on COMPASS Algebra.
In Math 1111 (College Algebra), students placed in LS and those near the cutoff but not placed in LS were predicted to get grades less than "C" and have a passing rate less than 50%. However, even this low performance for students placed in LS was higher than would have been expected without remediation. (The situation appears to have improved. In the most recent study, based on students taking core curriculum courses in FY2003, passing rates for Math 1111 increased 8 percentage points for students with System-mandated LS mathematics requirements and 8.8 percentage points for non-LS students.)
In most of the comparisons, placement in remediation appeared to have lowered the correlation between test scores and grades, a desirable result, suggesting that remediation was effective in helping students overcome deficiencies that would have resulted in low performance.
For more information, contact:
Dr. Kathleen Burk Consultant
Office of Strategic Research and Analysis Board of Regents of University System of Georgia
270 Washington St., SW Atlanta, Georgia 30334
Kathleen.Burk@usg.edu
Page 15 of 15 June 2005