Evaluating the Effectiveness of Learning Support Placement: Actual versus Expected Performance in College-Level Courses The purpose of most pre-college remedial programs, including the Learning Support (LS) program of the University System of Georgia, is to prepare students for success in college-level courses. A way to examine the success of Learning Support students and to evaluate the placement system is to compare the grades of students who completed LS with the grades they would have been expected to receive if they had not taken LS. Although we have not conducted experimental studies comparing the grades students with low scores would receive with and without remediation, we can use the relationship between placement test scores and grades in college-level courses to predict how students in LS would have performed without remediation. An effective remedial placement system has two essential components: a method of identifying students who would fail without remediation and the provision of effective remediation to increase the chances of success for the students who are predicted to fail. If a placement test is to be useful for identifying students who would fail without remediation, there must be a relationship between scores and grades. Higher test scores must predict higher grades in the college-level course, as illustrated in figure 1: 4 Grade 3 in Core Curriculum 2 Course 1 0 55 65 75 85 95 Placement Test Score Figure 1 Page 1 of 15 June 2005 If placement test scores are used to identity students who would fail without remediation, and if effective remediation is provided to those students predicted to fail, the failing grades predicted will not occur. In other words, effective remediation will serve to eliminate or reduce the relationship between placement test scores and actual grades. The relationship between scores and grades for students above the placement cutoff score can be used to predict success for students below the cutoff score. Figure 2 shows the relationship between placement test scores and grades in Math 1101 (Introduction to Mathematical Modeling) for students who received passing scores (75 or higher) on a mathematics placement test (CPE) and took Math 1101 in FY2000. CPE Math 4 3 Grade 2 1 0 50 55 60 65 70 75 80 85 90 95 CPE Score Math 1101 No LS Figure 2 Based on the relationship between placement test scores and grades for students who were not placed in remediation, the grades for students with failing scores can be predicted as shown by the dotted line in Figure 3. Page 2 of 15 June 2005 CPE Math 4 3 Grade 2 1 0 50 55 60 65 70 75 80 85 90 95 CPE Score Math 1101 No LS Math 1101 Hypothetical No LS Figure 3 Students below the placement cutoff score are predicted to earn failing grades (below 2.0 or "C") if they do not receive remediation. The actual relationship between placement test scores and grades is shown in Figure 4 for students who scored less than 75 and were placed in LS and those who scored 75 or above and were not placed in LS. CPE Math 4 3 Grade 2 1 0 50 55 60 65 70 75 80 85 90 95 CPE Score Math 1101 No LS Math 1101 LS Math 1101 Hypothetical No LS Figure 4 Page 3 of 15 June 2005 In this example, students placed in LS did not earn the failing grades they would have been expected to earn if they had not been placed in LS. Instead, they earned a predicted grade of approximately 2.0 ("C") regardless of placement test score. Thus, it appears that remediation was effective in increasing the probability of success in the college-level mathematics course for students who failed the placement test. Linear Regression Analyses Analyses were conducted for students enrolled in English 1101 (Composition I), Mathematics 1101 (Introduction to Mathematical Modeling), Mathematics 1111 (College Algebra) and Political Science 1101 (American Government) at University System of Georgia institutions. Linear regression analyses were conducted separately for students who took the Collegiate Placement Examination (CPE), the paper and pencil placement test developed by ACT, Inc. for use in the University System of Georgia, and COMPASS, the computerized placement test available from ACT, Inc. Results are based on students who took one of the college-level courses in FY2000 and had one of the placement test scores available. Students who passed the placement test at the System minimum cutoff score were included if they were not placed in LS; students who did not pass the placement test at the System minimum cutoff score were included only if they were placed in LS. Descriptive statistics for the samples are provided in Table 1. The results are shown on pages 5 through 8. In each graph, the line below the placement test cutoff score is based on students who completed Learning Support in the area; the line above the cutoff score is based on students who did not enroll in Learning Support in the area. For Political Science 1101, reading is the LS area used in the analysis. Page 4 of 15 June 2005 RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND GRADES IN CORE CURRICULUM COURSES BY LS PLACEMENT: LINEAR REGRESSION FY2000 Math 1101/CPE 4 3 Grade 2 1 0 55 65 75 85 95 CPE Score Math 1101/COMPASS 4 3 Grade 2 1 0 15 25 35 45 55 65 75 85 95 COMPASS Score Page 5 of 15 June 2005 RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND GRADES IN CORE CURRICULUM COURSES BY LS PLACEMENT: LINEAR REGRESSION FY2000 (CONTINUED) Math 1111/CPE 4 3 Grade 2 1 0 55 65 75 85 95 CPE Score Math 1111/COMPASS 4 3 Grade 2 1 0 15 25 35 45 55 65 75 85 95 COMPASS Score Page 6 of 15 June 2005 RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND GRADES IN CORE CURRICULUM COURSES BY LS PLACEMENT: LINEAR REGRESSION FY2000 (CONTINUED) English 1101/CPE 4 3 Grade 2 1 0 45 55 65 75 85 CPE Score English 1101/COMPASS 4 3 Grade 2 1 0 0 10 20 30 40 50 60 70 80 90 100 COMPASS Score Page 7 of 15 June 2005 RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND GRADES IN CORE CURRICULUM COURSES BY LS PLACEMENT: LINEAR REGRESSION FY2000 (CONTINUED) Political Science 1101/CPE 4 3 Grade 2 1 0 55 65 75 85 95 CPE Score Political Science 1101/COMPASS 4 3 Grade 2 1 0 15 25 35 45 55 65 75 85 95 COMPASS Score Page 8 of 15 June 2005 Table 1 Sample Sizes, Means and Standard Deviations of Grades, Correlations between Placement Scores and Grades, and Standard Errors of Estimates for Linear Regressions English 1101 No LS LS Math 1101 No LS LS CPE ___________________________ Grades _ N Mean SD R SEE 2474 2.37 1.18 .15 1.16 1393 2.03 1.07 .11 1.06 860 2.52 1.36 .37 1.26 574 1.85 1.28 -.02* 1.28 COMPASS ___________________________ Grades _ N Mean SD R SEE 2380 2.41 1.19 .21 1.16 1317 2.04 1.09 .10 1.08 734 2.29 1.38 .20 1.35 612 2.19 1.29 00* 1.29 Math 1111 No LS LS 735 1.76 1.35 .23 1.32 1072 1.60 1.25 .02* 1.26 Pol Science 1101 No LS 1729 2.26 1.2 .19 1.17 LS 779 1.86 1.1 .08 1.09 402 1.72 1.35 .32 1.28 503 1.56 1.28. .09* 1.28 1436 2.19 510 1.77 1.26 .23 1.22 1.13 -.06* 1.13 * Not significant at p< .05. Other correlations are significantly greater than 0. Note: Correlations are reduced because of restriction in range. Students with grades of A, B, C, D, F, or WF are included. Page 9 of 15 June 2005 Logistic Regression Analyses Another method used to examine the performance that would be expected of students in remediation compared with the performance that would be expected of the same students had they not been placed in remediation was logistic regression. In these analyses, the relationship between placement test scores and passing rates in core curriculum courses was examined. Results are provided on pages 10 through 13. Similar to the linear regression, the performance of students above the cutoff scores who were not placed in LS is shown. A separate regression is shown for students who scored below the placement cutoff score and who were placed in LS. As with the previous analyses, the relationship for those above the cutoff can be extrapolated to predict the performance of LS students had they not been placed in remediation. Table 2 indicates the sample sizes and passing rates for the samples used in the logistic regression analyses. The sample sizes for the logistic regressions are larger than those for the linear regressions because the logistic regressions included students who received grades of W (as not passing) in addition to those with grades of A, B, C, D, F, and WF. Table 2 Sample Sizes and Passing Rates for Logistic Regressions English 1101 No LS LS CPE ______________________ N % C or Better 2749 73.0 1574 65.1 COMPASS _______________________ N % C or Better 2706 71.8 1460 67.3 Math 1101 No LS 992 67.6 LS 734 48.4 854 63.6 780 55.6 Math 1111 No LS 918 46.3 LS 1391 41.1 510 43.9 688 39.2 Political Science 1101 No LS 1996 65.9 LS 911 57.1 1657 63.1 615 52.4 Page 10 of 15 June 2005 RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND PASSING RATES IN CORE CURRICULUM COURSES BY LS PLACEMENT: LOGISTIC REGRESSION FY2000 Math 1101/CPE 1 0.9 0.8 0.7 Passing 0.6 Rate 0.5 (>D) 0.4 0.3 0.2 0.1 0 55 65 75 85 95 CPE Score Math 1101/COMPASS 1 0.9 0.8 0.7 Passing 0.6 Rate 0.5 (>D) 0.4 0.3 0.2 0.1 0 15 25 35 45 55 65 75 85 95 COMPASS Score Page 11 of 15 June 2005 RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND PASSING RATES IN CORE CURRICULUM COURSES BY LS PLACEMENT: LOGISTIC REGRESSION FY2000 (CONTINUED) Math 1111/CPE 1 0.9 0.8 0.7 Passing 0.6 Rate 0.5 (>D) 0.4 0.3 0.2 0.1 0 55 65 75 85 95 CPE Score Math 1111/COMPASS 1 0.9 0.8 0.7 Passing 0.6 Rate 0.5 (>D) 0.4 0.3 0.2 0.1 0 15 25 35 45 55 65 75 85 95 COMPASS Score Page 12 of 15 June 2005 RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND PASSING RATES IN CORE CURRICULUM COURSES BY LS PLACEMENT: LOGISTIC REGRESSION FY2000 (CONTINUED) English 1101/CPE 1 0.9 0.8 0.7 Passing 0.6 Rate 0.5 (>D) 0.4 0.3 0.2 0.1 0 45 55 65 75 85 CPE Score English 1101/COMPASS Passing Rate (>D) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 10 20 30 40 50 60 70 80 90 100 COMPASS Score Page 13 of 15 June 2005 RELATIONSHIP BETWEEN PLACEMENT TEST SCORES AND PASSING RATES IN CORE CURRICULUM COURSES BY LS PLACEMENT: LOGISTIC REGRESSION FY2000 (CONTINUED) Political Science 1101/CPE Passing Rate (>D) 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 55 65 75 85 CPE Score Political Science 1101/COMPASS 1 0.9 0.8 0.7 Passing 0.6 Rate 0.5 (>D) 0.4 0.3 0.2 0.1 0 15 25 35 45 55 65 75 85 95 COMPASS Score Page 14 of 15 June 2005 Conclusion For each test and subject area, students who were placed in remediation as a result of the placement test had higher grades than they would have been predicted to earn without remediation. The logistic regressions indicated higher than expected passing rates for LS students in all subjects except English 1101 for students placed with the CPE. In many cases, students who were placed in remediation had higher performance than students near the cutoff score who were not placed in remediation. Especially in mathematics, the results provided evidence that students slightly above the cutoff score would have benefited from placement in LS. Based on this as well as other information indicating that the cutoff scores for mathematics were too low, the System minimum placement scores in mathematics were raised in 2002 from 75 to 79 on the CPE and from 30 to 37 on COMPASS Algebra. In Math 1111 (College Algebra), students placed in LS and those near the cutoff but not placed in LS were predicted to get grades less than "C" and have a passing rate less than 50%. However, even this low performance for students placed in LS was higher than would have been expected without remediation. (The situation appears to have improved. In the most recent study, based on students taking core curriculum courses in FY2003, passing rates for Math 1111 increased 8 percentage points for students with System-mandated LS mathematics requirements and 8.8 percentage points for non-LS students.) In most of the comparisons, placement in remediation appeared to have lowered the correlation between test scores and grades, a desirable result, suggesting that remediation was effective in helping students overcome deficiencies that would have resulted in low performance. For more information, contact: Dr. Kathleen Burk Consultant Office of Strategic Research and Analysis Board of Regents of University System of Georgia 270 Washington St., SW Atlanta, Georgia 30334 Kathleen.Burk@usg.edu Page 15 of 15 June 2005