Technical college readiness courses : spring 2017 pilot evaluation

Technical College Readiness Courses Spring 2017 Pilot Evaluation
July 2017
Prepared by Rosaline Tio Sam Rauschenberg

Technical College Readiness Spring 2017 Pilot Evaluation Part I
Course Overview
Each year students across Georgia graduate from high school but lack the English and mathematics skills required for admission into postsecondary institutions. Prior to graduation these same students are unable to qualify for participation in Move on When Ready (MOWR) courses through local technical colleges. With this challenge in mind in fall 2016 Hall County Schools proposed that the State Board of Education (SBOE) create new English and mathematics courses focused on ensuring that these students attain the basic skills required to enroll in technical college courses and prepare for careers.
In December 2016 the SBOE approved the creation of two courses Technical College Readiness (TCR) English and Mathematics allowing Hall County Schools to pilot them in spring 2017. Eligibility is limited to students who have struggled in middle and high school English Language Arts (ELA) and/or mathematics courses and who score below the diploma course thresholds on the ACCUPLACER exam for admission into technical college.1
In spring 2017 Hall County Schools piloted TCR English and TCR Mathematics with 56 students in three high schools (East Hall High School Johnson High School and Lanier Career Academy [LCA]). In the 2017-2018 school year Hall County as well as additional districts across the state will offer the courses.
The SBOE requested that the Governor s Office of Student Achievement (GOSA) monitor course implementation and evaluate outcomes. The following analysis provides a high-level overview of the Hall County spring 2017 TCR English and TCR Mathematics pilot enrollment and preliminary outcomes as well as lessons learned from Hall County teachers and district staff.
Student Eligibility Criteria
TCR English can be offered as a third or fourth course option to 11th and/or 12th grade students who need significant reading support and meet all the following criteria
Have earned credit in Ninth Grade Literature and Composition and Have earned credit or are concurrently enrolled in American Literature and Composition and Scored less than 55 on the Reading Comprehension ACCUPLACER Placement Test (minimum
score required to enroll in Technical College System of Georgia [TCSG] diploma-level courses).
TCR Mathematics can be offered as a third or fourth course option to 11th and/or 12th grade students who need significant math support and meet all the following criteria
Have earned math credit in Coordinate Algebra/Algebra I and Have earned math credit or are concurrently enrolled in Analytic Geometry/Geometry and Scored less than 34 on the Arithmetic ACCUPLACER Placement Test (minimum score required
to enroll in TCSG diploma-level courses).
1 The ACCUPLACER is a computer-adaptive placement test developed by College Board to assess student preparedness in reading writing and mathematics for introductory postsecondary courses.
1 Page

Technical College Readiness Spring 2017 Pilot Evaluation Part I
While the course can count as one of four required courses for high school graduation it does not count as a math credit for admission into a University System of Georgia institution. As such the course is intended to be limited to students who will graduate but lack requisite skills for admission into diplomalevel courses in technical college.
Monitoring/Evaluation Questions
Prior to course implementation GOSA identified the following five evaluation questions that it will monitor and evaluate annually.
1. Are schools adhering to the eligilibility criteria 2. Do ACCUPLACER scores increase for students who enroll in TCR English and/or Mathematics
What percentage of students become eligible for postsecondary coursework at the end of the course 3. Are students who enroll in TCR English and/or Mathematics more likely to earn Move On When Ready postsecondary credit compared to students with similar prior test scores but who were just above the ACCUPLACER score threshold and took MOWR courses 4. Are students who enroll in TCR English and/or Mathematics more likely to graduate from high school within four years compared to students from 2016-2017 with similar prior test scores (either Algebra I/Coordinate Algebra or 9th Grade Literatures) who did not take the course 5. Are there any patterns in performance between school systems student subgroups prior academic performance or grade level
Given the limited number of students who participated in the pilot and the preliminary nature of the outcome data this analysis will focus on answering questions 1 2 and 5. When the remaining data become available GOSA will complete the analysis for the remaining evaluation questions.
Course Standards and Curricula
In fall 2016 Hall County Schools staff developed standards and curricula in collaboration with Georgia Department of Education (GaDOE) and Northwest Georgia RESA staff. Both courses pull standards from various grade levels in the Georgia Standards of Excellence that align with the skills needed for success in technical college. TCR English focuses on reading comprehension identifying main ideas making inferences distinguishing between direct statements and supporting ideas and using various sentence structures. TCR Mathematics focuses on numeracy algebra and geometry in a variety of contexts including number sense linear and non-linear relationships functions and their graphs and measurement and geometry.
Hall County provided teachers with proactive instructional support and a blended learning platform to allow them to personalize students learning paths and focus on standards where the students needed the most support. Students took the ACCUPLACER before in the middle and at the end of the courses.
2 Page

Technical College Readiness Spring 2017 Pilot Evaluation Part I

Pilot Student Characteristics2

Table 1 TCR Enrollment by Course and School3

Course

East Hall HS

Johnson HS

LCA

English

5

13

6

Mathematics

15

12

6

Total 24 33

Overall 56 students enrolled in TCR English or TCR Mathematics during the spring 2017 pilot in Hall County. o 24 students enrolled in TCR English and 33 students enrolled in TCR Mathematics. o 19 students from East Hall High School (one student took both courses) 25 students from Johnson High School and 12 students from Lanier Career Academy (LCA) participated in the spring 2017 pilot.
The largest share of students who took TCR courses were in 10th grade (see Table 2). o Most students who took TCR English were in 10th and 11th grade. o Most students who took TCR Mathematics were in 10th and 12th grade.

Grade
10th 11th 12th Total

Table 2 TCR Enrollment by Grade Level

TCR English Enrollment Percentage

TCR Mathematics Enrollment Percentage

37.5%

45.4%

41.7%

6.1%

20.8%

48.5%

100.0%

100.0%

Overall Enrollment Percentage 42.9%
21.4% 35.7% 100.0%

Figure 1 below displays the demographics of the 56 students who took either course. Figure 1 Student Demographics

2 Hall County Schools provided the demographic information used for this evaluation. 3 One student enrolled in both TCR English and TCR Mathematics.

3 Page

Technical College Readiness Spring 2017 Pilot Evaluation Part I
The average Reading Comprehension ACCUPLACER pre-test score was 39.3 approximately 16 points below the eligibility cutoff score for TCR English.
The average Arithmetic ACCUPLACER pre-test score was 23.8 approximately 10 points below the eligibility cutoff score for TCR Mathematics.
Outcomes
Adherence to Eligibility Criteria
Both TCR English and TCR Mathematics were intended to be a third or fourth course option for 11th and/or 12th grade students. However 42.9% of TCR students were in 10th grade and all three pilot schools had 10th grade students who participated in the TCR pilot. This occurred for two reasons. Some students were in 10th grade in terms of credits earned but had been in high school for more than two years while others were considered 11th grade students after the fall 2016 semester credits were counted.
GOSA and the GaDOE are discussing removing this criterion since students grade classification is not necessarily related to whether they have completed the prerequisite mathematics and/or ELA courses for participation in Move On When Ready (MOWR). For example a 10th grade student who has completed both high school math course prerequisites would be eligible for MOWR if his/her ACCUPLACER score is above the diploma threshold. As such the requirement should not force a school to postpone enrollment in TCR Mathematics until the 11th grade year to meet this requirement.
Most students met the remaining three eligibility criteria for either TCR English or TCR Mathematics. There were only a few instances in which students had not earned credit or were not concurrently enrolled in Analytic Geometry/Geometry or American Literature and Composition. In two cases students were initially enrolled in both the TCR course and the required course but then dropped the required course during the semester. Due to the quick turnaround time for the pilot implementation Hall County believes these exceptions occurred because not all counselors were aware of the TCR course requirements. In another instance school staff misidentified an incorrect course as a prerequisite course.
In two other cases participating students had ACCUPLACER pre-test scores that were above the cutoff score to determine course eligibility. One instance was simply an oversight error. In the other case the school leader noted extraordinary circumstances involving the ACCUPLACER pre-test administration for the student and believed it was in the student s best interest to remain in the TCR course.
To prevent future enrollment errors Hall County plans to implement a supplemental process after the drop deadline to double check that TCR students are enrolled in the correct courses or have already received credit for prerequisite courses. Hall County will also validate that ACCUPLACER pre-test scores are below the eligibility thresholds during this process. Lastly Hall County plans to conduct additional staff training at TCR schools so all staff associated with scheduling will be aware of course requirements. GOSA recommends that similar procedures be implemented in all school districts planning to teach these courses during the 2017-2018 and subsequent school years. Future evaluation analyses will monitor and include reporting on these enrollment qualifications and data.
ACCUPLACER Performance
The ACCUPLACER is a computer-adaptive placement test developed by College Board to assess student preparedness in reading writing and mathematics for introductory postsecondary courses. Technical colleges in Georgia began using the ACCUPLACER in 2016 for admissions testing.
4 Page

Technical College Readiness Spring 2017 Pilot Evaluation Part I
TCR English students take the Reading Comprehension ACCUPLACER and TCR Mathematics students take the Arithmetic ACCUPLACER. TCSG has established ACCUPLACER cutoff scores for diplomaand degree-level coursework. During the pilot the TCR courses focused on helping students surpass the cutoff score for diploma-level courses (55 on the Reading Comprehension ACCUPLACER and 34 on the Arithmetic ACCUPLACER). The cutoff scores for degree-level coursework are 64 on the Reading Comprehension ACCUPLACER 34 on the Arithmetic ACCUPLACER or 57 on the Elementary Algebra ACCUPLACER. Although the course did not initially intend on having students take the Elementary Algebra ACCUPLACER a number of math students surpassed the diploma requirements by midsemester so the teachers moved them forward to try to attain degree-level course requirements through that exam by the end of the course.
Overall 87.5% of students in the pilot increased their ACCUPLACER score after taking TCR English or TCR Mathematics. o 83.3% of students who took TCR English increased their ACCUPLACER score from pre-test to post-test. o 90.9% of students who took TCR Mathematics increased their ACCUPLACER score from pre-test to post-test.
71.4% of pilot students became eligible for postsecondary coursework by scoring higher than the ACCUPLACER cutoff score for Reading Comprehension or Arithmetic. Overall mathematics students were much more successful at attaining the benchmark than English students. Figures 24 on the subsequent pages display the impact data. o Eleven of 24 (45.8%) English students scored at or above the Reading Comprehension ACCUPLACER threshold to qualify for diploma-level coursework. Eight of these students also surpassed the degree-level threshold. o Thirty of 33 (90.9%) mathematics students scored at or above the Arithmetic ACCUPLACER threshold for diploma-level coursework. o Fifteen (45.5%) mathematics students scored at or above the Elementary Algebra ACCUPLACER threshold to qualify for degree-level postsecondary coursework in technical college.
Teachers and staff attributed the lower progress for English students to the amount of time required to remediate struggling readers relative to teaching discrete math skills.
55 of 56 students earned a passing grade in the courses. The only student that did not pass did not take the post-test. In some cases students made little or no progress on the ACCUPLACER but earned grades of B or higher in the course. While ACCUPLACER performance is one factor in a student s grade the schools should keep in mind that students who earned credit for another core course but still are unable to enroll in technical college continue to require significant focus to ensure they are ready for college or a career upon graduation. To address this issue Hall County and school districts that offer the courses in the future should provide and enforce more specific teacher grading criteria that places a strong focus and requirement on student ACCUPLACER growth during the courses.
5 Page

Technical College Readiness Spring 2017 Pilot Evaluation Part I
Figure 2 Reading Comprehension ACCUPLACER Student Progress from Pre- to Post-Test
Diploma Degree
24
students took the course.
11
students are now eligible to enroll in diploma-level courses.
8
students are now eligible to enroll in degree-level courses.
Each bar represents a student s pre- and post-
test score.

Pre-Test Score Post-Test Met Degree Level Post-Test Met Diploma Level Post-Test Did Not Meet Threshold

20

30

40

50

60

70

80

90

100

110

120

ACCUPLACER Reading Comprehension Score (Possible Score Range is 20 to 120)

6 Page

Technical College Readiness Spring 2017 Pilot Evaluation Part I
Figure 3 Arithmetic ACCUPLACER Student Progress from Pre- to Post-Test
Diploma

33
students took the course.
30
students are now eligible for diploma-
level courses.
Each bar represents a student s pre- and
post-test score.

Pre-Test Score Post-Test Met Diploma Level Post-Test Did Not Meet Threshold

20

30

40

50

60

70

80

90

100 110 120

ACCUPLACER Arithmetic Score (Possible Score Range is 20 to 120.)

7 Page

Technical College Readiness Spring 2017 Pilot Evaluation Part I
Figure 4 Elementary Algebra ACCUPLACER Student Progress from Pre- to Post-Test
Degree

29 of 33
math students took the Elementary Algebra
ACCUPLACER.
15
students are now eligible for degree-
level courses.
Each bar represents a student s score. Pretests were not taken in Elementary Algebra.

Met Degree Level Did Not Meet Threshold

20

30

40

50

60

70

80

90

100 110 120

ACCUPLACER Elementary Algebra Score (Possible Score Range is 20 to 120.)

8 Page

Technical College Readiness Spring 2017 Pilot Evaluation Part I
Patterns in Performance
Due to the small sample of students who participated in the pilot the current report does not examine differences in performance by demographic subgroup. However GOSA used GAAWARDS data to disaggregate ACCUPLACER performance according to students prior performance in English or mathematics.
For TCR English of the students with prior Ninth Grade Literature scores 95% were below proficient.4 In addition students with lower Ninth Grade Literature EOC scores were less likely to increase their ACCUPLACER score or qualify for diploma-level coursework than students with higher Ninth Grade Literature scores. 85.7% of Beginning Learners on the Ninth Grade Literature EOC increased their ACCUPLACER score compared to 90% of Developing Learners. Moreover only 28.6% of Beginning Learners qualified for diploma-level coursework compared to 60% of Developing Learners.5
For TCR Mathematics of the students with prior Coordinate Algebra scores 96.4% were below proficient.6 92.6% of students who scored below proficient on Coordinate Algebra Milestones EOC increased their ACCUPLACER score and became eligible for postsecondary coursework after taking the course. Students with lower Coordinate Algebra scores performed just as well or better than students with higher Coordinate Algebra scores. All Beginning Learners on the Coordinate Algebra EOC increased their ACCUPLACER score and qualified for diploma-level coursework compared to 88.9% of Developing Learners. Of the students who did not meet expectations on the Coordinate Algebra EOCT prior to Georgia Milestones (11 students) even students with the lowest Coordinate Algebra scale scores increased their ACCUPLACER score and qualified for diploma-level coursework.
In both TCR English and TCR Mathematics one student had previously met expectations on the Ninth Grade Literature EOCT and Coordinate Algebra EOCT respectively. In each case the student increased his/her ACCUPLACER score and became eligible for diploma- and degree-level coursework.
Lessons Learned
To identify lessons learned from challenges and successes during pilot implementation GOSA conducted phone interviews with Hall County teachers and district staff who participated in the spring 2017 pilot. This section highlights major themes identified from the interviews.
Critical Instructional Supports for Teachers
During the pilot teachers received a variety of instructional supports for implementation of the TCR courses. Prior to the start of the semester teachers received two days of training for the TCR courses that included taking the ACCUPLACER tests. During the semester Hall County Schools hosted weekly
4 GOSA used students most recent score on the Ninth Grade Literature EOCT or EOC. Georgia transitioned from EOCTs to EOCs in 2015. EOCTs had three performance levels while EOCs have four performance levels. GOSA could not identify prior Ninth Grade Literature scores for four TCR English students. 5 Beginning Learner is the lowest performance band on the EOC assessment followed by Developing Learner. Two students only had EOCT scores prior to Milestones and were excluded from this comparison. 6 GOSA used students most recent score on the Coordinate Algebra EOCT or EOC. TCR Mathematics students did not have any prior Algebra I scores. GOSA could not identify prior Coordinate Algebra scores for five TCR Mathematics students. GOSA identified any student classified as "Does Not Meet Expectations" on the EOCT or "Beginning Learner" or "Developing Learner" on the EOC as "below proficient."
9 Page

Technical College Readiness Spring 2017 Pilot Evaluation Part I
webinars with teachers to provide updates and allow opportunities for questions and feedback. The district also made a comprehensive collection of lesson plans available for teachers to use and modify as needed. Teachers appreciated having a supply of prepared lessons from which they could choose and adapt to suit their students specific needs.
The program coordinator also conducted frequent classroom observations in the three pilot schools. These observations allowed teachers to receive coaching support when needed as well as provided feedback to the district on the TCR courses and the effectiveness of implementation. Finally teachers also appreciated having technical support from the district in the classroom to support the blended learning environment.
Benefit of TCR Courses for Students
Teachers and district staff were unanimous in the belief that TCR courses benefited students. All respondents believed the TCR courses are changing students lives by opening the door to alternative pathways that they were previously could not attain or of which they were unaware. Respondents felt that the TCR courses gave students an opportunity to have a future beyond high school that otherwise would not have been possible. Teachers also felt that the TCR courses helped boost students confidence to achieve new goals. With this in mind schools should be careful to keep class sizes small to allow for personalization and limit the course enrollment to students who need significant learning supports to meet the Accuplacer goals.
Effective Strategies with Students
Teachers identified several instructional strategies that were effective with the TCR pilot students. The most commonly identified strategy was providing students with one-on-one attention to support their needs in English or mathematics. Teachers appreciated having smaller class sizes that enabled them to personalize instruction for each student. As part of the blended learning environment teachers found that a balance of teacher-led and computer-based instruction helped maintain student engagement and allowed each student to learn at his/her own pace.
Teachers also mentioned several strategies to help increase student engagement including
Dedicating time at the beginning of the course to develop strong relationships with students Connecting skills taught with real world applications Encouraging students to set their own goals during the course and Planning field trips to the technical college to increase student motivation.
Future Course Adjustments to Increase Effectiveness
The challenge cited most frequently by teachers and district staff was keeping students engaged throughout the entire semester. Many students exceeded expectations and surpassed the ACCUPLACER cutoff score by the middle of the semester particularly in TCR Mathematics. In the future teachers and district staff plan to reframe the course goals for students at the beginning of the course so students can remain motivated throughout the entire course. For example rather than setting the ACCUPLACER goal for diploma-level courses teachers can encourage students to also pursue the degree-level threshold to expand their course options. Some participants recommended keeping the TCR courses single semester courses rather than a full-year course to maintain student engagement.
10 P a g e

Technical College Readiness Spring 2017 Pilot Evaluation Part I
Advice for Other Schools
Teachers and district staff provided several pieces of advice for schools who will be implementing TCR courses in the future.
Identify the right students for the courses by talking with students prior to enrollment to assess motivation and understanding of TCR s purpose.
If possible administer the ACCUPLACER during the prior semester to allow the placement process to be more deliberate.
Keep class sizes small to foster one-on-one interactions between teachers and students. Select teachers who have taught in various classroom environments had prior success working
with struggling students and are willing to be flexible. Be prepared to focus on basic discrete skills in TCR English and TCR Mathematics to build the
foundational skills required to master course standards. Use a block/semester schedule for the courses so that course completion aligns with technical
college schedules. Encourage teachers to establish strong student-teacher relationships to foster success. Provide strong administrative and district support to teachers through observations feedback and
reflection.
Supports Available to Other Teachers and Schools
Teachers and schools who plan to implement TCR courses in the future will have access to multiple resources. Hall County has provided all TCR lesson plans to the GaDOE for inclusion in the Teacher Resource Link during summer 2017. Along with the lesson plans Hall County is providing teacher resource guides with additional activities ideas and best practices for teachers.
Teachers also mentioned using publicly available online resources such as practice ACCUPLACER tests available through College Board and blended learning platforms that align with the course standards. A few teachers also suggested reaching out to middle school teachers who may be teaching content that is similar to the TCR courses for additional lesson and activity resources.
Next Steps
Overall the pilot provided an important opportunity for struggling students to become eligible for postsecondary coursework. In addition it provided important lessons learned that can inform implementation in Hall County and other districts as they implement the courses in school year 20172018.
Going forward when outcome data for postsecondary coursework and graduation data become available for the students in the pilot GOSA will publish an addendum to this evaluation. For subsequent years GOSA will monitor enrollment data and evaluate student outcomes including ACCUPLACER performance graduation rates and postsecondary performance to evaluate TCR course outcomes for enrolled students across the state.
11 P a g e