Special Examination Report No. 15-11

February 2016

Georgia Department of Audits and Accounts
Performance Audit Division
Greg S. Griffin, State Auditor Leslie McGuire, Director

Why we did this review
This special examination of virtual charter schools was requested by the House Appropriations Committee. The Committee asked that we provide information on selected accountability controls, selected measures of academic performance, and funding models. Specifically, we were asked about state controls over enrollment, teacher qualifications, and course rigor. The committee also requested information on the schools' retention of students, course completion and success rates, and rigor of course grading practices. Finally, the committee asked if there were other funding models for virtual schools that would be considered a best practice.
About Virtual Charter
Schools
The State Charter Schools Commission has provided charters to three fully online schools Georgia Connections Academy, Georgia Cyber Academy, and Graduation Achievement Charter High School. Two schools operate K-12 programs, while the third provides only high school courses. Combined, the three schools had over 19,000 students enrolled in October 2015 and received approximately $104 million in state funds for the 2015-16 school year. Like other state-chartered schools, they receive no local funding.

Virtual Charter Schools
Requested Information on Virtual Charter Schools
What we found The three state-chartered virtual schools have higher student turnover and lower academic performance than comparable brick and mortar schools. Our findings are largely consistent with studies by other state agencies and with research on fully online virtual schools in other states.
The virtual charter schools had higher student mobility between school years and during the school year. Virtual charter school students completing the 2013-14 school year were less likely than those in a comparable brick and mortar school to enroll in the same school the following year. They also had more turnover during the school year, with withdrawal rates between 27% and 46%. School officials noted that the reasons students enroll in virtual schools often lead to turnover. Some have health issues or have fallen behind academically and intend to return to their home school.
The virtual charter schools have had difficulty meeting some of the academic goals in their charters. The three schools were below their comparison district (state average) on their College and Career Ready Performance Index (CCRPI). Also, an analysis comparing their students' performance with that of similar students found that the virtual charter schools had less positive impact than expected for most subjects. However, a second analysis that compared the schools to similar schools found that two met expectations.
The grading practices of the virtual charter schools varied by school and course. All three virtual charter schools had lower course success rates (i.e., percentage of students with a passing grade) than comparable brick and mortar schools. By comparing course success rates and performance on state assessments, we found that two virtual charter schools had grading practices that were often similar

270 Washington Street, SW, Suite 1-156

Atlanta, Georgia 30334

Phone: (404)656-2180

www.audits.ga.gov

or more rigorous than comparable schools. The other school's practices were frequently similar or less rigorous.
Regarding the state's accountability controls, course rigor for all public schools is assured in a similar manner. The state sets academic standards, mandates the use of state standardized assessments, and uses the results to assess school and system performance. Like other charter schools, the virtual charter schools are subject to additional oversight--the charter can be terminated or not renewed based on failure to attain academic goals.
We did find that improvements were needed in the controls related to student enrollment counts and verification of teachers meeting federal standards.1
Student Enrollment To be included in the count used for state funding, GaDOE requires that a student be present at least one of the ten days preceding the "count day." However, the Georgia Department of Education (GaDOE) has not defined the term "present" for a virtual school setting. As a result, each school has developed its own definition and method of documentation.
Prior to the special examination, GaDOE had detected inconsistencies between the number of students in the FTE count and other student records. In late 2015, the Governor's Office of Student Achievement audited nine schools (two virtual charter schools) with the most inconsistencies. The two schools had incorrectly included students who enrolled but never attended school. Our review also found that the October 2014 FTE counts were not fully supported by available documentation.
Teacher Qualifications The state has no specific qualification requirements for virtual charter school teachers; however, GaDOE monitors the federal requirement that teachers in core subject areas be highly qualified. We found that the information system used to determine the status of these teachers does not contain reliable data. Specifically, there are teachers of core courses who were not included in the information system and others who were reviewed for the incorrect subject area. These errors were largely the result of the schools not updating information when new teachers were hired or teachers were reassigned to different subject areas.
We did not identify a best practice funding model for virtual schools. Like Georgia, most states continue to use some type of student enrollment count. While eight of the 31 states with a fully online virtual school have made course completion rates at least a part of their funding model, research has not yet proven that the model has improved education.
What we recommend Most of the report findings were intended to provide information requested by the House Appropriations Committee and did not result in recommendations. However, we do recommend that GaDOE define "present" in a virtual education setting for purposes of enrollment counts and ensure that methods to assess teacher qualifications relies on accurate data. The virtual charter schools should ensure that course and teacher assignment data provided to the state is accurate. See Appendix A for a list of recommendations.
Agency Responses: Two virtual charter schools stated that they should not be compared to brick and mortar schools because of differences in the student populations. The schools also noted that they operate with only a portion of funding received by the average public school. The state agencies generally agreed with the findings and recommendations. The State Charter Schools Commission stated that it is "committed to holding state charter virtual schools accountable for their academic performance as well as for their fiscal and operational compliance." Specific responses are included at the end of each finding.
1 The federal education law passed in December 2015 eliminated the requirement that core subject teachers be "highly qualified." GaDOE officials have not yet determined what type of oversight, if any, it will exercise over teacher qualifications.

Virtual Charter Schools

i

Table of Contents

Purpose of the Special Examination

1

Background

1

Virtual Education

1

Charter Schools

2

Financial Information

4

Selected Accountability Controls

6

Do sufficient controls exist to ensure that enrollment figures are accurate?

6

Do sufficient controls exist to ensure that virtual charter school teachers are

qualified?

9

What controls exist to ensure course rigor in virtual charter schools?

13

Selected Measures of School Performance

16

What are the student retention rates for the virtual charter schools?

16

What are the course completion rates for the virtual charter schools?

20

How do the virtual charter schools' course success rates compare to traditional

brick and mortar schools?

23

How does the rigor of the virtual charter schools' grading practices compare to

traditional brick and mortar schools?

27

Funding Models

32

How do other states fund virtual schools and are there any best practices?

33

Appendices

36

Appendix A: Table of Recommendations

36

Appendix B: Objectives, Scope, and Methodology

37

Appendix C: Georgia Connections Course Rigor

41

Appendix D: Georgia Cyber Academy Course Rigor

43

Appendix E: Graduation Achievement Center Course Rigor

45

Virtual Charter Schools

ii

Virtual Charter Schools

1

Purpose of the Special Examination
This review of virtual charter schools was conducted at the request of the House Appropriations Committee. This report answers the following questions:
1. Do sufficient controls exist to ensure that enrollment figures are accurate?
2. Do sufficient controls exist to ensure the rigor of virtual charter school courses?
3. Do sufficient controls exist to ensure that virtual charter school teachers are qualified?
4. What are the student retention rates for the virtual charter schools?
5. What are the course completion rates for the virtual charter schools?
6. How do the virtual charter schools' course success rates compare to traditional brick and mortar schools?
7. How does the rigor of the virtual charter schools' grading practices compare to traditional brick and mortar schools?
8. How do other states fund virtual schools and are there any best practices?
A description of the objectives, scope, and methodology used in this review is included in Appendix B. A draft of the report was provided to the State Charter Schools Commission, the Georgia Department of Education, and the three virtual schools chartered by the Commission (Georgia Connections Academy, Georgia Cyber Academy, and Graduation Achievement Charter High School). Pertinent responses were incorporated into the report.
Background

Synchronous is working together at the same time using such tools as chat rooms and online
conferences.
Asynchronous is the relay of information
with a time lag. Students may take online courses at unique times and communicate with teachers via email.

Virtual Education
Online or "virtual" learning is education in which both content and instruction are delivered primarily through the Internet. Public schools serving students in kindergarten through high school (K-12) typically provide online learning in one of the three following models.
Fully online schools In fully online schools, referred to as "virtual schools," students take their entire course load online. While students may be able to receive support at drop-in centers or other physical locations, they are not required to attend classes in physical school buildings. Communication between teacher and students typically occurs online or via telephone and may be both synchronous and asynchronous. Many virtual schools partner with private education management organizations (EMOs), such as K12 Inc. and Connections Academy.
Supplemental Online Programs Students may enroll in individual online courses to supplement or serve as part of a full-time program in a traditional school. Students may take these online courses at home or in designated spaces within physical school buildings. Many supplemental online courses

Virtual Charter Schools

2

are offered through state-run virtual schools such as the Georgia Virtual School.
Blended-learning models "Blended learning" combines online and face-toface instruction. Students may move between online and face-to-face course elements according to fixed rotations or on customized schedules and may spend varying portions of their time in either delivery format.
Public virtual schools are designed to expand learning options for K-12 students as well as to provide alternatives for students who may not be performing well in a traditional public school. Students attend virtual schools for a variety of reasons including illness, injury, behavioral issues, or extracurricular activities that make regular attendance in a single location difficult. In some cases, high school students attend because they are credit deficient or at risk of dropping out of school.

Charter Schools
A number of Georgia public K-12 virtual schools have been established as "charter" schools. A charter school is a public school of choice operating under the terms of a signed contract, or "charter," with either the state or local boards of education acting as an authorizer. Charter schools are publicly funded but organized and governed by a Georgia non-profit governing board. Charter authorizers are responsible for holding charter schools accountable to the terms of their charter contract.
State law (O.C.G.A 20-2-2065) exempts certain statutory and regulatory requirements in exchange for a higher degree of accountability in raising student achievement. Charter contracts contain "broad flexibility waivers" that exempt the schools from requirements related to such areas as teacher certification, maximum class sizes, and salary schedules. However, charter schools are not exempt from statutory and regulatory requirements related to health and safety, funding formulas, or accountability provisions. In addition, charter schools may not waive any aspect of federal law including the Elementary and Secondary Education Act (also known as the "No Child Left Behind Act,") the Individuals with Disabilities Education Act, and all applicable civil rights legislation.
This report focuses on the three public virtual schools in Georgia that serve students statewide and operate under a state charter. These schools include Georgia Connections Academy, Georgia Cyber Academy, and Graduation Achievement Charter High School. It should be noted that while Graduation Achievement Charter High School is classified as providing its curriculum fully online, curriculum is also delivered to a portion of its students in a blended-learning environment.
State Charter Schools
The State Charter Schools Commission of Georgia (SCSC) serves as the authorizer for the three virtual charter schools. SCSC has the power to approve or deny petitions for commission charter schools and renew, non-renew, or terminate commission charter school petitions in accordance with Georgia law. While SCSC's duties are set forth in law (O.C.G.A 20-2-2080 through 20-2-2091), SCSC describes its principal obligations as:
Reviewing charter school petitions for commission charter schools and assisting in the establishment of commission charter schools throughout Georgia;

Virtual Charter Schools

3

Developing and promoting best practices for charter schools and charter school co-sponsors in order to ensure that high-quality charter schools are developed and encouraged;
Promoting high standards of accountability for commission charter schools; and
Monitoring and annually reviewing the academic and financial performance, including revenues and expenditures, of commission charter schools and holding the schools accountable for their performance pursuant to the charter.
There are currently 25 state charter schools operating in Georgia. These schools are diverse in their structure, mission and service area. For example, state charter schools use a variety of curriculum models including online learning, expeditionary learning, and self-paced/individualized learning. Most schools serve students in a single district/county; however, the three fully online schools serve students from all districts in the state.
State-Chartered Virtual Schools
Similar to brick and mortar charters, virtual charter schools in Georgia are required to participate in state student assessments, are evaluated based on the state's performance framework, and are held accountable for their students' performance. Performance goals in charter contracts often use state assessments as a benchmark and frequently contain goals tailored to the online school's mission or student population.
There are currently three state-chartered virtual schools operating in Georgia: Georgia Connections Academy, Georgia Cyber Academy, and Graduation Achievement Charter High School. Each school contracts with a for-profit educational management organization (EMO) to provide services such as course curriculum, the online learning platform, and teaching staff.
Georgia Cyber Academy The school serves students statewide in grades K12. The mission of the school is "to provide an exemplary, individualized, and engaging education experience for students by incorporating school and community/family partnerships coupled with a rigorous curriculum along with a data-driven and student-centered instructional model." Prior to fiscal year 2015, the school operated as a virtual component of a "brick and mortar" charter school known as Odyssey School. Since July 1, 2014, Georgia Cyber Academy's activities has been governed by a separate governing authority and the school operated under the terms of a separate charter agreement with the SCSC. Georgia Cyber Academy contracts with K12 Inc. to provide all educational services including the online platform and curriculum, staffing, facilities, office support, management services, and technology services.
Georgia Connections Academy The school serves students statewide in grades K-12. The mission of the school is "to help each student maximize his or her potential and meet the highest performance standards through a uniquely individualized learning program." The school conducted its first year of operation in fiscal year 2012 as a state-chartered school. The school contracts with Connections LLC to provide educational materials, the online learning platform, staffing, purchasing, and other services for the school.

Virtual Charter Schools

4

Graduation Achievement Charter High School The school serves students statewide in grades 9-12. The mission of the school is "to provide historically underserved students with a flexible and highly individualized virtual high school experience." The school, previously known as Provost Academy, conducted its first year of operation in fiscal year 2013 as a state-chartered school. During fiscal years 2013-2015, the school contracted with, EdisonLearning Inc. for educational materials, the online learning platform, staffing, and other services. In fiscal year 2016, the contract with Edison Learning was terminated and the school entered into a new education services management contract with Edgenuity Inc. to provide similar services.

As shown in Exhibit 1, in fiscal year 2016, the three state-chartered virtual schools received state Quality Basic Education (QBE) funding based on 19,422 full-time equivalent (FTE) student enrollments in the 2015-16 school year. Over the past five years (fiscal years 2012-2016), enrollment in the three virtual charter schools had average annual increases of 32% from 6,686 students to 19,422 students.

Exhibit 1 State-Funded Full-Time Equivalent (FTE) Enrollments at the Virtual Charter Schools Increased an Average of 32% Annually, FY 2012-2016

Virtual School
Georgia Connections Academy

2012 FTE
287

2013 2014 FTE FTE
592 2,205

2015 FTE
2,944

2016 FTE
3,981

Avg. Annual % Increase
112%

Georgia Cyber Academy 6,399 10,188 12,288 12,431 13,699

23%

Graduation Achievement Charter High School

n/a

103

942 2,175 1,742

309%

Total

6,686 10,883 15,435 17,550 19,422

Source: 2012-2015 GaDOE QBE Allotment Sheets; 2016 from October 2015 FTE count

32%

Financial Information
Similar to traditional public schools, most charter schools receive funding from multiple sources including state Quality Basic Education (QBE) funding, local funding, federal grants, and fundraising or other charitable contributions. Statechartered schools, however, do not receive local funding.2 The majority of funding for state-chartered schools comes from state funds: QBE formula earnings and statefunded supplement to offset the loss of local funds.

QBE formula earnings The Quality Basic Education (QBE) Act of 1985 established the state's method for providing funding to local school systems through a series of calculations called the "QBE Funding Formula." Because the total amount of QBE funding is subject to funds appropriated by the General Assembly, the QBE funding formula, for practical purposes, is used to allocate funds among the state's local school systems. The amount of QBE funds earned by each school system, and therefore each charter school, is based on the number of full-time equivalent (FTE) students enrolled and on

2 One state-chartered virtual school has received approximately $51,000 in local funding resulting from contractual agreements.

Virtual Charter Schools

5

the certification levels and years of experience of the school's professional certificated staff. It should be noted that funding per student varies according to grade level and program categories (e.g., general education, students with disabilities, or gifted students).

State charter supplement Because no local money flows to state charter schools, additional state dollars are included in the QBE funding formula to partially offset the loss of local funds. The amount of the supplement for state charter schools is calculated using the following two components:
o State funds equal to the average amount of local revenue and state equalization grant funding for the five school districts with the lowest assessed valuation per student, with virtual state charter schools receiving only 2/3 of this calculated amount, and
o The statewide average of total capital revenue per full-time equivalent student. However, unlike brick and mortar state charter schools, virtual state charter schools do not receive this component of the supplement.

To fund operations for the State Charter Schools Commission, which manages the charter contracts, 2% of the state QBE earnings allotment is withheld from each state chartered school.

As shown in Exhibit 2, in fiscal year 2016, the three virtual charter schools were allotted approximately $104 million in state funds. This funding level was based on the reported total FTE count of 19,422 students and resulted in an average state funding level of $5,354 per FTE student. By comparison, the fiscal year 2015 average state and local funding per FTE for students not enrolled in a state charter school was approximately $8,263.

Exhibit 2 $104 million in State Funds Allotted to Virtual State Charter Schools, FY 2016

Virtual School
Georgia Cyber Academy
Georgia Connections Academy
Graduation Achievement Center

QBE Count 13,699
3,981
1,742

QBE

SCSC Charter SCSC Total State Funding/

Earnings Supplement Admin Fee Funding

FTE

$48,017,390

$26,813,938 ($1,500,759) $73,350,569

$5,354

$13,844,140

$7,781,313 ($433,247) $21,192,206

$5,323

$4,970,486

$4,662,008 ($192,548) $9,439,946

$5,419

Total

19,422 $66,832,016

$39,257,260 ($2,126,554) $103,982,721

$5,354

Source: GaDOE QBE Allotment Sheets

Virtual Charter Schools

6

Selected Accountability Controls

Do sufficient controls exist to ensure that enrollment figures are accurate?

Although the Georgia Department of Education (GaDOE) has developed a large number of automated controls to ensure the accuracy of Full-Time Equivalent (FTE) counts (i.e., student counts) reported by all public schools, additional improvements specifically associated with virtual charter schools are needed. Specifically, GaDOE should develop additional criteria related to attendance in virtual settings to ensure that the state-chartered virtual schools include only active enrolled students in the FTE count.

FTE count refers to the student count taken in October and March of each school year.
A school system's QBE funding is based on the
average FTE over three counts
(October, March October).

Virtual charter schools are required to adhere to the same standards as all public schools when reporting students in the FTE count. To be funded, students must be enrolled and present at least one of the 10 days preceding the count day. Because students enrolled in virtual schools generally do not interact with their teachers on a daily basis, other methods are necessary to determine if a student is "present." Currently, each school has developed its own criteria for a "present" student because GaDOE has not yet defined the term for the virtual school setting.
GaDOE's automated data controls
GaDOE's primary controls for ensuring enrollment accuracy are automated edit checks of FTE data submitted by school systems and year-to-year comparison of FTE changes within each program category. The edit checks identify missing data, invalid codes, and logical inconsistencies in the FTE data. They may reject incomplete or missing data or identify data abnormalities for review and resubmission by the school systems. In addition, year-to-year comparisons generate warning messages when a school system reports a significant change in the number of FTEs within a program.

GaDOE also compares FTE data to Student Record data (e.g., enrollments, withdrawals, absences, course grades) collected at the end of each school year. GaDOE requires that every student reported as active in the FTE count have at least one enrollment record in the Student Record data collection. However, these comparisons are intended to ensure that all FTE records have a matching, active enrollment at the time that the FTE count is finalized. The Student Record data is submitted after the school year ends, months after the two FTE counts are finalized.

While the data comparisons do not ensure the accuracy of the FTE data before it is used to determine enrollment-based funding, they do identify potential issues with a school system's FTE count that can be addressed in subsequent years. Based on these comparisons, GaDOE initiated a review of nine school systems with the largest number of these errors in the 2014-15 school year. Two of the three virtual charter schools (Georgia Cyber and Graduation Achievement) were among these nine systems. Both schools explained that these errors were due to students being "noshows" for the school year.

The "no-show" explanation for the high error rates indicates that neither school had sufficient procedures to identify actively enrolled students. Because only enrolled students who were "present" at least one of the 10 days preceding the count day should have been included in the FTE count, students who failed to "show up" after enrolling should have been automatically excluded.

Virtual Charter Schools

7

Varying Methods for Defining and Documenting Attendance
The structure of a virtual school necessitates a different method for defining and documenting student attendance; however, GaDOE has not defined what is meant by "present" for the purposes of the FTE count in a virtual school environment. We found that the three virtual charter schools have varying methods of determining whether a student has attended school and have varying methods of capturing attendance data. Georgia Connections has comprehensive requirements for a student to be considered present, and it also maintains documentation of student engagement. Graduation Achievement and Georgia Cyber have less stringent requirements.
Georgia Connections Academy Georgia Connections has established criteria for determining the attendance status of students specifically for the purposes of conducting the FTE count. For a student to be included in the FTE count, the student's learning coach must enter attendance hours into the online student management system and the student must complete a lesson during the 10-day count window. The primary documentation for attendance and for lesson completion is student management system logins. The login data includes the student's name, date and time of login, and the course content accessed and completed.
We analyzed login and lesson completion data from the 10-day period prior to the October 2014 FTE count and found that the data largely supported the number of students reported by Georgia Connections in that FTE count. Georgia Connections data shows that 3,765 students logged into the system and completed at least one lesson. This figure is within 1.7% of the number of students (3,833) reported in the FTE count.
Georgia Cyber Academy To be considered "present," Georgia Cyber only requires that a student's learning coach--typically their parent or guardian-- report their attendance into the Online Learning System (OLS). There is no requirement that students provide evidence of accessing educational material (e.g., logging into the OLS) or of completing lessons. Consequently, Georgia Cyber personnel stated that they do not maintain historical login or lesson completion data. As a result, we were unable to verify that students included in the FTE count showed evidence of attendance and logging into the OLS during the 10-day period prior to the count.
Because historic login data was not available, we attempted to obtain attendance data from the enrollment record for students included in the October 2014 FTE count. Similar to the GaDOE error report noted above, we found that 82 students included in the FTE count did not have an enrollment record. We reviewed student files for a sample of 10 of these 82 students and found that none of the files contained evidence that the students were present during any period of the 2014-15 school year. Additionally, the file for only one of the 10 students contained evidence of course enrollment for the 2014-15 school year. The files for the remaining students indicated that the last activity occurred in prior school years.
Information maintained in student files also indicates the attendance data included in the enrollment record reported to GaDOE may be inaccurate. For example, documentation for one student who was reported in the enrollment

Virtual Charter Schools

8

record as having no absences shows that the school withdrew the student because of excessive unexcused absences. Documentation for another student with no reported absences shows that the student had 19 unexcused absences in the first three months of the school year. Documentation for an additional two students with no reported absences indicates the students were never enrolled during the school year even though they were reported as active in both the enrollment record and the October 2014 FTE count.
Graduation Achievement Charter High School To be considered "present," Graduation Achievement requires students to log into the student management system in each course at least one hour per day. According to school officials, a student must complete one lesson to receive attendance credit. However, the school's management staff stated that for the purposes of the FTE count, students are only required to login to one course to be counted. In addition, minimum one-hour login times are not applicable for the purposes of the count.
We analyzed login data from the 10-day period prior to the October 2014 FTE count and found that almost 12% (221) more students were included in the FTE count than logged into the school's system. While Graduation Achievement reported 1,894 students in the October 2014 FTE count, the school's login data shows that only 1,673 students logged into the system at least once during the 10-day window. School officials stated that data integrity issues were one reason that it changed EMOs for the 2015-16 school year.
RECOMMENDATION
1. GaDOE should establish criteria for the term "present" in a virtual education setting.

GaDOE Response: "We will work with the State Charter School Commission and the Governor's Office of Student Achievement to define student attendance in virtual schools."
SCSC Response: "The SCSC will work with its state-level partners to review the recommendations of this examination and assist in the implementation of systematic improvements with the goal of increasing accountability for all state charter schools."
Georgia Connections Response: "The GACA board agrees with the recommendation that GADOE should establish criteria for the term "present" in a virtual education setting. As your report states, GACA has comprehensive requirements to define the term `present' and maintains documentation of student engagement. GACA's procedures and system meets the letter and spirt of GADOE's attendance policies. We will not deviate from them and recommend them to GADOE as the definition of `present' in a virtual school setting, whether in a public charter school or a school system virtual program."
Georgia Cyber Response: Georgia Cyber stated that the problem with the Fall FTE reporting were due to "one-time data migration issues" associated with a new student information system. It said that the errors were corrected in subsequent reports to GaDOE. It added that the data issues

Virtual Charter Schools

9

"resulted in the inaccurate reporting of 102 GCA FTE's as `nonfunded,' which would have [resulted in additional] funding had they been reported accurately. The net impact of these two errors combined was a loss of funding to GCA of approximately 58 FTE's." Georgia Cyber noted that an investigation by the Governor's Office of Student Achievement concluded that the school's actions were "appropriate to reduce the likelihood of future errors" and that "no further action on the part of the school is needed at this time."
Graduation Achievement Response: Graduation Achievement officials welcomed additional guidance from state officials regarding attendance and other issues that may be unique to virtual schools.

Do sufficient controls exist to ensure that virtual charter school teachers are qualified?
While broad flexibility waivers exempt the virtual charter schools from the requirement that their teaching personnel hold Georgia certificates, all three schools have policies that require their teachers to hold current Georgia teaching certificates. However, we found weaknesses in the state's primary method to determine whether teachers of core subjects are "highly qualified," making it difficult to confirm that teachers meet the requirements. We were unable to verify the "highly qualified" status for teachers of approximately 41% of the core courses taught in the 2014-15 school year. It should be noted that the methods used to make "highly qualified" determinations are used for all school systems, not just virtual charter schools.
Title II Part A of the No Child Left Behind Act requires that all teachers of core subjects be "highly qualified." Generally, this requires that teachers hold a bachelor degree, hold a valid Georgia teaching certificate, pass a content test (or otherwise demonstrate subject matter competence), and be teaching a course in the field in which they are certified. The state uses two methods to ensure that a system's core subject teachers meet the "highly qualified" requirement:
HiQ Data System The Georgia Professional Standards Commission (GaPSC) operates a data system that combines annual GaDOE data on teachers, subjects, and grade levels with its own teacher credentialing information. By comparing the information, the system determines whether the teacher is "highly qualified" for the reported subject area and grade level.

Recent Change to Highly Qualified Requirement
In December 2015, the new federal Every Student Succeeds Act (ESSA) eliminated the requirement that school systems use "highly qualified" teachers in core subjects to qualify for federal Title II funds.
State education officials have not yet determined what methods will be used, if any, to ensure that teachers are qualified. Most of the state's school systems have recently been granted additional operational flexibility, which includes determining the qualifications of their teachers. Most systems and charter schools already have the ability to hire teachers who do not have Georgia teaching certificates. When teacher qualification standards are set by a local school system, state oversight in this area would likely be minimal or eliminated.

Virtual Charter Schools

10

GaDOE's Title II-A Monitoring GaDOE provides oversight and technical assistance to school systems to ensure that they comply with various aspects of federal TitleII-A programs, including the "highly qualified" requirement for core subjects. GaDOE monitoring consists of desk and on-site reviews for a sample of schools each year.
We found weaknesses in the HiQ data system that make it difficult to determine if all core courses were taught by "highly qualified" teachers during the 2014-15 school year. Details of our review of these processes are discussed below.
HiQ Data System
While the HiQ data system indicated that 99.6% of the core courses in virtual charter schools were taught by "highly qualified" teachers, we found that the system did not include all necessary teachers and contained inaccurate information. As a result, we were unable to verify that teachers were "highly qualified" for 40.7% (510 of 1,253) of the core courses taught in the virtual charter schools during the 2014-15 school year. This does not indicate that the teachers were not "highly qualified," only that the state's oversight methods should be improved.
To determine "highly qualified" status of teachers, the HiQ data system relies upon information in GaDOE's Certified/Classified Personnel Information (CPI) system, updates to personnel assignments made by school systems, and its own teacher credentialing information. As described in Exhibit 3, a comparison of this information allows the system to determine if teachers in core subjects are highly qualified.

Exhibit 3 Highly Qualified Determination Largely Dependent on Data Entered by Virtual Charter School

Virtual Charter Scho ols
VCS enters personnel data in October/March

GaPSC's C ert if icat io n Infor ma tion
Database

GaPSC imports teacher credentials
into HiQ

GaDOE's Certified/ Classified Personnel Infor ma tion
(CPI)

GaPSC imports October teacher/ subject into HiQ

GaPSC's HiQ Data System

Teacher "Highly Qualified" Determination

Each school adds new teachers and corrects other errors (e.g., new teaching assignments since October CPI report)
Source: GaPSC interviews; GaDOE documentation and interviews

To test the accuracy of the HiQ, we obtained 2014-15 course data for the three virtual charter schools from GaDOE. Course data shows the teacher of record for each course taught during the school year. We compared the teachers and courses in this dataset

Virtual Charter Schools

11

with the data shown for the same teacher and subject in the HiQ system.3 (See Appendix B for a more detailed description of the analyses.)

As shown in Exhibit 4, a significant portion of courses were taught by teachers who could not be verified as "highly qualified" by the state's existing processes. We found that teachers of record were not always included in the HiQ system. We also found that teachers were assigned to courses in subjects for which they had not been reviewed for the "highly qualified" designation. These frequently appeared to be the result of the schools failing to edit HiQ information to reconcile conflicting information in the GaDOE CPI and course datasets. Details of our analyses are below.

Teacher Not Included in HiQ For 18% (221 of 1,253) of the core courses listed in the course data, we could not find a record for the teacher in the HiQ system. Specifically, the teacher's social security number was not present in the HiQ data. With the help of GaPSC, we determined that the missing data was frequently a result of the virtual charter schools failing to manually enter data or make corrections in the HiQ system to reflect scheduling or staffing changes in the HiQ system, though GaPSC also had not included contract teachers in its HiQ analysis. Details on the missing information and the reasons are discussed below.

o Graduation Achievement did not manually enter data for 14 of its 34 teachers, all of which were hired after the 2015 October CPI deadline. It also failed to make necessary corrections for three of its teachers.

o Georgia Cyber did not complete required manual entries for 13 of 277 core course teachers in 2015. It also did not make required corrections to the data after initial HiQ determinations were made. These corrections were

Exhibit 4 Missing or Inconsistent Data Limit Ability to Verify "Highly Qualified" Status

Reason HiQ Status Unknown
Teacher Not Included in HiQ
Teacher Determination for Wrong Subject

GA Connections

# Core Courses

% All Core Courses

GA Cyber Academy

# Core Courses

% All Core Courses

Grad Achievement

# Core Courses

% All Core Courses

Total

# Core Courses

% All Core Courses

4

1.3%

189

21.7%

28*

35.4%*

221

17.6%

9

3.0%

251

28.8%

18

22.8%

278

22.2%

Teacher Record Deleted from HiQ

5

1.7%

6

0.7%

0

0

11

0.9%

Total

18

5.9%

446

51.2%

46

58.2%

510

*These numbers include the 22 additional contract teachers GaPSC subsequently uploaded to HiQ for a determination. Source: GaDOE Course-Level Data from the Student Record and GaPSC HiQ data

40.7%

3 The HiQ system contains subject areas, not individual courses. For example, a HiQ record should indicate a subject listing of "Math" if GaDOE's course data record indicates Geometry. As part of our analysis, we developed a methodology to match the HiQ subject code and GaDOE's course code, which included manual verification when necessary.

Virtual Charter Schools

12

necessary to ensure an additional 33 core course teachers received a HiQ determination.
o Georgia Connections did not make necessary corrections for two of its 93 core course teachers.
o GaPSC did not include 22 teachers under contract with the Graduation Achievement's education management organization when it obtained CPI data for HiQ determination. GaPSC originally indicated that school systems were responsible for manually entering these teachers, but it has since changed its policy to include this type of teacher when pulling CPI data for future HiQ determinations. GaPSC retroactively ran these 22 teachers through HiQ to obtain a HiQ determination during the course of this special examination.
GaDOE Course and HiQ Subject Do Not Match For 22% (278 of 1,253) of the core courses taught, the course listed in GaDOE's course data did not match the subject listed in the HiQ data for the same teacher. For example, while GaDOE course data showed Teacher A was teaching a Geometry course, Teacher A was listed in HiQ as "highly qualified" for English Language Arts. While listed as highly qualified, the teacher's status was not evaluated for the correct subject. The HiQ determination depends on the accuracy of the subject included in CPI. If the subject is not correct, the HiQ determination will not be reliable. Both Georgia Cyber and Graduation Achievement had a significant percentage of core courses affected by this error.
Teacher HiQ Record Deleted For 0.9% (11 0f 1,253) of the core courses taught, the course listed in GaDOE's data had no matching HiQ record. The teachers associated with these courses had HiQ records that had been deleted by the school system during the correction process with no replacement entry created and no existing determination to verify teacher HiQ status.
GaDOE's Title II-A Monitoring Process
While documentation review to support a teacher's designation can occur, GaDOE's monitoring process is largely reliant on the results produced by the HiQ system. The "highly qualified" determinations generated by the HiQ system are presumed accurate by GaDOE and used in its monitoring processes. GaDOE's monitoring is not designed to check for the types of problems we identified.
GaDOE's Title II-A monitoring is designed to provide oversight of program implementation relating to Title II-A highly qualified teacher standards. Through an annual assessment, GaDOE categorizes school systems based on their risk for failing to comply with Title II-A requirements. It schedules monitoring visits and establishes other reporting or corrective action requirements based on this categorization. One component of this monitoring is to ensure school systems have documentation that supports the highly qualified status of their teachers, especially when the determination was based on the school's assertion (as opposed to GaPSC documentation.)
GaDOE could address these issues and obtain a more accurate assessment of school compliance with highly qualified requirements. Currently, GaDOE does not cross-

Virtual Charter Schools

13

check HiQ data with the most up-to-date teacher schedules or with its own course data, a process that would allow it to determine whether school systems made the required updates and entries to the HiQ system.
RECOMMENDATIONS 1. The virtual charter schools should complete required HiQ coding corrections and manual entries of core teachers not included in CPI cycle 1 data into the HiQ system.
2. GaDOE and PSC should ensure that they use the most up-to-date data provided by schools. Instead of the CPI data, the agencies should consider other sources that indicate the teacher of record for each course.

GaDOE Response: GaDOE said it will consider how best to respond to the recommendations.
GaPSC Response: GaPSC stated that it was in "substantial agreement" with the finding. It also agreed that "the accuracy of the information in the HiQ system depends on the quality of the CPI data submitted to it and on the diligent and accurate work of the monitored schools to ensure the HiQ system can make a true and accurate portrayal of the subjects taught by teachers. Since the HiQ system is intended to determine teachers' `highly qualified' status during the school year, the only data available at the time is the CPI report. Other sources are not available until after the school year has ended."
SCSC Response: As previously noted, SCSC said it would "work with its state-level partners to review the recommendations of this examination and assist in the implementation of systematic improvements with the goal of increasing accountability for all state charter schools."
Georgia Connections Response: "The GACA board agrees with the recommendation. While GACA's highly qualified teacher data had the highest accuracy rate at approximately 94%, we agree there is room for improvement and we will ensure fidelity to our processes when entering this data."
Georgia Cyber Response: "While the report identifies some data collection improvements that can be made to the HiQ process, we would like to emphasize that Georgia Cyber Academy has verified both through internal and external monitoring that 100% of its teachers during the FY15 school year were state-certified and Highly Qualified, as measured by the GA Department of Education."

What controls exist to ensure course rigor in virtual charter schools?
The methods used by the state to ensure rigor in virtual charter school courses are the same as those used for brick and mortar charter schools and similar to those used for traditional public schools. State agencies' controls are primarily associated with measuring student outcomes, instead of pre-approving or assessing the delivery of instruction. Like all schools, performance on state assessments is a significant part of a school's College and Career Ready Performance Index (CCRPI), but like all charter schools, virtual charter schools are also subject to academic goals and objectives that can affect the continuation of the charter.

Virtual Charter Schools

14

While there is no universally accepted definition, a rigorous course would likely result in students who demonstrate knowledge of state standards when taking standardized assessments. Rigor may also be associated with grading practices, specifically the relative difficulty of receiving a particular course grade in relation to the knowledge actually gained by the student. For example, if students in one school receive high grades and meet state standards for a course while students in a second school receive similar grades but fail to meet state standards on the test, the first school's grading practices would be more rigorous.
GaDOE and the State Charter Schools Commission (SCSC) do not subject virtual charter schools to additional controls regarding course rigor. Consistent with the treatment of traditional and other charter schools, the agencies do not review or approve the virtual charter schools' curriculum. The controls that are utilized focus on the expected outcomes of rigor performance on the Georgia Milestones (previously the CRCT and EOCT) and other national norm-referenced assessments (e.g., SAT, ACT). Like most schools, virtual charter schools are more likely to adopt controls that focus on inputs, such as comparing course curriculum to state standards and using prior year performance on assessments to adjust teaching methods or strategies in the following year.
State controls associated with course rigor include the following.
Charter's academic goals and measurable objectives The charters of all three virtual charter schools contain numerous academic goals and measures. For example, all three virtual charter schools have state assessment-related measures that require performance in all subjects and grade levels to improve each year or to exceed the statewide average. The schools also have academic measures associated with performance on norm-referenced tests, such as the ACT and SAT. The charters require each school to submit an annual report indicating its progress on the academic and other goals.
Based on many factors, including academic performance, the charters can be terminated or not renewed at the end of their term (June 2018 for Georgia Connections and Graduation Achievement; June 2019 for Georgia Cyber).
Assessments of school academic performance Like other public schools, the virtual charter schools receive a CCRPI score, much of which is based on student performance on the state assessments or other academic measures. As a charter school, the virtual charter schools have also been the subject of additional academic performance assessments. On behalf of the SCSC and GaDOE, the Governor's Office of Student Achievement has analyzed the performance of each charter school's students on state assessments, controlling for variables known to impact test scores. The results of these assessments can be used by the SCSC when considering whether a school's charter should be immediately terminated or renewed when the charter term ends.
The CCRPI scores and the GOSA analysis results are discussed in the finding on page 23.

Virtual Charter Schools

15

Accreditation The state requires that all three schools be accredited. Accreditation ensures the reciprocity of credits, course, and grade placements, but state officials stated that it does not provide assurance of rigor in a particular course. All schools are accredited by AdvancED, which has recently created standards specifically for virtual schools that consider the unique environment for teachers and students and the need for different types of resources for both. None of the three schools have yet to be subjected to a review based on the newer standards.
To ensure course rigor, all three schools indicated that their curriculum, while provided by a national education management organization, is aligned with the Georgia Performance Standards. One school stated its teachers were involved in the EMO's development of new courses for Georgia, while another noted that its board hired a consultant to ensure that the purchased curriculum was properly aligned with Georgia's standards. One school administers quarterly standard assessments that become predictors of a student's performance on the state assessment in the spring. The school official said that each year the school compares the results of the state assessments with its own benchmark assessments. Finally, all schools indicated that they compare the results of state assessments to the course grades given by their teachers.

Virtual Charter Schools

16

Selected Measures of School Performance

What are the student retention rates for the virtual charter schools?
Students in Georgia's virtual charter schools are less likely to return to the school the following year or to complete the current school year than students in a sample of comparable brick and mortar schools.4 The three virtual charter schools' retention rates and student mobility measures varied, as well as the reasons for student withdrawals.
Students can be negatively affected by changing schools, either during or between school years. Curriculum can vary among school districts causing a student's competencies or experiences to be misaligned with those of classmates in the new school. Additionally, different teachers and classrooms move at different paces, therefore, a student may come in ahead of, or behind, the new school's schedule. Schools and teachers may also have different teaching styles and methods, which may involve a learning curve for the new student which puts them at-risk of falling behind.
Mobility during the School Year
The Governor's Office of Student Achievement (GOSA) measured student mobility, or "churn rates," of all schools in Georgia during the 2012-13 and 2013-14 school years. This rate measures the percentage of a school's students who entered or exited during the school year and can be an indicator of the stability of a school's student population for the year. GOSA found that two of the three virtual charter schools5 had significantly higher rates of student mobility or "churn" than statewide averages.
Georgia Connections Academy During the 2012-13 year, the churn rate at Georgia Connections Academy (67.6%) was 50.7 percentage points higher than the statewide median rate of 16.9%. In the next year, the gap increased 70.8 percentage points as Georgia Connection's churn rate grew to 87.3% and the statewide median rate remained relatively unchanged at 16.5%.
Graduation Achievement Charter High School During the 2012-13 year, the churn rate at Graduation Achievement Charter High School (289.2%) was 272.3 percentage points higher than the statewide median rate. In 2013-14, this gap decreased to 74 percentage points as Graduation Achievement's churn rate decreased to 90.5%.
Our additional analysis of withdrawals found that students at the three virtual charter schools were less likely than students at comparable brick and mortar schools to complete the school year at the same school. As shown in Exhibit 5, 27% to 46% of students at virtual charter schools withdrew during the school year compared to 9% to 16% of students at comparable brick and mortar schools. This trend is consistent with research conducted in other states. For example, 49% of fully online students in Ohio withdrew during the year, compared to 20% in its eight largest school districts.

4 Comparable brick and mortar schools were identified for each virtual charter school based on similar percentages of economically disadvantaged students, students with disabilities, and students lacking English proficiency. Additional detail can be found in Appendix B on page 37.
5 Georgia Cyber Academy did not begin operations as a completely virtual school until 2014-15. It operated as part of Odyssey, which is a brick and mortar charter school, during the two previous years.

Virtual Charter Schools

17

Exhibit 5 Comparable Schools Have Lower Withdrawal Rates than Virtual Charter Schools, 2014-15 School Year

Georgia Connections Academy Comparison Schools

Georgia Cyber Academy Comparison Schools

Graduation Achievement Comparison Schools

0%

10%

20%

30%

40%

50%

Source: DOE Student Record Enrollment Data

While numerous studies have found that fully online virtual schools have high turnover, a Public Impact and National Association of Charter School Authorizers (NACSA) report commissioned by the Georgia State Charter School Commission found that none of the studies examined the reasons for these trends. Administrators at Georgia's virtual charter schools pointed to a number of reasons that may lead to a higher mobility rate for their students. Some students are not well-suited for a virtual education, lacking the discipline or the necessary education support from a learning coach (often a parent). Others enroll to satisfy temporary needs related to chronic health issues, work or sports activities, or family issues, and they may enroll in the virtual charter school with the intention to rejoin their zoned brick and mortar school the following year. One school serves many "at-risk" students who have fallen behind academically due to incarceration, drug or alcohol abuse, pregnancy, or having previously dropped-out of school.
In addition to the interviews, we compared the withdrawal reasons reported in student enrollment records for virtual charter schools and comparable brick and mortar schools. Two of the virtual charter schools and their comparable brick and mortar schools were most likely to cite transferring to another Georgia public school. However, as shown in Exhibit 6, the students at virtual charter schools are more likely to withdraw to attend home study and for reasons associated with dropping out of school such as lack of attendance and unknown reasons. Graduation Achievement is unique in that almost 80% of withdrawals were associated with lack of attendance and unknown reasons, while only 10% of withdrawals involved a transfer to another Georgia public school. School officials indicated that many of its students were already truant in their home districts before transferring to Graduation Achievement.

Virtual Charter Schools

18

Exhibit 6 Students Withdraw From Virtual Charter Schools for Different Reasons than Comparable Brick and Mortar Schools, 2013-14 to 2014-15

Virtual Charter Schools

19

Year-to-Year Retention Retention, which was also measured as the percentage of students who completed the 2013-14 school year and returned the next year,6 varied by school--64% at Graduation Achievement Charter High School, 62% at Georgia Connections Academy, and 73% at Georgia Cyber Academy. As shown in Exhibit 7, these retention rates are lower than the median retention rates of comparable brick and mortar schools.
Exhibit 7 Comparable Schools Have Higher Student Retention Rates than Virtual Charter Schools, 2013-14 to 2014-15 School Years

Georgia Connections Brick and Mortar Schools

Georgia Cyber Brick and Mortar Schools

Graduation Achievement Brick and Mortar Schools

0%

20%

40%

60%

80%

100%

Source: GaDOE Student Enrollment Data

Georgia Connections Response: The Georgia Connections board stated that a traditional school setting is not comparable to a fully online school and that comparing schools based on school demographics does not account for "student mobility, whether a student enrolled on-time in a virtual charter school, or the length of time a student spent enrolled in a virtual charter school." It stated that these factors "significantly impact the aggregate performance of a fully online school, whether it is operated as a charter school, non-charter school, or school system virtual program. These are factors that impact aggregate student performance across the online and blended learning spectrum."
The board added that brick and mortar schools are "not the appropriate comparison group" for an analysis of withdrawal rates. "A more appropriate comparison group would have been fully online school system virtual programs with open enrollment policies."
The board stated that "because there is a significant percentage of new students each year in a virtual charter school, a virtual charter school's aggregate performance is a reflection of students' prior educational setting. Student performance data should be disaggregated between new and returning students to reduce the lack of clarity regarding the effect of mobility." It added that the report should have disaggregated data by new and returning students to provide a more accurate picture of the school's performance.

6 Students who graduated during 2013-14 were not included in the calculation.

Virtual Charter Schools

20

Graduation Achievement Response: Graduation Achievement emphasized that the three virtual charter schools should not be compared to each other. Graduation Achievement targets at-risk students and operates only in grades 9-12. Officials also noted that virtual education is relatively new and that the lack of familiarity with the model affects student mobility. "It's great that families have the option of a virtual experience for their students...However, families are not always prepared to support their virtual learner, or understand the internal motivation involved in a virtual setting. Over time, as more and more students take courses virtually, whether via their brick and mortar school, a blended environment, or a totally virtual experience, there will be less mobility and more knowledge of the components needed for success."

What are the course completion rates for the virtual charter schools?

None of the virtual charter schools properly report the data necessary for a calculation of the percentage of course enrollments that result in a final course grade. If the state determines that a virtual school funding model should consider course completion rates, the data must be improved. We were able to calculate the percentage of students who completed a course segment (e.g., semester).7 During the 2014-15 school year, the three virtual charter schools had similar course segment completion rates, ranging from 72.3% at Graduation Achievement Charter High School to 83.5% at Georgia Connections Academy.

Course vs. Course Segment
A course may contains multiple segments. A year long course will often have two segments (first
semester and second semester), each with a grade reported to GaDOE. Completing a single segment in a multi-
segment course does not result in academic credit. A passing grade in the final segment is
necessary.

Course Completion Data Issues
While GaDOE's course-level records contain a field to indicate if a record is related to the final course grade, none of the virtual charter schools used the field correctly. Because many courses contain multiple segments (e.g., 1st and 2nd semester), more than one record may be created for a course. The "content completer" field should distinguish records associated with the final course grade from records associated with the partial course grade. Schools are instructed to enter "P" or "F" (pass or fail) if the record is associated with a completed course grade and "N" if the record is associated with a partial course grade. Although all of the virtual charter schools used two-semester marking periods for at least a portion of their courses, none of these records were marked as "N" to differentiate partial and completed course records.
While course data is not reliable to determine completion rates, GaDOE's data does allow for a determination of the number of enrollments in a course segment that were completed. Class records are created for each enrollment and indicate whether the student passes, fails, or does not complete the course segment. Depending on a school's scheduling practices and the course length, the record may represent the entire course or may represent only one of the multiple segments comprising a course. If a course is completed in one semester, the record indicates completion of the course. If a course takes an entire year to complete, two records will exist-one for each semester-for one course.

7 For example, a student who withdraws from school at the end of December may have grades recorded for six classes (course segments). However, if the courses require two semesters for completion, these grades do not signify course completion. Given limitations with GaDOE data, we were unable whether the grade indicated completion of an entire course or just one segment of the course.

Virtual Charter Schools

21

Course Segment Completion Rates
Due to the lack of reliable data, we were unable to determine the percentage of enrollments that resulted in completion of the entire course. However, we can determine the percentage of enrollments that resulted in the completion of a course segment, but these rates are likely higher than the percentage of enrollments that resulted in the completion of the whole course. Students may have enrolled in and completed only the first segment of a course without enrolling in the course's final segment. As previously noted, many students in virtual charter schools withdraw during the year. If a student withdrew after completing the first segment and prior to enrolling in the next, in this analysis the student would appear to have a 100% completion rate. In fact, the student's enrollment would have failed to produce a course completion.
As shown in Exhibit 8, the overall course segment completion rates at each of the three virtual charter schools were similar, ranging from 83.5% at Georgia Connections to 72.3% at Graduation Achievement. There was wider variation in the percentage of enrollments that were completed successfully (passing grade). The percentage of successful completions at Graduation Achievement was 26.1%, compared to 68.0% at Georgia Connections.
Exhibit 8 Three Virtual Charter Schools Have Similar Course Segment Completion Rates but More Variation in Successful Completion, 2014-15

Georgia Connections

Georgia Cyber

Graduation Achievement

0%

20%

Successful Course Segment Completions Incomplete Course Segments Source: DOAA Analysis

40%

60%

80%

100%

Unsuccessful Course Segment Completions

Similar trends and percentages were observed in the course segment completion rates for the core subject areas of English Language Arts, Math, Science, and Social Studies. The three virtual charter schools have similar rates, ranging from 84.3% at Georgia Connections, 73.0% at Georgia Cyber, and 72.5% at Graduation Achievement. Again, there is wider variation in the percent of course segments completed successfully, ranging from 68.9% at Georgia Connections to 25.0% at Graduation Achievement.

As shown in Exhibit 9, course segment completion rates for all courses and core subject area courses slightly decline as grade levels increase with the exception of 12th grade at Georgia Connections and Georgia Cyber. At Georgia Cyber, the 12th grade completion rate (97.8%) was 19 percentage points higher than the overall completion rate of 78.7%. At Georgia Connections, the 12th grade completion rate (93.6%) was 10 percentage points higher than the overall rate of 83.5%.

Virtual Charter Schools

22

Exhibit 9 With the Exception of Grade 12, Course Segment Completion Rates Slightly Decline As Grade Levels Increase, 2014-15

Source: DOAA Analysis
RECOMMENDATION 1. Virtual charter schools should correctly use the "content completer" field to differentiate course records associated with final course grades from records associated with partial course grades.

Virtual Charter Schools

23

Georgia Connections Response: "The GACA board agrees with the recommendation regarding the content completer field and is willing to collaborate with GADOE to ensure this data can be transferred between the school and state data systems so manual data entry is not required."
Graduation Achievement Response: As previously noted, Graduation Achievement emphasized that the three virtual charter schools cannot be compared to each other. Graduation Achievement targets at-risk students and operates only in grades 9-12.

How do the virtual charter schools' course success rates compare to traditional brick and mortar schools?
Each of the three virtual schools had consistently lower course success rates (defined as a grade of 70 percent or above) than a sample of comparable brick and mortar schools for all subject areas including the core subject areas of English Language Arts, Math, Science, and Social Studies. For all subject areas, students at Georgia Connections Academy and Georgia Cyber Academy were slightly less likely (12% and 13% respectively) to receive a passing grade than students at comparable brick and mortar schools. Students at Graduation Achievement Charter High School were significantly less likely (54%) to receive a passing grade.
While GaDOE data did not allow a reliable calculation of course completion rates (as discussed in the prior finding), we were able to determine course success rates. To determine course success, we included only those students enrolled in the virtual charter school or comparable brick and mortar schools for at least 65% of the school year.8 Because all students were enrolled for multiple marking periods, we were better able to identify the final course grade for analysis.
It should be noted that comparing course success rates across schools has limited value because course grades are inherently subjective. Different grading practices result in varying class grades, even for students who have the same knowledge of a course standards. As a result, standardized tests are commonly used to compare the knowledge gained by students taking the same course in different schools. In addition to course success rates, we have also included the three virtual charter schools' results on multiple outcome measures that use state assessments and other variables.
Course Success Rates
At each of the three virtual charter schools, the percentage of completed courses that were completed successfully was consistently lower in all grade levels and subject areas than at comparable brick and mortar schools. As shown in Exhibit 10, the course success rates for Georgia Connections, Georgia Cyber, and their comparison schools were relatively high. The rates for the virtual schools were typically above 80% and for the comparison schools typically above 95%. Course success rates at Graduation Achievement were significantly lower, ranging from 24% to 60%, while its comparison schools had median course success rates of 82% to 99%.

8 Limiting the analysis to those present for a "Full Academic Year," defined as 65% of the school year, ensures that course success (or lack of success) is associated with the appropriate school.

Virtual Charter Schools

24

Exhibit 10 For All Grade Levels, Virtual Charter School Course Success Rates Are Lower Than Comparable Brick and Mortar Schools

Source: DOAA Analysis
Each virtual charter school also had lower course success rates than comparable brick and mortar schools in the core subject areas. As shown in Exhibit 11, the rates for Georgia Connections and Georgia Cyber were lower but were typically within 15 percentage points. In all core subject areas, Graduation Achievement's course success rates were more than 50 percentage points lower than comparable schools.

Virtual Charter Schools

25

Exhibit 11 For All Core Subject Areas, Virtual Charter School Course Success Rates Are Lower Than Comparable Brick and Mortar Schools

Source: DOAA Analysis
Standardized Assessment Outcome Measures Through agreements with the GaDOE and the State Charter School Commission, the Governor's Office of Student Achievement (GOSA) has developed and implemented multiple measures that evaluate virtual charter schools' impact on student growth, proficiency, and college and career readiness the College and Career Readiness Index (CCRPI), the Value-Added Analysis (VAA), and the Beating the Odds Analysis.
The College and Career Ready Performance Index (CCRPI) measures a school's performance in academic achievement, student growth, and achievement gap reduction. Academic achievement is measured using scores from state standardized assessments, post-high school readiness (e.g., ACT/SAT exam performance, career pathways), and graduation rate. Student growth is measured using Student Growth Percentiles which compare a student's growth in state assessment scores with other students with similar

Virtual Charter Schools

26

past achievement. Achievement gap reduction is based upon the school's achievement gap size (difference between the school's bottom 25% of students and the state average) and the yearly change in that gap.

The Value-Added Analysis (VAA) assesses state charter schools based on their ability to positively impact the student populations they serve. The VAA utilizes a model that includes statistical controls for observable student characteristics and prior academic performance to generate an "impact score" for each school. Using estimated impacts of prior standardized assessment scores and student characteristics, predicted test scores are calculated. These predicted scores are compared to the student's actual score to obtain the school's impact on student achievement.

The Beating the Odds Analysis, which is applied to all charter schools, uses a linear regression to hold constant school characteristics and develops a "predicted" CCRPI score based on these characteristics and how similar schools performed. Each school's actual CCRPI score is compared to the predicted score to determine if the actual score is statistically lower, equivalent, or higher.
Using these analyses, GOSA found that none of the three virtual charter schools outperformed their comparison district9 on either the CCRPI or the VAA (see Exhibit 12) during the 2013-14 school year. However, according to the VAA both Georgia Connections and Georgia Cyber exceeded predicted outcomes in elementary, middle, and high school English subjects. The VAA found that both schools' impact on this subject area exceeded the impact of their comparison district (the state). In addition,
Exhibit 12 No Virtual Charter School Outperformed Comparison Districts in CCRPI or Overall Value-Added Measures; No School "Beat the Odds"; 2013-14 School Year

Georgia Connections Academy

Graduation Achievement

Georgia Cyber Academy

Charter High School

CCRPI

School Score: 51.3 Comparison District: 68.4

School Score: 47.4 Comparison District: 68.4

School Score: 33.2 Comparison District: 68.4

Value-Added Analysi s

The average impact score for all subjects combined is below the comparison district in elementary, middle, and high school grade bands. For individual subjects, the impact score exceeds the comparison district in 5 of 18 subjects.

The average impact score for all subjects combined is below the comparison district in elementary, middle, and high school grade bands. For individual subjects, the impact score exceeds the comparison in 6 of 18 subjects.

For individual subjects, the impact score is below the comparison district in 8 of 9 subjects and not statistically different in the remaining subject.

Beatin g the Odds

While the school did not "Beat the Odds" it did meet expectations of performance.

While the school did not "Beat the Odds" it did meet expectations of performance.

The school did not "Beat the Odds" nor did it meet expectations.

Source: GOSA and SCSC reports

9 Because these schools serve students throughout the state, their comparison "district" is the state.

Virtual Charter Schools

27

the Beating the Odds analysis found that while none of the three schools "Beat the Odds," Georgia Connections and Georgia Cyber met expectations.

Georgia Connections Response: "For many of the same reasons stated above, the GACA board disagrees with comparative data presented. Comparing fully online schools to "virtual twins" in brick-and-mortar schools is not appropriate when examining course success rates. Recent research indicates the proficiency of students enrolling in fully online schools tends to be lower than the comparative group of schools, but this is not examined in the report."
"Additionally, the GACA board recommends the report compare the course success rates with those of the Georgia Virtual School (GAVS). While GAVS provides supplemental courses and not full-time online instruction, a December 2010 report by your division showed GAVS had a course success rate of approximately 80%. GACA's course success rates were greater than 80% in almost every grade level and core subject area. The GAVS website does not provide course success rates but this comparison would provide the public additional information."
Graduation Achievement Response: As previously noted, Graduation Achievement emphasized that the three virtual charter schools cannot be compared to each other. Graduation Achievement targets at-risk students and operates only in grades 9-12.

How does the rigor of the virtual charter schools' grading practices compare to traditional brick and mortar schools?
Virtual charter school grading practices are not consistently more or less rigorous than those of comparable brick and mortar schools. While virtual charter school students are less likely than those in comparable brick and mortar schools to meet standards on state assessments, they are also less likely pass their courses. The difference between the passing rate on the state assessment and the final course grade was smaller for several virtual charter school courses, resulting in courses with more rigorous grading than those in the comparable brick and mortar schools. Other courses had less rigorous grading.
As noted on page 14, rigorous grading practices are associated with the relative difficulty of receiving a particular course grade in relation to the ability to demonstrate knowledge of course content on a standard assessment. Consistent with other gap analyses, we examined the relationship between course grades and performance on state assessments. Specifically, we compared the percentage of students failing a course to the percentage not meeting state assessment standards. The smaller the difference between the two measures, the more rigorous the grading practices. We compared the grading practices of each virtual charter school with their sample of comparable brick and mortar schools. See Appendix B or the box below for a more detailed methodology.

Virtual Charter Schools

28

Method for Assessing Rigor of Schools' Grading Practices

We compared each virtual charter school to a group of schools identified as having comparable percentages of students who were economically disadvantaged, disabled, and lacking English proficiency. For each virtual charter school and group of comparison schools, we:
Calculated the percentage of students that received a failing course grade in 2013-14, including only those courses with an associated state assessment.
Subtracted the percentage of students that did not meet standards on the associated Criterion-Referenced Competency Tests (CRCT) or End-of-Course Tests (EOCT).
The resulting gap shows the rigor of a school's grading practices. To determine whether the virtual charter school or comparison schools had a more rigorous grading practices for a course, we:
Subtracted the median difference for each group of comparison schools from the difference of the respective virtual charter school to determine which had more rigorous grading practices.

Example 1: Virtual Charter School % Failing Class: 20%
% Not Meeting EOCT Standard: 43% Gap: -23%

Comparison Schools % Failing Class: 5% % Not Meeting EOCT Standard: 35%
Gap: -30%

The VCS has a more rigorous grading practices because its gap is the higher number. The gap difference is +7% (-23% minus [-30%]).

Example 2: Virtual Charter School % Failing Class: 20%
% Not Meeting CRCT Standard: 24% Gap: -4%

Comparison Schools % Failing Class: 10% % Not Meeting CRCT Standard: 7%
Gap: 3%

The comparison schools have more rigorous grading practices because their gap is the higher number. The gap difference is -7% (-4% minus 3%).

While student performance among the three virtual charter schools varied, their students were generally less likely to meet standards on the state assessments. The schools were also more likely to issue failing grades. Details for each school are included below.
Georgia Connections Academy
Most Georgia Connections course grading practices in elementary, middle school, and high school were similar to or more rigorous than those in the comparable schools.
CRCT Subjects As shown in Exhibit 13, Georgia Connections' grading practices were more rigorous than the comparable schools in 11 of 24 CRCT subjects, and in seven subjects the percentage difference in the performance gaps was less than 5%. The comparable schools were more rigorous in six instances. Georgia Connections grading practices appear to be more rigorous than comparable schools in 6th, 7th, and 8th grades.

Virtual Charter Schools

29

Exhibit 13 Grading in Several of Georgia Connections' Grade 3-8 Subjects were More Rigorous than Comparable School Courses, 2013-14

Comparable Schools More Rigorous

Similar Rigor (+/- 5% Difference)

Georgia Connections More Rigorous

4th English

3rd Science

3rd Math

3rd English 4th Math

3rd Social Studies
5th Math

Source: DOAA Analysis

4th Science

4th Social Studies
5th English
6th Social Studies

5th Science
5th Social Studies
7th Social Studies

6th English 6th Science

6th Math

7th Math

7th Science

7th English

8th English 8th Math

8th Science
8th Social Studies

Appendix C shows the basis for the differences. Georgia Connections is more likely to fail students in all six grades. The failure rate exceeded 20% in all subjects by the sixth grade, and only in third grade were the failure rates comparable. The percentage of Georgia Connections students failing to meet proficiency standards on the CRCT was higher than comparable schools in all but one subject (6th grade English). Georgia Connections' rate was within five points of the comparable schools in 8 of 24 subjects, including 4th, 6th, 7th, and 8th English.

EOCT Subjects Grading in Georgia Connections' English classes was more rigorous than those in comparable schools, while it was significantly less so in U.S. History. As shown in Exhibit 14, grading rigor for the remaining five courses was similar, with a percentage difference of five or less.

Exhibit 14 Grading in Five of Georgia Connections' Eight EOCT Courses are Similar in Rigor to Comparable Schools

Comparable Schools More Rigorous

Similar Rigor (+/- 5% Difference)

Georgia Connections More Rigorous

U.S. History
Source: DOAA Analysis

Economics

Biology
Coordinate Algebra

Physical Science
GPS Geometry

9th Literature

American Literature

Virtual Charter Schools

30

For most courses, Georgia Connections' percentage of students failing courses and failing to meet the proficiency standard on the EOCT was relatively similar to that of comparable schools (see Appendix C). In the English courses, Georgia Connections' students were slightly more likely to meet standards on the EOCT but were more than twice as likely to receive failing grades. In U.S. History, Georgia Connections' percentage of students failing to meet standards on the EOCT was significantly higher (23% vs. 55%).
Georgia Cyber Academy
The grading practices in most of Georgia Cyber's elementary, middle, and high school courses were similar or less rigorous than those in the comparable schools.
CRCT Subjects The rigor of the grading in 12 Georgia Cyber courses was nearly the same as those of comparable schools, with percentage differences of 5% of less (see Exhibit 15). These included three of four courses in grades 5, 6, and 8. Grading was less rigorous in nine courses, including all courses in grade 3. Compared to other schools, Georgia Cyber's rigor appeared to be higher in English courses than in social studies and elementary math.

Exhibit 15 Grading in Half of Georgia Cyber's Grade 3-8 Courses was as Rigorous as Comparable School Courses; Most Others were Less Rigorous

Comparable Schools More Rigorous

Similar Rigor (+/- 5% Difference)

4th English

4th Science

Georgia Cyber More Rigorous

3rd English

5th English

5th Science

3rd Math

3rd Science

3rd Social Studies

4th Math

4th Social Studies

5th Math

6th Social Studies

7th Social Studies

Source: DOAA Analysis

5th Social Studies
6th Math

6th English 6th Science

7th Science 8th Science

8th Math
8th Social Studies

7th English

7th Math

8th English

As shown in Appendix D, Georgia Cyber students were more likely than students in comparison schools to receive failing grades in all subjects and to fail to meet standards on the CRCT in all but 6th grade English. In four of eight subjects in grades 7-8, the course failure rate exceeded 20%, compared to none above 14% in comparison schools. However, the gap between Georgia Cyber students and comparison school students on the CRCT was generally closer

Virtual Charter Schools

31

in the higher grades. Georgia Cyber's rate was within five points of the comparison schools in 5 of 24 subjects, including 6th, 7th, and 8th grade English.
EOCT Subjects The grading practices in half of Georgia Cyber's EOCT courses was similarly rigorous as those in comparison schools, while the practices in the other half was less rigorous. As shown in Exhibit 16, the two English courses and the two math courses were about as rigorous, while the science and social studies courses were less rigorous.
Exhibit 16 Grading Practices in Half of Georgia Cyber EOCT Courses were as Rigorous as Comparable School Courses; Others were Less Rigorous

Comparable Schools More Rigorous

Similar Rigor (+/- 5% Difference)

Georgia Cyber More Rigorous

Biology

Physical Science

9th Literature

American Literature

U.S. History

Economics

Source: DOAA Analysis

Coordinate Algebra

GPS Geometry

The percentage of Georgia Cyber students failing courses was similar to the brick and mortar schools in non-math courses (see Appendix D). In the English courses, Georgia Cyber students were slightly more likely to meet standards on the EOCT and were equally or more likely to receive failing grades. In the other six courses, Georgia Cyber students were less likely to meet EOCT standards.
Graduation Achievement Charter High School
Graduation Achievement is a high school; therefore, no students take the CRCT.
EOCT Subjects As shown in Exhibit 17, grading practices in four of Graduation Achievement's EOCT courses were more rigorous than those in comparison schools, while two were less rigorous and two were similar. Graduation Achievement's two math courses were graded more rigorously, as was one social studies and one English course.
Appendix E shows that Graduation Achievement's students performed as well as comparable schools' students on the English EOCTs, but they were more likely to receive a failing grade in the course. More than 75% of students in both groups failed to meet standards in the math courses, but Graduation Achievement was significantly less likely to give the student a passing grade. Finally, no students received a failing grade in biology, despite more than 65% failing to meet the EOCT standards. Comparison schools performed better on the EOCT but still had 15% of students fail the course.

Virtual Charter Schools

32

Exhibit 17 Grading Practices in Half of Graduation Achievement EOCT Courses were Similar to Comparable School Courses; Others were Less Rigorous

Comparable Schools More Rigorous

Similar Rigor (+/- 5% Difference)

Graduation Achievement More Rigorous

9th Literature

U.S. History

Biology

Economics

Source: DOAA Analysis

American Literature

Physical Science

Coordinate Algebra

GPS Geometry

Georgia Connections Response: "For many of the same reasons stated above, the GACA board disagrees with comparative data presented. Comparing fully online schools to "virtual twins" in brickand-mortar schools is not appropriate when examining grading practices, even though the report's analysis shows GACA has the most rigorous grading practices of the virtual charter schools."
Graduation Achievement: Graduation Achievement stated that some students move to a virtual school expecting course rigor to be easier and that they will progress quickly. However, "the level of self-discipline and rigor is higher than their expectations."

Virtual Charter Schools

33

Funding Models

How do other states fund virtual schools and are there any best practices?
Virtual schools are often funded using the same model as a state's brick and mortar schools. In most states, the funding model is student-based instead of completion or outcome-based. A student-based model focuses on a virtual school's number of students, which is captured through count days or averages based on class rosters or log-in information. An outcome-based model focuses on successful course completion rates. There are no clear best practices for funding among states with fully online virtual schools.
State Funding Models Virtual school funding models in the 31 states that provide fully online virtual schools can be broadly grouped into two types, those focusing on the number of students and those focusing on the number of successful course completions. The models focusing on students can be divided into various subgroups based on the procedures used by the state to capture participation. These include attendance averages based on log-in information or completed assignments, specified count days, or enrollment averages derived from active classroom rosters. Some states used mixed models, using both students enrolled and completion rates. Exhibit 18 shows a breakdown of the funding models in the 31 states with a fully online virtual school.
Exhibit 18 Most states use student counts to fund fully online virtual schools, 2015
Considers Course Completion 16%

Student-based: Enrollment+Attendance
23%

Student-based: Enrollment Only
61%

Source: DOAA analysis
Each of the model types used in the 31 states offering fully online virtual schools are discussed in detail below.
Student-Based Funding Models Twenty-six states fund fully online virtual schools based on the number of students enrolled or attending the schools.

Virtual Charter Schools

34

These student-based methods are also most common for funding traditional brick and mortar schools.
o Enrollment Only Funding Of the states that provide fully online virtual schools, more than half fund the schools based solely on the average number of enrolled students. These 19 states use the number of students on a school's roster for specific "count" days or for periods such as the first 100 days, specific months, or the entire school year. In South Carolina, virtual school funding is based on average student enrollment during two periods, school days 1-45 and school days 1-135. In North Carolina, virtual school funding is based on enrollment on two specific count days.
o Enrollment+Attendance Funding Seven states add an attendance or student participation requirement to the enrollment model. The requirements to demonstrate attendance or participation vary by states. Some states, including Georgia, allow each virtual school to devise its own system of keeping attendance rather than mandating a method, while others define attendance evidence as log-in data or assignment completion rates. The actual funding calculation methods also vary. For example, Tennessee determines attendance averages by dividing the total number of days in attendance by the total number of days taught during a 20-day window of accountability. Kansas counts the number of minutes that students are logged into courses during two count-days and uses the data to determine full- and part-time student attendance.
Course Completion-Based Funding Models Five states consider course completion rates in funding calculations. For example, in Texas, state funding is generated when a student successfully completes the course and is either promoted to the next grade level or earns credit for the course. If the student does not successfully complete the course, no state funding is generated. In Wyoming, course assignment/milestone completion rates are converted into attendance data which is used to calculate funding. For example, if only half of the course milestones are completed, a student is considered to be in attendance for only half of the year and, therefore, earns only 50% of the available funds. Utah uses a funding method that combines enrollment and course completion rates. In this method, online course providers receive 50% of funding based on enrollment and the remaining 50% upon successful course completion.
Best Practices for Funding Models
Research has not identified any of the virtual education funding models as a best practice. All are subject to manipulation and require a system of controls to ensure they are operating as intended. While a student-based funding model provides an incentive to enroll students (and delay withdrawals), a completion-based model provides an equal incentive for grade inflation. A student-based model should be accompanied by policies to ensure that enrolled students are actively participating in courses, such as completion of assignments and/or minimum log-in time. If school funding is based only on those students who complete courses, policies must address when a non-participating student should be withdrawn prior to the end of the semester or school year. If funding is dependent on students receiving a passing grades, additional controls are necessary to ensure that teachers are providing appropriately

Virtual Charter Schools

35

challenging assignments and that final grades are reflective of the student's knowledge of the course content.
Most states use the same funding model for virtual schools and brick and mortar schools, but those with completion or mixed models use the model only for their virtual schools. While the models are often the same, not all states provide funding levels commensurate with brick and mortar schools. Some states provide a set percentage of funding, while others exclude specific revenue streams available to brick and mortar schools, such as transportation, nutrition and capital grants.

Georgia Connections Response: Georgia Connections noted that Florida school districts are "subject to completion-based funding when students enroll in part-time virtual instruction, such as taking an Advanced Placement course. A student must pass the course in order to generate a proportional share of an FTE value, typically representing 1/6th or 1/12th of a 1.00 FTE. GACA is not a supplemental course provider; it is a fully online public school providing a complete school experience to its students; it is not a bundle of online courses. Georgia should implement a system of student-based funding equitably funding all students regardless of the type of school they attend before attempting a completion-based system for public education."

Virtual Charter Schools

36

Appendix A: Table of Recommendations

Do sufficient controls exist to ensure that enrollment figures are accurate? (p. 6) 1. GaDOE should establish criteria for the term "present" in a virtual education setting.
Do sufficient controls exist to ensure that virtual charter school teachers are qualified? (p. 9) 2. The virtual charter schools should complete required HiQ coding corrections and manual entries of core teachers
not included in CPI cycle 1 data into the HiQ system. 3. GaDOE and PSC should ensure that they use the most up-to-date data provided by schools. Instead of the CPI
data, the agencies should consider other sources that indicate the teacher of record for each course.
What are the course completion rates for the virtual charter schools? (p. 20) 4. Virtual charter schools should correctly use the "content completer" field to differentiate course records associated
with final course grades from records associated with partial course grades.

Virtual Charter Schools

37

Appendix B: Objectives, Scope, and Methodology
Objectives This report examines the accountability controls and effectiveness of State Chartered Virtual Schools. Specifically, our examination set out to determine the following:
1. Do sufficient controls exist to ensure that enrollment figures are accurate?
2. Do sufficient controls exist to ensure the rigor of virtual charter school courses?
3. Do sufficient controls exist to ensure that virtual charter school teachers are qualified?
4. What are the student retention rates for the virtual charter schools?
5. What are the course completion rates for the virtual charter schools?
6. How do the virtual charter schools' course success rates compare to traditional brick-and-mortar schools?
7. How does the rigor of the virtual charter schools' grading practices compare to traditional brick-and-mortar schools?
8. How do other states fund virtual schools and are there any best practices?
Scope This special examination covered activity related to state chartered virtual schools and comparable brick and mortar public schools during the 2013-14 and 2014-15 school years. Information used in this report was obtained by reviewing relevant laws, rules, and regulations; interviewing state chartered virtual school staff, officials from the State Charter Schools Commission (SCSC), Georgia Department of Education (GaDOE), Georgia Professional Standards Commission (GaPSC); obtaining and analyzing GaDOE student-level data, virtual school log-in and lesson completion data, and GaPSC HiQ data; reviewing student files at each virtual school; and reviewing research.
We reviewed GaDOE student-level data for virtual charter school students and students attending comparable brick and mortar schools for the 2013-14 and 2014-15 school years. Relevant student-level data sets included the FTE Data Collection, Student Record Data Collection, Student Class Data Collection, and standardized assessment results (e.g., Criterion Referenced Competency Tests (CRCT) and End Of Course Tests (EOCT)). Identifiable student information (e.g., student name, address, Student Identification Number and Georgia Testing Identification Number) was either not obtained or was encrypted by GaDOE prior to transmission. This data was used to test accountability controls and to assess outcomes. While we concluded that the information was sufficiently reliable for the purposes of our review, we did not independently verify the data.
We reviewed teacher qualification information from GaPSC's HiQ data system, for teachers delivering course content to virtual charter school students during the 201415 school year. Relevant data included the teacher's identification number, subject area, job code, and HiQ determination. This data was primarily used to test the sufficiency of controls ensuring that teachers have necessary qualifications. We assessed this data and determined it was sufficiently reliable for this purpose.

Virtual Charter Schools

38

We interviewed the chief officer (e.g., superintendent, director, etc.) of each virtual charter school, officials at the SCSC, GaDOE, Governor's Office of Student Achievement (GOSA), and the GaPSC. These interviews informed all of the objectives. We also conducted interviews with officials at departments of education in other states to identify alternate funding models.
Finally, we reviewed results of multiple student achievement assessments conducted by GOSA on behalf of GaDOE and SCSC. These assessments include the College and Career Ready Performance Index (CCRPI), the Value Added Analysis (VAA) and the Beating the Odds analysis. These results were used to inform objectives related student outcomes at virtual charter schools and comparable brick and mortar schools.
Methodology
To determine if sufficient controls exist to ensure that enrollment figures are accurate, we compared records in the FTE Data Collection with enrollment records in the Student Record Data Collection to determine if all students reported as active in the FTE data had corresponding enrollment records in the Student Record. When discrepancies were identified, we interviewed GaDOE and virtual charter school staff to determine causes. We also tested the validity of the field in the FTE record that notes if a student was present at the school during the 10-day period prior to the count day only students in attendance during that time are eligible for state funding. For this test, we reviewed student log-in data and lesson completion data from the virtual charter schools to determine if this data supports the schools' assertion of student attendance during the 10-day period. We also reviewed in student files to determine if reported attendance is supported by file documentation.
To determine if sufficient controls exist to ensure the rigor of virtual charter school courses, we reviewed relevant state laws, rules, and regulations; interviewed staff from the GaDOE, the SCSC, and each of the state chartered virtual schools; and, reviewed results of various student achievement outcomes reports (e.g., CCRPI, VAA, and Beating the Odds) produced by GOSA on behalf of GaDOE and SCSC.
To determine if sufficient controls exist to ensure that virtual charter school teachers are qualified, we reviewed relevant federal and state laws, rules, and regulations; interviewed agency officials and staff from the State Charter Schools Commission (SCSC), Georgia Department of Education (GaDOE), and the Georgia Professional Standards Commission (GaPSC); reviewed GaDOE documents summarizing Title II-A monitoring criteria and protocol, and monitoring reports for the virtual charter schools; and examined GaDOE Certified/Classified Personnel Information (CPI) and course record data, and information from the HiQ data system for the state virtual charter schools.
Given that the HiQ data system is used as a starting place to verify adequate teacher qualification for assignments, we designed an analysis to test the reliability of the HiQ data used to verify teacher "highly qualified" status. We analyzed GaDOE's 2014-15 school year course-level records from the Student Record Data Collection to identify the teacher of record for each completed course. Course records associated with core courses (e.g., those courses for which federal law requires teacher to be "highly qualified) were isolated for further review. These records (teacher/core course combinations) were then compared to records in the HiQ data system determine if: 1) all teachers of record existed in the HiQ system, 2) all teachers of record had a HiQ determination, and 3) the CPI subject code in HiQ (that informs the HiQ system of

Virtual Charter Schools

39

subject taught) matched actual course taught as indicated in the corresponding teacher course record.
To identify comparable brick and mortar schools for each state chartered virtual school to be used in various analyses, we reviewed GOSA data to identify brick and mortar schools with similar percentages of economically disadvantaged students (+/5%), students with disabilities (+/- 5%), and students with limited English proficiency (+/-2%). Based on these criteria, we identified 82 comparable schools for Georgia Connections Academy, 141 comparable schools for Georgia Cyber Academy, and 25 comparable schools for Graduation Achievement Center.
To determine the student retention rates for the virtual charter schools, we analyzed year-to-year retention rates and retention rates during the school year for both the virtual charter schools and comparable brick and mortar Georgia public schools. For year-to-year retention, we obtained and analyzed GaDOE enrollment data for virtual charter schools and the comparable brick and mortar schools to identify the percentage of students who completed the 2013-14 school year at a school and returned to the same school the following year (the 2014-15 school year). The median retention rate for each group of comparison schools was reported. For retention during the year we review GOSA's analysis of "churn rates" for the virtual charter schools and for the state as a whole. In addition we determine withdraw rates by analyzing GaDOE's enrollment data to identify the percentage of students enrolled in virtual charter schools and comparable brick and mortar schools during the 201415 school year that withdrew from the schools during the year. Lastly, we also reviewed the enrollment records to identify the reasons for the withdrawals.
To determine the course completion rates for the virtual charter schools, we compared the number of 2014-15 school year course segment enrollments (e.g., both semesters of a two-semester course) with the number of course segment completions for each of the virtual charter schools. Enrollment data was obtained from GaDOE's Student Class Data Collection. Course completion data was obtained from courselevel records in GaDOE's Student Record Data Collection. The Student Record only includes records for completed course segments. Due to data reliability issues, we were unable to determine the percent of courses started (e.g., student enrolled in the first semester of a two-semester course) that were completed (e.g., the student enrolled in and completed the second semester of a two-semester course).
To determine how the virtual charter schools' course success rates compare to traditional brick-and-mortar schools, we analyzed 2014-15 school year course-level records from the GaDOE's Student Record Data Collection for the virtual charter schools and for each group of comparison brick and mortar schools. Each course record includes the numeric course grade awarded to the student with grades 70 and above being considered "passing" grades. Course records for only those students who were enrolled in the virtual charter schools or the comparable brick and mortar schools for the "Full Academic Year" were included in this analysis. According to the Governor's Office of Student Achievement, a student is considered to have been enrolled for a full academic year in a particular school if they were enrolled in that school for at least 65% of the academic year. In addition, to the extent that the data allowed, duplicate course records (e.g., records associated with both segments of a two-semester course) were removed from the analysis with records associated with the latter marking period (e.g., the 2nd semester) being retained and considered the "final" course grade.

Virtual Charter Schools

40

To determine how the rigor of the virtual charter schools' course grading practices compare to traditional brick-and-mortar schools, we compared 2013-14 standardized assessment results (e.g., CRCT and EOCTs) for students enrolled in each virtual charter school and group of comparable brick and mortar schools with the grades awarded to these students for the related courses. The "performance gap" was calculated by subtracting the percent of students who were awarded failing course grades from the percent of students who "did not meet standards" on the related standardized assessment. For example, if 20% of students were awarded failing course grades but 80% of students did not meet standards on the assessment test, the performance gap is 60 percentage points. Larger performance gaps are associated with lower rigor in course grading practices. The performance gaps for the virtual charter schools and the median gaps for each group of comparable brick and mortar schools were compared to determine if course grading practices were more or less rigorous at the virtual charter schools.
Assessment test results for the 2013-14 school year were reviewed because, at the time of the analysis, those were the latest available results. We limited the review to students who were enrolled in the virtual charter schools or in the comparable brick and mortar schools for the "Full Academic Year" (e.g., at least 65% of the academic year.) In addition, duplicate course records (e.g., records associated with both segments of a two-semester course) were removed from the analysis with records associated with the latter marking period (e.g., the 2nd semester) being retained and considered the "final" course grade.
To determine how other states fund virtual schools and are there any best practices, we reviewed relevant laws, rules, and regulations for Georgia and other states with virtual charter schools; we reviewed professional literature; and, we interviewed staff members of education agencies in 22 states with virtual charter schools.
This special examination was not conducted in accordance with generally accepted government auditing standards (GAGAS) given the timeframe in which the report was needed. However, it was conducted in accordance with Performance Audit Division policies and procedures for non-GAGAS engagements. These policies and procedures require that we plan and perform the engagement to obtain sufficient, appropriate evidence to provide a reasonable basis for the information reported and that data limitations be identified for the reader.

Virtual Charter Schools

41

Appendix C: Georgia Connections Course Rigor

Elementary and Middle School

Percent Failing Course

3rd - English 3rd - Math
3rd - Science 3rd - Social Studies
4th - English 4th - Math
4th - Science 4th - Social Studies
5th - English 5th - Math
5th - Science 5th - Social Studies
6th - English 6th - Math
6th - Science 6th - Social Studies
7th - English 7th - Math
7th - Science 7th - Social Studies
8th - English 8th - Math
8th - Science 8th - Social Studies
0%

10% 20% 30% 40%

Comparisons Georgia Connections

Percent Not Meeting Standard - CRCT

3rd - English 3rd - Math
3rd - Science 3rd - Social Studies
4th - English 4th - Math
4th - Science 4th - Social Studies
5th - English 5th - Math
5th - Science 5th - Social Studies
6th - English 6th - Math
6th - Science 6th - Social Studies
7th - English 7th - Math
7th - Science 7th - Social Studies
8th - English 8th - Math
8th - Science 8th - Social Studies
0%

10%

20%

30%

Comparisons Georgia Connections

40%

Georgia Connections Academy

Course Less Rigorous

Course More Rigorous

3rd grade English 3rd grade Math
3rd grade Science 3rd grade Social Studies
4th grade English 4th grade Math
4th grade Science 4th grade Social Studies
5th grade English 5th grade Math
5th grade Science 5th grade Social Studies
6th grade English 6th grade Math
6th grade Science 6th grade Social Studies
7th grade English 7th grade Math
7th grade Science 7th grade Social Studies
8th grade English 8th grade Math
8th grade Science 8th grade Social Studies

-40%

-30%

-20%

-10%

0%

10%

20%

30%

Georgia Connections Comparisons

Source: DOAA analysis

Virtual Charter Schools

42

Appendix C: Georgia Connections Course Rigor (Continued)
High School Percent Failing Course

Ninth Grade Lit & Comp American Lit & Comp Biology Physical Science U.S. History Economics Coordinate Algebra GPS Geometry

0%

20%

40%

60%

80%

Comparisons Georgia Connections

100%

Percent Not Meeting Standard - EOCT
Ninth Grade Lit & Comp American Lit & Comp Biology Physical Science U.S. History Economics Coordinate Algebra GPS Geometry 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Comparisons Georgia Connections

Georgia Connections Performance Gap

Course Less Rigorous

Course More Rigorous

Ninth Grade Lit & Comp American Lit & Comp Biology Physical Science U.S History Economics Coordinate Algebra GPS Geometry

-60%

-40%

-20%

0%

20%

Comparison

Georgia Connections

Source: DOAA analysis

Virtual Charter Schools

43

Appendix D: Georgia Cyber Academy Course Rigor
Elementary and Middle School

Percent Failing Course

3rd - English 3rd - Math
3rd - Science 3rd - Social Studies
4th - English 4th - Math
4th - Science 4th - Social Studies
5th - English 5th - Math
5th - Science 5th - Social Studies
6th - English 6th - Math
6th - Science 6th - Social Studies
7th - English 7th - Math
7th - Science 7th - Social Studies
8th - English 8th - Math
8th - Science 8th - Social Studies
0%

10%

Comparisons

20% 30% 40% Georgia Cyber

Percent Not Meeting Standard - CRCT

3rd - English 3rd - Math
3rd - Science 3rd - Social Studies
4th - English 4th - Math
4th - Science 4th - Social Studies
5th - English 5th - Math
5th - Science 5th - Social Studies
6th - English 6th - Math
6th - Science 6th - Social Studies
7th - English 7th - Math
7th - Science 7th - Social Studies
8th - English 8th - Math
8th - Science 8th - Social Studies
0%

10% 20% 30%

Comparisons Georgia Cyber

40%

Georgia Cyber Performance Gap
Course Less Rigorous

Course More Rigorous

3rd grade English 3rd grade Math
3rd grade Science 3rd grade Social Studies
4th grade English 4th grade Math
4th grade Science 4th grade Social Studies
5th grade English 5th grade Math
5th grade Science 5th grade Social Studies
6th grade English 6th grade Math
6th grade Science 6th grade Social Studies
7th grade English 7th grade Math
7th grade Science 7th grade Social Studies
8th grade English 8th grade Math
8th grade Science 8th grade Social Studies
-40%

-30%

-20%

-10%

0%

10%

Comparisons

Georgia Cyber

20%

30%

Source: DOAA analysis

Virtual Charter Schools

44

Appendix D: Georgia Cyber Academy Course Rigor (Continued)

High School Percent Failing Course

Ninth Grade Lit & Comp American Lit & Comp Biology Physical Science U.S. History Economics Coordinate Algebra GPS Geometry

0%

20%

Comparisons

40%

60%

Georgia Cyber

80%

100%

Percent Not Meeting Standard - EOCT

Ninth Grade Lit & Comp American Lit & Comp Biology Physical Science U.S. History Economics Coordinate Algebra GPS Geometry

0%

20%

Comparisons

40%

60%

Georgia Cyber

80%

100%

Georgia Cyber Performance Gap

Course Less Rigorous

Course More Rigorous

Ninth Grade Lit & Comp

American Lit & Comp

Biology

Physical Science

U.S History

Economics

Coordinate Algebra GPS Geometry

-60%

-40%

-20%

0%

20%

Comparison

Georgia Cyber

Source: DOAA analysis

Virtual Charter Schools

45

Appendix E: Graduation Achievement Center Course Rigor
High School
Percent Failing Course
Ninth Grade Lit & Comp American Lit & Comp Biology Physical Science U.S. History Economics Coordinate Algebra GPS Geometry 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Comparisons Graduation Achievement

Percent Not Meeting Standard - EOCT

Ninth Grade Lit & Comp American Lit & Comp Biology Physical Science U.S. History Economics Coordinate Algebra GPS Geometry
0%

20%

40%

60%

80%

Comparisons Graduation Achievement

100%

Graduation Achievement Performance Gap

Course Less Rigorous

Course More Rigorous

Ninth Grade Lit & Comp American Lit & Comp Biology Physical Science U.S History Economics Coordinate Algebra GPS Geometry

-80%

-60%

-40%

-20%

0%

20%

Comparisons Graduation Achievement

Source: DOAA analysis

The Performance Audit Division was established in 1971 to conduct in-depth reviews of state-funded programs. Our reviews determine if programs are meeting goals and objectives; measure program results and effectiveness; identify alternate methods to meet goals; evaluate efficiency of resource allocation; assess compliance with laws and regulations; and provide credible management information to decision-makers. For more information, contact
us at (404)656-2180 or visit our website at www.audits.ga.gov.