Special Examination Report No. 19-15 December 2019 Georgia Department of Audits and Accounts Performance Audit Division Greg S. Griffin, State Auditor Leslie McGuire, Director Why we did this review This review of the Georgia Public Broadcasting Education Division was conducted at the request of the Senate Appropriations Committee. We were asked to evaluate GPB's educational offerings for teachers and students, identify where programs align to standards and evaluate investments, and provide information on teacher and student usage and outcomes. To answer these questions, we identified GPB's educational resources and the processes by which GPB determines content and level of investment for original and partnership programs. We also assessed to the extent possible, the impact of GPB's educational resources on Georgia's students and teachers. About Georgia Public Broadcasting Education Georgia Public Broadcasting is a state public media organization that provides media, digital, and educational programming to the citizens of Georgia. GPB operates nine public television stations and eighteen radio stations to deliver content across the state. The mission of GPB Education is to create, curate, and distribute quality educational programs and services. GPB Education provides digital educational content and streaming services to teachers and students through original content and partnerships with Discovery Education and PBS Learning Media. Georgia Public Broadcasting Requested Information on Educational Resources What we found Georgia Public Broadcasting (GPB) offers a wide range of original and partnership educational programs for students and teachers in Georgia. GPB's resources for students are aligned to state education standards and cover multiple subject areas for grades pre-kindergarten through high school. GPB also offers resources for teachers. GPB's decisions to create or update content are largely based on reasonable factors such as stakeholder input, usage data, and revised academic standards. GPB also considers resource availability, including staff time, and the need for external funding when making decisions. While GPB staff can anecdotally provide the rationale for individual content decisions, our review found that documentation for content decisions is not consistent nor readily available. Opportunities exist for better defining the content development and update process through consistent support documentation. In addition, GPB has experienced an upward trend in overall usage of its original and partnership resources. GPB has conducted surveys with teachers and received positive feedback on four original resources and some outreach sessions. However, additional information, such as more complete usage data, would be necessary to fully evaluate the impact of GPB's original and partnership resources. While we did not evaluate the methodologies used, several studies by digital education providers suggest positive student outcomes from using digital educational resources. Better usage data would allow GPB to make such outcome-based assessments. 270 Washington Street, SW, Suite 1-156 Atlanta, Georgia 30334 Phone: (404)656-2180 www.audits.ga.gov What we recommend This report is intended to answer questions posed by the Senate Appropriations Committee and to help inform policy decisions. While GPB uses reasonable factors, such as usage data, education standards, and teacher feedback, to make content decisions and has experienced an upward trend in usage, we identified opportunities for improvement. GPB could improve its documented support of content development decisions. In addition, GPB could consider ways to obtain more complete usage data and use outcomebased evaluations to assess the impact of educational content on students and teachers. Summary of Response: GPB indicated its agreement with the findings in the report. GPB stated that the findings show the "content development process has yielded high-quality digital learning resources that are aligned to academic standards and highly utilized by Georgia's students and educators." In addition, GPB noted the special examination "will help inform our existing procedures so we can continue to serve as Georgia's leading source for innovative digital learning resources." GPB Educational Resources i Table of Contents Purpose of the Special Examination Background Georgia Public Broadcasting 1 GPB Education 2 Educational Content 3 Financial Information 4 Requested Information Finding 1: GPB offers a wide range of educational programming to Georgia teachers and students that aligns to education standards. 7 Finding 2: GPB considers a variety of reasonable factors, such as needs and resource availability, when determining whether to create or update content, though the decisions are not always supported by documentation. 9 Finding 3: Usage levels for GPB's educational programs have increased and feedback GPB has collected from teachers for a few original resources and outreach sessions has generally been positive. Additional information would be necessary to fully evaluate the impact of its original and partnership educational resources. 11 Appendices Appendix A: Objectives, Scope, and Methodology 16 Appendix B: GPB Original Content Inventory 18 Appendix C: GPB's New Original Content and Content Updates 21 GPB Educational Resources ii This page left intentionally blank GPB Educational Resources 1 Purpose of the Special Examination This review of the Georgia Public Broadcasting's (GPB) teacher resources was conducted at the request of the Senate Appropriations Committee. Our examination focuses on the following questions: 1. What educational offerings does GPB provide to the citizens of Georgia and do they align with state educational standards; 2. What are GPB's processes for determining the content and the level of investment for original and partnership programs; and 3. To what extent do GPB's educational resources have an impact on students and teachers? A description of the objectives, scope, and methodology used in this review is included in Appendix A. A draft of the report was provided to GPB for its review, and pertinent responses were incorporated into the report. Background The Georgia Public Telecommunications Commission (GPTC) does business as Georgia Public Broadcasting (GPB). Georgia Public Broadcasting Georgia Public Broadcasting (GPB) is a state public media organization that provides radio, television, and educational programming to the citizens of Georgia. GPB delivers media content through nine public television stations that broadcast Public Broadcasting System (PBS) and locally-produced programs as well as eighteen radio stations that broadcast National Public Radio (NPR) news and information. GPB is administratively attached to the Board of Regents of the University System of Georgia (USG) and overseen by the GPB Board of Directors (the Board) . The Board is composed of nine members of the public who are appointed to four-year terms by the Governor. Since a 2001 amendment to O.C.G.A 20-13-2, Board members cannot be state or public school system employees or representatives. The Board meets four times a year and one meeting is focused on education and GPB staff present on the status of educational efforts. GPB's President & CEO reports to the Board and manages the daily operations (see Exhibit 1) . As of August 2019, there were approximately 125 full-time staff, and some additional number of part-time production staff. Reporting directly to the President & CEO is a Chief Content Officer, a relatively new position that oversees television programming, radio programming, production services, digital operations, and education. The role of the Chief Content Officer is to facilitate content alignment and integration across all platforms (radio, digital, etc.) . In March 2019, the Chief Content Officer reorganized the previously combined Digital and Education division into two separate divisions. GPB Educational Resources 2 Exhibit 1 GPB Organizational Structure GPB Commission Board Members President & CEO Sr VP External Affairs Chief Technology Officer Chief Content Chief Financial VP of Human Officer Officer Resources VP of TV Programming Director of Digital Content Director of Education VP of News (Radio) Director of Production Services Director of VP of Digital Operations & Ops & Sports Strategy Source: GPB Records Education GPB Education GPB Education is headed by a Director of Education who is a former educator and reports directly to the Chief Content Officer. The Director of Education oversees eight full-time employees, many of whom are former educators. The Education Project Manager develops original digital content and resources, including writing and editing live video events--Live Explorations--and managing social media and a monthly newsletter for teachers. An Early Childhood Education Manager oversees the Ready to Learn and Family Engagement Initiative and supervises the Early Learning Specialist who delivers PBS KIDS family workshops and professional development workshops to early learning educators. There are also two outreach specialists who travel around the state to conduct outreach and provide free technical support on GPB's resources and professional development training to Georgia educators and school administrators, as discussed on page 4. In addition, there are three staff within the division dedicated to production, design, and digital editing. The division may sometimes employ temporary staff, such as consultants, to provide additional support when developing specialized educational content such as the series on physics. The division also relies on staff from other units within GPB to produce original content. In 2018, the Education Division established an Education Advisory Group, which is composed of teachers from around the state who provide feedback to GPB Education. For the 2019-2020 school year, the advisory group included 18 members. GPB indicated that the members of the Education Advisory Group are required to attend a yearly meeting at GPB. GPB Educational Resources 3 Educational Content The mission of GPB Education is to create, curate, and distribute quality educational programs and services. To achieve this mission, GPB develops original content and partners with digital educational content providers, which Georgia students and teachers can access at no cost. GPB Original Content GPB Education has developed original educational content geared toward Georgia teachers and students since the 1990s. GPB Education develops content and instructional materials to align with state education standards, which have been adopted by the Georgia Board of Education as benchmarks for determining college and career readiness. These standards are revised and updated periodically. GPB coordinates with its community partners to create original content. These partners include the Atlanta Symphony Orchestra, the Humane Society, and state education entities such as the Department of Education (GaDOE), Governor's Office of Student Achievement (GOSA), and Department of Early Care and Learning (DECAL). According to GPB staff, these partnerships allow them to use their resources more effectively. In addition, the state partners are able to distribute their content through GPB's network. GPB has uploaded some of its original resources to the Department of Education's Learning Object Repository which serves as a digital library of educational content. As shown in Exhibit 2, GPB's content can be divided into four different resource types. Exhibit 2 GPB's Content Can Be Divided into Four Resource Types Source: GPB Records GPB Partnership Content GPB also delivers educational content through partnership programs. GPB contracts with Discovery Education (DE) to provide free access to educational content for all Georgia students and teachers across the state; this partnership has been maintained GPB Educational Resources 4 for approximately two decades. Since 2014, PBS Learning Media has been provided to GPB as part of its membership with PBS and is available for any member of the public to use. Exhibit 3 displays the different characteristics of these partnerships. Exhibit 3 GPB's Partnerships Provide Different Benefits and Costs Partnership Characteristic PBS Learning Media Discovery Education Content in all grade levels and all subject areas Aligned to state and national standards Content provided at additional cost Hosts GPB's resources Provides technical support to GPB Source: GPB, PBS, and DE documents Outreach Activities GPB engages in outreach activities to promote its educational content and provide technical assistance on GPB resources. As previously discussed, GPB has outreach specialists who conduct in-person training sessions that are tailored to the specified needs of educators, administrators, and other school staff (e.g., librarians, IT specialists). These sessions can cover original and partnership programs and include subject-specific topics. During the 2018-2019 school year, GPB conducted more than 160 outreach visits across Georgia; however, not all school districts have been reached. In addition to outreach sessions, GPB promotes the use of its original and partnership resources through social media posts on relevant content and upcoming events. In the last quarter of fiscal year 2019, GPB reported having nearly 5,000 followers on Twitter and over 1,800 followers on Facebook. GPB also operates a closed Facebook group for Georgia Studies teachers. Finally, it promotes content through its Education Matters blog and monthly newsletter for educators. Financial Information Funds from state appropriations, federal grants, and viewer contributions and donations, account for a significant share of GPB's annual operating income.1 During fiscal year 2019, GPB's revenues totaled approximately $35 million, which consisted of approximately $15 million (43%) in state general funds, $9 million (26%) in donations, and $4 million (11%) in grant awards from the Corporation for Public 1As an administratively attached agency, GPB receives state appropriations annually through the Board of Regents of the University System of Georgia. Contributions and donations are administered by the Foundation for Public Broadcasting in Georgia, Inc., which assists the GPTC in fulfilling its statutory responsibility. GPB Educational Resources 5 Broadcasting and U.S. Department of Education.2 Other income sources (such as other grant funds, rents and royalties, underwriting, etc.) make up the balance of GPB's revenue. As shown in Exhibit 4, GPB's total expenditures in fiscal year 2019 were approximately $39 million, and $2.5 million of GPB's total expenditures were attributed to the Education Division.3 Of the approximately $2.5 million, $1.96 million (78%) was state general funds. State funds were used to pay the annual licensing fee for Discovery Education (approximately $1.1 million), administrative expenses (approximately $641,000), and ongoing maintenance costs for the Georgia Studies Digital Textbook (approximately $264,000). The Education Division's remaining expenses were supported by federal grant funds and other funds, including approximately $61,000 in state grant expenditures from GaDOE and GOSA to support specific projects. Exhibit 4 GPB Education Division's Expenditures were Approximately $2.5 Million, FY 2019 Total: $39 million Production Total: $2.5 million Administration TV Broadcast Services Engineering Services News, Radio, & New Media Education Communication & Marketing Source: GPB financial records State Funds $1,962,139 Other Funds $418,933 Federal Funds $81,881 The $2.5 million expended by the Education Division represents only a portion of GPB's total expenditures on education-related activities. GPB indicated that there are additional production costs that are not accounted for when it develops educational resources. For example, GPB incurs expenses when the Education Division collaborates with other GPB divisions, such as digital and news, to develop its educational resources. Further, PBS Learning Media is provided as part of GPB's annual membership dues to PBS, but these costs cannot be isolated due to the way 2 The Corporation for Public Broadcasting is a private, non-profit corporation created by Congress that receives federal appropriations to invest in public broadcasting. 3 Total expenditures, which exceeds the approximation of total revenue for fiscal year 2019, includes prior year carry-over. GPB Educational Resources 6 membership costs are captured. In fiscal year 2019, total annual dues paid to PBS were approximately $4 million. GPB Educational Resources 7 Requested Information Finding 1: GPB's 24 original resources may include many individual assets. For example, the Georgia Stories video series includes over 125 videos and supplementary questions. GPB offers a wide range of educational programming to Georgia teachers and students that aligns to education standards. Through its original content and partnerships with PBS Learning Media and Discovery Education, Georgia Public Broadcasting offers educational resources in multiple subject areas for students in pre-K through high school and teacher professional development. All of GPB's original resources are developed to align with state education standards, though opportunities exist to make the standards alignment more visible on its website. Through its partnership with PBS Learning Media and Discovery Education, GPB provides thousands of free resources that are aligned to state and national standards. Other southeastern states provide both original and partnership content, but GPB is unique in providing Discovery Education content as a free resource to teachers and students statewide. GPB Original Resources Consistent with other southeastern states' public media organizations, GPB creates digital education resources for Georgia teachers and students (See Appendix B) . Of GPB's 24 original resources, 20 are intended for student use; the remaining four resources are for teachers and local communities.4 As Exhibit 5 shows, most of GPB's resources are video series and tend to focus on social studies. The emphasis on social studies allows GPB to meet a need for Georgia-specific history and social studies resources. In addition, GPB indicated that its science resources address a need related to the shortage of science teachers in the rural areas of the state. According to GPB, most of its original educational resources tend to be for middle and high school students because PBS provides numerous resources for pre-kindergarten and elementary school students through PBS resources. Exhibit 5 Most of GPB's Original Resources are Video Series and Focus on Social Studies Subject Total Social Studies 9 4 4 1 Science / STEAM 5 5 Multiple 3 1 1 1 Math 2 1 1 World Languages 2 2 English Language Arts 0 Total 20 6 12 1 1 Source: GPB Website 4 GPB's website currently hosts nine additional digital resources for students, which it plans to remove when its website redesign is completed in April 2020. GPB Educational Resources 8 All of GPB's educational resources for students align to state standards. For many of these resources, GPB has created detailed documentation on its website that indicates the standards each section of a resource is aligned to, allowing teachers to crosswalk the content to the standards. In other cases, information about standards alignment on GPB's website is not as visible. GPB's current website does not consistently show which standards a resource is aligned to. For instance, for some resources, the standards are not shown on GPB's website but found on PBS Learning Media's website, which means that the user may not be able to easily identify the standards. Partnership Resources In addition to original resources, GPB provides users with access to PBS Learning Media and Discovery Education (DE) partnership content. According to GPB, these partnerships supplement GPB's original content, helping to provide a more comprehensive set of educational materials for teachers and students. Both PBS Learning Media and DE provide thousands of resources covering a wide range of subject and grade levels. All partnership resources align to state and national standards. Unlike GPB's original content, these partnership resources are searchable online by state and national standards. Learning Media is a free resource repository managed by PBS. PBS resources are created by member stations, third-party producers or by PBS. PBS's website indicates that Learning Media provides over 118,000 digital resources to more than 1.8 million educators and 30 million students across the country. Some of GPB's original resources have been uploaded to Learning Media; these resources can be accessed from both GPB or Learning Media's website. PBS does not have a role in content creation from member stations, but it has conducted a review of GPB's resources on Learning Media to ensure that the resources are appropriate and aligned to standards. PBS offers an optional login which provides access to the additional professional development tools described below. Discovery Education is a paid service contracted for by GPB. The contract was originally with GaDOE and administered by GPB; now, the contract is managed and administered solely by GPB. Among southeastern states, GPB is the only public media organization to provide statewide access to DE. DE has a team dedicated to ensuring that all resources align to state and national standards. DE indicated that it provides GPB access to approximately 187,000 assets (which includes single videos and images instead of resource collections) . DE also provides GPB with additional technical support and training to GPB staff as requested. In order to access content, students and teachers must login. Other Resources In addition to student resources, GPB offers original and partnership teacher resources and educational community initiatives:5 Teacher Resources GPB offers digital resources for teachers. These teacher resources include the Education Matters blog which covers topics and trends in education as well as school-specific success stories. GPB also provides a 5 Because these resources are not intended for students, they do not necessarily have an education standards alignment. GPB Educational Resources 9 monthly education newsletter, which has a mailing list of approximately 44,000 educators. The newsletter provides lesson plans and information on free digital resources. In addition, GPB offers a teacher edition of its Georgia Studies Digital Textbook, for which approximately 5,000 teachers are registered users. PBS Learning Media and DE offer additional resources for teachers, including lesson plan creation tools, customizable dashboards, and discussion forums to connect with other teachers. For instance, PBS Teacher Line is a paid service that provides courses for teachers; DE also offers additional sessions on its content for teachers at an additional cost. Community Initiatives GPB conducts community initiatives for students, parents, and teachers, including competitions and workshops. In 2019, GPB hosted a PBS KIDS summer camp as well as community learning workshops for families. Theses workshops are part of the Ready-to-Learn Community Collaborative for Early Learning and Media Initiative which is funded by the U.S. Department of Education. Finally, GPB facilitates a PBS KIDS Writing Contest for students from kindergarten through third grade. Finding 2: GPB has received external funding from organizations, such as the Woodruff Foundation, SunTrust Foundation and Georgia Power, for its larger original resource collections. GPB considers a variety of reasonable factors, such as needs and resource availability, when determining whether to create or update content, though the decisions are not always supported by documentation. GPB's decisions to create or update content are largely based on factors such as stakeholder input, usage data, and revised academic standards. Often, the need for Georgia-specific content drives decisions about new content, while standards alignment drives decisions about content updates. Resource availability, including staff time, and the need for external funding are also considered when making decisions about content creation or updates. We found that documentation to support these decisions was not consistent nor readily available. Opportunities exist for better defining the content development and update process through consistent support documentation. As discussed below and shown in Appendix C, GPB described the factors it considered when making decisions about new original content or updating existing content. Content Development According to GPB, stakeholder input largely drives its decision to create original content that is Georgia-specific. This typically includes feedback from Georgia teachers through an education advisory group, social media, outreach sessions, and teacher conferences. For example, GPB's decision to produce virtual field trips was in response to an expressed need for bringing locations in Georgia to classrooms at no cost to teachers and students. GPB staff indicated that opportunities to secure additional funding through collaboration with different state and community partners such as the GaDOE can factor into decisions about content creation. Smaller projects--such as shorter video series--are typically funded through the Education Division's annual operating budget. Larger efforts that require GPB Educational Resources 10 expert consultants, third-party companies, or large content overhauls may necessitate external funding through grants. Finally, GPB indicated that it reviews educational resources available through its partnerships or in the general market to establish need and ensure it is not replicating existing material. Original Content Updates According to GPB, staff often will initiate content updates based on a review of usage data or following changes to state educational standards, in addition to stakeholder input. GPB also indicated that the availability of resources can dictate the extent to which GPB can update content. For example, GPB was able to overhaul its physics and chemistry series in part due to external funding from Georgia Power and GOSA which allowed it to hire teacher consultants and production staff who could create new content, scripts, and support materials. By contrast, GPB indicated that after observing declining usage levels, it decided it needed to update its Georgia Stories video series. However, due to its inability to secure additional external funding for the project, only the supplemental materials were updated but not video content. Opportunities for Improvement While staff can discuss the rationale for individual content decisions, GPB generally did not have documentation that was consistent and readily available. During our review, we identified several actions that GPB could implement to help ensure content decisions are consistent and sufficiently supported. These include: documenting its consideration of statewide educational initiatives and goals in relation to needs expressed by teachers and other stakeholders; documenting its identification of the appropriate level of investment for content development and its prioritization of funding to ensure the most important needs are addressed first; documenting its identification of the most appropriate mix of original and partnership content; tracking the cost of developing and updating content (including direct costs and staff time); and, evaluating the benefits obtained from its educational content, including content delivered through GPB's partnership with DE (see the finding on page 11 for a more detailed discussion). GPB's Response: "GPB agrees that opportunities exist for better defining the content development and update process through consistent support documentation. In fact, a byproduct of this audit process has been the establishment of new content strategy documentation in GPB's Education division, as well as other divisions." GPB Educational Resources 11 Finding 3: GPB uses Google Analytics (Analytics) to measure usage data. Analytics uses the following definitions: User: an individual device Pageviews: an instance when a page is loaded (or reloaded) in a browser Usage levels for GPB's educational programs have increased and feedback GPB has collected from teachers for a few original resources and outreach sessions has generally been positive. Additional information would be necessary to fully evaluate the impact of its original and partnership educational resources. GPB primarily assesses its impact based on usage data and teacher feedback. Usage data tells GPB approximate levels of usage, general trends over time, and which of its original resources are most used. Teacher feedback provides GPB with input on how often teachers use its educational resources and how helpful its resources are to students and teachers. Our review of studies from other digital education providers utilize similar methods for evaluating the impact of digital educational resources, but rely on more detailed usage data, surveys, and statistical analyses that allow them to link usage to student outcomes. For example, although we did not evaluate the methodologies, case studies using statistical analyses conducted or paid for by DE suggest positive learning outcomes in other states. With additional information, such as more detailed usage data, GPB would be able to conduct similar types of outcomebased assessments. Usage of Original and Partnership Educational Resources For GPB, usage is one of the most important indicators of its impact on teachers and students. As a result, GPB places an emphasis on increasing usage of original and partnership content. As shown in Exhibit 6, usage levels for GPB's original educational content have increased from fiscal year 2017 to fiscal year 2019. Both users and pageviews of GPB's educational content have increased, with the largest increase occurring from fiscal year 2017 to fiscal year 2018. It should be noted that there are limitations to GPB's usage data generated through Google Analytics. In addition, GPB indicated that while usage data can approximate the location of its users by city, it is not useful for identifying what schools or school systems are accessing its resources. GPB also indicated that the Google Analytics data only tracks usage of resources accessed through GPB's webpages and may not capture usage of its resources that are accessed by users through other social media platforms, such as YouTube and Facebook. GPB Educational Resources 12 Exhibit 6 Usage Has Increased for GPB's Original Educational Resources GPB Education Usage Pageviews Users 4,903,000 5,764,000 2,167,000 567,000 1,250,000 1,427,000 FY17 Source: Google Analytics FY18 FY19 As shown in Exhibit 7, usage for PBS Learning Media reflects trends in GPB's usage data, but the increase was not as significant. Similar to limitations of GPB's usage data, PBS's usage figures cannot differentiate use by type of individual user or by school. Exhibit 7 Usage Has Increased for PBS Educational Resources PBS Learning Media Usage Pageviews Users 749,000 838,000 857,000 241,000 258,000 225,000 FY17 Source: PBS records FY18 FY19 In contrast to PBS Learning Media and GPB, DE is able to provide more detailed information about usage, such as number of logins, usage by school and school district, GPB Educational Resources 13 and number of assets accessed because of user registration. For example, DE provides reports that show usage by school and school district, including the top twenty-five schools and school districts by usage. User registration allows DE to differentiate between student and teacher users and track which school and school district a user is from. Exhibit 8 demonstrates that overall usage of DE resources has increased from fiscal year 2017 to fiscal year 2019. (It should be noted that DE measures usage in terms of logins, not pageviews or users as PBS and GPB does; as a result, it is not possible to compare usage across providers.) This increase appears to be driven by student logins while the number of teacher logins has decreased. Exhibit 8 DE Data Shows Increasing Student Use, But Declining Teacher Usage Discovery Education Logins Student Logins Teacher Logins 1,282,000 1,362,000 1,392,000 964,000 970,000 782,000 318,000 FY17 Source: DE records 392,000 FY18 610,000 FY19 Feedback Surveys GPB has conducted feedback surveys with Georgia teachers for 4 of 24 original resources; these surveys are designed to obtain information about teacher satisfaction with its resources. For example, as of August 2019, 22 out of 24 teachers responded that using the Chemistry Matters series has led to improved learning outcomes/test scores for students. Similarly, 37 out of 54 teachers responded that the Georgia Studies digital collection has led to improved learning outcomes/test scores for students. GPB also conducts surveys on some of its outreach sessions. In an ongoing survey conducted by GPB, responses as of November 2019 indicated that 248 out of 317 teachers surveyed strongly agree that the information provided by outreach specialists would help them improve learning outcomes for their students. In addition, 250 teachers stated that information provided in the training session would help them teach the Georgia standards. GPB indicated that it would conduct surveys after every outreach session going forward. GPB Educational Resources 14 Previous Studies of Digital Education Resources Neither GPB nor an independent reviewer has conducted an assessment of the impact of its original and partnership educational content on student outcomes. However, several private digital learning media organizations have conducted their own studies that attempt to capture student outcomes resulting from use of their resources. Overall, existing studies on digital learning have reported positive results in student outcomes. Generally, these studies suggest that higher usage of digital learning resources lead to better student outcomes, which can be measured as assessment scores, attendance, or attitudes toward learning, as shown in Exhibit 9. While the difficulty of measuring outcomes from digital education resources is a challenge faced by the entire digital learning industry, the studies demonstrate that a variety of methods can be used to evaluate impact. It should be noted that some of these studies were self-funded by the digital providers and we did not evaluate the methodologies as part of our review. Exhibit 9 Existing Studies Indicate Positive Student Outcomes from Digital Learning Resources Results Methods Coursera Majority of users reported educational and career benefits Qualitative Discovery Education High usage resulted in greater interest and higher scores for students Statistical BrainPOP Higher test scores for users than non-users Statistical Khan Academy Positive correlation found between students' usage and state assessment scores PBS KIDS Increased positive attitudes in teachers, increased awareness in parents, increased interest and performance in students Source: PAD review of various outcomes studies Mixed Mixed Opportunities for Improvement While there appears to be no consensus in the digital media industry on the optimal way to measure usage, there are tools, such as user registration, that would allow GPB to gather more detailed and complete information on who is using the digital resources and how. User registration does not resolve challenges for measuring usage when a single log-in is used by multiple users--teacher log-in used for an entire classroom of students--but it generally provides greater detail of information on users. GPB indicated that additional funding would be necessary to implement a log-in system. More detailed usage data would allow GPB to conduct outcome-based assessments similar to those described in Exhibit 9. This would allow GPB to better understand the impact of its original and partnership resources on students and teachers and facilitate more data-driven decisions about content and level of investment in original GPB Educational Resources 15 and partnership programs. GPB indicated that it would need additional funding to replicate such assessments; however, we note that these types of assessments would not be needed every year, but on a periodic basis. As an alternative, GPB could evaluate the methodologies of studies conducted by DE to determine if the results can be applied to Georgia and considered as part of its decision to renew the partnership. In the absence of more complete usage data, GPB could employ more extensive qualitative methods, such as statewide surveys or focus groups, to obtain more data about teacher and student usage of its educational resources and level of satisfaction. While GPB has conducted limited surveys with generally positive results, it should be noted that these surveys provide information on user experience and satisfaction but do not provide information on why teachers may choose not to use GPB's resources. In order to better understand teacher usage, GPB could consider conducting a targeted survey with teachers about why certain resources are not used in their classrooms. GPB could then incorporate this feedback to help evaluate its efforts to increase usage or to identify needed content. GPB's Response: "GPB also agrees with your findings that more complete usage data would enable the agency to more fully evaluate the impact of GPB's original and partnership resources. As a state agency, the use of individual logins would allow us to collect this data. However, GPB is also a public media company federally mandated to provide barrier-free access to its content for all citizens of Georgia. The conflict between the interests of the state and those of the greater public media system is an area where GPB strives to find the correct balance. It is also worth noting that we are currently undergoing a website redesign that will provide better usage data and a more streamlined user experience for educators." GPB Educational Resources 16 Appendix A: Objectives, Scope, and Methodology Objectives This report examines Georgia Public Broadcasting's educational resources. Specifically, our examination set out to determine the following: 1. What educational offerings does GPB provide to the citizens of Georgia? 2. What are GPB's processes for determining the content and the level of investment for original and partnership programs? 3. To what extent do GPB's educational resources have an impact on students and teachers? Scope This special examination generally covered activity related to Georgia Public Broadcasting and its educational resources that occurred from fiscal year 2017 to fiscal year 2019, with consideration of earlier or later periods when relevant. Information used in this report was obtained by reviewing relevant laws, rules, and regulations, interviewing GPB management and staff, reviewing other state's public media organizations, analyzing usage data, and reviewing studies of the outcomes from digital learning. We interviewed staff involved with data analytics and managing partnerships at PBS Learning Media and Discovery Education. We reviewed GPB's contracts and amendments with Discovery Education. In addition to reviewing the websites of other public media organizations, we interviewed staff at Kentucky Educational Television. We interviewed staff at the Georgia Department of Education (GaDOE). We reviewed the by-laws and meeting minutes of GPB's oversight body, the Georgia Public Telecommunications Commission Board. We obtained expenditures related to the Education Division from GPB's internal budget documents and from TeamWorks reports. Government auditing standards require that we also report the scope of our work on internal control that is significant within the context of the audit objectives. We reviewed internal controls as part of our work on Objective 2. Specific information related to the scope of our internal control work is described by objective in the methodology section below. Methodology To identify GPB's educational offerings, we reviewed GPB's website to determine what educational resources were available for use by Georgia students and teachers. We interviewed GPB's education division management and staff about the different types of resources it offers, including community events and workshops. We reviewed the websites of public media organizations in other southeastern states. In discussing upcoming changes to GPB's website and the changes to some of its offerings, we interviewed GPB staff about its website redesign process. To obtain information on GPB's processes for determining the content and the level of investment for original and partnership programs, we interviewed GPB management and staff and reviewed documentation GPB provided about its processes and other controls around content development. The staff we interviewed include the Education Division director and staff, including the outreach specialists and staff involved in budgeting. We reviewed O.C.G.A. 20-13 and the GPTC by-laws to determine oversight authority. We also reviewed GPB's strategic plans for education, quarterly reports, grant GPB Educational Resources 17 proposals for external funding, production schedules and other documents related to content development. To determine the extent to which GPB's educational resources have an impact on students and teachers, we obtained and analyzed usage data from GPB's Google Analytics account. While we assessed the controls over data used for this examination and determined that the data used were sufficiently reliable for our analyses, we could not independently verify the data. For partnership data from PBS Learning Media and Discovery Education, the information was provided to us by GPB, and we could not independently access or verify the data. However, we note that this data cannot be altered because of the way analytics tracks and records usage metrics. We interviewed GPB staff about its processes for monitoring usage and measuring impact, including the feedback surveys it conducted with teachers. We reviewed several existing studies but did not assess the methodologies used by these studies to measure the outcomes from digital education resources. This special examination was conducted in accordance with generally accepted government auditing standards (GAGAS). Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. GPB Educational Resources 18 Appendix B: GPB Original Content Inventory As discussed earlier in the report, we identified a total of 24 GPB original resources for students and teachers. This count was determined from the larger resource collection, rather than individual videos or segments that compose an entire collection. For informational purposes, we included a column for usage levels in fiscal year 2019. Resource 1 Lights, Camera, Budget! Description Game and teacher guide Resource Type Launchable Resource 2 Chemistry: A Study of Matter 3 Georgia Stories Semester 1: 22 videos Semester 2: 21 videos, each with supplementary documents 136 episode video series Video Series Video Series 4 SALSA 42 episode video series with supplemental learning activities Video Series 5 Physics Fundamentals Semester 1: 29 videos Semester 2: 26 videos with supplemental documents Video Series 6 Education Matters Blog Blog Posts 7 Chemistry Matters 58 main videos, 21 Closer Look videos, teacher toolkit, supplementary guides and questions Video Series 8 Georgia Studies Digital 10 units and 30 chapters, including Launchable Resource Textbook (Student Edition) supplementary resources Subject Area Multiple Science Target Grade Level 6th -12th Year Published 2018 FY19 Usage (Pageviews) 733,224 7th -12th 2004 730,615 Social Studies World Languages Science 3rd -8th K-3rd 7th -12th 1994 1997 2004 507,933 379,641 310,310 N/A Science/STEAM Teacher 7th -12th Ongoing 2016 299,157 276,831 Social Studies 8th 2010 190,3391 1 This number was drawn from the same source as the virtual field trips and the teacher edition. However, we note that the usage number could be as high as 226,000. GPB Educational Resources 19 Resource 9 Physics in Motion Description 59 videos, 19 Closer Look videos, teacher toolkit and supplementary guides and questions 10 Virtual Journeys/Field Trips 34 Virtual Fieldtrip Presentations Resource Type Video Series Launchable Resource Subject Area Science Target Grade Level 7th -12th Year Published 2018 FY19 Usage (Pageviews) 213,692 Social Studies 3rd-12th 2016-current 172,865 11 Irasshai 138 videos and lessons with a teacher guide Video Series 12 Georgia Race Through Time 13 Count On It! 14 Fast Forward 2 games and a teacher guide Launchable Resource 20 episode video series with supplemental games and activities 32 episode video series with supplementary resources Video Series Video Series 15 Gasha Go! Game with 10 different math challenges Launchable Resource 16 Live Explorations 5 live video streams with supplementary resources Video Event 17 WWII Oral History Project 116 Testimonial videos Video Series 18 Georgia Studies Digital 10 units and 30 chapters, including Launchable Resource Textbook (Teacher Edition) supplementary resources World Languages Social Studies Math/STEAM Science/STEAM Math Multiple Social Studies Social Studies 9th -13th 8th K-3rd 3rd -12th preK-3 3rd -8th 6th -12th Teachers 1997 2017 2006 2013 2018 2017-current 2007 2015 61,768 53,867 53,723 38,196 33,144 21,9482 16,619 8,175 2 GPB indicated that the viewership for the Live Explorations is more accurately measured by applying a multiplier because it is a live event streamed across multiple audiences. For example, according to GPB, teachers will stream the Live Exploration on a smartboard or projection screen and could have up to 500 students watching at one time. GPB Educational Resources 20 Resource 19 Things Explained 20 Share Your Story 21 Peach State Politics 22 On the Campaign Trail 23 The Green Tree Frog 24 Education Matters Newsletter Description 11 videos with supplementary resources Text series written by teachers 12 videos with supplementary resources 10 episodes and 2 mini-episodes Resource Type Video Series Blog Posts Video Series Video Series Subject Area Multiple Target Grade Level 3rd -8th Year Published 2018-current FY19 Usage (Pageviews) 7,3283 N/A Social Studies Teacher 5th -12th Ongoing 2019 6,244 3,8403 Social Studies 6th -12th 2015-2017 846 Video Newsletter Video Blog Posts Social Studies N/A 8th Teacher 2008 Ongoing 641 N/A4 Source: GPB website and Google Analytics 3 For Peach State Politics and Things Explained, GPB indicated that these videos are also hosted on YouTube and social media. The usage numbers for these two resources do not include the usage data from the other platforms. For example, one episode of Things Explained had an approximately 7,000 views on YouTube. 4 The usage level is indicated as N/A because a different metric was used by GPB to track usage based on the type of resource. Between August 2018 and June 2019, GPB indicated the number of views for the newsletter ranged from approximately 3,000 to 4,000 per month. GPB Educational Resources 21 Appendix C: GPB's New Original Content and Content Updates New Original Content Georgia Studies Digital Textbook: Student Edition According to GPB, the idea to digitize the textbook came about as a result of conversations with representatives of Georgia Humanities and the University of Georgia's Carl Vinson Institute who indicated that the Georgia Studies textbook would no longer continue to be published. According to GPB, there was a need for a Georgia History textbook as well as a need to innovate the content and make it interactive for students. In addition, GPB indicated that by digitizing the textbook and making it available at no cost, it was saving school systems millions of dollars. GPB acquired the rights to the Georgia Studies Textbook in 2010 and used state funds to develop and distribute a digitized student version in 2012; a teacher edition was released in 2016. It took approximately two years from the time GPB acquired rights to the textbook to launch the student version of the digital textbook in 2012. GPB has obtained a grant through the Woodruff Foundation to add supplementary materials to the textbook, including a digital game, and 30 virtual field trips (see below) . GPB has also continued to update the textbook to maintain its historical relevance and update accessibility features. Georgia Studies Virtual Field Trips According to GPB, it had become "increasingly difficult for teachers to take students on out-of-class field trips due to the high cost of transportation and high demand for limited instruction time." It viewed virtual field trips as an alternative and "a nice ancillary addition" to the Georgia Studies digital textbook, which had been previously developed. Field trip locations (such as Andersonville, Warm Springs) were selected based on teacher input. According to GPB, once the locations were determined, GPB's Education Curriculum Specialists identified which Georgia standards should be addressed in each fieldtrip. Then, they began writing the content and curriculum, determining shot lists, identifying subject matter experts, aggregating and gathering permissions for primary source documents, images, and video footage. As previously mentioned, GPB obtained a grant from the Woodruff Foundation to fund thirty virtual field trips after it had produced the Andersonville virtual field trip. The virtual field trips generally took about two to four months from project start to launch. Live Explorations Similar to the rationale for virtual fieldtrips, GPB created Live Explorations in response to the financial and logistical constraints involved in planning field trips. According to GPB, many schools are unable to visit certain places in Georgia. GPB Education developed its Live Exploration production in order to bring these locations to the classroom, all at no cost and no planning for the school. GPB has partnered with University of Georgia, Georgia Institute of Technology, Okefenokee National Park, National Oceanic and Atmospheric Administration, and the Atlanta Symphony Orchestra on these live events. The idea for the Atlanta Symphony Orchestra (ASO) Live Exploration was initiated by a representative of ASO, who reached out to GPB about the possibility of a collaboration. The production took about four months to complete. GPB's Education Project Manager wrote the script in consultation with a teacher consultant from Atlanta Public Schools and ASO. The ASO funded half the cost of the production; GPB indicated that the production cost of other live explorations was generally shared by GPB and its collaborators. Peach State Politics According to GPB, it created the Peach State Politics series in response to feedback from teachers who utilize the Georgia Studies digital textbook who indicated a need for grade-level appropriate resources relating to Georgia's state government. GPB education staff collaborated with a GPB news reporter to develop this resource. This resource was created without external grant funds and relied on resources from GPB's radio and production staff. According to GPB, each video takes roughly a week from production to release, are approximately two minutes long, and are released every Tuesday during the legislative session. Lights, Camera, Budget! According to GPB, Lights, Camera, Budget! was developed in order to provide resources to increase Georgia students' financial literacy. The idea for the game came from a Georgia Council on Economic Education (GCEE) workshop, which was based on new education standards. Both GPB and GCEE applied for and received grant funds from the SunTrust Foundation to create the resource, which took around 6 months to complete. During the development process, GPB obtained input from teachers and students to guide production. GPB Educational Resources 22 GASHA Go! GPB indicated that the GASHA Go! game was developed in response to teacher feedback and Georgia Milestones test results which indicated a skills gap in math. In addition, GPB staff's own research indicated that children who have a well-developed number sense are able to succeed in early math and beyond. According to GPB, GASHA Go! was meant to be a pilot for a suite of early childhood math games and GaDOE had been consulted about the idea. GPB indicated that it used operational funds to develop the game. It contracted with two elementary math teachers to serve as consultants throughout the project and an outside production company to create GASHA Go!. GPB did not produce additional games because it was unable to secure funding. Things Explained According to GPB, "teachers are required to develop lessons that are rigorous and relevant to the lives of their students" and must identify local topics to provide relevant content. The idea for the series came from conversations with elementary school teachers, who are not trained to teach science and wanted a resource that could teach science concepts in less than twenty minutes. To assist teachers, GPB decided to launch its Things Explained video series, which GPB indicated highlights a range of Georgia-based phenomena that are tied to the Georgia Standards of Excellence. This resource is funded internally by GPB. According to GPB, each episode takes about a month or two from script-writing to release. Content Updates Chemistry Matters and Physics in Motion According to GPB, it decided it was necessary to update two older physics and chemistry series due to high usage of the older series (accounting for 10% of total GPB website traffic), a revision to Georgia's science educational standards, and technological changes. In addition, GPB indicated that for some public school districts without dedicated chemistry and physics instructors, the GPB courses were serving as stand-alone instructional tools. GPB hired outside teacher consultants recommended by the Georgia Science Teachers Association to review the previously existing content and its alignment with standards. Based on the review, GPB decided that a new approach, content, and scripts, would need to be developed to best align the content with revised standards. GPB secured funding for the updates from Georgia Power and the Governor's Office of Student Achievement. GPB also helped fund the projects. Both series were developed and produced by GPB with the assistance of production contractors, internal production staff, and teacher consultants. For each series, it took approximately two years from project kickoff to the series launch. Georgia Stories GPB indicated that the Georgia Stories video series was initially created in the 1990s to provide educational content to Georgia students in response to newly installed satellite dishes and a need for Georgia history content. According to GPB, education staff began to develop updated supplemental materials for the series in 2017 because of declining usage levels and changes to education standards. Supplemental content updates were created by educational staff and uploaded to its webpage using in-house resources. GPB also indicated that it would like to update the video content that includes animations but it has been unable to secure additional funding to do so. Fast Forward The Fast Forward series, which highlights STEM career opportunities in Georgia, was initially created in response to feedback that GPB received from Georgia teachers and Department of Education staff, and it was developed in partnership with the Department of Education and the Governor's Office. Similar to Georgia Stories, supplemental content for Fast Forward was updated in 2018 by Education Division staff using internal resources based on educational standards updates and declining usage. Source: GPB documents and interviews The Performance Audit Division was established in 1971 to conduct in-depth reviews of state-funded programs. Our reviews determine if programs are meeting goals and objectives; measure program results and effectiveness; identify alternate methods to meet goals; evaluate efficiency of resource allocation; assess compliance with laws and regulations; and provide credible management information to decision makers. For more information, contact us at (404)656-2180 or visit our website at www.audits.ga.gov.