Program evaluation. Department of Education, Elementary School Foreign Languages (ESFL) Pilot Program / prepared for the Budgetary Responsibility Oversight Committe

Program Evaluation
Prepared For The Budgetary Responsibility Oversight Committee
Department of Education
Elementary School Foreign Languages (ESFL) Pilot Program
March 2003

Russell W. Hinton, State Auditor Performance Audit Operations Division 254 Washington St. S.W.

Department of Audits and Accounts

Atlanta, GA 30334-8400

The Georgia Elementary School Foreign Languages (ESFL) Pilot Program was started in fiscal year 1993 through a special initiative of the General Assembly. The original goal of the "pilot" program was development of curriculum and standards for an elementary school foreign language program that could be utilized by schools throughout the state. Documentation obtained from the Department of Education (DOE) indicated that the Program was intended to be expanded statewide at the conclusion of a five-year pilot period. Information indicating whether this expansion was to be state funded or locally funded was not available. At some point after the Program was originally implemented, DOE documents started to refer to the Program as a "model" program. The goal of the Program was changed to developing model programs at various elementary schools to serve as demonstration sites for other schools in Georgia that may wish to implement a foreign language program. At the elementary school sites where the Program has been implemented, the Program's underlying goals are to produce fluent users of foreign languages and to support the regular instructional program in kindergarten through fifth grade. The schools are allowed to select the language to be offered.
The Program provides grants to school systems to help fund foreign language instruction in designated elementary schools. The original implementation of the Program started with kindergarten and expanded to an additional grade level each year, until all children at participating elementary schools received foreign language classes five days a week, thirty minutes each day from kindergarten through fifth grade. Program guidelines have been developed which include: using the second language 98-100% of the time, teaching around a theme, helping students understand without translating, including meaningful culture in every lesson, and providing the students with opportunities to talk about things that are meaningful to them.
Program teachers can only teach foreign language and teachers are normally limited to teaching a maximum of eight classes per day. Usually, the teachers, rather than the students change classrooms. The teachers have carts with teaching materials and supplies that they move between classrooms.
The Program is coordinated by the Program Specialist for Foreign Languages in the Georgia Department of Education's (DOE) Office of Student Learning and Achievement. About 25% of the Program Specialist's time is involved with ESFL activities. The Program Specialist's ESFL

Elementary School Foreign Languages Pilot Program

Page 1

duties primarily involve consulting with school systems, managing program grants to local school systems, coordinating annual training seminars, and Program public relations. Each participating school system has an ESFL contact person that supervises the Program at the local level. The duties of these contact persons primarily involve submitting quarterly reimbursement requests to DOE, maintaining program records, staff development and recruiting, and serving as a liaison between DOE and the local school system.
Exhibit 1 identifies the 15 school systems and 26 schools that are currently participating in the Program as well as the language offered at each school, the grades for which instruction is offered, the number of students participating in the Program at each school and the amount of grant funds provided to participating school systems in fiscal year 2003. The map (Appendix) on page 22 shows the locations of the school systems that are currently participating in the Program as well as the locations of school systems that no longer participate in the Program.

Exhibit 1 Schools Currently Participating in ESFL Program
As of January 2003

Language1

School System

Elementary School Name

Grade

Year Grant Started K 1

2

3

# of 4 5 Students

Grant Amount (FY2003)

Atlanta Public Schools

Fickett Centennial Place

F S

DeKalb County

Austin Rowland

G G

Dougherty County

Lake Park Sherwood

S S

Fulton County

Mimosa

J

Gwinnett County

Rockbridge

S

Columbia County

South Columbia Steven's Creek

S S

Lithia Springs

F

Douglas County

Chapel Hill

F

Holly Springs

F

Catoosa County

Graysville

S

Cherokee County

Little River Canton

S S

Decatur City Schools

5th Ave. at Oakhurst Westchester

S S

Forsyth County

Big Creek Chestatee

F S

Richmond County

Lake Forest Hills

S

Taliaferro County

Taliaferro County

S

Whitfield County

Varnell

S

Cherokee

S

Sumter County2

Sumter Co. Primary

S

Sumter Co. Elementary S

2001 2003
1993 1993
1993 1993 1993 1993
1998 1998 1998 1998 2001
2001 2001 2001 2001 2001
2001 2001 2001 2001 2001 2002 2002 2002

XXX X X X XXX X X X
XXX X X X XXX X X X
XXX X X X XXX X X X XXX X X X
X X XXX X
XXX X X X
XX X X X
XX X
XXX X X X XXX X X X XXX X X X
XXX X X X XXX X X X
XXX X X X XXX X X X
XXX X X X XXX X X X XXX X X X
X

1 F=French, S=Spanish, G=German, J=Japanese 2 Sumter County contracted for three schools, but could only staff one school for first grade.
(on 2/10/03 an amendment to the contract was requested reducing the grant to $26,651)
Source: Program Records and Evaluation Team's Survey of Participating School Systems

787 667 519 501
655 480 630 280 369 753
242 190 232
572 549 457 189 202
1,448 971 427 112 582
210 0 0
12,024

$258,510
154,906
154,906 128,755
77,453 181,557
79,953
103,604 155,906 104,604
309,812 77,953 26,651 77,953
79,953
$1,972,476

Elementary School Foreign Languages Pilot Program

Page 2

Financial Information
Program grants to participating school systems in fiscal year 2003 totaled approximately $2.0 million consisting of $25,651 per teacher and $500 (for established programs) or $1,000 (for new programs) to each participating school for materials. In fiscal year 2003, school systems will receive state grants ranging from $26,651 (one school, one teacher) to $309,812 (two schools, 12 teachers). (See Exhibit 1.) The remaining Program funds (approximately $225,000) are used by DOE for teacher training/staff development, curriculum development, program assessment contracts (in some years), supplies, and printing costs. Local school systems are responsible for paying the difference between the grant award and the actual cost of the Program (primarily the total amount of the teacher's salary and benefits).

The Program was originally funded with $361,000 ($21,740 per teacher) to start providing foreign language instruction in 24 schools within 15 school systems. Exhibit 2 shows the Program funding for fiscal years 1993 through 2003.

Exhibit 2 Elementary School Foreign Languages Program
Funding History
Fiscal Years 1993 - 2003

Admin/Other Costs

School System Contracts

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003

$.4million Information Not Available

$.8million

$1.3million

$1.7million $2.0million

$2.2million $2.2million $2.2million $2.2million $2.2million

0 $.5 $1.0 $1.5 $2.0

Source: Program Grant and Contract Records

Millions

Grant payments are made to school systems under the provisions of a contract between the State Board of Education and the local school systems. The local school systems submit requests for reimbursement of Program expenditures.

Elementary School Foreign Languages Pilot Program

Page 3

Evaluation Scope and Methodology
This evaluation was conducted in compliance with O.C.G.A. 45-12-178 and in accordance with generally accepted governmental auditing standards for performance audits. The evaluation was conducted at the request of the Budgetary Responsibility Oversight Committee (BROC). The scope of the evaluation focused on the following issues:
Should the Program be continued as it is currently operated, continued with identified steps to remediate deficiencies, or discontinued?
To what extent has the Program been evaluated?
How many different elementary schools have received funding since the inception of the Program?
Are there policies and procedures in place for awarding state funds to local systems?
How much have the local school systems contributed to the Program?
To what extent could distance-learning technologies be used to complement or replace the Program?
The evaluation team reviewed available records going back to the inception of the Program. Grant applications, contracts with local school systems and reimbursement requests were reviewed and the evaluation team visited three participating elementary schools and observed 13 classes. Various Georgia Department of Education, U.S. DOE, and local school system personnel were contacted for information. In addition, the fifteen school systems currently participating in the Program were surveyed for information on their programs.
The entire report was discussed with appropriate personnel at the DOE. Personnel from the Department of Education were invited to provide a written response to the report, and DOE was also invited to indicate areas in which it planned to take corrective action. Pertinent responses from DOE are reflected in the report as appropriate.
It should be noted that at the time this report was written, the Governor's Fiscal Year 2004 Budget Report recommended that funding for the program be eliminated. As a result, our recommendations for corrective actions address actions that should be taken if the Program is continued.
In its response to the evaluation, DOE noted that according to the Program's original Program Specialist, the term "pilot" was removed from the title of the Program at the request of the Office of Planning and Budget before the Program was ever funded initially. In addition, DOE noted that according to two former Program Specialists, "the intent of the Program was always to develop fluent users of the language and to provide models of excellent elementary foreign language instruction throughout the state." Documentation obtained by the evaluation team including Budget Tracking documents from the Legislative Budget Office and DOE's press release announcing implementation of the Program referred to the Program as a pilot program.

Elementary School Foreign Languages Pilot Program

Page 4

Issue: Should the Program be continued as it is currently operated, continued with identified steps to remediate deficiencies, or discontinued?
Finding No. 1
The Elementary School Foreign Languages (ESFL) Pilot Program should be discontinued. The Program achieved its goals as a pilot program years ago and the need to continue funding 26 model programs when only two non-ESFL funded programs (20 schools) appear to have been developed based on the Program's models is questionable. To continue subsidizing only 26 (2%) of the state's 1,218 elementary schools would also seem to be inequitable. While education experts feel there is a benefit to offering foreign language instruction to young children, information from the Education Commission of the States (ECS) only identified six states that had any state requirements for foreign language instruction at the elementary school level. The evaluation team estimated that it would cost the state an additional $125 million annually to expand the Program statewide by providing the same subsidy to all the other elementary schools in the state. In comparison, the Federal government only provides about $14 million of start-up "seed money" annually for elementary school foreign language programs for all of the United States. More detailed information on the Program's operations is provided in the remaining findings in this report.
In its response to the evaluation, DOE noted that it felt that dismantling the Program at this point would effectively waste the approximately $19.2 million invested in the Program to date. DOE indicated that it has long advocated the expansion of elementary foreign language instruction statewide and it continues to support expansion as a long-term goal. The Department noted the global context in which Georgia's students will work and recommended that the General Assembly continue to work toward the goal of an extended sequence of language learning for all students by investing in elementary foreign language. DOE recommended that statewide implementation should be started by funding program services for all fourth and fifth grade students in the state and then expanding program services to the remaining students in Georgia's elementary schools as funds allow. The Department estimated that the annual state cost of providing foreign language instruction statewide in grades four and five would be approximately $42 million. It should be noted that local schools systems would also have to fund a portion of the cost of the Program.
Issue: To what extent has the Program been evaluated?
Finding No. 2
The Department of Education has not evaluated the Program's effectiveness as a pilot program or as a model program. The evaluation team concluded that the Program achieved the goals of the pilot program within the first five years of implementation. The team also concluded that the need to continue funding 26 model programs when only two non-ESFL funded programs (20 schools) appear to have been developed based on the Program's models is questionable.
The General Assembly originally funded the Program as a "pilot" program to develop curriculum and standards for an elementary school foreign language program that could be

Elementary School Foreign Languages Pilot Program

Page 5

utilized by schools throughout the state. Documentation obtained from DOE indicated that the Program was intended to be expanded statewide at the conclusion of a five-year pilot period. At some point after the Program was originally implemented, DOE documents started to refer to the Program as a "model" program. The goal of the Program was changed to developing "model" programs to serve as demonstration sites for other elementary schools in Georgia that may wish to implement a foreign language program. While DOE has never evaluated Program's effectiveness as a pilot program or as a model program, the evaluation team's observations on its effectiveness in these areas are discussed below.
During the five-year pilot period, the ESFL Program developed the core curriculum and teaching techniques that were the basis of the foreign language programs offered at elementary schools. DOE had no information identifying why the program was not eliminated at the end of the pilot period or why the program was not expanded throughout the state. Only six elementary schools have continuously participated in the Program since its inception in fiscal year 1993.
While the state has continued to provide funding to 26 elementary school foreign language programs which serve as models for the Program, local school systems have not made significant use of these model programs to develop locally funded foreign language programs. DOE's Program Specialist for Foreign Languages identified nine school systems with elementary school foreign language programs in the state that were not funded through the Program and the evaluation team identified three additional non-ESFL funded programs through a survey of current ESFL Program participants. As shown in Exhibit 3 only two of these programs appeared to follow the basic model program guidelines (i.e., provide 30 minutes of daily instruction to all students in grades K-5).
The Program achieved the goals of the pilot program within the first five years of implementation. The need to continue funding 26 programs as models when only two non-ESFL funded programs (20 schools) appear to have been developed based on the Program's models is questionable.
In its response to the evaluation, DOE disagreed with the conclusion that the Program has been unsuccessful as a model program. The Department noted that it was indeed unfortunate that state funds had not been appropriated to expand the Program into more schools, but contended that this situation was scarcely the fault of the Program itself. The Department also did not think that the evaluation took into account the many school programs that have been influenced by the Program in many respects. DOE indicated that it felt that the Program was quite successful as a model and noted that in addition to the 20 schools that adopted the model in every respect, it thought that 126 additional schools had programs based closely on the model. The Department indicated that it did not think that these additional programs would exist if the Program had not been established and thought it doubtful that they will continue to exist if the ESFL Program is dismantled.
It should be noted that the 146 schools (20 schools fully adopting the model and 126 schools reportedly closely based on the model) would be at most only approximately 12% of the 1,218 elementary schools in the state. Since most of the 126 schools with programs based on the model appeared to only provide instruction for grades four and five, a significantly smaller percentage of elementary school students in the state are participating in foreign language

Elementary School Foreign Languages Pilot Program

Page 6

6programs based on the Program's model. In addition, a press release by the Department of Education indicated that 20 school systems had elementary foreign language programs (number of schools unknown) prior to implementation of the ESFL Program. As shown in Exhibit 3, currently only 12 school systems appear to have elementary foreign language programs that are not funded by the Program.

Exhibit 3

Local Elementary School Foreign Language Programs in Georgia

That Are Not Funded Through the ESFL Program

School System
Atlanta City Decatur City Forsyth County Breman City DeKalb County Calhoun City Douglas County Fulton County Morgan County Catoosa County Cherokee County Gwinnett County TOTAL

(as of January 2003)

# Schools

Doesn't

Follows Follow

Comments

Model Model

16

46 Only grades 4 & 5 for 46 schools.

4

Follows model.

10 Only 30 minutes every other day.

1 Only 30-45 minutes per week.

40 Less than 5 days per week and/or only grades 4 & 5.

2 Only 3045 minutes per week.

15 Only grades 4 & 5.

15 Only grades 4 & 5.

2 Only 45 minutes per week for one semester.

2 Only 30 minutes twice per week for grades K 2.

18 No time requirement and no special sessions.

6 Only 45 minutes per week.

20

157

Source: Program Records, Evaluation Team Survey, and Interviews with local school system personnel

Finding No. 3
The Program has conducted some evaluations of its effectiveness at achieving its underlying goals related to teaching foreign languages at participating schools. However, the independent study that is widely quoted as indicating that the Program is the best elementary school foreign language program in the United States only compared Georgia's Program to the scores of 214 students at seven schools in four other states. The main evaluation citing higher standardized test scores for ESFL students did not attempt to identify if the higher test scores were actually attributable to the ESFL Program or to some other factor.
The Department of Education has contracted for independent evaluations of the Program's effectiveness in teaching foreign languages to elementary school students and it has evaluated the effect Program participation has had on standardized test scores. The results of these evaluations are discussed below.
The Center for Applied Linguistics (CAL), a private, non-profit organization of scholars and educators has conducted four evaluations of the Program since 1995. In the latest evaluation, completed in June 2001, CAL evaluated the oral fluency

Elementary School Foreign Languages Pilot Program

Page 7

and listening comprehension abilities of ESFL Model Program students in kindergarten, third, and fifth grades. CAL evaluated 506 students (168 kindergartners, 189 3rd graders, 149 5th graders) in 13 Georgia schools who had been in the Program since kindergarten or first grade.
When discussing the ESFL student scores, CAL stated in their report that "These results are higher than any known average ratings in other foreign language elementary programs in the United States." While this statement has been broadly quoted by various parties, no qualifying information has been included to acknowledge that a very small comparison group was used to draw this impressive conclusion. The scores of the Georgia ESFL students were compared to only 214 students at seven schools in four other states.
To illustrate the difference in the skill levels of Georgia ESFL students to students in the other states, CAL compared Georgia ESFL students to 30 third grade students in one program in another state and 60 fifth grade students in a different program in another state. As shown in Exhibit 4, Georgia's ESFL students scored higher than the students in other states. However, as also shown in Exhibit 4, the Georgia ESFL students also had considerably more instruction time per week than was provided to the students in the other states. The CAL study did not attempt to evaluate the relationship between the amount of instruction time provided and evaluation scores in order to determine the relative effectiveness of the various programs.
In September 2000, DOE's Research, Evaluation and Testing Division reviewed third and fifth grade Reading Comprehension Iowa Test of Basic Skills (ITBS) scores from 1993 through 2000 for ESFL students in comparison to non-ESFL students in the state. This review found that third grade ESFL students scored higher than non-ESFL students on the Reading Comprehension sub-test and that ESFL fifth graders also scored higher on Reading Comprehension than non-ESFL fifth graders. The evaluation also noted that the students at the schools with ESFL programs scored higher on these tests than the non-ESFL school students prior to the Program's implementation; however the study did not attempt to identify if the higher test scores could be attributed to the ESFL instruction or if the scores were merely a continuation of the schools' prior relative performance.
In 1998, a consultant to the Program completed a dissertation (and published an article on the dissertation in 1999) comparing student ITBS scores for students who were in the 3rd grade in 1996 (and that had participated in the ESFL Program) to 3rd graders at the same schools in 1995 (who had not participated in the ESFL Program). The conclusion of the dissertation (and of the article) was that taking a foreign language in elementary school "...is not detrimental to, and in some instances may increase, scores in basic skills areas." This somewhat tentative conclusion did not provide significant insight into the Program's effectiveness.

Elementary School Foreign Languages Pilot Program

Page 8

Exhibit 4 Elementary School Foreign Languages Program Comparison of Ratings and Weekly Minutes of Instruction

Georgia Program

Program in Other State

Intermedate- Mid 5 Program Rating

Intermedate- Low 4 Novice - High 3 Novice - Mid 2

2.74 1.83

3.37 2.80

4.40 3.87
2.19 1.48

Novice - Low 1

Oral Fluency

Listening Comprehension

Third Grade

Oral Fluency

Listening Comprehension

Fifth Grade

Minutes of Instruction per Week

150

150

150

100

90

50

40

Third Grade
Source: 2001 CAL Report on Georgia ESFL Program

Fifth Grade

If the Program continues, the Department of Education should establish measurable goals and objectives for ESFL Program performance relating the oral fluency of program participants and the expected impact on standardized test scores. Evaluations should be periodically conducted to determine if the Program is meeting those goals. The Program should ensure that measurements of performance are compared to appropriate control groups to ensure that Program performance is accurately benchmarked and to ensure that Program effectiveness (and not other factors) is being isolated and measured.

Elementary School Foreign Languages Pilot Program

Page 9

In its response to the evaluation, DOE indicated that it felt that more information should have been provided in the report regarding the first three CAL studies of the Program. The Department indicated that it felt that when taken together these studies provided convincing evidence of the high quality of the ESFL Program. The Department cited the following paragraphs from the introduction to the 2001 evaluation as summarizing the three prior evaluations:
In 1995, the Center for Applied Linguistics (CAL) responded to a request from the Georgia Department of Education to conduct an evaluation of their ESFL Model Program. This evaluation was based on classroom observations and a survey of model program teachers. The evaluation found that the ESFL Model Program was a vital, strong program, enjoying support from parents, administrators and teachers.
As a follow-up to the 1995 evaluation, CAL assessed the listening and speaking skills of third graders studying French, German, Japanese, and Spanish in the ESFL Model Program. The Student Oral Proficiency Assessment (SOPA), a speaking and listening instrument designed for Grades 1-4, was administered to the third grade students in November 1996 by a team of CAL staff and Georgia teachers. Before the assessment, CAL staff trained Georgia ESFL teachers to interview students using the SOPA and to rate the students' foreign language proficiency. The students interviewed were from seven schools representing the ESFL Model Program across the state. This assessment found that students at the beginning of the third grade in Georgia ESFL Model Program were reaching the desired levels of proficiency for foreign language learners in the elementary classroom setting.
In the fall of 1997, the Georgia Department of Education contracted with CAL to provide additional assistance in the assessment of fifth grade students in their ESFL Model Program. The fifth grade ESFL Model Students who participated in this assessment were the first to have gone through the entire program since it began in 1992. The purpose of the contract was for CAL to (1) develop and field test a version of the SOPA for fifth grade students, (2) train interviewers and raters to assess the students, (3) jointly administer the assessment with Georgia teachers at both ESFL Model schools and at FLES schools that are not part of the Model Program, (4) analyze and interpret the data, and (5) report the results. Findings from the 1998 assessment demonstrated that the listening and speaking skills of fifth grade students in the ESFL Model Program were higher than those of third grade students assessed the previous year, confirming that the ESFL Model Program students' proficiency increased with the number of years of instruction. Also, it was found that the ESFL model program students' skills were higher than those of fifth grade students in other FLES programs, confirming the assumption that frequency of instruction is a key factor for increasing language proficiency.
DOE also pointed out that they felt that the dissertation on ITBS scores concretely demonstrated that students who receive 2.5 hours of foreign language instruction per week, thereby receiving less instruction in Reading and Math, scored just as well or better on the ITBS in those areas as their peers who had additional hours of instruction.

Elementary School Foreign Languages Pilot Program

Page 10

Finally, DOE indicated that if the Program is continued, it would be willing to establish performance benchmarks not only for oral fluency, but also for reading comprehension, vocabulary and grammar. The Department did not feel that it would be appropriate to establish benchmarks relating to standardized tests; however, the Department would monitor standardized test scores periodically to make sure that the Program does not have a negative impact on these scores.
It should be noted that the dissertation on ITBS scores only indicated that ESFL students received less instruction in other subjects (it did not determine that there was reduced instruction in math and reading classes). In addition, while the dissertation found that the average ITBS percentile score for the 18 schools that were reviewed went from 53.8 to 57.7 for math and from 50.1 to 50.8 for reading, seven schools had math scores that stayed the same or declined and 10 schools had reading scores that declined.

Issue: How many different elementary schools have received funding since the inception of the Program?
Finding No. 4
Since the inception of the ESFL Program in fiscal year 1993, a total of 47 elementary schools have received funding. In addition, one middle school received funding in fiscal year 2000. Currently, 26 elementary schools are participating in the Program.
Exhibit 5 on the next page identifies the school systems and schools that have received funding from the Program and the years for which funding was received. The map (Appendix) on page 22 shows the locations of the systems that have received funding since the Program's inception. Program personnel indicated that they did not know why a middle school received ESFL funding in fiscal year 2000 (there have been changes in staff at DOE over the years and current personnel were not involved with the Program when the middle school was funded). However, Program personnel speculated that the school district may have had a change in the number of teachers needed late in the year, and the system may have received special permission to use the teacher(s) in the middle school.
As shown in Exhibit 5 on page 12, in fiscal year 1993, 15 school systems (24 schools) were awarded contracts to provide funding for elementary school foreign language instruction. In fiscal year 1998, one school system (two schools) dropped out of the Program, and three systems (five schools) were added to the Program. In fiscal year 2001 when all school systems were required to apply for funding, eight school systems (12 schools in the eight systems and one additional school in another system) dropped out of the Program. Seven systems (10 schools in the seven systems and three additional schools in two other systems) were added to the Program in fiscal year 2001. In fiscal year 2002, two school systems (three schools) dropped out of the Program and one system (three schools) that was originally approved for a grant in 2001 entered the Program. Six elementary schools have continuously participated in the Program since its inception in fiscal year 1993.
In its response to the evaluation, DOE agreed with the information provided in the report on the number of elementary schools that have received funding since the inception of the Program.

Elementary School Foreign Languages Pilot Program

Page 11

Exhibit 5 School System and School Participation in ESFL Program

Participated in Program

School System

Elementary School Name

School Year Ending

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003

Atlanta Public Schools
Coweta County
DeKalb County
Dougherty County
Elbert County Fulton County Gwinnett County Habersham County
Lumpkin County 1 McIntosh County Paulding County Pulaski County Rome City
Telfair County
Walker County Cartersville City Columbia County
Douglas County
Catoosa County Cherokee County Decatur City Schools
Forsyth County Richmond County Taliaferro County Whitfield County
Sumter County 2

Morris Brandon East Rivers Connally Fickett Centennial Place Newnan Crossing Western Austin Rowland Lake Park Sherwood Beaverdam Blackwell
Mimosa Rockbridge Baldwin Demorest Lumpkin County Long Branch Todd Grant McGarity Pulaski County West Central West End Telfair County Primary Central 3 Chattanooga Valley Fairyland Cartersville Primary South Columbia Steven's Creek
Lithia Springs Chapel Hill Holly Springs Graysville Little River Canton 5th Ave. at Oakhurst Westchester
Big Creek Chestatee Lake Forest Hills Taliaferro County Varnell
Cherokee Sumter Co. Primary Sumter Co. Elementary
Total Number of Schools

24 24 24 24 25 27 27 27 27 27 26

1 FY00 contract included Lumpkin County Middle School grades 6 and 7. 2 Sumter County was awarded funding in FY01; however, they could not staff the Program. In FY03, they were
contracted for three schools, but could only staff one school for the first grade. 3 Central Elementary was only included in Telfair County's contracts for FY97; however, Program personnel believe there
were errors in the contracts and the school participated in the Program from FY96 through FY00.

Source: ESFL Contracts

Elementary School Foreign Languages Pilot Program

Page 12

Issue: Are there policies and procedures in place for awarding state funds to local school systems?
Finding No. 5
The Department of Education has created a grant application that provides a numerical score to rank applicants. As discussed below, DOE did not appear to make all grant awards based on applicant scores in 1992 or when three empty slots were filled in 1998. It does appear that DOE awarded grants based on the score that the applicants received in 2000 when all applicants that agreed to meet Program requirements received grant funds.
In 1992, 52 school systems applied for grants, and 15 grants were awarded for the first year of the Program. One grant was awarded in each of the ten existing Congressional Districts in the state and five additional statewide grants were awarded. Four (27%) of the 15 systems that were awarded grants were rated lower than other school systems that did not receive a grant. There was no documentation explaining why all the highest rated systems were not awarded grants.
In 1997, the Program identified that there would be three openings in 1998 (one school system in the 10th Congressional District needed to be replaced and the Department was adding two additional slots). Three school systems applied for the 10th District grant. Columbia County was awarded the grant, although McDuffie County was the highest rated system. Eighteen systems applied for the other two grants. Cartersville City, the system that received the top score, received one of the grants; however, Douglas County was awarded the other grant, although Morgan County had a higher rating. There was no documentation explaining why all the highest rated systems were not awarded grants.
In 2000, DOE required all interested school systems to apply for grant funding (including school systems that were already participating in the Program). DOE received 21 applications. The 20 highest rated school systems were awarded grants; however four systems declined the grant award and did not participate in the Program. (One of the systems did not actually take advantage of the funding until the next year.) The lowest rated system was denied a grant award because the system did not agree to meet all of the Program's requirements.
If the Program continues, the Department of Education should define how often systems should be required to reapply for grant funding. Grant awards should be based on the rating scores of the applications. If the systems receiving the highest scores are not awarded grants, approval and justification for not utilizing the rating system scores should be documented.
In its response to the evaluation, DOE agreed with the finding. The Department indicated that if the Program is continued, it would define how often systems must reapply for funding and take steps to ensure that the systems receiving the highest ratings receive the awards. The Department also indicated that it would document justification for any decisions which resulted in grants not being awarded to the systems that received the highest ratings.

Issue: How much have the local school systems contributed to the Program?
Finding No. 6 Information was not available on the amount contributed by local school systems to fund their programs. However, while Program personnel commonly report that local school systems contribute about 50% of the cost of the Program, a model developed by the evaluation team indicated that the local contribution percentage was only approximately 24%. In addition, while it may be debatable, local school systems could also benefit from

Elementary School Foreign Languages Pilot Program

Page 13

"freed-up" resources resulting from the Program. The potential value of these "freed-up" resources could be estimated based on the state-paid portion of the regular classroom teacher's salary whose time is available to perform additional duties that might otherwise require local funds during the time foreign language instruction is being provided. If the value of these "freed-up" resources is considered, the local school systems may be receiving up to almost 38% more state and federal funding than the cost of the Program to the local school system.
The Department of Education does not require local school systems to provide information on their total ESFL Program costs. Local school systems submit reimbursement requests for ESFL grant funds; however, no information is provided on how much local systems spend in excess of their ESFL grant funding. The local school systems are required to make up the difference between the grant amount ($25,651 per teacher) and the total salary and benefits paid to ESFL teachers.
The evaluation team created a model to estimate the amount local school systems contribute to the Program per teacher. The model was based on salary revenue source information from three (20%) of the participating local school systems. Exhibit 6 shows the relative amount of local funding per teacher based on three different sets of assumptions.

Exhibit 6

Estimate of Local Contribution to ESFL Program for One Teacher

Program Cost
Salary Benefits1 Other (Supplies and Equipment)2
TOTAL PROGRAM COST

(Fiscal Year 2003)

Theory

$

%

Actual

$

%

Actual Including

Freed-Up Resources3

$

%

$38,453 11,536
unknown $49,989

100%

$38,453 11,536
unknown $49,989

100%

$38,453 11,536
unknown $49,989

100%

State and Federal Contribution State ESFL Grant - Salary State ESFL Grant - Materials State QBE Salary Funds Federal Salary Funds State Cost of Freed-up Resources TOTAL STATE/FED. CONTRIB.

$25,651 364 0 0 0
$26,015

52.04%

$25,651 364
10,243 1,902 0
$38,160

76.34%

$25,651 364
10,243 1,902
30,600 $68,760

137.55%

LOCAL CONTRIBUTION

$23,974 47.96% $11,829 23.66%

$0

0%

State/Federal Contribution in

$18,771 37.55%

excess of Cost
1 Benefits estimated at 30% based on DOE recommendation. 2 Amount unknown; however amount is estimated to be minimal since teachers do not have their own classrooms. 3 "Freed-up" resources is the estimated state paid portion of the regular teacher's salary during the time that ESFL

teachers are teaching. Local school systems might be able to utilize this freed-up time to perform duties that the

local system might otherwise have to pay for with local funds.

Source: Program Records and Survey of Participating School Systems

Elementary School Foreign Languages Pilot Program

Page 14

The columns labeled "Theory" are based on DOE's assumption that the local systems receive no additional state or federal funding for their ESFL programs. Based on the average salary and benefits of the teachers at the three systems reviewed by the evaluation team, local systems would appear to be contributing approximately 48% of the Program cost. This estimate is very close to DOE's widely quoted statement that "local systems contribute about half the cost of the Program".
The columns labeled "Actual" show that at the three systems reviewed by the evaluation team, some school systems were coding salary expenses as using state Quality Basic Education (QBE) and federal funds to pay teachers in the Program. DOE financial personnel indicated that ESFL should not be a QBE Program and that QBE funds should not be used to pay ESFL teachers. Program personnel were not aware of any other state and federal funds being used to pay any ESFL teachers. Based on the way expenses were coded at the systems in our review, the local contribution is actually only about 24% of the Program cost.
Finally, the columns labeled "Actual Including Freed-Up Resources" take into account the fact that the regular classroom teachers are resources that the local school systems have available during the time foreign language instruction is being provided by ESFL teachers. The school systems continue to receive state funds for these teachers, and could use these teachers to perform additional duties that might otherwise require local funds. While the real value of these "freed-up" resources to local school systems may be debatable, based on the total state-paid portion of the "freed-up" resources, the local school systems might be receiving up to almost 38% more than the cost of the Program.
The evaluation team also identified that one school system (Columbia County) received a special $50,000 grant of federal funds in October 2002 to provide additional funding for the county's ESFL program at one school. The Columbia County School System also reported that other grants to help fund its ESFL programs were received in December 2001 ($50,000) and September 1998 ($50,000). The October 2002 grant was approved by the former State Superintendent of Schools and the first two grants (2001 and 1998) were approved by a Deputy State Superintendent of Schools. The last two grants (2002 and 2001) were paid out as lump sums before any contracts were signed and the payments were not based on reimbursement requests. In our Management Report for The Department of Education (for the Year Ended June 30, 2002) the Department of Audits and Accounts questioned whether the $50,000 grant received in December 2001 was allowable under federal provisions. The report indicated that the grant did not appear to address an authorized activity under Public Law 103-382 (Title VI Innovative Education Program Strategies). The report recommended that the U.S. Department of Education review the contract to determine the manner in which the questioned cost should be resolved. The additional funding provided by these grants was not included in our model since the Columbia County School System was not one of the three systems used to develop our model.
If the Program continues, the Department of Education should require local systems to submit information on their actual Program expenditures and verify that the local systems are not receiving any unintended state funding. The Department may also need to reevaluate the funding formula to determine if any adjustments are needed to achieve a 50% local contribution rate as has been reported to be the intent of the Program.
In its response to the evaluation, DOE indicated that it had grave reservations regarding the amounts identified as being paid by QBE funds in the model estimating local contributions to the

Elementary School Foreign Languages Pilot Program

Page 15

Program. The Department felt that additional study was needed before it could be stated that local systems are using QBE funds to offset the cost of the Program. DOE contacted four school systems (none of the systems included in the model were contacted) and two of the systems reported that they charged at least some portion of their ESFL teacher salaries to QBE account codes. According to the Department of Education, financial personnel at local systems reported that QBE codes are routinely used for all teachers, although a large portion of teacher salaries are covered by local funds. One Accounting Supervisor summarized the situation as follows: "Douglas County School System, as most counties do, codes all teachers in QBE coded program numbers, such as 1011 and 1013 for a kindergarten teacher, whether or not they are paid from QBE funds. This does not mean that the salary is actually paid from state funds, rather, that the teacher teaches in that particular area."
DOE indicated that if the Program is continued, it would require local systems to submit information on the actual program expenditures and will verify that the districts are not receiving any unintended state funding.
It should be noted that the evaluation team's discussions with personnel in the Education Audits Division of the Department of Audits and Accounts indicated that it is not uncommon for school systems to charge much more to QBE account codes than could ultimately be paid for with QBE funds. Any amount remaining in these accounts after exhaustion of the available QBE funding is usually paid with local funds. However, additional analysis of the QBE accounts would be necessary to verify that there was sufficient local funding to cover the charges for ESFL Program teachers charged to QBE account codes. It was recommended that charges for ESFL teachers not be charged to QBE account codes so this additional analysis would not be necessary.
Finding No. 7 If the Program is continued, the Department of Education needs to establish better controls over grant reimbursement payments made to local school systems for Program costs.
The Program reimburses local school systems for Program expenses based on amounts submitted on "Request for Reimbursement" forms and does not require any supporting documentation. The Program Specialist responsible for approving expense requests for payment indicated that she primarily checks the request form to verify that it has been approved by the local school system Superintendent or Chief Financial Officer. The following case provides an example of problems resulting from control weaknesses in the current system.
On November 4, 2002 the Program Specialist approved a check for $28,931.62 (36%) of the Sumter County School System's total grant of $79,953. When the evaluation team questioned why more than 25% was to be reimbursed for the first quarter of the grant period, the Program Specialist indicated that she thought that local school system expenses were significantly more than grant amounts; therefore, many local school systems drew down grant funds early in the grant period and then funded the remaining portion of the year with local funds.
On November 6, 2002 the evaluation team questioned the amount paid to Sumter County again after the Program Specialist mentioned that she heard that Sumter County was having problems filling their approved teaching positions. While the System's contract called for three teachers, the Program Specialist indicated that she was not sure if Sumter County had one, two or three filled teaching positions. The Program Specialist also

Elementary School Foreign Languages Pilot Program

Page 16

indicated that she still thought the system had sufficient expenses to justify the expense request that was previously approved.
Around December 2, 2002 the Program Specialist contacted the Sumter County School System and the system confirmed that only one of the teaching positions was filled. The school system reported that the original reimbursement request erroneously included prior year expenses (when three teachers were employed). The system submitted a revised request for $4,368.74.
On December 16, 2002 the check for the original reimbursement was voided.
On January 17, 2003 a new check for $4,368.74 was issued based on the revised request by the school system.
On February 10, 2003 the Program Specialist requested that the grant contract for the Sumter County School System be amended to reflect only one teacher and the amount of the grant reduced from $79,953 to $26,651.
If the Program is continued, DOE needs to improve controls over grant reimbursement payments. Supporting documentation should be required to accompany reimbursement requests from the local school systems. Since Program expenses are almost entirely salaries, the supporting documentation should include the names of the teachers being paid. In addition, local school systems should be required to inform the Program of vacancies in teaching positions included in the grant so the impact of any vacancies can be considered when expense requests are reviewed and so the Program can evaluate the significance of the vacancies in order to determine if a grant needs to be amended.
In its response to the evaluation, DOE agreed that controls over grant reimbursement payments should be strengthened. The Department indicated that it would require the following documentation to process future reimbursement payments.
A progress report signed by the ESFL coordinator for the school system including: the number of students receiving instruction, the names of teachers providing instruction, all pertinent details regarding the delivery of contracted services and a description of any departures from program guidelines.
A print-out documenting salaries paid for the time period covered by a reimbursement request (such as a printout from the General Ledger or Journal including the names of teachers being paid).
A list of any teacher vacancies and the last date on which a teacher was employed by the system.
The amount of any other state and/or federal funds that were used to offset ESFL model program costs.
In addition, future annual Project Completion Reports would require:
A list of teachers that taught any part of the school year.
The number and grade level of all ESFL classes taught by each teacher.
The number of students who participated in the Program.
The total cost of the Program to the local system

Elementary School Foreign Languages Pilot Program

Page 17

Documentation of any other funds, either federal or state, that were used to offset program costs.
A summary of the services provided under the contract for the year, including all pertinent details of program delivery.
Details of any departure from established Program guidelines.
Issue: To what extent could distance-learning technologies be used to complement or replace the Program?
Finding No. 8 DOE personnel reported that an informal survey of foreign language professionals indicated that the use of distance learning (having a teacher in one location broadcasting the class to another location or to multiple locations) to teach foreign language to elementary school children would not be appropriate. The distance learning format is not sufficiently engaging for elementary school students. DOE personnel recommended that any technology-based solution for delivering foreign language instruction be video-based. Several existing technology-based foreign language instruction programs such as SALSA, MUZZY, and IRASSHAI were identified.
The evaluation team calculated the current annual cost of the ESFL Program to be approximately $183 per student. While the following programs are not necessarily equivalent to the level of instruction provided by the ESFL Program, they are examples of existing technology-based programs for teaching foreign language that were identified during the course of this evaluation.
SALSA, an award-winning Spanish language video series for children, was produced by PeachStar Education Services, a Division of Georgia Public Broadcasting (GPB). Each 15-minute episode is introduced in English, and the remainder is presented in Spanish. GPB contracted with Emory University in 1998 to evaluate the effectiveness of the SALSA program in regards to generating viewer interest, promoting cultural appreciation, and facilitating Spanish-language acquisition. Emory's study found that younger students expressed more interest than older students, non-Hispanic children gained appreciation of and respect for Hispanic people and cultures, and students with limited Spanish-language skills acquired core Spanish vocabulary. No information is available on how much SALSA is used in elementary schools in Georgia. In fiscal year 2000, 38% of the elementary school media specialists that responded to a GPB survey indicated that teachers in their schools used SALSA to some degree. SALSA is mandated in Wyoming for kindergarten through second grade students and has been in use for one year in about 60% of Wyoming's school districts.
DOE personnel estimated the following statewide implementation costs for SALSA. The estimated costs were not tested or validated by the evaluation team. DOE personnel estimated that 80% of Georgia's elementary schools would elect to offer Spanish. It would cost around $2.5 million dollars to implement the SALSA program at these schools in grades one through three (since SALSA is already in existence and would not require development costs), and approximately $600,000 per year thereafter. This implementation cost estimate includes updating the SALSA website with additional interactive materials, developing an assessment tool, maintaining the website, and training teachers around the state on the program. This estimated cost would be

Elementary School Foreign Languages Pilot Program

Page 18

approximately $9 annually per student during the initial implementation period and $2 annually per student thereafter. Currently there are no SALSA programs that are applicable to grades four and five.
MUZZY, a video series produced by the British Broadcasting System, offers multiple languages and is an alternative available to school systems that do not want to teach Spanish. For systems choosing to use MUZZY, Early Advantage, the firm that markets MUZZY in the United States, would provide teacher training and materials, as well as student assessments.
DOE personnel estimated the following statewide implementation costs for MUZZY. The estimated costs were not tested or validated by the evaluation team. DOE personnel estimated that 20% of Georgia's elementary schools would elect to offer a language other than Spanish with MUZZY. The cost to implement MUZZY at these schools would be approximately $7.95 million for grades one through three. The cost to implement MUZZY would be approximately $115 per student for the first year. After MUZZY is implemented, the estimated annual cost would be a $1.5 million licensing fee, which would be about $21 per student.
MUZZY also has program materials for instruction for grades four and five in French, German, Italian and Spanish. The estimated cost to implement MUZZY for grades four and five at all elementary schools would be $8.5 million. The cost to implement MUZZY would be approximately $37 per student in the first year and $32 per student in the second year. After MUZZY is implemented, the estimated annual cost would be a $2.4 million licensing fee, which would be about $10 per student.
IRASSHAI is not specifically designed for elementary school foreign language instruction and the developers of the program felt that is was far too advanced for elementary school students; however, it is another example of a technology-based program for providing foreign language instruction. IRASSHAI is an award-winning Japanese distance-learning program that is also produced by PeachStar Education Services. IRASSHAI was developed in 1996 and is designed to meet the needs of high schools, colleges and business. It combines satellite television instruction, web/multimedia, and telephone interaction. Students can call a toll-free number twice weekly for small-group conversations with Japanese native speakers (15 to 20 minutes each session).
IRASSHAI is free to Georgia schools and is also offered to Public Broadcasting Service (PBS) affiliates nationwide. The cost to out-of-state schools is $650 per student, which includes textbooks, website access, and audio interaction.
If the Program continues, DOE needs to evaluate the relative effectiveness of the various alternative delivery methods to identify the mix of techniques that provides the most efficient and effective means of providing foreign language instruction to elementary school students.
In its response to the evaluation, DOE agreed with the information provided in the report regarding the use of video-based programs. The Department wanted to emphasize that videobased programs do not produce the same results as programs taught by qualified, certified language teachers.

Elementary School Foreign Languages Pilot Program

Page 19

The Department of Education's state-level monitoring of the Program was not within the scope of this evaluation; however, the evaluation team identified indicators of monitoring problems that should be studied and evaluated in greater detail if the Program is continued. More state-level monitoring is needed to ensure school-level compliance with Program standards identified in contract provisions. In addition, the Program has not done any evaluation of the relative efficiency of the programs offered by the various school systems. The average costs per student at the school systems ranged from $120 to $381 annually. These average costs seem to indicate extremely large differences in the efficiency of programs provided by the school systems.
Reviews of Program records and information provided through a survey of participating school systems that was conducted by the evaluation team identified the following examples where systems did not appear to be in compliance with Program standards identified in the Program's contracts with the local school systems.
None of the 15 systems were submitting quarterly progress reports detailing the services provided by the systems (including the number of students participating and other pertinent details) that were required by their contracts.
Three of the 15 systems had schools that were teaching fewer grades than required in their contracts. In addition, nine of the 15 systems had schools that were teaching more grades than required by their contracts.
Six of the 15 systems had not submitted their Quarterly Request for Reimbursement more than five months after the end of the first quarter of fiscal year 2003.
Four of the 15 systems had one or more teachers that exceeded contract provisions limiting teachers to a maximum of eight classes per day. (These teachers did not appear to meet the requirements for exception to this provision.)
Two of the 15 systems did not seem to offer an opportunity to continue the sequential study of the same foreign language in middle school as required by their contracts.
The following table (Exhibit 7) shows the large range in average Program cost per student at the school systems which seems to indicate extremely large differences in the efficiency of programs provided by the school systems.
In its response to the evaluation, DOE noted that it has already taken steps to increase the amount of state-level monitoring to ensure school-level compliance with Program Guidelines stipulated in contract provisions. Reimbursements will no longer be processed without accompanying Progress Reports and a Project Completion Report will be required at the end of the fiscal year. If the Program continues, DOE is considering conducting periodic unannounced site visits to further assure compliance.
DOE also noted that it did not feel the range of average cost per student was as dramatic as it initially appeared. The cost of $380.73 per student in Sumter County will be reduced to approximately $126 per student once that contract has been amended. The Department also noted that other districts with higher than usual per student costs (Taliaferro County and the City of Decatur) are very small school systems with smaller than typical class sizes. In addition,

Elementary School Foreign Languages Pilot Program

Page 20

teachers at these systems were teaching six classes per day rather than eight classes per day. DOE indicated that while these situations may lessen overall "efficiency", the decision to offer smaller classes or to limit teacher workload are local district decisions.
Finally, DOE indicated that if the Program continues, it would use the data collected in Project Completion Reports to analyze Program efficiency and will request additional documentation if student costs are not adequately explained in the reports.

Exhibit 7

Average ESFL Grant Per Student By School System

(Fiscal Year 2003)

Avg. Number Avg. Number

Avg. Amount of

of Classes per of Students

Grant

Grant Per

School system

Teacher

per Class

Amount

Student

Douglas County

8.00

27.67 $ 79,953

$ 120.41

Forsyth County

9.25

21.79

309,812

128.07

Whitfield County

9.00

21.56

77,953

133.94

Dougherty County

8.00

20.27

154,906

136.48

DeKalb County

8.33

20.40

154,906

151.87

Cherokee County

8.17

20.53

155,906

154.98

Columbia County

7.71

20.78

181,557

161.82

Atlanta Public Schools

7.64

17.31

258,510

177.79

Catoosa County

6.75

21.19

103,604

181.13

Richmond County

8.00

17.79

77,953

182.56

Fulton County

6.00

21.00

128,755

204.37

Taliaferro County

6.00

18.67

26,651

237.96

Decatur City Schools

5.50

17.77

104,604

267.53

Gwinnett County Sumter County1

7.33 10.00

12.73 21.00

77,453 79,953

276.62 380.73

TOTAL

7.84

20.17 $1,972,476

$ 164.04

1 Sumter County's grant funding was set for three teachers; however they were only able to staff

one position. (On 2/10/03 an amendment to the contract was requested reducing the grant

to $26,651.)

Source: Program Records and Evaluation Team's Survey of Participating Systems

Elementary School Foreign Languages Pilot Program

Page 21

Appendix School Systems Participating in the ESFL Pilot Program

Catoosa

11

2

Whitfield

Walker

2
Lumpkin1

Rome
2

Carte1rsv ille

2
Cherokee

Forsyth
2

1

1

Gwinnett

Paulding

Decatur

Douglas
3

Atlanta
32
1

22
DeKalb

Fulton

2
Coweta

Habersham
2

Elbert
2

Num ber of schools that currently participate show n in circle
Num ber of schools that no longer participate show n in circle

Taliaferro
1

Columbia
2 1
Richmond
Bacon (OA)

3
Sumter2
2
Dougherty

1
Pulaski

2
Telfair

McIntosh
1

1 FY00 contract included Lumpkin County Middle School, grades 6 and 7 (not reflected on the map). 2 Sumter County w as contracted for three schools in FY03, but could only staff one school.
SOURCE: Program Records

For additional information, please contact Paul E. Bernard, Director, Performance Audit Operations Division, at 404.657.5220.

Elementary School Foreign Languages Pilot Program

Page 22