PROGRAM EVALUATION
Prepared for the Budgetary Responsibility Oversight Committee
Department of Defense Youth Challenge Academy August 2001

Performance Audit Operations Division Russell W. Hinton, State Auditor Department of Audits and Accounts

254 Washington St. SW Atlanta, GA 30334

The purpose of the Youth Challenge Academy is to provide Georgia's high school dropouts with academic and life skills training aimed at improving employment potential and creating productive citizens. The Academy, which is operated by the Georgia National Guard of the Department of Defense, has two facilities in Georgia. One facility is located in Fort Stewart (Hinesville) and was established in 1993 and the other facility is located at Fort Gordon (Augusta) and was established in September 2000.

This program provides an alternative for Georgia's high school dropouts. However, it does not serve the entire dropout population. During the 1999/2000 school year, approximately 27,000 of the 378,799 students enrolled in Georgia's public high schools dropped out prior to graduating1. During the same period, the Academy accepted 593 students.

In 1993, Georgia was selected as one of 10 states to receive funds for the Youth Challenge Academy pilot program. The pilot was designed to target at-risk youths and determine whether life skills and employment potential of high school dropouts could be significantly improved through militarybased training. In 1997, Congress provided permanent authorization to continue operation of the Youth Challenge Academies throughout the nation. At present, there are 27 Academies in 24 states and Puerto Rico and each graduates two classes per year.

Exhibit 1 Student Living Quarters--Fort Stewart (Hinesville)

To be eligible for admission to the Acad-

emy, a candidate must: (1) be a Georgia resi-

Source: Youth Challenge Academy, Fort Stewart

dent; (2) be between the ages of 16 and 18; (3) be a high school dropout; (4) be unemployed or un-

(1) This is the number of dropouts for grades 9 through 12 as reported by the Department of Education. Youth Challenge Academy

Page 1

deremployed; (5) be drug-free; and, (6) have no current involvement with the legal system and not be on parole or probation for anything other than juvenile offenses. Attendance at the Academy is completely voluntary and participants cannot be sentenced into the program. Applicants are generally referred by the local school system or by other local public agencies or officials. Applicant selections are made by a state screening committee which includes state agency personnel. In an effort to obtain a diverse group of students, the committee attempts to select one applicant from each county represented by the pool of applicants. As part of the application process, the Academy reviews prospective student's scores on the Test of Adult Basic Education (TABE) to identify their math and reading ability in terms of grade level (grade 1-12).

The program consists of three phases: Pre-Challenge, Challenge, and Post-Residential. Students live on the campuses during the first two phases, which last a total of 22 weeks. Military personnel, functioning as platoon leaders and mentors, teach and train the students during this time.

Pre-Challenge (Hard Core): In this two-week phase, participants learn to adapt to a military

lifestyle, including haircuts, physical training, and military customs. This phase of the Academy, de-

signed to teach cadets to successfully function as members of a group, includes activities that are

based on the Georgia Army National Guard leadership training programs. Participants who cannot

or will not adapt to this phase of training are dismissed from the Academy and are not included in

the Academy's completion statistics. Historically, about 20% of the students who are initially ac-

cepted drop out during this phase of the program.
Challenge: During this 20-week phase, participants are

Exhibit 2 Eight Core Components

involved in a series of training activities that encompass the eight core components of the Youth Challenge Academy,

Academic Excellence

as defined by the National Guard Bureau. Students must reach a specified minimum level in each of the core compo-

Responsible Citizenship

Community Service

nents to graduate from the Academy. The components focus on improving the students' life and job skills as well as their physical fitness, and understanding of concepts such

Physical Fitness

HOLISTIC GROWTH

Health & Hygiene

as leadership and community service (see Exhibit 2). The

belief is that growth in these eight areas leads to holistic

Life Skills

Job Skills

growth, improving an individual's chances of operating effectively in the world. Specific activities are designed to

Leadership & Followership

address each component. For example, to complete the

Source: National Guard Bureau

community service component, students must work 40 hours on community

projects; to complete the academic excellence component, students must obtain a General Educa-

tional Development (GED) diploma or show improvement in their Test of Basic Adult Education

(TABE) scores.

Youth Challenge Academy

Page 2

GED Notes
The GED tests knowledge in five parts: writing, math, social studies, science, and liberal arts. Passing the test indicates a level of knowledge equivalent to the average high school graduate.
In Georgia, individuals are allowed to take the test one part at a time and take the test up to three times per year.

States are permitted to focus their programs on a particular area as long as all eight components are addressed. Georgia's program focuses on helping students pass the GED test (which address the academic excellence component), because it provides students with the option of enlisting in the military or enrolling in college or technical institutions, in addition to the possibility of joining the job market. The Academy provides self-paced instruction to prepare students to

take the test. While studying for the test, students continue to be in-

volved in activities related to the remaining seven components. Once students obtain the GED, they

may move on to job shadowing or attending training courses provided by a local technical institute

(e.g., Savannah Technical Institute or Augusta Technical Institute).

Throughout the 20-week period, students are subject to random drug screenings and are required to adhere to standards of discipline. Those who fail the drug screening or are considered disciplinary problems are removed from the program. In addition, because the program is voluntary, students may opt to leave at any time. Those who obtain a GED and complete the remaining seven components receive an adult high school diploma. Those who do not obtain a GED, but meet the requirements of the eight core components, receive a Youth Challenge Academy Certificate, acknowledging successful completion of the 22-week program.
As shown at right, attrition occurs at each level of the Program. Using Fort Stewart's Class 15 as an example, 229 (64%) of the original 358 students who showed up for the first day of the program ultimately graduated.

Exhibit 3 Attrition by Phase YCA Class 15 (July to December 2000) Fort Stewart, Hinesville

Grad2u2a9ted RePsoisdte-ntia l

Enro2l8le2d

Challenge

Regis3t5e8red App6l2ie7d

Pre-Challenge Application

Source: YCA files

Post-Residential: This phase begins after graduation, lasts for 12 months, and takes place in the
community. During this time, graduates work at achieving the career and educational goals they developed while at the Academy. Each participant is guided through this period by a mentor, chosen by the cadet and approved by the Academy. Mentors are required to be in contact with the participant at least four times a month, and must submit a monthly report to the Academy detailing the student's current education or work activities. As stated by the Academy, the goal of the PostResidential phase is to ensure that cadets attain their goals of gainful employment or education while remaining free from criminal activity or substance abuse.

Youth Challenge Academy

Page 3

The U.S. National Guard has contracted with Social Consultants Incorporated (SCI), a management consulting firm, to conduct analysis on the design, conduct and effectiveness of the Challenge programs in each state. The compiled information for all states is provided to Congress on an annual basis. To facilitate their review, SCI developed and provided each Academy with a standardized data collection system called the Challenge Monitoring and Evaluation Information System (CHAMEIS). All of the Academies are required to use CHAMEIS, which allows them to track information such as GED scores and the placement status of the graduates. In addition, individual management reports can be generated to track students' progress throughout the Academy.

Program staff reports that since its establishment in 1993, a total of 2,495 participants representing 130 counties have graduated from the Fort Stewart campus. Of those graduates, 2,275 (91%) have earned their GED. This is well above the national Youth Challenge average of 64%. The Fort Gordon campus opened in September 2000 and graduated its first class of 96 students in March 2001. Seventy-two (75%) of these students received their GED. It should be noted that, in Georgia, students are permitted to take the GED test up to three times each year and may test on the individual five sections separately. Staff reported that this flexibility increases the likelihood that students will be successful in passing the exam. Other states may or may not have the same flexibility to test by section or retake the exam.
As shown below, the Academy accepts approximately 59% of those who apply, and of those accepted, 58% actually make it through the program to graduation.

Class by Dates

Number of Students by Stage of Program

Registered Enrolled in

Applied to Accepted into for Hard Challenge

Received

the Academy the Academy Core Phase Phase Graduated(1) GED

6/99-12/99
Fort Stewart 1/00-5/00 6/00-12/00
1/01-5/01 Fort Gordon 11/00-3/01
TOTALS for the period

537 567 627 648 237 2616

306

278

228

187

153

287

267

205

175

138

359

358

282

229

167

399

356

260

212

161

205

171

120

96

72

1556

1430

1095

899

691

Source: YCA records (1) See Appendix A for the number of graduates by county

Youth Challenge Academy

Page 4

Budget and Expenditures: Youth Challenge Academies receive federal funds which must be
matched with state money. The match requirement for states has been increased to 40% during state fiscal year (FY) 2002. It will remain at this level for the duration of the program.
In FY2000, the Academy expended $4.17 million, of which $1.3 million (31%) were state funds and $2.86 million (69%) were federal funds. In FY2001, the Academy expended $5.3 million dollars, of which $1.76 million (33%) were state funds and $3.51 million (67%) were federal funds. The campus at Fort Gordon was added in FY2001, which increased the funding. The FY2002 budget for Fort Stewart is $5.57 million, of which $3.34 million (60%) is federal funds and $2.23 million (40%) is state funds. The FY2002 budget for Fort Gordon is $2.8 million, of which $1.68 million (60%) is federal funds and $1.12 million (40%) is state funds(2).
Cost Per Student: Georgia's average cost per student is budgeted at $10,000 based on the fed-
eral fiscal year funding and a target enrollment of 560 students. The national average cost per student is approximately $14,000. The cost per graduate for state fiscal year 2000 was $12,698. We were not able to obtain verifiable information on number of graduates from other states to calculate a national cost per graduate amount.
The Academy at Fort Gordon is housed in buildings which were donated by the National Guard and are therefore not part of the cost per student. The facilities at Fort Stewart are leased from the National Guard Unit. According to Academy staff, the difference in the state's cost per student versus the national average is due in part to Georgia's choosing to eliminate the stipend which can be provided to students in favor of accepting more cadets into the program.

As discussed earlier, the National Guard Bureau stipulates that youth cannot be adjudicated (i.e, sentenced by a judge) into the Youth Challenge Academy. We contacted 24 states and asked whether they had a program modeled on the Youth Challenge Academy for adjudicated youth; only one state reported having such a program. Oklahoma's Bravo Company focuses on youth from 13 to 18 year olds(3). The older students may take the GED exam; however, the focus for the younger students is getting back into the traditional school environment. The program is completely state funded.
It should be noted that Arkansas also reported a separate military style training program for adjudicated youth, known as the Civilian Student Training Program; however, staff specified that it was not modeled on the Youth Challenge Academy. Students range from 11 to 17 and receive academic and life skills training, as well as vocational training. The program is completely state funded.

(2) Minor discrepancies in the percentage match may occur due to overlapping state and federal fiscal years and the carryover of fed-
eral monies. (3) Oklahoma also operates the Thunderbird Youth Academy which is its Youth Challenge Academy and, therefore, does not admit
adjudicated youth.

Youth Challenge Academy

Page 5

This evaluation was conducted in compliance with O.C.G.A. 45-12-178 enacted in 1993. The evaluation was conducted under the Budgetary Responsibility Oversight Committee's calendar year 2001 theme "Public Safety and Sentencing Offenders." The scope of the evaluation focused on fiscal year 2000 with additional reviews conducted as necessary and the methodology included interviews with YCA staff; reviews of YCA policies and records; and observations of activities.
The evaluation focused on the following objectives: determine if the Academy has established adequate measures for determining its success in
achieving its goals and objectives and if its data collection method is valid and reliable;
determine the extent to which the Academy is achieving its goals and objectives; and
determine to what extent the Academy compares to similar Programs in other states.
The entire report was discussed with appropriate personnel at the Youth Challenge Academy and a draft copy was provided for their review. The Academy was invited to provide a written response to the report and to indicate areas in which they planned to take corrective action. Pertinent responses from the Academy are reflected in the report as appropriate.

The Academy should take the steps necessary to establish an effective method of tracking graduates in order to evaluate its effectiveness in fulfilling the mission. The Academy currently relies on monthly reports from graduates' mentors to gather information on graduates; however, it has not established procedures to be followed if these reports are not received. In addition, a review of the files for graduates indicated that the information on their activities may be inaccurate.
The reports from mentors are used to determine whether graduates are working, enrolled in the military or school, or incarcerated. This information allows the Academy to monitor the progress of graduates and measure the level of success of the program. As discussed below, if reports are not received, the Academy must develop alternative methods for obtaining information and it must ensure that the information collected is accurate.

The audit team reviewed mentors' reports for 21 (10%) of the 189 students who graduated in December 1999 and found that only one mentor had submitted all the required reports. Of the remaining 20, none submitted a report during the last six months of the post residential year, and very few submitted reports during the first six months, even though guidelines require mentors to report every month. According to staff, the Academy has an unwritten policy of attempting to contact mentors if no report is received. However, no documentation could be located to indicate the Academy had contacted the mentors for the months in question. According to the SCI, without documentation of contact with mentors or graduates, the Academy's statistics on placement would be considered invalid.
We attempted to locate graduates through the Department of Labor (DOL), the Department of Technical and Adult Education

Youth Challenge Academy

Page 6

(DTAE), the Board of Regents, and the Department of Corrections and found that:
According to DOL, 139 were employed for at least a portion of two quarters of the post-residential year. However, of those 139, only 60 were employed for some portion of all four quarters;
Sixty-eight were enrolled in a technical school according to the DTAE;
Three graduates were enrolled in twoyear colleges according to the Board of Regents; and,
Seven had been arrested and placed on probation according to Corrections.
Without complete and accurate information about what graduates are doing following their completion of the program of study, the Academy cannot determine if its efforts have been successful in helping these young people become responsible, productive members of society.
It should be noted that Arkansas reported having similar problems tracking graduates. To address their problems, the Academy pays a quarterly post-residential stipend and requires graduates to contact the Academy (directly or through a mentor) in order to receive the money. Additionally, the National Guard Bureau recommends a ratio of one post-residential case worker per 40 graduates. At present, Fort Stewart employs two case workers to oversee 441 graduates.
In its response, the Academy noted its agreement that hiring additional caseworkers would facilitate the tracking of graduates. However, at present, the Academy has not received funding for additional positions. The Academy also noted that it does not have access to the records of DOL, DTAE, DOE or DOC to track graduates.
The Academy has adequate measures for determining its success in meeting the objec-

tives of the residential phase of the program. The National Guard Bureau has developed objectives for the residential phase which focus on the eight core components. As discussed below, the Academy has outlined the specific activities to be conducted for each component and is using the Bureau's standards for what constitutes successful completion. However, it also needs to ensure that the standards are consistently applied and adhered to.
(A) Objectives have been established which can be used to measure the Academy's effectiveness in teaching cadets based on the eight core components. For example, academic excellence is measured by the student obtaining the GED or improving his/ her Test of Basic Adult Education (TABE) scores; physical fitness is measured by receiving the Presidential physical fitness award. The Academy has developed activities to ensure that students have an opportunity to achieve the standard set. For example, to complete the leadership/followership component, the Academy states that students will have an opportunity to rotate through a squad leadership position.
(B) While the standards are appropriate to measure the program's effectiveness, a review indicated that some graduates of the program did not successfully meet all applicable standards. For example, 10 of the 96 graduates from the first class at Fort Gordon did not meet the standards of academic excellence because they did not either obtain a GED or show a satisfactory improvement in their TABE scores.
The objectives for the residential phase of the program appear measurable and can be used to assess the Academy's effectiveness in teaching the eight components. However, the Academy should ensure that standards are applied consistently to ensure the results are accurate.

Youth Challenge Academy

Page 7

In its response, the Academy stated that the tracking of student success during the residential phase is accomplished through a computerbased database furnished by Social Consultants, Inc. Many improvements have been made to this software over the past six years. Another upgrade to the system is scheduled in the fall of 2001.
The Academy cannot ensure the reliability of the information found in the CHAMEIS database, due to limited computer controls. The Academy does not have adequate management, physical, or operating system/application controls over its current data collection system. The audit team conducted a review of the computer system controls and found the following:
Management Controls. The Academy does not have a formal process in place for authorizing access to the CHAMEIS database. Our review found that management verbally authorizes the network specialist to provide network access. A written policy and standardized form to regulate access to the database should be developed to ensure that only authorized persons can use the system.
Physical Controls. Currently, the computers are in an open office environment, rather than in a locked office. In addition, although the Academy backs up the database files frequently, the backups are stored in an unlocked drawer next to the computer. Should a disaster destroy the computer system, the backups would be necessary to recover the system. However, as they are currently stored, the Academy runs the risk of losing the backups as well. The Academy needs to ensure that the computers are housed in secure offices accessible only to the system administrator and that backups are stored offsite.
Operating System/Application Controls. Utilizing the current system, the Academy

cannot implement the necessary controls. The database is currently operating under a Windows 95 operating system, which does not provide the level of access controls necessary to prevent and/or detect unauthorized users. Access to the files cannot be limited, and as a result, unauthorized persons can alter the database and/or view confidential information. Consideration should be given to upgrading the systems to Windows NT/2000, or a compatible operating system, so that stronger security measures can be implemented thereby providing better control over the system and its data.
It should be noted that because the computer controls could not be relied on, the team conducted an analysis of data authenticity to determine if the computer-based data accurately reflected the source records. The team compared GED scores in the CHAMEIS database to the GED transcripts in the Academy's files for two classes. The comparisons revealed a 91% match rate for all entries. Although it appears that the data is correct, no assurances exist that past or future data can be relied on due to the lack of the appropriate controls.
The Academy stated in its response that the CHAMEIS information and database are currently limited to the two computers in a single office. Two people enter all information into this database. The new upgrade to this system will enable CHAMEIS to be accessed by virtually all computers in the complex. Each functional area, i.e., medical, counselor, teacher, cadre, operations, admissions, will be able to manipulate the data in their domain. This will require specific authorizations, codes, and passwords, which are currently under development. The anticipated upgrade to Windows NT/2000 will allow these protections.

Youth Challenge Academy

Page 8

The Results Based Budgeting (RBB) goals and objectives are not adequate to determine whether the Academy is successful in accomplishing its mission. As stated in the 2002 Governor's Budget Report, the Academy's RBB goal is to intervene with at-risk youth who are high school dropouts, to produce employable, self-sufficient and productive citizens. While the goal reflects the overall purpose of the program, the desired result (graduate 360 students per year) is not an effective measure of how well the Academy has accomplished its goal. Measuring the number of graduates only provides information on the output of the Academy, not its effectiveness in providing successful intervention, or the impact of this intervention. In 1998, the Academy submitted its state plan to the National Guard Bureau; the plan includes measurable objectives which reflect the goals and mission of the program, such as: 80% of graduating students will be placed in employment or educational institutions within 30 days of graduation from the resident phase; and, 80% of graduates will remain on the job or in school after the end of the post residential phase. Such objectives allow the Academy to determine whether its program is having the desired long-term impact on students and should be incorporated into the RBB as more effective measures of the Academy's effectiveness.
In its response to this finding, the Office of Planning and Budget stated that it is currently working with the Academy to develop more specific goals. The Academy also noted its intention to include additional measurable goals in its next RBB submission.

While the Program is effective at helping cadets obtain their GEDs (approximately 2,275 (91%) of the 2,495 graduates to date have obtained a GED), attention needs to be given to ensuring that those who do not pass the GED test benefit from the program. A review of 189 graduates revealed that 34 did not receive a GED, and of these, only 14 (41%) were employed during the first three months following graduation. Additionally, only 16 held a job for more than two quarters of the first year following graduation.
Obtaining a GED increases a student's options following graduation; he or she now has the choice of enlisting in the military, continuing his or her education at a technical institute or traditional college, or entering the job market. However, those who complete the program by completing the life skills portion only have the path into the job market. The Academy should take steps to ensure that those graduates of the program who do not obtain a GED acquire sufficient skills to also achieve the goals of the program by becoming productive, responsible members of society.
The Academy's response stated that achievement of the GED is paramount to graduates' future success. As a result, students who struggle with attainment of the GED are afforded additional time to study for the test, even if the time comes at the expense of job shadowing or technical college coursework. Those graduates who have not attained a GED are directed to the Job Corps for additional assistance.

Action should be taken to ensure that students who do not receive a GED are adequately prepared to succeed following graduation.

Youth Challenge Academy

Page 9

Appendix A
YCA Graduates by County First 16 Classes

Dade Catoosa

Murray

Fannin

Towns

10

Union

Rabun

Walker 2

Whitfield 3

Gilmer

1 WhiteHabersham

Chattooga

Gordon 14

5

Lumpkin 1

Pickens

2

Dawson

3

5 Stephens 4
Banks 1 Franklin

Floyd

Bartow Cherokee Forsyth Hall

4

Hart 2

4 Polk

7

17

14

4 Jackson

2

Madison

Barrow

2

Elbert 4

HaraH11lesCaoarn5rdroPllauD7lCdooi3un7w3g7gelatasC1oFF2b1u8ab23ylt48eoCtntle1aS1yp7tDoa1eln12dH7Kin0e2Ragn6Globrcwy9k4indB0nauN4elteettstw4tWoJnaa6lst1o2pneMr Oo1rPcgou2a1Cntnn0elaae7rmkGeOregelenH1tehaonrTcpoaeclikafWerWir3lkoGae2lrsaresMcnoLccdi2nku2cfofCilenol2u8mRbiicah8m8 ond

One From County Two to 14 15 to 30 31 to 99 100 to 199 Over 200 No graduates
Fort Gordon

Pike Lamar

Baldwin

Troup Meriwether 3

8

1

4 Monroe 20

Jones 45

7

Jefferson

Washington

5

Burke 10

MCHShuat1asre1rtcwit6soaagh2reot eochWTMeaeea1lb1rbsioottenSrUcThpa1lsSeyouylonmr tMeCra1rcao1wn8foPrDed1oa64coHhlyoB2u8i4b0s0btPouTnlw6a1is2gBkgilsec1WklDieloky84idngseoLnau6renWs5h1eJeMolTehorr3nnestu2go1tolnemnTEeomroy7am8nbuselCaT1nJa3dettnlne2Ekarivln1las1Bnsu2ll2oScchr4evBern3y8aEnffin3g8Chhaamtham

Quitman

Randolph Terrell

1

6

6
Lee 16

Clay

Calhoun

Dougherty 68

Wilcox

Crisp

2

Telfair

10 Turner

Ben Hill 22

Jeff Davis Appling 10

2

Worth

6

Tift

Irwin 3

Coffee 10

Bacon 4

33
Long 57
Wayne 13

Liberty 372
Mcintosh 6

173
Fort Stewart

Early

Baker

5 Berrien Atkinson

Pierce

1 Miller 2

Mitchell 11

Colquitt 7

Cook 16

3

3

Lanier

9

Ware 29

Brantley 2

Glynn 19

Seminole Decatur 4

Grady Thomas 2

1

Clinch

Brooks Lowndes

1

2

11

Echols

Charlton

Camden 20

Source: Program files

For additional information, please contact Paul E. Bernard, Director, Performance Audit Operations Division, at 404.657.5220.

Youth Challenge Academy

Page 10