Vocational Rehabilitation Program: greater focus on client outcomes and improved operational practices needed [2016]

Performance Audit Report No. 14-18 162016

October 2016

Georgia Department of Audits and Accounts
Performance Audit Division
Greg S. Griffin, State Auditor Leslie McGuire, Director

Why we did this review
More than $70 million in state and federal funds are annually appropriated to the state's Vocational Rehabilitation (VR) Program to help individuals with disabilities become employed. We reviewed the extent to which the Program is: 1) serving those eligible and in need of its services and 2) accomplishing its mission of employment and independence for individuals with disabilities, including delivering services in a cost-effective manner.
About the VR Program
The VR Program is administered by the Georgia Vocational Rehabilitation Agency. In fiscal year 2016, $76 million in state and federal funds were appropriated to the Program, primarily in support of its 39 field offices and services to eligible clients with disabilities to help them prepare for, locate, and retain employment. During fiscal year 2016, the Program served approximately 34,000 clients with a variety of disabilities, including both mental and physical impairments. Services such as vocational counseling, job placement, and mental health and substance abuse treatment are provided by VR staff, nonprofit community rehabilitation programs, or other vendors.

Vocational Rehabilitation Program
Greater focus on client outcomes and
improved operational practices needed
What we found The primary purpose of the VR Program is to help clients with disabilities obtain employment. However, the Program only tracks the outcomes of the clients that end up receiving training services and has not actively monitored the length of time it takes clients to move through the rehabilitation process to reach their employment goals. In addition, we identified various operational issues that likely impact clients' experiences and the Program's efficiency and effectiveness.
Rather than evaluating the outcomes of all clients that it comes into contact with, the Program tracks and reports on the employment-related outcomes of clients that make it to the final phases of a lengthy screening, assessment, and training process. To understand the experience of VR clients, we followed a cohort of all clients who entered the Program in fiscal year 2008 through exit and found:
Of the 11,678 clients who entered the Program, 61% exited without an employment outcome (unsuccessful); 36% exited as "successfully rehabilitated" as of August 2014.This is in contrast to the Program having reported a federal indicator of approximately 60% exiting the Program with an employment outcome in four of the last five years.1
Of the 7,124 unsuccessful clients, 37% exited the Program prior to receiving any training services.
1 The discrepancy is a result of two different methods for calculating outcomes. Our analysis looks at the number of clients that reached their employment goal as a percentage of all clients who entered the Program. The federal indicator looks at the number of clients who received training that reached their employment goals.

270 Washington Street, SW, Suite 1-156

Atlanta, Georgia 30334

Phone: (404)656-2180

www.audits.ga.gov

Our analysis also revealed that clients in the 2008 cohort averaged 22 months (ranging from 2 days to 7 years) in the Program before exiting with or without an employment outcome. While the length of time a client is in the Program is expected to vary because of individualized client training plans, the Program does not actively monitor clients' progress throughout the rehabilitation process. Clients who exited unsuccessfully represented a greater percentage of clients who were in training status for one or more years.
Management's efforts to measure the outcomes of the Program have been limited to ensuring compliance with federal performance indicators. These indicators, however, ignore a significant percentage of clients served and fail to measure all aspects of the Program's effectiveness.
The following issues were identified as having the potential to impact client outcomes and the Program's effectiveness and efficiency.
Services: Clients and counselors lack information and tools needed to ensure clients can make informed decisions about providers and services. The implementation of performance-based contracts with service providers could assist in evaluating the cost-effectiveness of various services and providers.
Accessibility and outreach: The Program's efforts to promote awareness of the vocational services available through the Program have been limited and have been conducted in an ad hoc manner as district staff time permits.
Counselor turnover: Turnover rates for counselors in the VR Program have been around 30%, which is higher than the state government rate of 19%. High turnover among counselors increases the caseload of remaining counselors and can have a negative impact on clients' experience with the rehabilitation process.
Information management: GVRA's case management system has not been an efficient or effective tool to assist case managers in managing their caseloads. In addition, the system's design and implementation impede management's ability to monitor Program activities and client outcomes and identify areas for improvement.
Administration: Improvements in several administrative areas--submitting reimbursement requests to the Social Security Administration, conducting client financial needs assessments, and inventory controls over equipment purchased for clients--are needed to improve efficiency, consistency and fairness, and stewardship over resources. In addition, GVRA lacks adequate policies and procedures for ensuring conflicts of interest are identified and resolved and that adequate safeguards are in place to protect sensitive client information.
At the time of our review, GVRA officials were aware of some of these issues. As new issues were brought to their attention during the audit, they began taking steps to address many of them.
What we recommend To improve client outcomes and the Program's operational efficiency, GVRA should: take steps to monitor and evaluate clients' progress toward their employment goals, including an assessment of outcomes associated with individual services and providers; continue efforts to promote awareness of and expand services to eligible members of its target population and recruit and retain adequate counseling staff to serve its clients; and improve administrative processes to maximize its efficiency and ensure operational controls are adequate. See Appendix A for a detailed listing of recommendations.
GVRA Response: In its response, GVRA indicated that it has no major disagreements with the report. In addition, GVRA stated it "has already taken some corrective steps to address the findings and recommendations, and...will provide greater detail in [a] subsequent corrective action plan, which will specifically address each finding and recommendation."
Pertinent aspects of GVRA's corrective action plan have been incorporated into the report.

Vocational Rehabilitation Program

i

Table of Contents

Purpose of the Audit

1

Background

Mission and History

1

Organization and Staffing

2

Vocational Rehabilitation Process

4

Program Services

6

Activity Data and Performance Measures

7

Workforce Innovation and Opportunity Act (WIOA)

8

Financial Information

9

Findings and Recommendations

The outcomes of VR clients have not been tracked, monitored, or evaluated to

ensure the Program effectively serves its clients.

11

GVRA should evaluate and improve monitoring activities to ensure clients move

through the rehabilitation process in a reasonable time frame and that areas

needing improvement are identified.

14

GVRA should take action to produce information clients need to ensure they

make informed decisions about the services they receive.

17

GVRA should continue efforts to establish performance-based contracts to

ensure CRPs deliver quality, cost-effective services to VR clients.

19

The Program should improve its outreach efforts to ensure persons with

disabilities are aware of the vocational services available to assist them in

obtaining employment.

21

GVRA should continue its efforts to reduce counselor turnover and streamline

the hiring process to fill vacancies in a timely manner.

23

GVRA should take steps to improve its administration of the Ticket to Work

Program.

27

GVRA should take steps to ensure that it consistently conducts financial needs

assessments as required by Program policy.

29

GVRA should improve controls over equipment purchased for clients and ensure

that equipment is properly inventoried and recovered.

31

GVRA should ensure that its next case management system is acquired and

implemented according to best practices.

34

GVRA should ensure that conflicts of interest are reported and managed

appropriately.

35

Vocational Rehabilitation Program

ii

Additional guidance is needed to enhance protection of confidential client

information.

37

GVRA should ensure that it has documented policies and procedures for key

business operations.

39

Appendices

Appendix A: Table of Recommendations

41

Appendix B: Objectives, Scope, and Methodology

44

Appendix C: Federal Performance Indicators for GVRA

47

Vocational Rehabilitation Program

1

Purpose of the Audit
This report examines the Vocational Rehabilitation (VR) Program within the Georgia Vocational Rehabilitation Agency. Specifically, our audit set out to determine the extent to which the Program is: 1) serving those eligible and in need of its services and 2) accomplishing its mission of employment and independence for individuals with disabilities, including delivering services in a cost-effective manner.
A description of the objectives, scope, and methodology used in this review is included in Appendix B. A draft of the report was provided to GVRA for its review, and pertinent responses were incorporated into the report.
Background

State law defines a person with disabilities as "an individual having a
physical or mental impairment that
substantially limits one or more of the major life activities."

Mission and History
The federal Rehabilitation Act of 1973 was enacted to 1) empower individuals with disabilities to maximize employment, economic self-sufficiency, independence, and inclusion and integration into society1 and 2) ensure the federal government plays a leadership role in promoting the employment of individuals with disabilities, especially individuals with significant disabilities, and in assisting States and providers of services in fulfilling the aspirations of such individuals with disabilities for meaningful and gainful employment and independent living. The Act provides for federal grants to assist states in operating statewide comprehensive, coordinated, effective, efficient, and accountable programs, each of which is (a) an integral part of a statewide workforce investment system; and (b) designed to assess, plan, develop, and provide vocational rehabilitation services for individuals with disabilities, consistent with their strengths, resources, priorities, concerns, abilities, capabilities, interests, and informed choice, so that they may prepare for and engage in gainful employment. The Act also sets forth requirements pertaining to the administration of programs receiving funding under the Act.
The Vocational Rehabilitation (VR) Program within the Georgia Vocational Rehabilitation Agency (GVRA) is the state's effort to comply with the Act.2 Consistent with the Act, the mission of the VR Program is employment and independence for individuals with disabilities. To achieve its mission, the Program provides vocational rehabilitation services to "compensate a person with disabilities for his or her disability to employment and to enable such individual to engage in a remunerative occupation." These services range from counseling to purchasing assistive technology devices and equipment to paying costs associated with postsecondary education. During fiscal year 2015, 30,590 clients were served by the Program.

1 Per the Act, this would occur through statewide workforce investment systems, independent living centers and services, research, training, demonstration projects, and the guarantee of equal opportunity.
2 The Program was originally established as the Office of Rehabilitation Services in the Department of Education (GaDOE). Responsibility for the Program moved from GaDOE to the Department of Human Resources in 1972 and then to the Department of Labor in 2001. On July 1, 2012, the Georgia Vocational Rehabilitation Agency (GVRA) was created by the General Assembly.

Vocational Rehabilitation Program

2

Organization and Staffing GVRA is administratively attached to the Department of Human Services. As shown in Exhibit 1, a nine-member Georgia Vocational Rehabilitation Services Board provides oversight of the executive director. The board, whose members are appointed by the governor, recommends 1) changes related to vocational rehabilitation services, 2) improvements in coordination between public agencies providing such services, and 3) improvements in the condition of those individuals in need of services. The VR Program director and approximately 415 employees are responsible for day to day operations of the Program.3
The VR Program consists of six units or offices, including Field Services, Assistive Technology, Business Relations, Fiscal Services, Statewide Quality and Innovation, and Business Applications. The offices are described in detail on the next page.
Exhibit 1 Organization of the Vocational Rehabilitation Division, September 2015

Georgia Vocational Rehabilitation Board

Georgia Vocational Rehabilitation Executive
Director

Vocational Rehabilitation Division

Field Services

Business Relations

Business Applications

Statewide Quality and Innovation

Fiscal Services

Assistive Technology

West 11 Offices
East 9 Offices
South 11 Offices
Metro 8 Offices
Source: GVRA Records

3 Staffing figures included in the report represent positions filled as of September 2015.

Vocational Rehabilitation Program

3

Field Offices Program staff are located in 39 field offices throughout the state. As shown in Exhibit 2, each office is assigned to one of four regions: East, West, South, and Metro.
Exhibit 2 VR Regions and Office Locations (as of November 2015)

Newton

Source: GVRA records
Each region is supervised by a regional director. Each field office is staffed by a regional unit manager, counselors, counselor assistants, and an office assistant. The number of counselors and assistants depends on the size and location of the office. The responsibilities of each position are discussed below.
The 31 regional unit managers manage office operations, implements quality assurance standards and establishes/maintains relationships with community and business partners.

Vocational Rehabilitation Program

4

The 187 vocational rehabilitation counselors work directly with clients, performing assessments, determining eligibility, developing client work plans, certifying clients as job ready, and providing follow-up services after placement.
The 87 counselor assistants assist the counselors by completing intakes for clients and helping administratively move cases through the rehabilitation process. Generally, one Counselor Assistant assists two counselors.
The 40 office assistants provide administrative support to the field office including working the front desk, maintaining office equipment, and ordering supplies and equipment.

Other Units
The remaining units provide support services to field offices and clients, as described below.
The Assistive Technology team is composed of rehabilitation engineers, technicians, technologists, and occupational therapists and works with counselors to assess the need for assistive work technology for the client and the employer. Staff provide technology assessments and acquisition and training services to clients as well as job modification, job analysis, ergonomic consultation, and complex accessibility survey services to employers.
The Business Relations unit was created in October 2014 and works with businesses and clients to place clients in jobs after they have completed their training.
The Fiscal Services unit is responsible for budgeting, payments, and contracts. The staff works with vendors and counselors to ensure that vendors are identified and that services provided are paid for at the appropriate rate.
The Statewide Quality and Innovation unit consists of coordinators for specialty areas such as the deaf, blind, deaf-blind, behavioral health, developmental disabilities and autism, and veterans. The staff provide technical expertise regarding the specific population they cover. In addition, they assist with the on-boarding, initial training and on-going training of staff, and are responsible for program evaluation.
The Business Applications unit supports the Georgia Rehabilitation Agency Client Information (GRACI) system, which is the VR Program's current case management system. The staff assists with running reports and making necessary changes to the system, such as adding and deleting staff access.

Vocational Rehabilitation Process The vocational rehabilitation process begins with a referral. While individuals are referred through a variety of sources, they primarily self-refer or are referred by schools. In fiscal year 2015, approximately 66% (15,112 of 22,7424) of active clients whose disability had been determined had cognitive or psychosocial impairments
4 Figures are based on summary information which may not include all clients and may contain duplicates of clients with multiple disabilities.

Vocational Rehabilitation Program

5

such as learning, processing information, interpersonal and/or behavioral impairments, while 33% (7,615) had physical disabilities.5 Individuals referred to the Program may be employed at the time or unemployed. Regardless of their employment status, VR counselors help referred individuals complete an application, and discuss how the disability affects their ability to work, and how they may benefit from VR services.
If deemed eligible for services, the counselor and individual (client) discuss the desired employment outcome and the counselor conducts various assessments to determine the services necessary to reach the desired employment goals. The goal is based on a client's strengths, resources, priorities, concerns, abilities, capabilities, interests, and informed choice. Goals may include full- or part-time work in all types of vocations ranging from cashier to dentist. Clients who are already employed may choose goals such as maintaining current employment or becoming capable of employment beyond their current level.
The counselor creates an individualized work plan outlining the client's responsibilities and the services needed to reach the employment goal. The counselor also conducts a financial eligibility assessment to determine if the client should help fund particular services. Such assessments are required in each case where VR Program funds are to be utilized. Individuals who have been determined eligible for social security benefits or Temporary Assistance to Needy Families (TANF) are not required to contribute to the cost of services. The client receives the specified services outlined in the plan and, upon completion, is deemed "Ready for Work." Once a client is employed, and has maintained that employment for 90 days, the case is marked closed. The Program categorizes the various steps in the rehabilitation process as "statuses," which are shown in Exhibit 3.
Each of these statuses is tracked in GRACI, GVRA's case management system. GRACI contains all client case records as well as some vendor and payment information. All counselors are required to maintain case documentation in GRACI.

5 The remaining 1% (15) of clients had no impairment listed or the impairment was unknown.

Vocational Rehabilitation Program
Exhibit 3 Vocational Rehabilitation Program Process

Status 00 "Referral" Complete application.

Status 02 "Applicant" Make eligibility determination.

Status 20 "Ready for Work" Client has completed services outlined in IPE and is ready for job placement.

Status 18 "Training" Client is receiving services outlined in Individualized Plan for Employment (IPE).

6
Status 10 "Work Plan Development"
Establish work goal and develop Individualized Plan
for Employment (IPE) to achieve work goal.
Status 12 "Work Plan Completed"
Work plan has been developed but services have
not yet started.

Status 22 "Working" Client has begun employment.

Status 26 "Closed Rehabilitated" Client maintained employment for a
minimum of 90 days.

Source: GVRA Data
While the Program does not maintain this data, we found that clients who entered in 2008 spent an average of 673 days (2 years) in the Program, ranging from 2 days to 7 years. The average length of time in the Program for clients who maintained employment for 90 days and those whose cases were closed without an employment outcome was virtually the same at 671 days and 675 days, respectively. The total length of time clients spend going through the rehabilitation process varies because training services are individualized.

Program Services The VR Program provides a range of services to its clients to help improve the likelihood of employment or satisfaction with employment, such as:
vocational counseling and job placement services;
goods, services, and technology to live and work independently or become more satisfied with a job;
tuition and supplies for higher education and training;
equipment and tools needed to learn a particular vocation or perform a particular job; and
mental health and substance abuse treatment.

Vocational Rehabilitation Program

7

These services are provided through vocational rehabilitation counselors and numerous third-party vendors located throughout the state. Community Rehabilitation Programs (CRPs) are non-profit organizations that provide employment services to individuals with disabilities under contract with GVRA.6 Other vendors provide services through individual agreements on an as-needed basis.

Exhibit 4 shows the types of goods and services the Program provided participants and the expenditures associated with those goods and services in fiscal year 2015.

Exhibit 4 Services Provided to VR Clients in Fiscal Year 20151

Service Description

Amount

Aids, Equipment, Tools, Prosthetics

$1,416,657

Evaluations

$2,260,735

Interpreter/Reader Services

$590,489

Job Placement

$256,600

Medical/Psychological Evaluation/Treatment

$1,398,509

Post-Secondary Education

$3,922,753

Training

$8,233,240

Van/Car/Residence Modification

$278,863

Support Services

$1,675,507

Supported Employment Services

$967,950

Other1

$4,500,000

Grand Total

$25,501,303

1 The amount could not be broken out by service type because the payments were made to CRPs and were made outside of GRACI, VR's client information system.
Source: GRACI data

In addition to providing services to clients, the Vocational Rehabilitation Division also provides services to prospective employers, without obligation to ultimately hire a person with a disability. These services include: consulting with businesses to help them improve access for people with disabilities; educating managers, supervisors, and other employees about the potential impact of stereotypical attitudes towards people with disabilities; providing technical assistance, information, and clarification to employers about the Americans with Disabilities Act and how it relates to their businesses; and, helping to improve a job function by redesigning the work environment, changing the sequencing of tasks, or providing assistive devices/equipment.

Activity Data and Performance Measures The number of clients served by the VR Program declined by 15% over a six-year period. As shown in Exhibit 5, the decrease in the number of clients served began in fiscal year 2013 and continued into fiscal year 2014 with 25% fewer clients served than the year prior. Program officials attribute the decline to budget cuts resulting from the Program's separation from the Georgia Department of Labor on July 1, 2012. While
6 There are 19 CRPs located throughout the state.

Vocational Rehabilitation Program

8

numbers served have trended upward over the last two years, these figures are still well below numbers served in fiscal year 2011.

Exhibit 5 Activity Data, Fiscal Years 2011-2016

Fiscal Year

Total Served

2011 2012 2013 2014 2015 2016

40,346 41,988 36,833 27,514 30,590 34,095

Source: GRACI records

Percent Change -4%
-12% -25% 11% 11%

The U.S. Department of Education's Rehabilitation Services Administration (RSA) establishes seven performance indicators for state vocational rehabilitation programs. (See Appendix C.) It also sets the states' performance targets for indicators and requires that states meet these targets for at least five of the seven indicators. State vocational rehabilitation programs are required to submit an annual strategic plan to RSA detailing how they will meet the targets. Annually, RSA evaluates the state's performance. States that fail to meet the targets in the required number of performance indicators must develop a performance improvement plan outlining specific actions they will take to improve program performance. In fiscal year 2014, the VR Program met performance requirements in four areas, falling short of the requirement.

Workforce Innovation and Opportunity Act (WIOA)
The Workforce Innovation and Opportunity Act (WIOA) enacted on July 22, 2014, made significant changes to the Rehabilitation Act of 1973. The amended Act seeks to empower individuals with disabilities to maximize employment, economic selfsufficiency, independence, and inclusion and integration into society. Federal agencies are currently working to create new rules to implement WIOA. Proposed regulatory changes will align vocational rehabilitation programs with other workforce development programs by unifying strategic planning requirements, developing common performance accountability measures, and revising requirements for the onestop delivery system. The changes will emphasize competitive integrated employment for individuals with disabilities, including those with the most significant disabilities, and expand services to support transition of students and youth with disabilities to postsecondary education and employment.
Examples of specific programmatic changes include a requirement that at least 50% of the state's supported employment program funds be reserved to provide services to youth with the most significant disabilities. Under WIOA, expanded services may be provided to these youth for up to four years and states will be required to provide a nonfederal 10% match for the reserved funds. In addition, the population of students with disabilities and the types of services VR agencies may provide to this population as they transition from school to postsecondary education and employment will be expanded. States will also be required to reserve 15% of their VR allotment to provide

Vocational Rehabilitation Program

9

pre-employment transition services to students with disabilities who are eligible, or potentially eligible. These pre-employment transition services are designed to provide job exploration and other services, such as counseling and self-advocacy training, in the early stages of the transition process. Additional changes include the establishment of new performance measures, elimination of uncompensated employment outcomes (i.e., homemaker and unpaid family worker); and a requirement that supported employment outcomes be in competitive integrated employment.
As a result of this new legislation, GVRA in coordination with state agencies such as the Department of Labor, Department of Economic Development's Workforce Division, the Office of Adult Education of the Technical College System of Georgia, and the Department of Human Services, submitted a unified state plan for a statewide workforce development system that includes how it will serve vocational rehabilitation clients.

Financial Information
The VR Program is funded primarily through federal and state funds. In order to draw down federal funds, the state must match the funds at a rate of 21.3% in non-federal funds. In fiscal year 2016, the state provided $14.5 million and drew down $58 million in federal funds. For funds to be used as matching funds, the funds must not be included as matching funds for any other federally-funded program and must be from a sustainable source.
The state also received funds through the Social Security Administration's (SSA) Ticket to Work, which is a voluntary federal program for persons with disabilities who are receiving Social Security Disability Insurance (SSDI) and/or Supplemental Security Income (SSI) and who want to work and participate in planning their employment. The VR Program can apply to the SSA for reimbursement of the cost for services provided to Ticket to Work Program participants if certain requirements are met. Over the past five years, the state's reimbursement under this program has ranged from a high of $3 million in fiscal year 2010 to a low of $439,000 in fiscal year 2015.
As shown in Exhibit 6, GVRA expended $70.3 million on the VR Program in fiscal year 2015. Of that, approximately $26.7 million (38%) was expended on personal services, $18.1 million (26%) on regular operating, and $25.5 million (36%) on grants and benefits to clients.

Vocational Rehabilitation Program

10

Exhibit 6 Budget and Expenditures (Fiscal Years 2015-2016)1

Fund Sources
State General Funds Federal Funds Other Funds2 Total

FY 2015

FY 2016

$ 13,914,809 $ 52,181,136 $ 4,235,978 $ 70,331,923

$ 14,586,826 $ 58,280,746 $ 3,765,283 $ 76,632,855

Expenditures by Activity Personal Services Regular Operating3 Client Services Total

$ 26,677,611 $ 18,104,305 $ 25,546,307 $ 70,331,923

$ 30,405,907 $ 18,920,620 $ 27,306,328 $ 76,632,855

1Figures were provided by GVRA and represent actual (FY 2015) and budgeted (FY 2016) amounts as of August 2015.
2Other funds is comprised of incentive funds from the Social Security Administration's Ticket to Work Program, funds from MOU's with other agencies, and private donations.
3Regular Operating includes real estate rentals, capital outlay, contractual services, travel, motor vehicle expenses, computer charges, and telecommunications.

Source: GVRA Records

Vocational Rehabilitation Program

11

Findings and Recommendations

Service Outcomes

The outcomes of VR clients have not been tracked, monitored, or evaluated to ensure the Program effectively serves its clients.

The primary purpose of the VR Program is to assist disabled clients in obtaining employment. However, the extent to which the Program is aware of the outcomes of its clients is limited to a subset of those served. Rather than evaluating the outcomes of all eligible clients that it comes into contact with, the Program focuses its tracking and reporting on the employment-related outcomes of clients who receive training services. Our analysis of 11,678 eligible clients who entered the Program in 2008 revealed that 36% (or 4,200) reached their employment goal for at least 90 days after receiving services.7 The remaining two-thirds exited the Program without an employment outcome, including approximately 37% (4,341) of whom left the Program prior to receiving training services. Meaning, they left while they were going through more detailed eligibility assessments or while their work plans were being developed by VR counselors.

A cohort is a group of subjects sharing a
particular event together during a particular time span, are often studied over time to evaluate outcomes, understand the cause and effect of certain interventions, or compare
to other groups. For example, high school graduation rates are often calculated using a cohort of students who entered at the same time and are expected to finish within a
set time frame.

Management's efforts to measure the outcomes of the Program appear to be a result of its focus on compliance with reporting federal performance indicators. However, a 2005 U.S. Government Accountability Office (GAO) study cited several problems with the employment indicators that are required to be reported by states' VR Programs (including Georgia) to the federal government.8 For example, the calculation of the percentage of clients exiting the Program with an employment outcome does not capture clients who have not yet made it all the way through training, a group that accounts for a significant portion of clients served in any given year. Examining a fraction of clients served limits an evaluator's ability to comprehensively evaluate the Program's performance in serving all VR clients. The report also revealed that the employment rate had been manipulated by some state vocational rehabilitation agencies by not closing out certain unsuccessful cases so that they could meet performance targets for the year. According to GVRA staff, this was also occurring in Georgia. See Appendix C for a listing of federal indicators.
To more comprehensively evaluate the VR Program's effectiveness, we obtained data on a cohort of clients who entered the Program in fiscal year 2008 and tracked them through August 2014 (6-7 years after entering the Program). Our analysis revealed the following:
Employment outcomes. As shown in Exhibit 7, we found that of 11,678 clients who entered the Program in fiscal year 2008, 61% (7,124) exited the Program without an employment outcome. Of the remaining clients, 36% (4,200) were successfully rehabilitated and 3% (354) were still active clients

7 Status of clients as of August 2014.
8 Vocational Rehabilitation: Better Measures and Monitoring Could Improve the Performance of the VR Program, GAO-05-865.

Vocational Rehabilitation Program

12

in the Program as of August 2014.9,10 This is in contrast to the Program having reported a federal indicator of approximately 60% exiting the Program with an employment outcome over the last four out of five federal fiscal years.11
Exhibit 7 2008 Clients Exiting the VR Program over a Seven-Year Period (as of August 2014)1

Clients Entering
Status

Eligibility Determination 11,678

Work Plan Development 10,068

Work Plan Completed 7,375

Training 7,191

Successful 4,200 (40%)

Clients Exiting Program During Status

1,608 (14%)

2,693 (23%)

40

2,027

Unsuccessful

(3%)

(17%)

6,368 + 756 = 7,1241

(60%)

Clients

Still Active

0

in Status

Open

2

0

328

330 + 24 = 3542

(3%)

1 In addition to 6,368 clients' cases closed unsuccessfully from the statuses shown here, 756 cases were moved to a holding status and were ultimately closed without employment outcomes. 2 In addition to 330 clients' cases that were still active as of August 2014, 24 cases were moved to a holding status and remained active.
Source: PAD analysis of GVRA client data
Premature exits. We examined the 7,124 unsuccessful clients in more detail and found that the majority had exited the Program prematurely. At least 61% (4,341) of unsuccessful clients exited the Program prior to receiving training services. Exhibit 7 shows the various phases of the rehabilitation process and the number of clients who exit at each phase, including during eligibility determination (14%), after working with a VR counselor to develop a work plan (23%), and after receiving training services (17%).
Closure reasons. We also found that large numbers of clients who left the Program prematurely reportedly left for reasons that included a "failure to cooperate" and "no longer interested". As shown in Exhibit 8, 74-80% of clients who left during eligibility determination, work plan development, and training were reported as leaving for these reasons. Further examination of

9 Successful rehabilitation is defined as clients who maintained employment for 90 consecutive days after receiving services through the Program. Includes those clients who exited as homemaker, a status that qualified them as having an employment outcome at the time the data were captured.
10 Some open cases may have closed since our review. Our review of 54 client cases in November 2015 found that 25 had closed (19 successful and 6 unsuccessful) and 29 remained in various stages of the rehabilitation process.
11 The federal government requires that states report the percentage of clients exiting the Program during a given 12-month period with an employment outcome and allows the calculation to include only those clients who had received training in the computation.

Vocational Rehabilitation Program

13

client case files or discussions with clients themselves would be useful for identifying areas of improvement.

Exhibit 8 2008 Cohort Clients' Reasons for Prematurely Exiting the VR Program (as of August 2014)

Closure Reasons

Eligibility Determination

Work plan Development

After Receiving Training Services

Failure to cooperate

775 48%

1,349 50%

1,098 54%

No longer interested in receiving services or further services

514 32%

797 30%

405 20%

All other reasons

185 12%

287 11%

227 11%

Unable to locate or contact

126 8%

242

9%

280 14%

Transportation not feasible/available Closed in extended employment Total1

8 1,608

0.5% 0%
100%

18 -
2,693

1% 0% 100%

16 1
2,027

1% 0% 100%

1 Percentages may not total due to rounding.

Source: GVRA records.

The absence of a systematic evaluation of all outcomes can lead to undetected problems or misplaced problem identification within the VR Program. A Program of this magnitude--30,590 clients12, 415 staff, and 39 district offices--needs to be managed more effectively by tracking, monitoring, and evaluating its success in assisting clients. Systematic evaluation would allow the Program to better identify the services or attributes that increase/decrease the likelihood of a successful employment outcome.
As discussed in the remainder of the report, our review revealed service-related and case management concerns that may contribute to unsuccessful client outcomes or the length of time those that successfully completed took to achieve their employment goals. We also identified concerns that impact the Program's operational efficiency. Key areas are noted below and discussed in detail throughout the remainder of the report.
The Program does not actively track and evaluate clients over time to ensure they are moving through the rehabilitation process in a timely manner and that the Program is operating efficiently and effectively. (See page 14.)
The Program does not provide clients information needed to ensure they make informed decisions about the services they receive. However, efforts to expand the use of performance-based contracts would assist in this effort. (See pages 17 and 19.)
The Program's efforts to promote awareness of VR services have been limited and have been conducted in an ad hoc manner. (See page 21.)
The Program has experienced high turnover of its VR counselors, which could negatively impact clients' experiences with the rehabilitation process and clients' outcomes. (See page 23.)

12 Figure represents number of clients served in fiscal year 2015.

Vocational Rehabilitation Program

14

Certain administrative functions--Social Security Administration reimbursement requests, financial needs assessments, inventory controls over purchased equipment--need improvement to ensure efficiency, consistency and fairness, and stewardship over resources. (See pages 27, 29, and 31.)
The Program's poorly implemented case management system has not been an effective tool to assist case managers in managing their caseloads and allow management to identify areas in need of improvement. (See page 34.)
The Program lacks adequate policies and procedures for ensuring conflicts of interest are identified and resolved and that adequate safeguards are in place to protect sensitive client information. In addition, policies and procedures for key business operations are missing or outdated. (See pages 35, 37, and 39.)
RECOMMENDATION
GVRA should develop performance indicators and a process for regularly assessing its effectiveness in helping clients achieve their employment goals. Conducting analyses of client outcomes over time and by services provided would allow GVRA to further examine the VR Program to identify areas that potentially cause clients to exit without achieving their employment goals. Additional expertise may be needed to design evaluation methods, inform data collection methods, and ensure that the Program data and processes are evaluated on an on-going basis to assist with a greater focus on client outcomes.
GVRA's Response: GVRA stated that it intends to identify and track Key Performance Indicators such as conversion rate, cycle time, retention rate, number of clients served, and clients who successfully prepare, find and maintain employment. It also noted that staff will analyze the results to identify trends and "create Quality Improvement Plans that address negative outcomes and provide recommendations and strategies to improve the quality of services and service delivery."

GVRA should evaluate and improve monitoring activities to ensure clients move through the rehabilitation process in a reasonable time frame and that areas needing improvement are identified.
Overall, clients in the 2008 cohort stayed in the VR Program an average of 22 months (ranging from 2 days to 7 years) before exiting with or without an employment outcome. While there is no standard for the overall length of time someone should take to complete his/her training in the Program (because each client's circumstances and work goals are unique), time frames have been established for the first four phases of the rehabilitation process. However, our review found that clients had exceeded established timeframes. Neither the VR Program nor RSA have conducted a recent analysis of time spent in the Program13, but RSA thinks it is important to evaluate clients' length of time in the Program to understand agency operations and identify "agency inefficiencies" and "clients being shortchanged".

13 The most recent analysis of the VR program's timeliness in serving its clients was conducted by RSA in 2011. At the time, the VR program had reported that clients who had exited with an employment outcome had left within about 24 months while those who had not achieved an employment outcome left within 18 months.

Vocational Rehabilitation Program

15

Our review found that clients had exceeded established timeframes in certain aspects of the rehabilitation process leading up to training. Based on requirements established in federal law and VR Program policy, it is estimated that the four phases leading up to training should take no more than 210 days (approximately 7 months). Clients generally completed these phases within the 210 days, but client's experiences in individual phases of the process varied significantly as discussed below and shown in Exhibit 9.
Exhibit 9 2008 Clients' Time in Phase Prior to Receiving Training Services (as of August 2014)

Referral 30 days

Eligibility Determination
60 days

Work Plan Development
90 days

Work Plan Completed
30 days

Total 210 days

95%

5%

67% 33%

54% 46%

95%

5%

All clients

0 5.74 X

Successful clients

0 4.48 X

Unsuccessful 0 6.34

clients

X

Met timeframe

Exceeded timeframe

269 0 73.62 X
258 0 41.38 X
269 0 92.82 X

2,064 1,037 2,064

0 164.16 X
0 71.67 X
0 233.70 X

2,155 0 7.50 X
1,435 0 4.91 X
2,155 0 10.93 X

1,408 798 1,408

Minimum

Average X

Maximum

Source: PAD analysis of GVRA client data
Referral Of the 11,678 clients in the 2008 cohort who entered the referral phase, 95.1% (11,105 clients) exited within the 30 days required by VR Program policy, and 4.9% (573 clients) spent longer than 30 days in referral status. While the number of clients in referral status longer than expected is not significant, approximately 60 individual clients were in this phase from 90 days to 269 days (2 to 8 months longer than expected).
Eligibility Determination Of 11,678 clients who entered eligibility determination, 67.3% (7,856 clients) exited within the 60 days required by federal law, but 32.7% (3,822 clients) were in this phase longer than expected (ranging from 61 to 2,064 days). Although the applicant (client) and counselor

Vocational Rehabilitation Program

16

can request an extension of the 60 day requirement, our review found that only 56 clients had a documented extension in place.
Work Plan Development Of the 10,068 clients who entered work plan development, 54.0% (5,436 clients) exited within 90 days, per VR policy,14 and 46.0% (4,632) of clients spent more than 90 days in this phase (ranging from 91 to 2,155 days). Similar to eligibility determination, the client and counselor may request an extension. However, extensions can only be identified by reviewing individual case files for the 4,632 clients because they are not separately tracked in GRACI.15
Work Plan Completed VR has established a 30-day time limit for clients to exit the work plan completed phase, which largely consists of the client and counselor signing the work plan. Of the 7,375 clients who entered this phase, 95.0% (7,008 clients) exited within 30 days. The remaining 5.0% (367) of clients were in the phase longer than expected (ranging from 31 to 1,408 days).
In addition, we found that unsuccessful clients were disproportionately represented in the number of clients whose time in these phases exceeded expectations. For example, during eligibility determination, of the 3,729 clients who exceeded the timeframe, 77% (2,878) of them eventually left the Program without an employment outcome. Similarly, of the 4,458 clients who exceeded the timeframe for work plan development, 77% (3,417) of them exited the Program unsuccessfully. And, while training programs are individualized and the length of time required to complete them is expected to vary, we found that clients who exited unsuccessfully represented a greater percentage of clients who were in training status for one or more years, as shown in Exhibit 10.
Exhibit 10 VR Clients Exiting without Employment Outcomes Spent More Time in Training

60%

50%

40%

30%

20%

10%

0% < .5 years .5 to 1 year 1 to 2 years 2 to 4 years
successful unsuccessful Source: PAD analysis of GVRA client data, as of August 2014

4 or more years

14 As a result of WIOA, the 90-day timeframe for completion of work plan development is now a federal requirement.
15 A note is supposed to be added if more than 90 days is needed to complete work plan development.

Vocational Rehabilitation Program

17

Without an extensive review of individual clients' case files, we are unable to identify reasons why clients remained in certain statuses longer than expected. And, while the counselors assigned to these cases are required by federal law and VR Program policy to monitor progress through the various statuses (which may identify reasons why cases are not moving forward) and are in the best position to know how long training plans should take, we found this was not happening consistently. We reviewed certain processes in place for monitoring progress of cases and found that 1) counselors had not documented contact with clients every 90 days as required by VR policy and 2) employment plans had not been reviewed annually as required by federal law. In both instances, we found that VR counselors had not conducted the activities as required in approximately one-third of the cases reviewed. For example, our review of approximately 19,000 active cases as of May 2015 found that 33% did not have documentation reflecting counselor contact within the previous 90 days. At the same time, we reviewed approximately 11,000 active cases with an employment plan in place and found that 32% of cases had not had a documented review in more than a year.
RECOMMENDATION
1. GVRA should evaluate and revise its process for monitoring clients' progress through the rehabilitation process. This includes identifying a way to flag clients with approved extensions in the data system.
2. GVRA should regularly evaluate clients' overall length of time in the Program to establish baselines for categories of cases and identify areas for improvement in various aspects of the process.
3. GVRA should ensure that counselors conduct monitoring activities as required by Program policy and/or federal law.
GVRA's Response: According to GVRA, it has developed a new service delivery system "which includes evaluation points to monitor and analyze client outcomes from point of entry to point of closure." GVRA noted it will conduct systematic reviews of outcomes to "identify trends in client barriers to achieving a successful employment outcome" and "to isolate training needs for individual counselors as well as statewide."

GVRA should take action to produce information clients need to ensure they make informed decisions about the services they receive.

Informed choice is the process of choosing from
options based on accurate information and
knowledge. These options are developed by a partnership consisting of the consumer and the
counselor that will empower the consumer
to make decisions resulting in a successful vocational rehabilitation
outcome.

Federal regulations emphasize the importance of informed choice for VR clients and require that the Program provide clients with information necessary to make an informed choice about services and providers. This information is to include the outcomes achieved by individuals working with the service providers, to the extent that such information is available. However, because the Program has not captured and analyzed data on services and outcomes, this information has not been available to clients.
Prior to November 2014, analysis of services by type or by provider was difficult to do given the manner in which data was maintained by GVRA. Data about the services clients received was maintained in vendor payment data captured using two different

Vocational Rehabilitation Program

18

processes.16 Payments made to 19 Community Rehabilitation Programs (CRPs) were recorded in individual service provider spreadsheets while payments to all other providers were recorded in GRACI. As a result of capturing the information in a decentralized manner, GVRA would have to go through a cumbersome and timeconsuming process to compile data on services by client from both data sources. Effective November 2014, all client services data is now recorded and managed in GRACI. However, GVRA is not planning to enter data from prior years into GRACI. As a result, it will lack trend data on providers and services for a number of years.
We found that North Carolina, which currently uses the same system as GVRA, provides its counselors and clients with information about services and related outcomes using a dashboard in its system. For example, the dashboard indicated that clients receiving On-the-Job Training experienced a 63.9% success rate, which is higher compared to other services clients received. Although GVRA had purchased the module that allows them to analyze client services and outcomes, it had not been installed at the time of our review. In addition, with the move to a new data system, as discussed in the finding on page 34, this type of analysis will likely experience further delays.
It should be noted that GVRA is moving toward performance-based contracts for specific services, which may also help identify those services and providers that are related to positive client outcomes. Performance-based contracts are discussed further in the next finding.
RECOMMENDATION
1. GVRA should begin analyzing client outcomes against service and provider data. The results of this analysis should be made available to counselors and clients so they can make informed decisions about which services to pursue.
2. GVRA should ensure that it has a method for tracking and evaluating service and provider outcomes. In doing so, GVRA should assess whether it has the expertise needed to design evaluation methods, inform data collection methods, and ensure that the information is evaluated on an on-going basis.
GVRA's Response: GVRA stated that it has engaged a consultant to review provider services and "provide strategies for performance-based contracts to ensure that providers deliver high quality, cost effective services (i.e., identify provider outcomes, identify a provider management tool, develop Provider Dashboard (predictive model), etc.)." In addition, GVRA noted that it intends to incorporate provider data into its case management system "to allow professional staff the ability to share...performance data of the provider in real time" with clients.

16 Prior to July 1, 2014, payments to CRPs (approximately $9.5 million to 19 core service providers in fiscal year 2014) were recorded and maintained in individual spreadsheets while payments to all other providers were recorded and maintained in the Client Information System (i.e., GRACI).

Vocational Rehabilitation Program

19

GVRA should continue efforts to establish performance-based contracts to ensure CRPs deliver quality, cost-effective services to VR clients.

Performance-based contracts clearly spell out
the desired end-result expected of the contractor.
They encourage contractors to be innovative and to find costeffective ways of delivering services. By shifting the focus from process to results, they also promise better
outcomes.

At the time of our review, the 19 CRPs were operating under traditional, negotiated service contracts, whereby vendors received payment for services provided to clients referred by the VR Program. Some CRPs had operated under this contracting method for approximately 40 years with little accountability for the outcomes of those served. Due to concerns about client outcomes, the RSA encouraged GVRA officials to consider a milestone performance system, such as through performance-based contracts. The Program began piloting efforts to transition CRPs into performancebased contracts during our review. While it is making progress, the Program should ensure the contracts include key attributes of performance-based contracting needed to hold CRPs accountable for client outcomes.
In an effort to generate better outcomes for clients and place more responsibility for outcomes and service costs on CRPs, the Program has started to shift from its traditional contracting method of paying by the service/service hour to making periodic payments as clients move through various phases of the rehabilitation process, with a maximum payment amount per client served. Based on our review of the initial performance-based contract, the maximum payment is $6,000 of which 60% would be paid to the CRP when they first meet with the client, 20% to be paid when the client is placed in a job, and the final 20% payment to be made when the client has been employed for 90 days. According to GVRA officials, CRPs would have the flexibility to decide the most appropriate specific services clients would need to reach their employment goal. In addition, officials anticipate that under this contracting method, the percentage of clients successfully rehabilitated would increase to approximately 60%. It should be noted that, according to GVRA officials, the State Accounting Office has concerns about the step payment model described above because it allows the Program to pay CRPs before services are delivered.
While GVRA officials have expectations for how well clients will do under performance-based contracts, it has not yet defined these expectations in the initial contracts, nor has it detailed potential consequences for failing to meet expectations. According to guidance on performance-based contracting, these are among the attributes that performance-based contracts should contain.17 However, GVRA officials indicated that performance standards were not included initially in an effort to encourage the CRP to agree to the performance-based contracting approach. The parties agreed to use the initial contract period as a baseline and, going forward, officials intend to include base year performance levels in the contracts and require CRPs to meet or exceed those results.
Benefits of performance-based contracts were cited by RSA and other states. Interviews with federal RSA personnel indicated that the use of performance-based contracts is a best practice. In addition, North Carolina replaced its fee-for-service contracting approach with performance-based contracts as a way to reduce billing irregularities, paperwork, and the need for documentation and monitoring, and to

17 GAO Report 02-1049, Contract Management, cites guidance on performance-based contracting developed by the Office of Federal Procurement Policy. According to the guidance, performance-based contracts should describe the requirements in terms of results required rather than the methods of performance of the work; set measurable performance standards; describe how the contractor's performance will be evaluated in a quality assurance plan; and identify positive and negative incentives, when appropriate.

Vocational Rehabilitation Program

20

hold CRPs accountable for VR client outcomes. North Carolina cited that the change to performance-based contracts helped improve CRPs in the state.
RECOMMENDATION 1. To fully implement the performance-based contracting approach, the Program should ensure its contracts include all the necessary elements of performance-based contracts.
GVRA's Response: In its response, GVRA indicated that "over the next 12 months, GVRA will release Requests for Proposals for all current and new providers in order to expand the network of providers, increase the quality of cost effective services, and provide contract language that includes additional accountability for providers as it relates to performance and outcomes and annual review criteria."

Vocational Rehabilitation Program

21

Accessibility and Outreach

The Program should improve its outreach efforts to ensure persons with disabilities are aware of the vocational services available to assist them in obtaining employment.
The Program's outreach efforts have been limited and have been conducted in an ad hoc manner as district staff time permits. As shown in Exhibit 11, we estimate that the Program served between 30% and 44% of Georgia's disabled population who were looking for work in fiscal years 2010-2014. As discussed below, there are a number of factors that may limit awareness and access to the Program.

Exhibit 11 VR Program Clients Served Compared to Disabled and Looking for Work Population in Georgia, Ages 21-64 (Fiscal Years 2010-2014)1

90,000 80,000 70,000 60,000 50,000 40,000 30,000 20,000 10,000
0

33% 2010

33% 2011

44% 2012

37% 2013

30% 2014

Disabled and Looking for Work

Served

1 Figures on clients aged 21 to 64 are estimates derived from total clients served by the program in fiscal years 2010 to 2014 and may be slightly underestimated. Figures on the number of individuals who are disabled and looking for work are estimates based on samples of the population surveyed annually for the American Community Survey.
Source: PAD analysis of 2010-2014 Cornell University Disability Statistics data reports and GVRA client data

Historically, the Program had not maximized opportunities to partner with other entities serving a similar population to increase awareness and access to vocational rehabilitation services. Specifically, its relationships with local school systems and other state agencies serving persons with disabilities limited the extent to which populations served by these groups are referred to the VR Program. The nature of these relationships are described in detail below.
Partnerships with local school systems. Local school systems had the option to enter into collaborative agreements with GVRA to provide dedicated, concentrated vocational rehabilitation services to students with disabilities in the schools. School systems were required to provide the equivalent of 21.3% of funds to match 78.7% of federal VR funds, which averaged $10,800 per system in fiscal

Vocational Rehabilitation Program

22

year 2016.18 The goal was to increase successful transition outcomes for students with disabilities. At the time of our review, GVRA had agreements with approximately one-third (61 of 181) of local school systems, which was a decline from 82 systems with agreements in fiscal year 2013. According to GVRA staff, the decrease in the number of agreements was related to GVRA's fiscal year 2013 budget issues which prevented them from serving new clients, including students enrolled in local school systems that had an existing agreement with GVRA. In addition, GVRA staff told us some school systems did not have the funding needed to enter into a collaborative agreement and that other systems believed GVRA was required to provide services to their students regardless of the system's ability to pay.
Partnerships with other state entities: With the exception of one agreement with the Department of Behavioral Health and Developmental Disabilities (DBHDD),19 GVRA did not have formal relationships with state agencies serving similar populations prior to 2015. To encourage greater coordination between agencies, the General Assembly included language in the fiscal year 2015 Appropriations Act instructing GVRA to enter into a Memorandum of Understanding with DBHDD to maximize federal funding for supported employment services. GVRA was also directed to create a plan for establishing agreements with DBHDD, University System of Georgia, Technical College System of Georgia, and Georgia Student Finance Commission to maximize financial assistance for VR clients. At the time of our review, GVRA had established agreements with DBHDD (one related to supported employment for behavioral health clients and one related to vocational rehabilitation services for developmental disability clients). According to GVRA, its efforts to establish agreements with the remaining entities is ongoing.
A more coordinated approach to outreach could improve overall participation in the Program and the new WIOA legislation (see page 8) compels state vocational rehabilitation agencies to coordinate efforts with entities serving similar populations. Passed in 2014, the Act emphasizes serving youth with disabilities as they transition from school to employment and requires states to use at least 15% of federal vocational rehabilitation funds (approximately $13-14 million of the state's federal allotment20) to provide pre-employment transition services to these students. As a result, the 21.3% match requirement for local school systems will no longer exist as of July 1, 2016, according to GVRA officials. In addition, the Act requires the VR Program to coordinate with other state entities to submit a Unified State Plan that describes how the entities will work together to prepare potential employees for the workforce and meet employers' needs.
RECOMMENDATION
1. GVRA should continue its coordination efforts to improve access to VR services for persons with disabilities, particularly students with disabilities.
GVRA's Response: GVRA indicated that it has taken action to improve its outreach efforts.

18 The amount contributed by school systems ranged from $700 to $26,000 in fiscal year 2016.
19 The agreement with the Department of Behavioral Health and Developmental Disabilities included services to its mental health clients.
20 For federal fiscal year 2016, Georgia was allotted $90,381,181.

Vocational Rehabilitation Program

23

GVRA created a dedicated unit "to develop, implement, and evaluate strategies to increase access to the VR Program, particularly students with disabilities."
GVRA has "developed strategic partnerships statewide to improve awareness of the services available as well as increase outreach to students with disabilities." For example, GVRA notes that it is a "core partner for each of the 19 One-Stops", partners with the Department of Education (including Charter Schools), and has "set up collaborative communities through the Georgia Learning Resource Centers in every school district in Georgia."
GVRA is currently rebranding the agency "under one, unified `Able Georgia' brand to increase agency visibility and improve statewide recognition." It is also undertaking marketing strategies such as target marketing to special populations (i.e., veterans and transition-aged populations), broadcast advertising, and website and social media redevelopment."
GVRA staff are engaged in community outreach activities such as participation "at disability community events and job and resource fairs, and staff deliver informational multimedia presentations to raise awareness about the agency's services."

GVRA should continue its efforts to reduce counselor turnover and streamline the hiring process to fill vacancies in a timely manner.
At the time of our review, turnover rates for the VR Program were significantly higher than the state government rate.21 In addition, the number of days it took to fill vacancies created by departed employees exceeded the desired timeframe established by GVRA. For counselor positions22, these vacancies potentially created higher caseloads. Also, this could have the potential to negatively impact clients' experiences with the rehabilitation process. Although GVRA has taken steps to address turnover and hiring concerns, it is too early to tell whether the changes have had an impact.
Employee Turnover. As shown in Exhibit 12, our review of turnover rates found that turnover within the VR Program significantly exceeded the state government rate by approximately 10% in fiscal years 2013 and 2014. In addition, we found that turnover among VR counselors was slightly higher than the VR Program's overall rate. While turnover rates had decreased between fiscal years 2013 and 2014, the fiscal year 2014 turnover rate for the counselor series positions was 30.1%, compared to the Program-wide rate of 28.5% and 18.9% state government rate. It should be noted that turnover rates are based on terminations resulting from resignations, retirements, dismissals, and transfers.

21 Includes only executive branch agencies.
22 Counselor series positions include Certified Rehabilitation Counselors, Provisional Rehabilitation Counselors, Counselors for the Deaf, and Rehabilitation Casework Associates.

Vocational Rehabilitation Program

24

Exhibit 12 VR's Turnover Rate Exceeds State Government Rate1 (FY 2013-2014)
40%

35%

30%

25%

20%

15%

10%

5%

0% 2013

2014

State Government

VR Program

Counselor Series Positions

1 Retirements accounted for 21-26% of terminations in counselor series positions in fiscal years 2013 and 2014 Source: Georgia Department of Administrative Services and Audit Team analysis of GVRA data

A potential cause for the turnover among those who voluntarily resigned their positions was identified in the October 2014 employee satisfaction survey conducted by GVRA. Of the 328 employees completing the survey, most identified 1) frequency of pay raises (84% of respondents) and 2) entry salary (64% of respondents) as "the worst things about their experience" with GVRA.
Hiring Process. We identified delays in the hiring process that could exacerbate problems created by vacancies in the counselor positions. A limited review of the Program's hiring data showed that it took 143 days (20 weeks) on average to fill five counselor positions compared to GVRA human resources staff's goal of 60 days.23 Certain parts of the process appear to account for the majority of the delay. For example, our review found that it took from 1 to 128 days for the five positions to be posted once the request had been received, compared to the 14 days expected by GVRA. Our review also found that it took an average of eight weeks from the point at which the applicants were interviewed to the hire date. According to GVRA, posting delays were due to the number of approvals required prior to posting positions, administrative issues created by the state's implementation of a new applicant tracking system, and an increase in the number of new positions. In addition, it noted a number of factors potentially causing hiring delays, such as background checks, inexperienced hiring managers, candidates relocating for the job, and hard to fill positions (e.g., specialty counselors) which often have to be re-posted.

23 Based on GVRA's analysis of time to hire, it has taken an average of 143 days from requisition to hire date for counselor positions and three to six months for other positions.

Vocational Rehabilitation Program

25

Vacancies and hiring delays have potentially resulted in high counselor caseloads. According to GVRA officials, its ideal caseload size is 80-100 cases per counselor. The caseload in 15 of 39 offices are too high to allow for 1:100 cases per counselor even if cases were evenly distributed among counselors. For example, one office had 1,843 cases and only 11 counselors. Distributed evenly, the average caseload per counselor would be 179, 80% above the Program's maximum caseload target.24 Our discussions with Program staff indicated that high counselor caseloads reflect insufficient staffing resulting from vacancies. In addition, RSA has in the past determined that when counselors carried higher caseloads than needed, it had a direct impact on the quality of services provided to VR clients.
Over the course of our review, GVRA implemented several strategies/improvements to current processes that are intended to reduce counselor turnover and improve various aspects of the hiring process. However, it is too early to tell if the steps taken will have an impact. These actions are discussed below.
Revised salary structure In September 2014, GVRA implemented a new salary structure for critical positions, including counselors. According to GVRA officials, all counselors were brought in at entry-level salary regardless of their experience. The new salary structure, which accounts for experience, increased entry-level pay from $32,418 to $35,660 for a counselor with less than two years' experience and to $43,345 for a counselor with 9-10 years' experience, and to $52,686 for counselors with 17 or more years of experience. Counselors employed at the time of the adjustment had their salaries adjusted, if necessary, to put them in line with the new pay structure.
Consistent exit surveys According to GVRA officials, they are in the process of formalizing its policy related to employee exit surveys, which serve to identify possible causes of high employee turnover. At the time of our review, GVRA had piloted a new exit survey, but few surveys had actually been administered. Our review of employee data found that of 16 counselor positions vacated from November 1, 2014 (when exit surveys began) through July 7, 2015, only two counselors were administered the survey.25 According to GVRA officials, the individual responsible for administering the survey was not consistently notified when employees vacated their positions and, as a result, did not administer surveys. GVRA's improvements to the process places responsibility for administering the survey with the Operations and Benefits Unit, which receives termination notices and issues the separation notices.
Other actions taken In March 2015, GVRA hired a recruiting coordinator to oversee the hiring process up to the time the position is posted. It also secured an additional vendor to conduct background checks. According to GVRA, the latter has reduced the amount of time for a background check by one week. Finally, GVRA has added a continuous posting to several job sites (e.g., LinkedIn, Monster) to announce positions that are hard to fill. GVRA officials hope this will offset hiring delays.

24 Based on an analysis of caseload data as of November 2014.
25 Of the 16 counselors, 7 resigned, 6 retired, 1 was dismissed, 1 failed to return from leave, and 1 forfeited his or her position.

Vocational Rehabilitation Program

26

RECOMMENDATION
1. If turnover remains an ongoing concern, GVRA should continue its efforts to survey existing and departing employees to identify other factors that influence employees' decisions to leave their positions. In addition, GVRA should monitor the actions taken so far to determine if additional action is necessary to address causes of counselor turnover.
2. Similarly, GVRA should continue to evaluate hiring time and determine if actions taken have addressed delays or if additional actions are needed.
3. GVRA should evaluate counselor caseloads to ensure improvements in counselor caseloads are commensurate with any reductions in turnover and hiring delays.
GVRA's Response: In its response, GVRA indicated it will review the feedback it receives through employee satisfaction surveys and exit interviews "to identify factors that may influence employees' decisions to stay or leave....and determine if action is needed." In addition, GVRA noted that it has implemented several strategies and is monitoring the impact of the strategies on employee turnover. According to GVRA, such strategies include "(1) streamlin[ing] responsibilities of [certified rehabilitation counselors]; (2) increasing the number of paid internships; (3) supporting the costs of value-added certificates and licensures; and (4) on-going review of competitive salaries." Finally, GVRA stated that it "added a Recruiting Coordinator to the team that has reduced the time to fill positions."

Vocational Rehabilitation Program

27

Resource Management

GVRA should take steps to improve its administration of the Ticket to Work Program.

Administered by the Social Security
Administration, Ticket to Work is a voluntary program for persons with disabilities who want to work and
participate in planning their employment.

At the time of our review, Ticket to Work was not being administered efficiently or in a manner that maximized reimbursements from the Social Security Administration (SSA). Of 4,830 payment requests submitted to SSA from July 2009 to June 2014, approximately 83% (3,988) had been allowed and 14% (694) were disallowed.26 Disallowed requests included 475 that could have been avoided by ensuring clients met criteria for payment prior to submitting the requests. An additional 31 disallowed requests could have potentially resulted in additional reimbursements to the VR Program had they been filed properly or in a timely manner.
As shown in Exhibit 13 below, SSA makes two payment methods available-- Employment Network and Cost Reimbursement--each with its own set of requirements that must be met before SSA will approve payment requests. If approved, SSA reimburses the VR Program up to $27,060 for SSI recipients and up to $28,116 for SSDI recipients, depending on the payment method selected.

Exhibit 13 Ticket to Work Payment Methods

Model

Payment Basis

Payment Trigger

Limit on Amount

Example

Employment Network

Set milestone and outcome payments

SSA tracks and automatically pays when milestones are met

Cost Reimbursement

Reimburses for actual costs of services provided

GVRA submits claim within 12 months of the last day of the client's 9th month of SGA.1

$24,919 over a six year period if all milestones and outcomes are met
All costs up to the amount that SSA would pay in benefits to a client

Ideal for those whose return to work is incremental and episodic and beneficiaries more likely to work part-time and at lower wages.
Ideal for clients wanting to pursue full-time employment and who have the capacity to earn at or above the SGA level.

1 Substantial Gainful Activity (SGA) is generally measured at a specified earnings level which changes each calendar year. Source: Ticket to Work website and GVRA documents

Our review of the 694 disallowed payment requests found that some were denied for reasons that were avoidable. As shown in Exhibit 14, 358 disallowed requests were associated with clients not achieving earnings levels required for payment. In addition, 165 payment requests submitted under Employment Network had been denied because GVRA had previously been compensated for the clients under one of the two payment methods.

26 An additional 147 payment requests were still pending payment or pending a decision of the SSA at the time of our review.

Vocational Rehabilitation Program

28

We also found that 31 payment requests were denied (primarily under the Cost Reimbursement payment method), due to late applications, meaning GVRA had not submitted the requests within one year and one day from the first time the client achieved Substantial Gainful Activity (SGA). These requests included approximately $800,000 in claimed service costs, some portion of which might have been reimbursed to the VR Program if they had been submitted timely and subsequently approved. According to GVRA personnel, this was the result of differences in how they track client wages after case closure and how SSA tracks. GVRA indicated that they have changed their tracking to more closely follow SSA's tracking method.
Exhibit 14 Payment Requests Denied (July 2009 to June 2014)

n=694

Earnings Criteria Not Met

358

Prior Payment Made via Another Method

165

Client Still Receiving SSI/SSDI Benefits

55

SSI/SSDI Eligibility Cannot be Determined

33

VR Program Did Not File Timely Application

31

Insufficient Documentation 29

Other 23

Source: PAD analysis of GVRA payment request data
Based on our discussions with GVRA, staff responsible for processing these payment requests had not been properly trained in SSA's reimbursement processes and/or did not have access to sufficient information on participants in Ticket to Work at the time of our review. These are described more fully below.
We found that staff may not be using all available data to validate clients' SSI/SSDI status, prior participation in Ticket to Work, and earnings history. SSA makes this information available upon request to the clients themselves or GVRA staff (with client's consent). For example, if GVRA staff have access to SSA's Ticket Portal system, they can check a client's ticket assignability, query tickets assigned or formerly assigned to the Program, and query payments already made and pending payments.
Staff indicated they rely on information contained in the GRACI data system to make such determinations about which payment options to pursue for individual clients. However, as discussed in the finding on page 34, qualitative data needed to assess a client's ability to generate earnings at or above SGA may not be in a format readily accessible in GRACI, such as in a case notes section. In addition, our review found that staff do not have direct contact with clients or assigned counselors to obtain information necessary

Vocational Rehabilitation Program

29

to 1) determine the most appropriate payment model for the client's individual circumstances and 2) better understand each client's work/earnings history and past experiences with vocational rehabilitation programs.
At the time of our review, GVRA staff responsible for submitting payment requests had not attended training provided by SSA to understand how the Program works, including how SSA calculates earnings for the various disability types (SSI, SSDI, etc.).
Our discussions with staff found that guidance documents relating to Ticket to Work payments had not been updated since 2008 when VR services were provided under the Department of Labor. With frequent changes occurring in SSA's process, some of the information included in the document has become obsolete.
RECOMMENDATION To ensure that the Program maximizes its ability to collect reimbursement funds from the Social Security Administration, GVRA should:
1. Ensure Program staff have access to all available resources needed to make appropriate decisions related to payment requests. This includes a. ensuring staff are adequately trained to effectively administer the Ticket to Work reimbursement process, and b. developing a system to ensure that SSI/SSDI recipients are effectively identified and that the most appropriate Ticket to Work payment model is selected for each client.
2. Establish adequate procedures to ensure that reimbursement requests are submitted in a timely manner.
GVRA's Response: In its response, agency officials stated "In FY16, GVRA's SSA's Initiatives staff began participating in the annual training opportunities offered by the Social Security Administration on the Ticket to Work program, subsequently improving the administration of this program and increasing the agency's revenue by $2 million over the past year." In addition, officials noted that staff responsible for the "Ticket to Work program has updated the procedures manual to be reflective of the training that has been received by the Social Security Administration that includes specific procedures to ensure that reimbursement requests are submitted timely." Further, GVRA staff indicated that it "provides statewide training annually and as new staff are on-boarded."

GVRA should take steps to ensure that it consistently conducts financial needs assessments as required by Program policy.
VR clients' ability to contribute toward service costs are not routinely evaluated. Our review found that some client files do not include financial needs assessments as required by Program policy. In addition, when financial needs assessments are present, some do not include supporting documentation as required by the policy. This inconsistency creates an issue of fairness for clients who completed the assessment and, subsequently, had to pay a portion of their service costs. In addition, by not ensuring all clients have a current and complete financial needs assessment on file, GVRA may have missed opportunities to share the cost of services with those clients.

Vocational Rehabilitation Program

30

Though not required, federal regulations allow state VR agencies to consider clients' ability to contribute toward the cost of certain services, which GVRA opted to do in order to free up funding available for client services. As stated in the VR Program's policy, financial need should be assessed and documented in case files when it is anticipated or known that VR Program funds will be used for services, which occurred for approximately 40% (13,000) of the 34,000 clients served in 2016. A client's ability to pay is based on a sliding scale which assigns clients a contribution rate depending on the amount of household income exceeding the Program's allowance (150% of federal poverty level). 27 For example, a client whose household income is $10,000.01 to $12,000 would be required to pay 4% of the cost of individual services he/she receives. Federal law exempts certain individuals and services from financial needs participation, such as those receiving federal cash benefits and individuals receiving services not directly related to obtaining employment.28

Using GRACI, we reviewed a non-random, non-statistically valid sample of 29 clients who received services eligible for financial participation29 in fiscal year 2014 and found that financial needs assessments had either not occurred in all cases or had not been documented in the case files as required by Program policy. Of the 29 clients reviewed, 9 had completed financial needs assessments included in their case files. In addition, 3 of the 9 case files containing financial needs assessment forms included documentation to support information provided in the forms (e.g., tax records, confirmation of SSI/SSDI benefits). The remaining 20 case files reviewed did not include a financial needs assessment form, including those clients identified as receiving federal cash benefits30, as shown in Exhibit 15.

Exhibit 15 Documentation of Financial Needs Assessment (Fiscal Year 2014)

GVRA Requirements

Total

Yes

No

Reviewed

Financial Needs Assessment Included in Case File

9

20

29

Supporting Documentation Included in Case File

41

25

29

1Includes one client whose file did not contain the financial needs assessment form. Source: PAD analysis of GVRA records

GVRA personnel indicated that client case files are subject to periodic review. As part of the review, reviewers note the presence or absence of financial needs assessments.

27 The contribution rate for a client whose household income is within 150% of federal poverty level (FPL) is 0% while a client whose income exceeds 150% of FPL by $54,000 or more would be required to pay 100% of the cost of individual services. Per VR Program policy, contributions cannot exceed 50% of a household's total income.
28 Federal regulations prohibit states from applying financial needs tests or requiring financial participation of individuals as a condition of providing rehabilitative services if the individual receives federal cash benefits (e.g., SSI, SSDI, TANF) or as a condition of providing certain types of services to any individual (e.g., eligibility determination assessments, counseling and guidance, referral services, jobrelated services).
29 We limited our assessment to assistive technology (specifically, computers), college tuition, and maintenance services.
30 When a client is a recipient of federal cash benefits, a financial needs assessment form must be completed with supporting documentation confirming current eligibility for those benefits.

Vocational Rehabilitation Program

31

However, these reviews have been put on hold indefinitely as GVRA continues its efforts to address issues related to its case management system.
It should be noted that there is a cost associated with conducting the assessments. Program staff must assist clients in filling out the financial needs assessment form, obtain, review, and validate supporting documentation, and evaluate each client's overall circumstances before making a determination of financial need. Due to the way the data is maintained in GRACI, we were unable to determine the amount clients have been asked to contribute over time. As a result, the cost-benefit of the Program's financial needs assessment policy could not be determined.
RECOMMENDATION 1. GVRA should evaluate the cost-benefit of its financial needs assessment policy and use the information to determine if the policy should be continued or discontinued.
2. If GVRA finds the financial needs assessment policy is cost beneficial, it should:
a. ensure Program staff consistently conduct financial needs assessments as required by its policies, and
b. resume case file monitoring activities to ensure all required documentation (e.g., financial needs assessment, supporting documentation) is included in each client's electronic case file.
GVRA's Response: GVRA indicated its plans to "establish a committee to conduct both a costbenefit review and a qualitative review of the Financial Needs Assessment (FNA) to make the determination if a Financial Needs Assessment should be continued and at what capacity." In the event FNA is continued, GVRA noted that 1) "the committee will establish a baseline of data to monitor the consistency of its completion...[including] procedures for monitoring that all supporting and required documents are included in the client's case file" and 2) "procedures will be revised to monitor the annual review of the FNA and/or at the time the client reports a significant change."

GVRA should improve controls over equipment purchased for clients and ensure that equipment is properly inventoried and recovered.

The Program buys assistive work technology devices, equipment, and tools needed to prepare clients to prepare for or perform entry level requirements of an occupation supported by their work goal. Such purchases may include computers, software, wheelchairs, closed circuit televisions, tools-of-thetrade, durable medical equipment,
etc. Also, the Program may purchase modifications (hand controls, lifts, floor mounted restraints, etc.) for vehicles and
other mobility devices.

The VR Program lacks a complete inventory of equipment purchased on behalf of its clients. While the Program has established a policy to 1) guide staff decisions related to tools and equipment purchases and 2) specify inventory requirements, procedures to support some aspects of the policy are not in place. And, where procedures do exist, there appears to be inconsistency in implementation among VR staff. While current policies are applicable to purchases exceeding $350, the value of purchased items range from a few dollars to tens of thousands of dollars. For example, the Program purchased a cane for a client at a cost of $25 and it paid $25,000 for a special printer for another client. By not ensuring equipment is properly inventoried, the Program may be unnecessarily purchasing equipment it already has available or that could be recovered from former or existing clients. In addition, the lack of an inventory makes equipment purchases subject to fraud or abuse.

Vocational Rehabilitation Program

32

Currently, a centralized listing of purchased equipment is not being maintained by the Program. Instead, the Tools/Equipment Responsibilities form is the only record of equipment assigned to individual clients. According to Program policy, the form is used to document each item purchased (valued over $350), model/serial numbers, cost, dates that apply to the authorized equipment (for depreciation purposes) 31, and the client's responsibilities. The policy also requires that a copy of the completed form be given to the client, a copy be placed in the case file, and a copy be provided to the regional tool and equipment coordinator.
However, this information is not easily obtained from GRACI--basically, the paper form is scanned and included as an attachment to the client's case file. There is no data field in GRACI that allows VR Program staff to quickly identify all equipment assigned to clients. In addition, we searched individual case files and found that these forms are not always present in a client's case file. Our review of a non-random, nonrepresentative sample of 18 clients for whom the VR Program had purchased equipment found that 7 clients' files contained a completed Tools/Equipment Responsibilities form.32 The remaining four client files contained no evidence of them having received equipment.
Given the shortcomings in how equipment purchases are managed/documented, it is unclear how the Program can adhere to various provisions in its policy related to assignment and depreciation of equipment. Some examples are described in detail below:
According to the policy, when a client is not using items for their intended purposes (e.g., clients who exit the Program unsuccessfully), VR staff can request the items be returned and made available to other clients. However, in order for Program staff to identify whether clients have had equipment purchased for and assigned to them, staff would have to read through each individual case file, which is a time-consuming process and contributes to missed opportunities to reassign unused equipment.
In addition, the policy indicates that purchased items are to remain property of the Program until the established depreciation date has expired or the items are transferred to another state agency or to the state Surplus Property Warehouse. However, the Program has no mechanism to track the movement of equipment across clients or transfers to other entities. While serial/model numbers are to be documented in the Tools/Equipment Responsibilities form, the manual format makes the information not useful for this purpose.
While the policy places responsibility for maintenance of a regional/centralized inventory with regional tool and equipment coordinators, our review found that this position no longer exists in the VR Program's organizational structure and the responsibilities have not been transferred to other positions. And, while the Program's Assistive Technology team appears to be responsible for tracking certain equipment (e.g., assistive work devices) and has begun designing a database to compile a list of

31 The Program has established time frames to determine depreciation dates. For example, occupational tools-of-the-trade, hand tools, and small electrical tools are considered 2-year property; large electrical tools, manual wheelchairs, and computer software/hardware are considered 3-year property; and power wheelchairs, removable adaptive equipment for vehicles, etc. are 5-year property.
32 We identified these clients from a listing of authorized purchases made by VR staff in fiscal year 2014.

Vocational Rehabilitation Program

33

equipment available to be distributed to clients33, their efforts do not cover the full range of equipment that might be provided to clients (e.g., wheel chairs, tools-of-thetrade) or cover items already distributed to clients. It should be noted that, according to Assistive Technology staff, counselors do not consistently consult with the team before assistive work technology devices are purchased, as required by the policy.
RECOMMENDATION To ensure proper stewardship of state (and federal) funds allocated for the purposes of purchasing equipment for VR clients, the Program should:
1. Update policies to reflect its current organizational structure. In addition, it should reassign responsibility for inventory management to positions that currently exist within its structure.
2. Develop a statewide inventory of purchased equipment for tracking the issuance, return, and reissuance of equipment purchased for clients.
3. Integrate inventory data with its client/case management system to ensure equipment is assigned to individuals who need it.
GVRA's Response: In its response, GVRA noted that it "will put together a committee to identify what purchased equipment needs to be inventoried and develop procedures to [conduct] statewide tracking of the issuance, return and reissuance of the equipment. This will include procedures to [reissue] equipment based on client need." GVRA stated that "this committee will also look at the financial practicality for recovering equipment and ...will draft procedures...[to] include protocols for communicating to the client the policies for returning unused equipment and the enforcement of the policy moving forward."
GVRA indicated that "the committee will have open dialogue with the developers of the case management system to determine if the inventory data can be integrated into the new system and costbenefit for enhancing the system if the functionality is present."

33 This effort was still in the developmental stage at the time of our review.

Vocational Rehabilitation Program

34

Information Management and Other Matters
GVRA should ensure that its next case management system is acquired and implemented according to best practices.
GVRA's current system (GRACI), has been a largely ineffective management tool due to design and functionality issues. In addition, the vendor has decided to discontinue the case management product starting in June 2017. To assist the VR Program, an effective case management system should (1) involve stakeholders in outlining system requirements before it is acquired; (2) enable counselors to manage client caseloads in an efficient manner; (3) generate reports for federal compliance, financial management and individual case management; (4) allow client services, outcomes, and costs to be tracked and analyzed; and, (5) enable vendor payments to be made and reconciled.
GRACI was purchased by the Department of Labor (DOL) in 2012 for $1.1 million and the system went live in March 2014. According to GVRA officials, users (e.g., counselors, managers, financial personnel) had very little input into the planning and design of the system. Based on our discussions with a representative of the Georgia Technology Authority (GTA), DOL did not work with them to acquire the system and was not required to do so. In addition, DOL requested the system duplicate existing paper-based forms rather than automate or streamline the case management process. To address problems and inefficiencies with GRACI post-implementation, GVRA has had a representative from the vendor on-site to help troubleshoot problem areas and make changes, such as automating certain processes and adding data accuracy controls. It also hired a consultant in March 2015 to strategically assess the on-going issues with the system. GVRA estimates that it will have incurred approximately $930,000 in vendor and personnel costs related to GRACI since its implementation.
The issues associated with GRACI's design, functionality, and implementation are discussed below.
Incomplete Case File Data. Counselors indicated that client work plans from GVRA's previous system were not retained during the conversion to GRACI, requiring counselors to manually re-enter all previously entered case data. As a result, counselors do not trust the data contained in GRACI and some offices continue to use paper files due to concerns about its completeness.
Duplicate data entry. Because the system was designed to mirror paper-based case files, there is a significant amount of manual data entry and duplicate information being entered because the same information is captured on multiple forms. As a result, the system has generated little time savings for counselors.
Inconsistencies in data generated by management reports. During our review, management reports yielded different results even when all query variables remained the same, raising questions about the reliability of the reports. Upon investigation, we found that not all data had been uploaded and that, at times, duplicated data was causing the inconsistencies. In addition, the documentation needed to understand the relationships between data elements and tables has not been developed by the vendor.
Difficult data extraction and analysis. Memo fields (e.g., case notes) in GRACI are used to capture large quantities of information that are important for

Vocational Rehabilitation Program

35

monitoring and managing various aspects of the Program, such as counselor caseloads. The use of more standardized data fields would allow cases to be efficiently and effectively tracked and monitored and for greater analysis of outcomes and various aspects of the Program's operations.
Payments functionality. It was not until fiscal year 2015 that payments to CRPs were captured in GRACI. Prior to that time, payments for services (delivered by CRPs) were maintained in Excel spreadsheets. GVRA is now able to track service costs in GRACI, but is limited in its ability to identify total costs associated with individual clients served by the Program prior to fiscal year 2015. In addition, the system was designed without a reconciliation feature which caused issues related to paying vendors and required a manual reconciliation of all payments made to CRPs.
RECOMMENDATION
Faced with another system conversion or implementation to replace GRACI, GVRA should ensure issues of design, functionality and implementation experienced with GRACI are not repeated going forward.
1. GVRA should work with GTA and the Department of Administrative Services (DOAS) to evaluate the vendor's recommended replacement system and to identify potential alternatives that will meet the needs of the VR Program.
2. In so doing, GVRA should ensure that it follows state policies and standards for information technology and procurement to ensure that it transitions to a system that is well-designed and well-implemented. Significant focus should be placed at the front-end of the process which involves users carefully outlining the required specifications for the new system prior to going through an RFQ or RFP process.
GVRA's Response: GVRA has "initiated the process of purchasing a different case management system at the beginning of this year that involved stakeholders at the onset of the project. The process included researching what other case management systems were available, working closely with the Georgia Technology Authority (GTA) and the Department of Administrative Services (DOAS), which resulted in a sole source approval." According to GVRA, "GTA/Independent Verification and Validation (IV&V) is involved in the current case management implementation project...to ensure the right product is built and it's correct."

GVRA should ensure that conflicts of interest are reported and managed appropriately.
While GVRA has a conflict of interest policy, it has not ensured the process for identifying conflicts of interest has been carried out consistently across its more than 470 employees. Disclosure requirements instituted since GVRA became a separate agency (and administratively attached to DHS) appear to have been applied to employees hired after July 2012, but not retroactively to employees hired before. As a result, management may be unaware of its employees' potential conflicts of interest and employees may not be aware of what constitutes a conflict of interest.
Human resources/personnel policies outline requirements which allow management to determine whether or not a conflict of interest exists as a result of relationships

Vocational Rehabilitation Program

36

between employees or employees and service providers contracting with GVRA. Employees are required to 1) sign a form acknowledging their review of and compliance with standards of conduct, including conflicts of interest and 2) complete a conflict of interest questionnaire to disclose potential conflicts. 34
However, our review found that not all conflict of interest requirements have been met by GVRA employees. We reviewed a non-random, non-representative sample of 19 employees' personnel files (out of 470) to document the presence or absence of the required documents.35 Of the 19 employees reviewed, all had completed the acknowledgement form. However, only 6 of 19 employees had completed the questionnaire. Those employees were all hired after GVRA became an independent agency while the 13 employees who had not completed the form were employed with GVRA prior to that time.
In addition, the policy governing conflicts of interest does not require employees to submit disclosures of potential conflicts annually. The conflict of interest questionnaire currently states that "GVRA holds its employees accountable for avoiding conflicts of interest as well as the appearance of conflicts of interest; therefore it is critical that employees report any relational circumstances that have potential to create conflicts or the appearance of conflicts of interest within the Department." However, without a process to obtain this information on a routine basis, GVRA has no assurance that employees are reporting these conflicts as circumstances change.
It should also be noted that when conflicts are reported, it is not clear how they are acknowledged and addressed by GVRA management. Currently, when an employee completing the questionnaire responds affirmatively to questions about relationships with GVRA employees or service providers, it is incumbent upon him or her to return the form to their supervisor (versus HR) who, presumably, would take action as needed. However, nowhere on the questionnaire, or any other document, is it documented that a supervisor 1) acknowledges receiving the form and 2) took action to address the disclosed conflict. For example, one employee included in our sample disclosed a conflict of interest related to employment with a VR service provider, but no evidence of supervisory review, notes, or other explanation related to the disclosure were included in the file. When the audit team inquired about the questionnaire, GVRA staff contacted the employee for an explanation and learned that the employee had resigned his employment with the provider upon accepting the position with GVRA.
RECOMMENDATION
1. GVRA should ensure that employees hired prior to the creation of GVRA complete the conflict of interest questionnaire.
2. GVRA should implement a process to require an annual update of the conflict of interest questionnaire.

34 DHS Human Resource Policy #1201 - Standards of Conduct and Ethics in Government. 35 Our sample consisted of employees who were employed with GVRA prior to its becoming an independent agency and after. Our sample also included GVRA executives hired both before GVRA and after.

Vocational Rehabilitation Program

37

3. GVRA should ensure that conflict of interest questionnaires are properly reviewed and any actions taken in response to disclosed conflicts are documented.
GVRA's Response: In its response, GVRA indicated that it "will implement a new conflict of interest questionnaire that will be completed during the annual update training, new employee orientation, or as an employee's situation changes." In addition, GVRA stated it will implement a review process that "establish[es] a communication path between Human Resources' staff, the employee and the employee's supervisor to resolve, correct or monitor any disclosed conflicts of interest...[and] all actions that are taken will be documented by the Human Resources Department as part of the employee's personnel file." GVRA also noted that "during the annual update training, the process for reporting a conflict of interest will be reviewed."

Additional guidance is needed to enhance protection of confidential client information.
Over the course of our review, we detected vulnerabilities in the management of paper and electronic case files. These seem to relate to a lack of policies and procedures instructing employees on how to handle client information or weaknesses in the implementation of GVRA's existing policies. Such weaknesses could lead to unauthorized individuals reading, disclosing, or tampering with confidential information.
To determine an individual's eligibility and vocational rehabilitation needs and to develop the individualized plan for employment of eligible individuals, GVRA has collected and maintains personally identifiable information on approximately 30,000 clients in fiscal year 2015. Such information includes names, addresses, demographics, and may include financial, medical, and psychological information resulting from various assessments. Any information collected from clients or other sources is maintained in a case file that is protected by federal and state law.36 To comply with the laws, GVRA has policies relating to confidentiality of client information. These are outlined in its Client Services Manual which provides requirements related to the protection, use, and release of personal information.
Our review found indications that GVRA's own policies may not be consistently implemented. In addition, we found that GVRA has not adopted policies and procedures that specify how such information should be physically safeguarded. We observed a variety of practices over the course of our review that seem inconsistent with existing guidance or commonly accepted practices for safeguarding confidential information. Examples of situations we observed are described below.
Some GVRA staff have read access to electronic case files in GRACI who may not be on a need-to-know basis. Per GVRA's own Client Service Manual, direct access to an individual's case file shall be limited to VR Program staff on a need-to-know basis, unless access has been administratively authorized. . In addition, guidance provided by the Georgia Technology Authority (GTA) suggests that, depending on job functions, some employees should not be authorized to know the information contained in client case files.
36 State Vocational Rehabilitation Services Programs, 34 C.F.R. 361 and O.C.G.A. 49-9-18.

Vocational Rehabilitation Program

38

Controls in GRACI to prevent unauthorized access were not always active. For purposes of the audit, members of the audit team were granted access to GRACI. During this time, members of the audit team observed that the GRACI application did not automatically timeout or terminate after a period of inactivity. Per GTA guidance, a timeout (or session lock) control and termination feature should be activated in applications containing confidential information that would require users to re-enter credentials prior to resuming a session. GVRA's IT staff were notified of the possibility that the control had not been activated in GRACI.
GVRA staff sent unencrypted confidential information via unsecured email. During our review, we noted instances in which GVRA staff transmitted confidential client information over email in plaintext format which is not secure. GTA guidance related to electronic communications requires that originators of electronic data or correspondence be responsible for the appropriateness of message content, awareness of data classification, and confirming authorization and need-to-know of the recipients before transmitting any electronic correspondence from a state information system on behalf of the state.
Some GVRA locations could not locate some clients' paper files. During site visits to eight VR offices, the audit team selected a non-random, non-representative sample of 82 hardcopy client files to review. VR Program staff at three offices could not locate 7 of the 33 files selected for their locations. In addition, while the other five offices were able to locate all of the 49 files requested by the audit team, three files were located at an office other than the one the clients were assigned to.
Without specific guidance and/or proper training in these areas, staff are left to make their own decisions about how sensitive client information is to be handled.
RECOMMENDATION
1. GVRA should develop and implement a security plan that includes mandatory periodic training in computer security and restricting users to the minimum access or type of access necessary to perform their jobs. A security plan would also outline techniques for safeguarding the security of information.
a. GVRA should ensure that its staff understand what information is confidential and how confidential information should be transmitted.
2. GVRA should establish a centralized policy for the management of physical client files. Offices should be held accountable for management of client information.
GVRA's Response: In its response, GVRA identified several actions it intends to take "to ensure that clients' confidential information is clearly protected."
"GVRA's Legal Department in coordination with Facilities and IT are reviewing the procedures that are in place for securing and transmitting both electronic and paper confidential information. Based on the findings, a Security Plan will be developed and implemented for monitoring how all records are maintained."

Vocational Rehabilitation Program

39

"GVRA is implementing "AWARE", a new software system that will comply with GTA guidance on access to client files that is limited to assigned staff and has automatic timeouts to control access to the system."
"GVRA's Training Department in coordination with the Legal team is updating their policy training to be reflective of the changes to policies and procedures being made. Training on proper handling of confidential information and record retention will be held for all staff."

GVRA should ensure that it has documented policies and procedures for key business operations.
At the time of our review, policies and procedures for certain VR Program functions were either out-of-date or had not yet been developed. Policies and procedures provide the framework for how organizations operate and support effective decision making to meet their original objectives. Without clear, up-to-date policies and procedures, the VR Program cannot ensure that work is managed consistently throughout the state and that expectations are clear.
We identified examples of missing or outdated policies and procedures. These are described below.
GVRA established the Business Relations Unit in October 2014 without documented policies and procedures. As previously noted, the unit works to develop relationships with businesses (potential employers) to facilitate job placements once VR clients complete training. At the time of our review, the unit was staffed with 35 full-time employees.
The Regional Initiatives Coordinator Unit, which administers the Ticket to Work Program, does not have up-to date policies and procedures. As noted in the finding on page 27, the manual in use at the time of our review was last updated in 2008 when the VR Program was administered under the Department of Labor and did not reflect the most up-to-date guidance from the Social Security Administration. At the time of our review, the unit was staffed with 4 full-time employees.
RECOMMENDATION 1. The VR Program should ensure that up-to-date policies procedures are established and clearly communicated to staff.
GVRA's Response: In its response, GVRA indicated it "has appointed a Director of Compliance & Accountability who is reviewing current policies and procedures for key business operations related to the agency's Business Operations Unit & Ticket to Work Program." In addition, GVRA noted it has retained a consultant to update policies related to client services. Revisions will address changes in federal guidelines, and policies related to accepting and processing client referrals, Ticket to Work, equipment purchases and inventory management, conflicts of interest, information technology, and handling of confidential information.
In addition, GVRA noted "with the changes being made to the service delivery system ..., a new procedural manual and trainings are being created simultaneously. As programmatic changes are

Vocational Rehabilitation Program

40

being implemented, the manual and trainings will serve as tools to support field staff in a more comprehensive way. Annual trainings will also be instituted to ensure ongoing staff compliance."

Vocational Rehabilitation Program

41

Appendix A: Table of Recommendations

The outcomes of VR clients have not been tracked, monitored, or evaluated to ensure the Program effectively serves its clients. (p. 11)
1. GVRA should develop performance indicators and a process for regularly assessing its effectiveness in helping clients achieve their employment goals. Conducting analyses of client outcomes over time and by services provided would allow GVRA to further examine the VR Program to identify areas that potentially cause clients to exit without achieving their employment goals. Additional expertise may be needed to design evaluation methods, inform data collection methods, and ensure that the Program data and processes are evaluated on an on-going basis to assist with a greater focus on client outcomes.
GVRA should evaluate and improve monitoring activities to ensure clients move through the rehabilitation process in a reasonable time frame and that areas needing improvement are identified. (p. 14)
2. GVRA should evaluate and revise its process for monitoring clients' progress through the rehabilitation process. This includes identifying a way to flag clients with approved extensions in the data system.
3. GVRA should regularly evaluate clients' overall length of time in the Program to establish baselines for categories of cases and identify areas for improvement in various aspects of the process.
4. GVRA should ensure that counselors conduct monitoring activities as required by Program policy and/or federal law.
GVRA should take action to produce information clients need to ensure they make informed decisions about the services they receive. (p. 17)
5. GVRA should begin analyzing client outcomes against service and provider data. The results of this analysis should be made available to counselors and clients so they can make informed decisions about which services to pursue.
6. GVRA should ensure that it has a method for tracking and evaluating service and provider outcomes. In doing so, GVRA should assess whether it has the expertise needed to design evaluation methods, inform data collection methods, and ensure that the information is evaluated on an on-going basis.
GVRA should continue efforts to establish performance-based contracts to ensure CRPs deliver quality, costeffective services to VR clients. (p. 19)
7. To fully implement the performance-based contracting approach, the Program should ensure its contracts include all the necessary elements of performance-based contracts.
GVRA should improve its outreach efforts to ensure persons with disabilities are aware of the vocational services available to assist them in obtaining employment. (p. 21)
8. GVRA should continue its coordination efforts to improve access to VR services for persons with disabilities, particularly students with disabilities.
GVRA should continue its efforts to reduce counselor turnover and streamline the hiring process to fill vacancies in a timely manner. (p. 23)
9. If turnover remains an ongoing concern, GVRA should continue its efforts to survey existing and departing employees to identify other factors that influence employees' decisions to leave their positions. In addition, GVRA should monitor the actions taken so far to determine if additional action is necessary to address causes of counselor turnover.
10. Similarly, GVRA should continue to evaluate hiring time and determine if actions taken have addressed delays or if additional actions are needed.
11. GVRA should evaluate counselor caseloads to ensure improvements in counselor caseloads are commensurate with any reductions in turnover and hiring delays.

Vocational Rehabilitation Program

42

Appendix A: Table of Recommendations (continued)

GVRA should take steps to improve its administration of the Ticket to Work Program. (p. 27)
12. Ensure Program staff have access to all available resources needed to make appropriate decisions related to payment requests. This includes: a. ensuring staff are adequately trained to effectively administer the Ticket to Work reimbursement process, and b. developing a system to ensure that SSI/SSDI recipients are effectively identified and that the most appropriate Ticket to Work payment model is selected for each client.
13. Establish adequate procedures to ensure that reimbursement requests are submitted timely.
GVRA should take steps to ensure that it consistently conducts financial needs assessments as required by Program policy. (p. 29)
14. GVRA should evaluate the cost-benefit of its financial needs assessment policy and use the information to determine if the policy should be continued or discontinued.
15. If GVRA finds the financial needs assessment policy is cost beneficial, it should: a. ensure Program staff consistently conduct financial needs assessments as required by its policies, and b. resume case file monitoring activities to ensure all required documentation (e.g., financial needs assessment, supporting documentation) is included in each client's electronic case file.
GVRA should improve controls over equipment purchased for clients and ensure that equipment is properly inventoried and recovered. (p. 31)
16. Update policies to reflect its current organizational structure. In addition, it should reassign responsibility for inventory management to positions that currently exist within its structure.
17. Develop a statewide inventory of purchased equipment for tracking the issuance, return, and reissuance of equipment purchased for clients.
18. Integrate inventory data with its client/case management system to ensure equipment is assigned to individuals who need it.
GVRA should ensure that its next case management system is acquired and implemented according to best practices. (p. 34)
19. GVRA should work with GTA and the Department of Administrative Services (DOAS) to evaluate the vendor's recommended replacement system and to identify potential alternatives that will meet the needs of the VR Program.
20. In so doing, GVRA should ensure that it follows state policies and standards for information technology and procurement to ensure that it transitions to a system that is well-designed and well-implemented. Significant focus should be placed at the front-end of the process which involves users carefully outlining the required specifications for the new system prior to going through an RFQ or RFP process.
GVRA should ensure that conflicts of interest are reported and managed appropriately. (p. 35)
21. GVRA should ensure that employees hired prior to the creation of GVRA complete the conflict of interest questionnaire.
22. GVRA should implement a process to require an annual update of the conflict of interest questionnaire.
23. GVRA should ensure that conflict of interest questionnaires are properly reviewed and any actions taken in response to disclosed conflicts are documented.

Vocational Rehabilitation Program

43

Appendix A: Table of Recommendations (continued)

Additional guidance is needed to enhance protection of confidential client information. (p.37)
24. GVRA should develop and implement a security plan that includes mandatory periodic training in computer security and restricting users to the minimum access or type of access necessary to perform their jobs. A security plan would also outline techniques for safeguarding the security of information. a. GVRA should ensure that its staff understand what information is confidential and how confidential information should be transmitted.
25. GVRA should establish a centralized policy for the management of physical client files. Offices should be held accountable for management of client information.
GVRA should ensure that it has documented policies and procedures for key business operations. (p.39)
26. The VR Program should ensure that up-to-date policies procedures are established and clearly communicated to staff.

Vocational Rehabilitation Program

44

Appendix B: Objectives, Scope, and Methodology
Objectives
This report examines the Vocational Rehabilitation (VR) Program within the Georgia Vocational Rehabilitation Agency. Specifically, our audit set out to determine the extent to which the Program is:
1. serving those eligible and in need of its services, and
2. accomplishing its mission of employment and independence for individuals with disabilities, including delivering services in a cost-effective manner.
Scope
This audit generally covered activity related to the VR Program that occurred in fiscal years 2014 to 2015, with consideration of earlier or later periods when relevant. Information used in this report was obtained by reviewing relevant state and federal laws, rules, and regulations; interviewing GVRA officials and staff, service providers, and the Rehabilitation Services Administration's (RSA) state liaison, and representatives of other states' programs; reviewing GVRA/VR Program documents; researching existing audit reports, academic studies, industry studies, and news articles relating to vocational rehabilitation services; and conducting site visits to GVRA field offices. We also analyzed data and reports obtained from GVRA, and RSA. Data obtained from GVRA's case management system, GRACI, was used to inform several objectives. Data obtained from these sources was deemed sufficiently reliable for the purposes of our review.
Government auditing standards require that we also report the scope of our work on internal control that is significant within the context of the audit objectives. We reviewed internal controls as part of our work under both objectives, and other areas as we saw fit. Deficiencies in internal control are discussed in the findings on pages 11, 14, 17, 19, 27, 29, 31, 34, 35, 37, and 39 of this report. Specific information related to the scope of our internal control work is described by objective in the methodology section below.
Methodology
To determine the extent to which the Program is serving those eligible and in need of its services, we estimated the number of individuals served by the Program and assessed the Program's efforts to promote services and make them accessible to the target population. To determine numbers served, we ran queries in GRACI and analyzed the data to identify the number of clients aged 21-64 served by the Program during fiscal years 2010 to 2014. Using disability statistics compiled by Cornell University, we identified the number of persons with disabilities aged 21-64 and number of individuals in that group looking for work during calendar years 2010 to 2014.38 We used this information to calculate the percentage of disabled adults looking for work and compared results to numbers served by GVRA. We interviewed GVRA officials and staff about outreach efforts and collaboration with entities serving similar populations. We obtained and reviewed collaborative agreements between GVRA, Department of Behavioral Health and Developmental Disabilities (DBHDD), and local school systems. We also interviewed Department of Education staff to understand

38 Figures on disabled adults are estimates based on samples of the population surveyed by the U.S. Census Bureau for the annual American Community Survey (ACS).

Vocational Rehabilitation Program

45

transition services available in local school systems and their coordination with GVRA. We reviewed proposed federal regulations related to the federal Workforce Innovation and Opportunity Act (WIOA) to document requirements related to transition-aged youth.
To gain an understanding of staffing concerns that might limit GVRA's ability to provide services to those in need, we interviewed GVRA officials and staff, reviewed Program documents related to caseloads, salaries, hiring, and turnover (including employee satisfaction survey results). To identify the number of active cases being handled by each of the 39 field offices, we used GRACI to identify active cases and assigned offices as of November 2014. We also obtained GVRA data on number of counselor positions--filled and vacant certified and provisional counselor positions-- by office location. We obtained and analyzed GVRA data on employee terminations, which was used to calculate turnover rates for counselor positions. For comparative purposes, we requested and obtained data from the Department of Administrative Services on turnover across executive branch agencies, including GVRA. We reviewed academic and other professional literature to identify impacts of counselor turnover on clients. The audit team also reviewed GVRA hiring data on a non-statistically valid sample of new employees to assess timeliness of the hiring process and researched human resources industry information to identify industry standards and best practices.
To determine the extent to which the Program is accomplishing its mission of employment and independence for individuals with disabilities and delivering services in a cost-effective manner, we evaluated client outcomes and the length of time clients spent in the Program and reviewed processes in place to ensure services were being delivered in a cost-effective manner.
To evaluate client outcomes and length of time in the Program, we interviewed GVRA officials and staff and Georgia's federal RSA liaison. We reviewed federal regulations, performance indicators on service outcomes, and various reports compiled by RSA on the status of Georgia's VR Program. We reviewed information provided by GVRA relating to its mission and purpose and various policies and procedures manuals. We also reviewed various research and audit reports conducted by the Government Accountability Office (GAO) and other states to identify best practices. To determine outcomes of VR clients and time spent in the rehabilitation process, we obtained a full backup copy of data from GVRA's GRACI system as of August 2014 and analyzed the data to identify a cohort of clients who initiated a case with the VR Program in fiscal year 2008. Of the 12,707 unduplicated individuals identified, we excluded 1,029 individuals who ultimately were not served by the Program due to death or because they were transferred to another agency or institution, too severely disabled, did not have a disabling condition, or did not have an impediment to work. We conducted various analyses to assess the outcomes and length of time in the Program for the 11,678 eligible cohort clients. We also analyzed data in GRACI to document counselor case management activities for clients active as of May 2015, which provided possible reasons for timeliness issues identified in the report.
To evaluate cost-effectiveness of service delivery, we interviewed GVRA officials and staff and reviewed relevant Program policies and procedures. We contacted representatives in six other states (Alabama, Illinois, Mississippi, Ohio, Pennsylvania, Tennessee) and Puerto Rico and conducted industry research to identify best practices related to vocational rehabilitation services. We analyzed client data

Vocational Rehabilitation Program

46

provided by GVRA on Ticket to Work claims/payment requests submitted in federal fiscal years 2008 to 2014 to document requests paid and denied during that time period and reviewed guidance provided by the Social Security Administration on submitting claims. We accessed client data in GRACI to document compliance with GVRA policies relating to financial needs assessments and inventory of purchased equipment.
In addition to work under these objectives, we evaluated the effectiveness of 1) controls over the GRACI client information system, 2) controls in place to ensure conflicts of interest are identified, and 3) controls in place to ensure confidential and sensitive client information is protected. To accomplish this, we interviewed GVRA officials and staff, DOAS staff, and Georgia Technology Authority (GTA) staff to identify processes in place at the time GRACI was procured. We reviewed policies and procedures related to conflicts of interest and confidential information and conducted site visits to GVRA field services offices. We consulted with Department of Audits and Accounts Technology Resources Assurance Services Division, reviewed information on GTA's website, and conducted research to identify best practices and industry standards with regard to procurement of IT systems, conflicts of interest, and information security.
We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Vocational Rehabilitation Program

47

Appendix C: Federal Performance Indicators for GVRA (Federal Fiscal Years 2010-2014)

Indicator

Goal

2010 2011 2012 2013 2014

Difference between number exiting program

Equal or

Pass Pass Pass

Fail

Fail

1.1

with employment outcome during current period and number exiting program with

Exceed Previous

employment outcome during previous period Performance

161

239

418 1,469 2,038

Percentage exiting program during current

1.2

period with employment outcome after

receiving services

55.80%

Pass Pass Pass

Fail Pass

62.45% 62.57% 58.72% 35.04% 64.81%

Percentage exiting program in integrated employment with hourly wages of at least

Pass Pass Pass Pass Pass

1.3

minimum wage, based on those exiting with

72.60%

employment outcome after receiving services

95.32% 95.17% 96.93% 96.30% 92.13%

1.4

Percentage of those identified in Indicator 1.3 who have significant disabilities

62.40%

Pass Pass Pass Pass

Fail

78.23% 84.76% 85.61% 89.48% 51.68%

Ratio of average hourly earnings of all

1.5

individuals in competitive employment to average hourly earnings of all employed

individuals in state

For those in Indicator 1.3, difference in

percent who at program entry reported their

1.6

income as largest source of support, and

percent that reported their income as largest

source of support at program exit

Ratio of the percent with a minority

2.1

background to the percent without a minority background exiting program who received

services

0.52 53 0.8

Fail

Fail

Fail

Fail

0.45

0.43

0.44

0.47

Pass Pass Pass Pass

73.27 74.44 72.92 70.16

Pass Pass Pass Pass

0.94

0.89

0.93

0.9

Fail 0.48 Pass 76.4 Pass
0.9

Source: GVRA Data

The Performance Audit Division was established in 1971 to conduct in-depth reviews of state-funded programs. Our reviews determine if programs are meeting goals and objectives; measure program results and effectiveness; identify alternate methods to meet goals; evaluate efficiency of resource allocation; assess compliance with laws and regulations; and provide credible management information to decision-makers. For more information, contact
us at (404)656-2180 or visit our website at www.audits.ga.gov.