Vocational Rehabilitation Program: action has been taken, but further client and provider monitoring is needed

Follow-Up Review Report No. 17-20

July 2018

Why we did this review
This follow-up review was conducted to determine the extent to which the Georgia Vocational Rehabilitation Agency (GVRA) addressed recommendations in our October 2016 performance audit (Report #14-18).
The 2016 audit examined the extent to which the Vocational Rehabilitation (VR) Program was: 1) serving those eligible and in need of its services and 2) accomplishing its mission of employment and independence for individuals with disabilities, including delivering services in a cost-effective manner.
About the VR Program
GVRA administers the VR program to provide vocational rehabilitation services that enable individuals with disabilities to achieve employment and independence. Services such as vocational counseling, job placement, and mental health and substance abuse treatment are provided by VR staff, non-profit community rehabilitation programs, or other vendors.
In fiscal year 2019, the VR program was appropriated approximately $111 million in primarily state and federal funds to support its field offices. In fiscal year 2018, the Program served an estimated 35,000 clients (as of April 2018) with a variety of disabilities, including physical and mental impairments.

Follow-Up Review
Vocational Rehabilitation Program
Action has been taken, but further client
and provider monitoring is needed
What we found Since our 2016 performance audit, the Georgia Vocational Rehabilitation Agency (GVRA) has addressed several recommendations from our original report. In addition to restructuring internal processes and acquiring a new case management system, Vocational Rehabilitation (VR) staff have made improvements based on new rules in the federal Workforce Innovation and Opportunity Act (WIOA), which went into effect in October 2016.
As discussed in the original report, GVRA's previous case management system was an ineffective management tool due to various issues, including incomplete case file data, inconsistent reporting, and duplicate data entry. Staff indicated GVRA's new case management system--which became operational in May 2018--is expected to improve functionality in workflow management, financial management, and reporting, thereby enabling the agency to address a number of audit recommendations.
The original audit noted that GVRA did not have a system to track clients who did not receive an employment placement or monitor the length of time required for a client to move through the rehabilitation system and into employment. At the time of the audit, approximately 61% of clients were identified as exiting the VR program without an employment outcome; 37% of those exited before receiving training services.
GVRA has begun reporting the number of cases that have exceeded federal- or state-prescribed durations for each program phase (e.g.,

270 Washington Street, SW, Suite 1-156

Atlanta, Georgia 30334

Phone: (404)656-2180

www.audits.ga.gov

Vocational Rehabilitation Program

2

referral, training) to monitor timeliness and ensure clients are moving through the process. GVRA staff members review monthly reports and identify staff training opportunities when a large proportion of cases exceed time in a particular phase. The agency's new case management system will also flag cases that have been open for more than a specified amount of time.
While the agency has not yet incorporated performance measures similar to those identified in the original audit, plans are in place to begin monitoring client outcomes. According to staff, WIOA requires the agency to track the percent of participants employed 6 and 12 months after their exit, along with their median earnings and measurable skills gained. The new case management system will report these and other metrics identified during the VR strategic planning process.
GVRA has also not reviewed outcomes at the provider level, which the original audit noted could assist clients with making informed decisions and help GVRA evaluate the cost-effectiveness of its contracts. GVRA staff indicated they are working to gather data that could be used in provider report cards to clients as well as performance-based contracting. Potential metrics include the type of services offered, client utilization rates, and job placement outcomes similar to those required by WIOA.
Changes in WIOA have also led to actions related to the original report's recommendation to improve outreach efforts, particularly for students with disabilities. GVRA now has memoranda of understanding with multiple entities, including postsecondary education institutions, county school systems, the Georgia Department of Education, and the Department of Juvenile Justice to ensure eligible clients are aware of and have access to VR services.
The original audit also identified several areas within central office administration that should be improved to increase efficiency, consistency, and stewardship over resources. GVRA has taken the following steps to address these areas, though some additional action is needed.
GVRA has increased training for employees in the Ticket to Work program, which staff stated has led to improved processes for obtaining reimbursements from the Social Security Administration. According to staff, more timely and appropriate reimbursement requests increased revenue by $1.1 million between state fiscal years 2016 and 2017.
Counselor turnover rates have decreased from 30% at the time of the original audit to 20% in fiscal year 2017. Staff attribute the lower rate and a decreased time to hire to numerous new initiatives, including a new position, a paid internship program, division of case duties across multiple counselors (which staff stated has lowered caseload burden), and supplemental pay increases for certain counselors.
GVRA has worked to improve policies and procedures related to areas identified in the original audit, including financial needs assessments (FNA), inventory management, conflicts of interest, and confidential information. However, some practices have remained unchanged. VR staff, for example, continue to apply FNAs inconsistently, and the agency has yet to create an inventory management system. According to staff, the increased functionality of the new case management system should assist in further improvements, though it should be noted additional steps may be necessary depending on the system's capabilities.
GVRA's Response: GVRA stated it agrees with the audit team's assessment and indicated it has taken steps to make immediate improvements. Additional actions GVRA intends to take in areas not fully addressed are summarized in the table.
The following table summarizes the findings and recommendations in our 2016 report and actions taken to address them. A copy of the 2016 performance audit report #14-08 may be accessed at http://www.audits.ga.gov/rsaAudits.

Vocational Rehabilitation Program

3

Vocational Rehabilitation Program

Follow-Up Review, July 2018

Original Findings/Recommendations

Current Status

Partially Addressed The original audit noted GVRA was not tracking employment-related outcomes, including the percent of clients who exited without an employment outcome, the percent of unsuccessful clients who exited before receiving training, and reasons clients left the program prematurely.

The outcomes of VR clients have not been tracked, monitored, or evaluated to ensure the Program effectively serves its clients.
We recommended that the Program develop performance indicators and a process for regularly assessing its effectiveness in helping clients achieve program goals.

While these or similar metrics have not yet been incorporated into program monitoring, GVRA staff indicated they will begin tracking additional metrics to comply with new requirements in the federal Workforce Innovation Opportunity Act (WIOA). These include the percent of participants employed 6 and 12 months after their exit and their median earnings, as well as metrics showing measurable skills gained and effectiveness in serving employers. GVRA staff also indicated they will expand follow-ups and customer satisfaction surveys to include clients who exited without training.
GVRA hired its first program evaluator to help design evaluation methods and incorporate performance measures into the program's strategic plan. Staff also indicated the program's new case management system, which became operational in May 2018, is able to create customized reports based on management requests in addition to those federally required.

GVRA Response: GVRA agreed and stated it has additional plans for monitoring and evaluation. This includes developing performance metrics for the program; reintroducing a process to monitor client plan development, services received, and the success of services; and tracking WIOA performance indicators quarterly. GVRA also stated its case management system will be used to track the metrics identified in the original report.

GVRA should evaluate and improve monitoring activities to ensure clients move through the rehabilitation process in a reasonable time frame and that areas needing improvement are identified.
We recommended that GVRA evaluate and revise its processes for monitoring clients' progress through the various phases of the rehabilitation process. We also recommended GVRA regularly evaluate clients' length of time in the Program to establish baselines and identify areas for improvement. Finally, we recommended GVRA ensure counselors are conducting required monitoring activities.

Fully Addressed GVRA implemented comprehensive monthly reporting in 2017 to document the number of active cases and the timeliness of service delivery. These reports include a tally of the number of cases that exceed the federal- or state-prescribed duration for each status without a "legitimate reason" (e.g., client rescheduled appointment, provider waiting list). In the March 2018 monthly report, nearly 3,000 of the 23,500 total active cases (12%) were exceeding status without a legitimate reason, though more than a third of active cases in work plan development and work plan completion exceeded time frames.
GVRA's WIOA Compliance and Policy Unit identifies cases based on status duration to investigate reasons for delayed completion. While GVRA has not yet established benchmark percentages, the unit will determine whether additional staff training is needed if a particular status has a large proportion of cases exceeding benchmarks. Additionally, according to staff, managers are required to monitor a report that shows cases that have had no documentation in three months and follow up to ensure issues have been addressed.
Finally, GVRA's new case management system is expected to track milestones and flag cases that have been open for more than a specified length of time. Alerts will be sent to counselors, team leads, and supervisors.

Vocational Rehabilitation Program

4

Vocational Rehabilitation Program

Follow-Up Review, July 2018

Original Findings/Recommendations

Current Status

GVRA should take action to produce information clients need to ensure they make informed decisions about the services they receive.
We recommended that GVRA analyze client outcomes against service and provider data, and make the results available to counselors and clients. We also recommended that GVRA have a method for tracking and evaluating said outcomes and assess whether or not the necessary skills and expertise exist within the organization to ensure the information is collected and evaluated properly.

Not Addressed GVRA has not yet compiled provider information and outcomes into a report card or dashboard available for counselors to share with clients. As noted in the original report, this information would enable clients to make informed choices based on outcomes achieved by individuals working with the service providers, which federal regulations have emphasized.
GVRA staff stated they are working to gather the appropriate data and decide on metrics that would reflect provider performance in the report cards. Information may include the types of services offered, client utilization rates, and job placement outcomes. Staff plan to implement the provider dashboards or report cards at the end of calendar year 2019; however, this depends on the new case management system's functionality. It should be noted, however, that some aspects of vendor management and performance were not included in the system's initial implementation and may be added in the future.
GVRA Response: GVRA agreed with the current status. While GVRA works to develop provider report cards (which is expected to be completed by November 1, 2019), it plans to provide written reports to clients on each provider that will include services offered, number of GVRA clients served in the past year, and the length of time as a provider.

Not Addressed GVRA has not yet implemented performancebased contracting for major providers. Rather, the contracts established between GVRA and service providers allow for payment based on the service or hours provided, rather than client outcomes (as noted in the original audit). Similar to the provider dashboards discussed above, performance-based contracts for service providers would help to ensure the quality of services delivered to clients, as well as client outcomes.

GVRA should continue efforts to establish performance-based contracts to ensure CRPs deliver quality, cost-effective services to VR clients.
We recommended that GVRA fully implement the performance-based contracting approach and ensure contracts include the necessary elements of a performance-based contract.

GVRA is currently negotiating with Community Rehabilitation Programs (CRPs), the largest providers of VR services, to add performance measures to contracts. As part of the negotiation process, GVRA has asked providers to begin producing quarterly reports that include metrics similar to those required by WIOA, such as credential attainment, employment rates, average wage, and measurable skill gains. GVRA is also asking CRPs to verify the names and case numbers of the individuals served, as well as report on reasons for accepting or rejecting clients and the average length of time for clients to progress.
According to staff, implementing performance-based contracting is a challenge because CRPs do not necessarily collect data in the same way. As such, they must first agree on common collection methods to ensure valid comparisons among providers. Additionally, while the previous case management system did not produce reliable reports that would enable provider monitoring, staff indicated the new system would better support performance-based contracting.

GVRA Response: GVRA agreed and stated it intends to establish performance-based contracts by June 1, 2019.

Vocational Rehabilitation Program

5

Vocational Rehabilitation Program

Follow-Up Review, July 2018

Original Findings/Recommendations

Current Status

The Program should improve its outreach efforts to ensure persons with disabilities are aware of the vocational services available to assist them in obtaining employment.
We recommended that GVRA continue its coordination efforts to improve access to VR services for persons with disabilities, particularly for students with disabilities.

Fully Addressed GVRA has conducted broad outreach campaigns through radio, television, and social media, and worked to improve intake and outreach procedures. Additionally, the agency has taken steps to target populations that may be eligible for VR services, particularly by increasing its presence in high schools to improve outreach to students with disabilities (based on WIOA mandates). Changes in WIOA also require GVRA to coordinate with other groups, such as the Georgia Department of Education and the Department of Juvenile Justice, to identify and provide services to students and other individuals who may be eligible.

Fully Addressed In an effort to reduce counselor turnover, GVRA sought state funds in the summer of 2017 to increase pay for staff who work with clients. Though the funding request was not granted, GVRA has worked to decrease turnover rates using other strategies.

GVRA should continue its efforts to reduce counselor turnover and streamline the hiring process to fill vacancies in a timely manner.
We recommended that GVRA continue to survey existing and departing staff to learn why employees leave, and monitor actions taken to determine whether additional efforts are needed to address turnover. We also recommended that GVRA continue to evaluate hiring time and determine whether they have done enough to address delays, while evaluating caseloads to ensure they are reduced appropriately given staff turnover and hiring delays.

According to agency staff, counselor turnover in fiscal year 2017 was 20%, down from 30% at the time of the original audit. Staff also indicated the average time-to-hire has decreased from five months to three to four months. Staff attribute the increased retention and improved hiring to various efforts and initiatives that followed the 2016 audit. These include:
Providing supplemental pay to staff who have completed a certification program and work with specific caseloads;
Funding exams for counselors seeking rehabilitation certificates;
Creating a new position that requires a bachelor's degree (unlike the certified counselor position that requires a master's degree);
Dividing case-related tasks across multiple positions rather than a single counselor;
Creating a paid internship position that can more easily transition to full-time employment;

Hiring a recruitment coordinator and restructuring the human resource department's talent acquisition team; and

Providing training and development opportunities for all staff through a memorandum of understanding with the Carl Vinson Institute of Government.

Vocational Rehabilitation Program

6

Vocational Rehabilitation Program

Follow-Up Review, July 2018

Original Findings/Recommendations

Current Status

GVRA should take steps to improve its administration of the Ticket to Work Program.
We recommended that GVRA ensure program staff have the appropriate resources available for making TTW decisions-- specifically that staff are trained to administer the reimbursement process and a system is developed to ensure eligible recipients are identified and the best payment model selected for each client. We also recommended that GVRA establish procedures to ensure reimbursement requests are submitted on time.

Fully Addressed The original audit found that staff were not administering the Ticket to Work (TTW) program efficiently and failed to maximize possible reimbursements from the Social Security Administration because they did not ensure clients met the criteria and failed to submit reimbursement requests in a timely manner.
In state fiscal year 2017, GVRA collected approximately $5.2 million in TTW revenue--up from $4.1 million in fiscal year 2016. Staff attribute this increase to newly implemented training that has helped ensure staff are properly filing client payment requests and meeting submission deadlines. GVRA also created a document educating staff on appropriate policies and procedures related to determining the correct reimbursement model and client eligibility.

GVRA should take steps to ensure that it consistently conducts financial needs assessments as required by Program policy.
We recommended that GVRA evaluate the cost-benefit of its FNA policy and determine whether the policy should be continued. We also recommended that if GVRA determined the FNA was beneficial, it should ensure Program staff consistently conduct the assessments according to policy and resume case file monitoring to ensure all required documentation is included in each client's electronic case file.

Partially Addressed GVRA has taken steps to address the weaknesses and inconsistencies in the Financial Needs Assessment (FNA) policy and process identified in the original audit, which led to inconsistent application of the FNA. According to GVRA staff, FNAs were determined to be necessary because they help ensure the agency is the payee of last resort by identifying clients who can pay a portion of their costs. As such, the agency determined that all client cases should contain the FNA. Staff were trained on FNAs in 2017 and will receive further training in 2018.
Staff stated that though there continues to be inconsistent application of the FNA, the new case management system includes significant controls that will ensure counselors complete FNA forms during client enrollment.
GVRA Response: GVRA agreed and indicated it will take additional steps to further ensure FNAs are consistently applied. These include more trainings in the coming year and an audit to identify specific problem areas. The agency expects that with the additional training and ongoing quality assurance reviews, FNA compliance will be improved within the next year (with an established target date of January 1, 2019).

Vocational Rehabilitation Program

7

Vocational Rehabilitation Program

Follow-Up Review, July 2018

Original Findings/Recommendations

Current Status

GVRA should improve controls over equipment purchased for clients and ensure that equipment is properly inventoried and recovered.
We recommended that GVRA update policies to reflect the current organizational structure and reassign responsibility for inventory management to positions that currently exist within its structure. We also recommended that GVRA conduct a statewide inventory of equipment purchased for clients and integrate data with case management system to ensure equipment is assigned to individuals who need it.

Not Addressed The current state of inventory management remains largely unchanged from the processes reviewed during the original audit, which, as noted in the report, could result in the program unnecessarily purchasing equipment and leaving inventory vulnerable to fraud and abuse. However, staff members stated they are committed to enhanced enforcement of current inventory management standards.
A policy and procedure review committee consisting of representatives from several GVRA divisions began meeting in April 2018 to develop a plan for tracking and storing equipment, which will be completed in September 2018. This includes selecting a centralized storage location and purchasing an automated inventory management software. Inventory management was not included in the new case management system's functionality, though it may be added in the future.
GVRA Response: GVRA agreed and reiterated its plan to evaluate policies, implement an electronic or automated accountability system, and establish a central storage facility for purchased items that clients may return or are currently not in use (expected by June 1, 2019).

GVRA should ensure that its next case management system is acquired and implemented according to best practices.
We recommended that as GVRA replaces its current case management system, it work with the Department of Administrative Services (DOAS) and the Georgia Technology Authority (GTA) and follow state policies and standards for information technology procurement. This includes clearly outlining requirements.

Fully Addressed GVRA's new case management system became operational at the beginning of May 2018. GVRA began acquiring the system in 2016 through a sole source process approved by GTA and DOAS. During development, GVRA utilized Independent Verification and Validation services to ensure the correct product was being built and identify issues.
GVRA outlined business requirements for the new system, which include functionality related to workflow management, financial management, reporting, and vendor management. According to staff, this functionality has improved over the previous system. Additionally, staff indicated the recent system's data conversion was more structured than what was observed in the previous audit to ensure complete client files were transferred.

GVRA should ensure that conflicts of interest are reported and managed appropriately.
We recommended that GVRA ensure all employees complete the conflict of interest questionnaire and implement a process to require an annual update of the questionnaire. We also recommended that GVRA implement a process to review all questionnaires and document any actions taken in response to disclosed conflicts.

Fully Addressed While GVRA had a conflict of interest policy at the time of the original audit, the agency had not enforced the process for identifying conflicts of interest across all employees. Since the audit, GVRA's human resources department has created an agency-wide policy that governs conflicts of interest for all employees, increased training, and, according to staff, ensured that all employee files contain a completed conflict-ofinterest questionnaire.
All new hires now receive training and complete the conflict of interest questionnaire during orientation. Human resources staff then review the documentation to ensure the employee does not receive an assignment that may cause a conflict. Employees recently promoted to a managerial or supervisory position also receive conflict of interest training, as well as all managers and supervisors at an annual leadership summit. Potential conflicts that arise after an employee is hired can be reported during the annual training or following major life changes.

Vocational Rehabilitation Program

8

Vocational Rehabilitation Program

Follow-Up Review, July 2018

Original Findings/Recommendations

Current Status

Additional guidance is needed to enhance protection of confidential client information.
We recommended that GVRA develop and implement a security plan that includes training users in computer security and restricting access to client information. We also recommended GVRA develop a policy for managing physical client files.

Fully Addressed In accordance with GTA requirements, GVRA staff are currently preparing a security plan to describe how they intend to handle confidential data in the new system. This includes more clearly defining security permissions for counselors (during the original audit, all had similar access ability).
GVRA also revised its policy regarding securing hardcopy client files. Offices are now required to secure files in a designated locked room or file cabinet, with access only permitted during business hours. The policy also now restricts access only to GVRA staff who also have electronic access to the case files, which, as mentioned above, has been further limited.
Finally, GVRA developed training on how to handle confidential information, which is provided to new hires and new managers. GVRA staff also indicated its Office of General Counsel worked with all departments to ensure record retention policies complied with state archives mandates.

GVRA should ensure that it has documented policies and procedures for key business operations.
We recommended that the VR Program ensure that up-to-date policies procedures are established and clearly communicated to staff.

Fully Addressed GVRA has updated its VR client services policy manual several times since the original audit, which found that policies and procedures for certain VR functions were either out-of-date or missing. The agency has created a Standing Policy Committee, consisting of senior leadership, which meets monthly to review policy and determine changes. Updates are communicated to staff via formal policy communications and policy memos.
One of the two programs found to lack policies in the original audit was absorbed into a separate unit that is governed by division-specific policies and processes along with the general client services policy manual. Policies and procedures related to the other program--Ticket to Work--have been removed from the manual and are updated based on federal regulations, which are communicated to staff as needed.

13 Findings

8 Fully Addressed 2 Partially Addressed 3 Not Addressed

The Performance Audit Division was established in 1971 to conduct in-depth reviews of state-funded programs. Our reviews determine if programs are meeting goals and objectives; measure program results and effectiveness; identify alternate methods to meet goals; evaluate efficiency of resource allocation; assess compliance with laws and regulations; and provide credible management information to decision makers. For more information, contact
us at (404)656-2180 or visit our website at www.audits.ga.gov.