Annual quality management report 2016

FY 2016 QUALITY MANAGEMENT
ANNUAL REPORT

TABLE OF CONTENTS
Section 1: Executive Summary_________________________________________________________4 Section 2: Introduction _____________________________________________________________10 Section 3: Behavioral Health Quality Reviews ____________________________________________14
Section 3a: Sample selection ________________________________________________________________ 15 Individual Records and Billing Review _______________________________________________________ 15 Individual and Staff Interviews _____________________________________________________________ 17
Section 3b: BHQR Overall Review Scores _______________________________________________________ 17 BHQR Overall Scores _____________________________________________________________________ 19 BHQR Overall Scores by Tier _______________________________________________________________ 21 BHQR Billing Validation Scores _____________________________________________________________ 25 BHQR Assessment and Treatment Planning Scores _____________________________________________ 28 BHQR Compliance with Service Guidelines Scores _____________________________________________ 30 BHQR Focused Outcome Areas (FOA) Scores__________________________________________________ 33 BHQR Staff and Individual Interview Scores __________________________________________________ 36
Section 3c: Assertive Community Treatment ___________________________________________________ 38 ACT BHQR Billing Validation Scores _________________________________________________________ 41 ACT BHQR Assessment and Treatment Planning Scores _________________________________________ 43 ACT BHQR Compliance with Service Guideline Scores __________________________________________ 45 ACT BHQR Focused Outcome Areas Score ____________________________________________________ 48
Section 3d: Crisis Stabilization Unit Quality Reviews _____________________________________________ 50 Sample Selection________________________________________________________________________ 51 CSU Quality Review Overall Scores__________________________________________________________ 51 CSU Individual Record Review Scores _______________________________________________________ 53 CSU Compliance with Service Guidelines Scores _______________________________________________ 56

Page 1

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

CSU Focused Outcome Areas Scores ________________________________________________________ 58 CSU Staff and Individual Interview Scores ____________________________________________________ 60 Section 3e: Behavioral Health Summary of Findings and Recommendations __________________________ 62 Summary of Findings ____________________________________________________________________ 62 Recommendations ______________________________________________________________________ 64
Section 4: Intellectual and Developmental Disabilities ___________________________________ 66
Section 4a. Background (IDD) ________________________________________________________________ 66 Section 4b: Sample Selection ________________________________________________________________ 68
Person Centered Review (PCR) _____________________________________________________________ 68 Quality Enhancement Provider Review (QEPR) ________________________________________________ 68 Section 4c: Review Processes ________________________________________________________________ 69 Section 4d. Overall Review Scores ____________________________________________________________ 71 Person Centered Review (PCR) _____________________________________________________________ 73 Quality Enhancement Provider Review (QEPR) ________________________________________________ 75 Section 4e. Person Centered Review __________________________________________________________ 77 PCR Scores by Focus Outcome Area (FOA)____________________________________________________ 77 PCR Scores by Tool ______________________________________________________________________ 78 PCR Scores by Tool and Focused Outcome Area (FOA) __________________________________________ 81 Section 4f: Quality Enhancement Provider Review (QEPR) _________________________________________ 85 QEPR Scores by Provider Size ______________________________________________________________ 85 QEPR Scores by Focused Outcome Area (FOA) ________________________________________________ 89 QEPR Review Scores by Tool _______________________________________________________________ 91 QEPR Scores by Tool and FOA______________________________________________________________ 94 Section 4g: ISP QA Checklist_________________________________________________________________ 97 ISP Written to Support ___________________________________________________________________ 97 ISP Expectations ________________________________________________________________________ 99 Section 4h: Quality and Technical Assistance Consultation (QTAC) _________________________________ 100

Page 2

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Section 4i: IDD Summary of Findings_________________________________________________________ 104 Community Life ________________________________________________________________________ 106 Choice _______________________________________________________________________________ 107 Person Centered Practices _______________________________________________________________ 107 Provider Documentation ________________________________________________________________ 108
Section 4j: Recommendations ______________________________________________________________ 108
Section 5: Behavioral Health and Intellectual Developmental Disabilities Comparisons __________109
Overall Results by Focused Outcome Area--Behavioral Health and IDD _____________________________ 109 Record Review Results by Indicator/Question and FOA __________________________________________ 111 Individual and Staff Interviews IDD and Behavioral Health Comparison ____________________________ 112
Section 6: Conclusion______________________________________________________________115 Appendix A: Abbreviations and Acronyms _____________________________________________116

Page 3

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

SECTION 1: EXECUTIVE SUMMARY
On June 30, 2016, The Georgia Collaborative Administrative Services Organization (ASO) completed the first year of the new and innovative quality management contract with the Georgia Department of Behavioral Health and Developmental Disabilities (DBHDD). The ASO was tasked with incorporating disparate service delivery systems into one collaborative and integrated quality program: behavioral health, crisis stabilization, and intellectual and developmental disability services. Throughout the implementation period (September 2014 to June 2015), ASO staff worked collaboratively with DBHDD to create an integrated quality management approach, sharing ideas across programs to develop and coordinate tools and processes used to evaluate provider systems and assess the quality of services for individuals receiving behavioral health and intellectual and developmental disability services.

There are many similarities between the behavioral health and intellectual and developmental disability program. Therefore, similar tools and processes are used to monitor quality and compliance within each area including interviews with individuals and providers/staff and record reviews maintained by providers for individuals receiving services. Similarities include, for example, review of the individual support plan (ISP) for intellectual and developmental disabilities and the individualized recovery plan (IRP) for behavioral health. In both systems, focused outcome areas (FOA) critical to a person's quality of life are assessed: person centered practices, choice, community, rights, whole health, and safety. However, there are some distinct differences as well. Intellectual and Developmental Disability (IDD) reviews include onsite observations of licensed residential and day programs, as well as administrative review of staff qualifications and training requirements, while Behavioral Health (BH) reviews include extensive claims review to identify and report billing validation and discrepancies.

Page 4

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

During the first year of the contract, the ASO completed behavioral health quality reviews (BHQR) for 141 providers. At the request of DBHDD, Assertive Community Treatment (ACT) was analyzed separately from other behavioral health services. BHQRs are used to measure provider performance through four different components or categories of the review process. Findings are shown in the following table for BH and ACT reviews and indicate that ACT results were generally higher. BH showed lower compliance with assessment and planning standards than in other categories.

BHQR and ACT Results by Category

Billing

Assessment

Service

Validation FOA and Planning Guidelines

BH

81%

85%

79%

90%

ACT

92%

88%

85%

85%

Overall 84% 88%

Detailed findings of the lowest-scoring areas within the broader categories may direct quality improvement initiatives for the ASO. These include:
Whole Health was the lowest-scoring FOA for both BH and ACT reviews with some specific indicators that included: o Ongoing assessment to determine the need for external referrals for health services, supports, and treatment o Communication with external referral sources to determine results of testing and treatment o Medical conditions assessed, monitored, and recorded o Documented safeguards utilized for medications

Page 5

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Co-occurring health conditions were only included in 36 percent of the IRPs reviewed and generally without explanation as to why
Specific billing issues most often cited included: o Missing/incomplete service orders o Individuals not meeting admission criteria o Progress notes missing
In January 2016, the ASO began review of Georgia's 19 crisis stabilization units (CSU), reviewing 285 records. The focus was primarily to assess compliance with DBHDD and Medicaid requirements. CSUs showed an overall score of 83 percent, similar to BH reviews, with the FOAs showing the highest scoring area (88%). Findings suggest relatively high compliance with documentation of admission criteria, timely assessments, daily status updates, and required staffing and physician access. However, documentation lacked evidence that individuals routinely participate in the treatment planning process throughout their course of stay and treatment plans are not always tailored to the individual's presenting needs.
During the year, the ASO completed 484 Person Centered Reviews (PCR) and 100 Quality Enhancement Provider Reviews (QEPR) as part of the IDD quality assurance reviews for intellectual and developmental disabilities. The focus of the QEPR is to monitor providers to ensure they meet requirements set forth by the Medicaid waiver and DBHDD and to evaluate the effectiveness of their service delivery system. The focus of the PCR is to assess the individual's quality of life, as well as the effectiveness of and satisfaction with the service

Page 6

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

delivery system. Data indicate relatively high scores on most aspects of both processes. The following table lists results by tool and shows:
Relatively high compliance rates across all tools Documentation results (Provider Record Review [PRR] and Support Coordinator
Record Review [SCRR]) are lower than findings from observations and interviews Qualifications and Training requirements is the second lowest scoring tool which may
suggest the need for additional training for providers in this area

IDD PCR and QEPR Results by Tool

Tool

QEPR PCR

Individual Interview

95.1% 95.1%

Staff Interview

96.9% 96.4%

Provider Record Review

73.6% 79.0%

Observation

98.2% 98.4%

Support Coordinator RR

N/A 79.9%

Support Coordinator Interview

N/A 90.8%

Qualifications & Training

82.9% N/A

Overall

88.4% 93.8%

More detailed findings indicate lower scoring areas in community, choice and person centered practices. Individuals are not always:
Learning about, identifying or developing social roles Exposed to new experiences in the community or able to identify something new they
have experienced or learned Provided opportunities to develop community employment Involved in life's decisions

Page 7

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Offered informed choice for competitive employment or community participation
Recommendations from the BH and IDD data include: Adding more questions in BH to address qualitative issues as well as compliance Adding the Quality Technical Assurance Consultation to the BH reviews as needed Continuation of the Quality Training Series in FY 2017 Develop trainings that target community integration, promotion of supported employment and problematic areas in provider documentation
These and other results are discussed in more detail throughout the report, including a summary of review processes and sampling procedures. The report also includes a summary of findings for each FOA; in-depth analysis of data from the BH, IDD and CSU reviews; an initial comparative analysis of similar items across BH and IDD; as well as tabular and graphic displays of findings. Throughout the report, abbreviations and acronyms are employed for brevity and efficiency. Appendix A is a reference of all abbreviations and acronyms used in this report. Finally, throughout the report, we have included quotes from front line staff and individuals served, to give a voice to those often unheard.
"I can't even believe I am the same person I was three years ago. If someone told me life could be this way I would not believe it."

Page 8

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

The observations, findings, and recommendations in this report will be presented to leadership of DBHDD for consideration in identifying issues that need additional analysis, investigation, and interpretation to inform quality improvement initiatives. The responsibility for the use of the information within this report is that of the directors of Divisions of Performance Management and Quality Improvement, Behavioral Health, Intellectual and Developmental Disabilities, and Accountability and Compliance. The division directors will consider these and other quality data and recommendations collected by the department to improve the quality of services. DBHDD's reorganization provides a platform for clarified roles and responsibilities in addressing quality of services, including further analysis, implementation of targeted action steps, and determination of the impact of selected activities. The programmatic divisions of Behavioral Health and Intellectual and Developmental Disabilities will drive improvement strategies and be supported by the Divisions of Accountability and Compliance, and Performance Management and Quality Improvement, along with the resources and support of the Georgia Collaborative.
-DBHDD, Division of Performance Management and Quality Improvement

Page 9

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

SECTION 2: INTRODUCTION
The Georgia Department of Behavioral Health and Developmental Disabilities (DBHDD) created the Georgia Collaborative Administrative Services Organization (ASO) to integrate disparate systems for data, service delivery, finance, and quality oversight into one, cohesive, unified organization. Historically, quality reviews for both behavioral health providers and providers offering services for individuals with intellectual and developmental disabilities had been completed by different contracted organizations. The ASO's Quality Department was tasked with incorporating these two distinct systems (IDD and BH) into one collaborative and integrated program.
July 2015 marked the implementation of the new quality reviews developed by the ASO, in collaboration with and approved by DBHDD. In the first year, several goals were accomplished, including:
Coordinating and integrating, where applicable, BH and IDD review tools and processes Incorporating feedback from providers, individuals, families, advocates and
stakeholders into the review process Minimizing and alleviating some provider administrative burden by:
o Conducting joint BH and IDD reviews simultaneously when possible and appropriate
o Conducting person-centered reviews (PCR) during the quality enhancement provider reviews (QEPR) for providers rendering services for the intellectual and developmental disabilities population

Page 10

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

o Including a quality review of crisis stabilization unit (CSU) services during the behavioral health quality reviews (BHQR), if offered by the provider
Incorporating the Substance Abuse Mental Health Services Administration's (SAMHSA) National Behavioral Health Quality Framework recommendations into behavioral health review tools; and most importantly
Implementing a stronger focus on quality so that individuals receiving services from the provider network have easy access to high-quality care that leads to a life of recovery and independence

By aligning IDD and BH review tools and processes, the ASO's Quality Department created a more seamless and uniform system to assist DBHDD with oversight of their programs. Through the joint BH/IDD review process, the ability to share best practices across the historically disparate service systems was enhanced. Other important enhancements made under the new Georgia Collaborative ASO system include the following:
The Division of Developmental Disabilities adopted a more aggressive review schedule, similar to the BH reviews, to review more providers each year
For the intellectual and developmental disabilities reviews, an overall score for the provider is calculated, similar to the behavioral health reviews, to provide a quantitative way to measure overall provider performance
Behavioral Health adopted individual and staff interviews, similar to the IDD reviews, to hear directly from individuals receiving services as well as providers offering services
Behavioral Health incorporated focused outcome areas (FOAs) into the review process, similar to the IDD reviews, to provide the ability to measure congruent outcomes across populations.

Page 11

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

This report is the first annual quality performance report completed for the new and innovative ASO Quality Management System. The report includes BH, IDD and CSU findings as well as a detailed explanation of the review process, analysis of the network performance across providers, and information from individuals about the services received. While there are similarities in the review processes, there are also distinct differences. These differences primarily relate to the needs of the IDD and BH communities and the demands of the service delivery system (e.g., service array, diagnoses, types and levels of care, national reporting standards, and accounting for and reporting of billing and claims); therefore, BH, IDD, and CSU results are reported separately.
The body of the report is divided into sections: Behavioral Health Quality Reviews, including o Assertive Community Treatment Crisis Stabilization Unit Quality Reviews Intellectual and Developmental Disabilities o Person Centered Reviews o Quality Enhancement Provider Reviews o Quality Technical Assistance Consultations Behavioral Health and Developmental Disabilities Comparisons

Page 12

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 1 depicts the overall number of Quality Reviews, records reviewed and interviews completed. The Quality Department conducted more than 700 reviews during the first year that included review of almost 6,000 individual records. The voices of more than 3,600 individuals and staff were heard and for the first time, including almost 1,500 individuals and staff from the behavioral health reviews.

Review
Type
BHQR CSU PCR QEPR TOTAL

Table 1. Overview of Review Data

Number of

Records

Individuals

Reviews

Reviewed

Interviewed

141

3,687

19

285

484

558

100

1,337

744

5,867

737 94 483 667 1,981

Staff
Interviewed
753 96 447 353 1,649

At the conclusion of each section, a summary of findings and evidence-based recommendations are offered to address areas that may need improvement or training/education programs to help improve the quality of services provided to Georgians.

"I got in the program. It was one of the best decisions I ever made - a life changing experience."

Page 13

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

SECTION 3: BEHAVIORAL HEALTH QUALITY REVIEWS
The purpose of the BHQR is to determine adherence to DBHDD standards and to assess the quality of the service delivery system through individual record and claims reviews.1 Review questions are based on DBHDD and Medicaid requirements, and are organized into four categories or review components: billing validation, assessment and treatment planning, compliance with service guidelines, and focused outcome areas (FOAs). The score for each category represents the percent of applicable questions that were met or present. The BHQR overall score is calculated by averaging the four categories. Each category accounts for 25 percent of the overall score.
When a BHQR is completed, the lead assessor convenes the team to analyze data gathered and plan the exit conference. The exit conference is designed to give immediate, preliminary feedback of the BHQR findings to the provider. A report of these preliminary findings is left with the provider in the form of an exit conference report which outlines the provider's identified strengths and any areas of concern from the four primary categories of the review. Assessors also include any items of concern that fall outside the parameters of the review yet are determined to be an area of concern or risk.
Within 30 days of completion of a BHQR, a final assessment report is posted to the Georgia Collaborative website, and the provider is notified via electronic mail of the posting along with their final scores. Like the exit conference report, the final assessment identifies strengths and

1 Please refer to the following link to access a full description of the review process and review tools. http://georgiacollaborative.com/providers/prv-BH.html

Page 14

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

opportunities for growth in the four categories. Additionally, the final assessment includes recommendations for improvement. Much like the exit conference report, the final assessment highlights any areas of concern or risk that fall outside the scope or scoring of the BHQR in the form of additional comments on practices.
Section 3a: Sample Selection
Individual Records and Billing Review
During the 2016 fiscal year, all 141 providers eligible for review, as determined by DBHDD, received a behavioral health quality review (BHQR). A sample of individuals was selected for the record reviews and a sample of those individuals' claims was used for the Billing Review. To be eligible for the sample, each individual must have had at least three claims in the three months (or six months, if necessary) preceding the BHQR.
To select the individuals as part of the record review, an unduplicated list of individuals receiving services from the provider was stratified by payer source: Medicaid, state contracted, and fee-for-service claims submissions. From this list, a stratified random sample of individuals receiving services from the provider was selected, proportionate to the provider's ratio of individuals served by payer source. The sample for each provider consisted of up to 30 individuals for non-intensive and specialty service providers, with an additional 15 individuals for providers who also offered assertive community treatment. The number of individuals selected per provider was based on the number of individuals served. For FY 2016, a total of 3,687 individuals were sampled for record reviews, an average of 26 per provider.

Page 15

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 2. Records Reviewed per Provider

Median

Mean

Standard Deviation

30

26

10.1

For each individual chosen for record review, a random sample of up to 10 paid claims was selected for a billing review. The number of claims reviewed per person was based on services provided and claims submissions. The total number of claims reviewed for FY 2016 was 31,213, an average of 221 claims per provider. When providers did not have adequate claims submissions in the three months preceding their review, individuals and claims samples were pulled from claims submissions up to six months preceding the review.

Table 3. Claims Reviewed per Provider

Median

Mean

Standard Deviation

240

221

100.6

At the direction of DBHDD, the BHQRs focused on specific services. The services reviewed are listed in Figure 10 on page 29 of this report. All providers were reviewed once during the fiscal year with at least one occurrence per billed service included in the claims review, ensuring the complete array of services provided and billed were included in the BHQR. Per DBHDD request, Assertive Community Treatment (ACT) was included in the approved services and is highlighted separately in the report.

Page 16

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Individual and Staff Interviews
Samples used for the Individual and Staff Interviews were selected by the provider and quality assessors conducting the BHQR; services received or provided were not considered in the selection of interviewees. Quality assessors attempted to complete a minimum of five individual and five staff interviews per BHQR; however, the actual number interviewed fluctuated based on individual and staff availability, their agreement to participate in the interview process, the number of staff, and the number of individuals the provider served at the time of the review. If an individual or staff declined to be interviewed, assessors selected an additional individual or staff to be interviewed.
In FY 2016, the Quality Department completed 737 individual interviews and 753 staff interviews during the course of the BHQRs; this is not inclusive of the interviews conducted during the Crisis Stabilization Unit (CSU) reviews which are discussed later in the report. Results from the interviews conducted as part of the BHQR are not included in the BHQR overall scores.
Section 3b: BHQR Overall Review Scores
In this report, data are aggregated and presented by overall provider scores, by category (billing validation, assessment and treatment planning, compliance with service guidelines, focused outcome areas [FOAs], and individual and staff interview). Each focused outcome

Page 17

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

area subcategory (choice, person centered practices, whole health, safety, rights, and community life) also has scores displayed and discussed.2
Graphical distributions for each category show the number of providers who scored within the specified range of scores, as shown on the horizontal/x-axis. With each distribution, the median, mean, and standard deviation (SD) are provided. The median represents the score that falls at the middle of a distribution (50th percentile). The mean represents the average score. The standard deviation refers to the amount of variation or dispersion there is in a distribution of scores, or how much scores tend to spread-out from the mean. A smaller standard deviation (relative to the mean score) indicates the majority of scores tended to be very close to the mean. In this case, scores may cluster around the mean (or average) score, with only a few scores farther away from the mean (outliers). By contrast, a larger standard deviation (relative to the mean score) indicates that scores were more widely spread-out from the average score (mean). 3

2 Unless otherwise stated, category scores were determined by dividing all the "Yes" answers by the sum of the "Yes" and "No" answers in the category. Questions scored as "Not Applicable" (N/A) were not factored into the overall category score. Each subcategory's score was determined similarly based on the questions contained within each subcategory.
3 Standard deviations are sometimes used to determine significant differences between scores within a distribution; however, this application requires a distribution to be normally distributed (similar to a bell curve). Most of the distributions presented in this report do not meet the requirements of a normal distribution, meaning they do not look like a normal bell curve; therefore, standard deviations may be used to determine the variation of scores around a mean but not to determine significant differences between scores, or between the score and the mean.

Page 18

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

BHQR Overall Scores
Figure 1 shows the average overall score for the 141 providers who received a BHQR and the scores for each category. Table 4 shows the mean, median and standard deviation for each. The mean of overall scores was 84 percent. The mean of compliance with service guidelines scores was the highest at 90 percent, followed by focused outcome areas (85%), and billing validation (81%). The lowest scoring category was assessment and treatment planning, averaging 79 percent. The following are highlights of the overall category scores:
The lower assessment and treatment planning scores were primarily driven by the lack of documentation supporting the incorporation of whole health and wellness goals and objectives into individual resiliency plans (IRPs).
The billing validation score indicates that approximately 20 percent of claims reviewed were unjustified and subject to recoupment.
The focused outcome areas represent a new area of review and contain questions for which providers had not previously been assessed; however, these reflected higher scores for most providers.

Page 19

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 1. FY 2016 BHQR Overall Review Scores (N = 141)

Category Overall Billing Validation Assessment & Tx Planning Service Guidelines Focused Outcome Areas

Table 4. Overall Scores Data

Number of Reviews

Median

141

87%

141

88%

141

79%

140

92%

141

87%

Mean 84.1% 81.4% 79.3% 90.3% 85.1%

Standard Deviation
10.4% 23.0% 11.7% 10.7% 12.5%

Page 20

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 2: BHQR Provider Overall Review Scores Distribution (N = 141)

40

32

31

17

1

2

12316

5

Median 87%

Mean 84.1%

Standard Deviation 10.4%

BHQR Overall Scores by Tier
In July 2014, DBHDD implemented the Community Behavioral Health Provider Network Structure in which providers were classified using a three-tiered structure.4 Tiers are defined as Tier 1: Comprehensive Community Providers, Tier 2: Community Medicaid Providers, and Tier 3: Specialty Providers. A more detailed definition of each tier is provided below, along with a distribution graph of the overall scores for each tier.
Table 5 provides a snapshot of the BHQR overall average review scores by tier. Scores were generally lower for Tier 3 providers (78%), with the highest average score for Tier 1 providers (86%), which is higher than the State Average of 84 percent. The distributions of providers across scores, by Tier, are shown in Figures 3-5.

4 Policies regarding the implementation and definition of the DBHDD Community Behavioral Health Provider Network Structure can be found at DBHDD's PolicyStat website: https://gadbhdd.policystat.com

Page 21

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Tier 1 (N = 24)
86%

Table 5. BHQR Overall Scores by Tier

Tier 2

Tier 3

Overall Average

(N = 96)

(N = 21)

(N = 141)

85%

78%

84%

Tier 1 providers function as the safety net for behavioral health services in the state. These providers serve individuals from multiple payer sources (Medicaid, state contracted, and feefor-service), must provide mental health and addiction services for adults, adolescents, and children; must provide at least one intensive in-clinic service, such as psychosocial rehabilitation group or mental health peer support group; and must provide at least one intensive out-of-clinic service, such as assertive community treatment (ACT) or community support team (CST). Tier 1 providers may also provide crisis services such as CSU and specialty services such as intensive family intervention (IFI), but are not required to do so.

Figure 3. Tier 1 Overall Provider Scores Distribution

(N = 24)

9

45

6

Median 88%

Mean 86.3%

Standard Deviation 4.8%

Tier 2 providers make up the largest number of providers in Georgia. There are 96 Tier 2 providers, compared to 45 Tier 1 and Tier 3 providers combined. Tier 2 providers may serve

Page 22

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

adults, children and adolescents, or both, but must have the capacity to provide the entire array of non-intensive services including both mental health and addictive disease services. Tier 2 providers also have the opportunity to provide specialty services, but are not required to do so.

Figure 4. Tier 2 Overall Provider Scores Distribution

(N = 96)

29

22

20

13

1

314

3

Median 87%

Mean 84.9%

Standard Deviation 8.8%

"The treatment has helped me a lot, I love my counselors. They are nice and friendly. They are strict about recovery and that is why I am here."

Page 23

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Tier 3 providers provide an array of specialty services to individuals from multiple payer sources (depending on the service provided). Demographics of individuals who receive services from Tier 3 providers may also fluctuate depending on the specialty service provided.

Figure 5. Tier 3 Overall Provider Scores Distribution (N = 21)
6

4

2

2

2

2

1

1

1

Median 83%

Mean 77.7%

Standard Deviation 17.8%

Page 24

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

BHQR Billing Validation Scores
Figure 6 shows the total dollar amount reviewed through claims analysis across all providers during the BHQRs ($3,417,902.28) and the dollar amount found to be unjustified ($807,050.16). Information in Table 6 indicates the specific billing discrepancy reasons found during the BHQRs. When a claim was found to be unjustified, assessors select all applicable reasons a reviewed claim was identified as a discrepancy; therefore, one claim may have multiple discrepancy reasons identified. The most prevalent billing discrepancy reasons were:
Missing/incomplete service orders (1,612)
Individuals did not meet admission criteria (1,132)
Progress note missing (779)

Figure 6. BHQR Billing Validation Amount Reviewed

$3,500,000.00 $3,000,000.00 $2,500,000.00 $2,000,000.00 $1,500,000.00 $1,000,000.00
$500,000.00 $0.00

$807,050.16 $2,610,852.12

Total $ Justified Total $ Unjustified

Total Amount Reviewed in FY 2016: $3,417,902.28

Page 25

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 6. BHQR Billing Discrepancy Reasons
Eligibility Missing/incomplete order Does not meet admission criteria Quantitative Progress note is missing Code is missing/different than code billed Staff credential missing Signature missing Date of entry missing Units billed exceed time / units documented Consistency requirements missing Credential not supported by documentation Record not submitted within timeframe Location missing (out-of-clinic) Time in / time out missing Date of service incorrect / missing Printed name missing Performance Standards Content does not match service definition Content does not support code billed Content does not support units billed Intervention outside staff's scope/practice Content is not unique to the individual Multiple services billed at the same time Non-billable activity Diversionary activities billed Mutually exclusive services billed

Occurrences 1,612 1,132
Occurrences 779 381 299 275 263 236 202 199 197 166 92 63 42
Occurrences 735 428 375 348 137 95 76 48 2

Page 26

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

The billing validation score for each BHQR is the percentage of justified billed dollars divided by the total paid/billed dollars for the reviewed claims. The distribution of providers across billing validation scores is shown in Figure 7. Four providers scored 100 percent and an additional eight providers scored 99 percent. Thirty-five providers scored below the FY 2016 DBHDD Performance Monitoring Report threshold of 80 percent and three providers scored 0 percent in billing validation.5

322

Figure 7. BHQR Billing Score Distribution

(N = 141)

25 31 28 21

9

111121134

5

Median 88%

Mean 81.4%

Standard Deviation 23.0%

5 DBHDD's Performance Monitoring Report billing validation score threshold increased to 85% effective August 22, 2016. For more information, please refer to CCP Standard 13 - Administrative Services Organization and Audit Compliance, 01-213 and CMP Standard 9 - Administrative Services Organization (ASO) & Audit Compliance, 01239 on DBHDD's PolicyStat website: https://gadbhdd.policystat.com.

Page 27

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

BHQR Assessment and Treatment Planning Scores
The assessment and treatment planning category consisted of nine questions that were answered once per record reviewed. The questions and percent "yes" on each are presented in Figure 8 and the distribution of the 141 providers across scores is shown in Figure 9.
The lowest-scoring question indicated that co-occurring health conditions were often not included in the individuals' plans of care (36% yes); therefore, in 64 percent of the plans, cooccurring health conditions were omitted from planning and without any explanation. Likewise, where other needs were identified (housing, employment, childcare, higher education, etc.), "needs assessed are addressed," were included in 59 percent of the plans. Additionally, 59 percent of individuals had goals, objectives, or interventions in their plans to address wellness.
These trends in scoring reflect a deficit in a comprehensive, whole-person, whole-health approach in treatment planning with a number of the providers in the network. The average score for this category was 79.3 percent. Although scores for individualized language and discharge planning were higher than those already discussed (78% and 80%, respectively), data suggest areas for growth as all individuals require highly-individualized and tailored plans to meet their unique needs.

Page 28

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 8. BHQR Assessment & Treatment Planning (N = 141)

Core Customer Criteria Medical Screening
Individualized Language Goals honor achievement of the individual and/or family
Interventions/objectives are related to goals Needs assessed are addressed
Whole-health wellness goals and interventions Co-occurring health conditions included Discharge plan has step-down service

95% 98% 78% 91% 96% 59% 59% 36% 80%

Figure 9. Assessment & Treatment Planning Scores Distribution

(N = 141)

32

10 12 11

24 20
14 10

1

1

33

Median 79%

Mean 79.3%

Standard Deviation 11.7%

Page 29

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

BHQR Compliance with Service Guidelines Scores
The Compliance with Services Guideline questions were answered once per individual record reviewed. The number of questions answered varied, specific to the service scored. The service reviewed for each individual record was based on which services were reflected in the billing claims sample for that individual; therefore, multiple services could be reviewed within one individual's record. There were a total of 18 services reviewed across all providers in FY 2016, as shown in Figure 10. The n size in Figure 10 represents the number of providers who were reviewed for that service, based on the claims sample.
The Compliance with Service Guidelines is the highest scoring category of the four, with an average of 90 percent. The most frequently provided services scored the highest (e.g., Individual Outpatient Services n = 113, and Family Outpatient Services, n = 92). The Specialty Service of Psychosocial Rehabilitation-Group was also a higher scoring service (95%) but was not reviewed as frequently (n = 25). The lowest scoring service was Addictive Disease (AD) Peer Support Individual at 79 percent; however, only one provider reviewed was providing this service. Other lower scoring services included Addictive Disease Support Services (ADSS) at 81 percent and Community Support - Individual (84%). The lower scores in both of these services were driven by a lack of documentation reflecting coordination of care with natural supports and needed community resources and lack of documentation reflecting staff making required monthly contacts.
One of the 141 providers reviewed in FY 2016 was not reviewed for Compliance with Service Guidelines because the provider only billed a service that was not reviewed programmatically

Page 30

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

(Opioid Maintenance Treatment). Over half (57.9%) of providers reviewed scored above 90 percent in this category (Figure 11).

Figure 10. BHQR Service Guidelines Scores by Service (N = 140)

AD Peer Support - Individual (n = 1) Addictive Diseases Support Services (n = 48)
Assertive Community Treatment (n = 19) Case Management Services (n = 69)
Community Support - Individual (n = 84) Community Support Team (n = 7)
Family Outpatient Services (n = 92) Group Outpatient Services (n = 63) Individual Outpatient Services (n = 113) Intensive Case Management (n = 10) Intensive Family Intervention (n = 30) MH Peer Support - Group (n = 33) MH Peer Support - Individual (n = 7) Nursing Assessment & Health Services (n = 85) Peer Support - Whole Health & Wellness (n = 7)
Psychiatric Treatment (n = 91) Psychosocial Rehab - Group (n = 25) Psychosocial Rehab - Individual (n = 75)

79% 81% 85% 85% 84%
94% 96% 93% 97% 95% 86% 95% 87% 85% 92% 95% 96% 90%

Page 31

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 11: Frequency Distribution of Compliance Scores

(N = 140)

45

36

28

20

1

136

Median 92%

Mean 90.3%

Standard Deviation 10.7%

"I am beyond satisfied and I am working on becoming a Certified Peer Specialist"

Page 32

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

BHQR Focused Outcome Areas (FOA) Scores
Focused outcome areas questions are answered once per record reviewed. Each subcategory has a different number of questions for a total of 24 questions scored in this category.
Figure 12. FY 2016 BHQR Focused Outcome Areas Overall Scores (N = 141)

Rights 90%
Choice 92%

Whole Health
63%

Safety 81%

85%
Community 87%

PersonCentered Practices
92%

Choice questions address how and if individuals receiving care were provided with options of services and were encouraged to make educated choices with respect to supports and services provided. When barriers to services were identified, Assessors looked for documentation as to how the individual was engaged in addressing the identified barriers (93%).

Page 33

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Person Centered Practices questions are used to measure whether documentation shows individuals have a voice and participate in creating their care plans and are active participants in modifying them as needed and desired. One of the higher-scoring questions indicated individuals were active participants in the planning and receiving of services (94%).
Whole Health questions address whether the records reviewed demonstrated that individuals were treated as a whole person with their physical health needs being assessed, documented, and monitored, as evidenced by documented communication with outside healthcare providers. Of the six FOA subcategories, Whole Health was the lowest-scoring (63%), which is consistent with the scoring of questions related to Whole Health and Wellness in the Assessment and Treatment Planning category.
Although item-level detail is not comprehensively provided in this report, The Georgia Collaborative determined that it was imperative to include the four questions from the Whole Health subcategory as it was the lowest FOA score (Table 7).

Table 7. FOA Whole Health Questions: 63% (N = 141)

Ongoing assessment to determine need for external refers for health services, supports and treatment

71%

Communication with external referral sources to determine results of testing and treatment

61%

Medical conditions assessed, monitored and recorded

64%

Documented safeguards used for medications

71%

Safety questions address whether providers were attending to certain risk factors for individuals, such as their tendency to experience a crisis, have suicidal or homicidal thoughts or actions, and whether individuals had received information and education about the risks

Page 34

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

and benefits of prescribed medications. This area represents the second lowest-scoring area of the FOAs. Of the three questions in the subcategory, the one most often scored "No" indicated individuals (or their legal guardians) often had not signed medication consent forms along with the prescriber (52%).
Rights questions address whether individuals had been apprised of their rights while in treatment, at the point of entry into the system and on an annual basis thereafter, and whether they had been informed of their rights under Federal HIPAA laws. The area of greatest concern in this group of five questions was whether individuals were apprised of their rights and responsibilities on an annual basis (58%).
Community Life questions address how individuals were engaged in their communities of choice and whether they held valued social roles. The five questions in this subcategory are used to measure whether individuals had been assessed for their need to make changes in their living, learning, working, and/or social environments (96%), and whether they had been assisted with making these changes (92%).
"The groups I facilitate are like a mini-United Nations. I present the topic and then interpreters sit with small groups of people
who speak languages like Arabic, Farsi, and Swahili. They discuss and then the interpreters relay the information back to
the rest of the large group."

Page 35

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 13: Focused Outcome Areas Score Distribution (N = 141) 41

30 25

17

12

1

3

147

Median 87%

Mean 85.1%

Standard Deviation 12.5%

BHQR Staff and Individual Interview Scores
The interview questions were divided into the six FOA subcategories. Individual Interviews served to assess the individual's quality of life and also the perception of care with the provider and services rendered. Staff Interviews helped determine whether a personcentered approach was used in providing services and empowering individuals. The data and anecdotal information gathered from individual and staff interviews supplemented the record review and enhanced the review process by providing a tailored perspective to illustrate the quality of care the agency provided.
Data in Figure 14 and Table 8 represent the average scores for the individual and staff interviews. It is notable that all areas scored in the 90th percentile which may indicate a high level of satisfaction for individuals who were served by the providers. Likewise, provider staff

Page 36

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

interview scores seem to indicate knowledge regarding providing services that align with standards assessed by the FOAs as well as knowledge of the individuals with whom they work. While not directly assessed via the interview questions, provider staff frequently made statements that indicated a high level of employment satisfaction with the reviewed agency.

Figure 14: BHQR Interview Subcategory Data

Overall
Community Life
Rights
Safety
Whole Health Person Centered
Practices Choice Staff Interviews (N = 753)

98.1% 95.6%
99.2% 95.0%
99.5% 97.6%
97.6% 96.2%
97.4% 92.7%
97.9% 95.8%
98.5% 97.4%
Individual Interviews (N = 737)

Page 37

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 8. BHQR Interview Subcategory Data

Individual Interviews

Staff Interviews

Focused Outcome Area Median Mean

SD Median Mean

SD

Choice

100% 97.4% 10.2% 100% 98.5% 9.2%

Person-Centered Practices 100% 95.8% 11.8% 100% 97.9% 9.2%

Whole Health

95% 92.7% 11.5% 100% 97.4% 9.9%

Safety

100% 96.2% 10.3% 100% 97.6% 9.4%

Rights

100% 97.6% 9.4% 100% 99.5% 9.0%

Community Life

100% 95.0% 13.5% 100% 99.2% 9.0%

Overall

97% 95.6% 9.6%

99% 98.1% 8.9%

Section 3c: Assertive Community Treatment
At the direction of DBHDD, The Georgia Collaborative conducted reviews of 19 Assertive Community Treatment (ACT) providers, embedded within the routine behavioral health quality reviews (BHQR). For the 17 providers who offered both non-intensive outpatient and ACT, up to 15 additional individuals receiving ACT services were selected for review, and for the two providers who offered only ACT services, 30 individuals were selected for each review. This totaled 314 ACT records reviewed; and with up to 10 billing claims per record reviewed, a total of 3,029 claims were analyzed across all ACT reviews. The data below represents findings from reviews of ACT services only.

"ACT showed me that there was a reason to live. They never gave up."

Page 38

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 9. ACT Records Reviewed per Provider

Median

Mean

Standard Deviation

15

16.5

4.8

Table 10. ACT Claims Reviewed per Provider

Median

Mean

Standard Deviation

148

159

46.7

Figure 15. ACT Statewide Averages (N = 19)

Page 39

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

It should be noted that all four category scores for ACT reviews are higher than the corresponding BHQR scores for non-intensive services.

Table 11. FY 2016 BHQR vs. ACT FY 2016 Average

Assessment/ Service

Billing

FOA

Planning Guidelines

BHQR Averages

81%

85%

79%

90%

ACT Averages

92%

88%

85%

85%

Overall 84% 88%

Table 12. ACT Overall Scores Data (N = 19)

Category

Median Mean SD

Overall

90% 87.6% 4.6%

Billing Validation

92% 92.3% 6.7%

Assessment & Tax Planning

84% 85.2% 6.0%

Service Guidelines

89% 84.6% 9.1%

Focused Outcome Areas

90% 87.6% 7.6%

Figure 16: ACT Overall Score Distribution

(N = 19)

8

4

5

2

Median 90%

Mean 87.6%

Standard Deviation 4.6%

Page 40

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

ACT BHQR Billing Validation Scores
Figure 17 shows the total dollar amount reviewed through claims analysis during the ACT BHQRs ($306,628.32) and the dollar amount found to be unjustified ($23,422.57). Information in Table 13 indicates the specific billing discrepancy reasons found during the BHQRs. Assessors select each reason a reviewed claim was identified as a discrepancy; therefore, one claim may have multiple discrepancy reasons identified. The most prevalent billing discrepancy reasons were as follows:
Staff credential missing (69 progress notes) Location was missing for out-of-clinic claims (39 progress notes) Content did not support the units billed (27 progress notes)
Figure 17. ACT Billing Validation (N = 254)

Page 41

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 13. ACT BHQR Billing Discrepancy Reasons

Eligibility

Occurrences

Missing/incomplete order

3

Quantitative

Occurrences

Staff credential missing

69

Location missing (out-of-clinic)

39

Code is missing/different than code billed

25

Units billed exceed time / units documented

17

Progress note is missing

16

Date of entry missing

1

Signature missing

1

Performance Standards

Occurrences

Content does not support units billed

27

Content does not support code billed

17

Intervention outside staff's scope of practice

12

Non-billable activity

9

Multiple services billed at the same time

7

Content does not match service definition

5

Content is not unique to the individual

4

The billing validation score is the percentage of justified billed dollars divided by the total paid/billed dollars for the reviewed claims. The statewide average ACT billing score of 92 percent was higher than the BHQR statewide average of 81 percent.

Page 42

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 18. Frequency Distribution of ACT Billing Scores

(N = 19)

7

5

4

1

2

Median 92%

Mean 92.3%

Standard Deviation 6.7%

ACT BHQR Assessment and Treatment Planning Scores
The ACT assessment and treatment planning scores were similar to the BHQR statewide averages and reflected a deficit in a comprehensive, whole person, whole health approach in treatment planning. Questions assessing whether co-occurring health conditions, whole health and wellness, and all assessed needs were addressed on the IRP were also the lower scoring questions statewide. However, IRPs within the ACT program scored higher than for the BHQRs regarding individualized language and discharge planning (92% and 90%, compared to 78% and 80% respectively).

An ACT staff member stated that she really enjoys being able to "encompass the whole health of an individual" and link individuals to the necessary
resources in the community.

Page 43

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 19. ACT Assessment and Treatment Planning Scores (N = 19)

Core Customer Criteria Medical Screening
Individualized Language Goals honor achievement of the individual and/or family
Interventions/objectives are related to goals Needs assessed are addressed
Whole-health wellness goals and interventions Co-occurring health conditions included Discharge plan has step-down service

100% 100% 92% 94% 99% 59% 75% 47% 90%

Figure 20. ACT Assessment & Tx Planning Distribution (N = 19)

4

6

5

2

1

1

Median 84%

Mean 85.2%

Standard Deviation 6.0%

Page 44

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

ACT BHQR Compliance with Service Guideline Scores
The ACT compliance with service guidelines overall average is 90 percent. Documentation supported that 100 percent of the 314 individual records reviewed met admission or continuing stay criteria. Additionally, 100 percent of individuals received at least one symptom assessment and medication management contact per month. The lower-scoring questions regarding working with natural supports monthly and treatment plan reviews (43% and 58%, respectively) indicate a deficit in providing person-centered, holistic care that included supports from the individual's environment. Also of note is that the question assessing the completeness of the ACT team scored 76 percent, indicating that many ACT teams were lacking one or more required team members.
The ACT compliance with service guideline score is calculated using 13 questions or indicators. Table 14 shows the item-level detail to illustrate the degree of variation across indicators within the ACT compliance with service guidelines mean score of 84.6 percent.

Page 45

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 14. ACT Compliance with Service Guidelines Scores (N = 19)

Documentation supports that the individual meets admission or continuing stay criteria.

100%

The ACT team completes a treatment plan review with the staff, the individual, and his/her family/informal supports prior to the reauthorization of services.

58%

There is documentation to support when substance use services are needed and are integrated into the treatment plan.

94%

There is evidence that the ACT team is working with informal support

systems/collateral contacts at least 2-4 times per month with or without the individual present (and it is documented) to provide support and skills training to

43%

assist the individual in his/her recovery. (Review authorization period.)

The ACT team is working with the individual towards educational or vocational needs, interests, per IRP (once per authorization).
Following admission to a psychiatric facility, the ACT team is involved in each individual's discharge planning.
There is documentation of individual's involvement in transition planning.
One of the contacts per month addresses the symptom assessment and management of medications (once a month).

95%
87% 82% 100%

The ACT team has all required staff.

76%

For discharged individuals, there are multiple documented attempts to locate and make contact with the individual prior to discharge (Over a 45 day period).

100%

Progress notes contain documentation of the individual's progress (or lack of) toward specific goals/objectives on the treatment plan.

91%

The staff interventions reflected in the progress notes are related to the staff interventions listed on the treatment plan.

98%

The progress notes document individual response to the staff intervention provided. 99%

Page 46

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 21. ACT Compliance with Service Guidelines

Scores Distribution

(N = 19)

7

5

2

3

1

1

Median 89%

Mean 84.6%

Standard Deviation 9.1%

Page 47

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

ACT BHQR Focused Outcome Areas Score
Focused Outcome Area (FOA) questions are answered once per record reviewed. Each FOA has a different number of questions for a total of 24 questions scored in this category. The overall score for ACT FOA is 90 percent. Similar to the BHQR statewide scores, the lowest category scores for ACT are Whole Health (76%) and Safety (78%). Person Centered Practices received the highest score (99%).
Figure 22. FY 2016 ACT BHQR Focused Outcome Area Overall Scores (N = 19)

Page 48

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 23. ACT FOAs Score Distribution

(N = 19)
6

2

1

4

3

3

Median 90%

Mean 87.6%

Standard Deviation 7.6%

Page 49

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Section 3d: Crisis Stabilization Unit Quality Reviews
At DBHDD's request, a special review of all Crisis Stabilization Unit (CSU) providers was initiated during FY 2016, as routine quality reviews of this specialty service had not been conducted previously. The purpose of the CSU review was to assess the provider's overall practices and quality of service delivery, and to determine adherence to DBHDD standards through individual record reviews.6 While these reviews were conducted in conjunction with a BHQR when the CSU provider also provided essential routine services, the CSU reviews resulted in separate scores and final assessment reports.
Review questions are based on DBHDD and Medicaid requirements, and were organized into three categories or review components: Individual Record Review (IRR), Compliance with Service Guidelines and FOAs. The score for each category represents the percent of applicable questions that were met or present. The CSU overall score is calculated by averaging the three categories. Each category accounts for 33.3 percent of the overall score. Billing validation was not a component of the CSU reviews for FY 2016.
During FY 2016, all 19 eligible CSU providers were reviewed. Two were freestanding CSUs and 17 occurred in conjunction with behavioral health quality reviews (BHQRs). Of the CSUs reviewed, three were conducted jointly with a BHQR and a QEPR at one provider location to reduce administrative burden. Three of the CSUs provided services to adolescents and 18 of the CSUs provided services to adults.

6 Please refer to the following link to access a full description of the review process and review tools. http://georgiacollaborative.com/providers/prv-BH.html

Page 50

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Sample Selection
A random sample of 15 individuals who had received services within the three months preceding the review was selected for record reviews. When providers did not have an adequate number of individuals served in the three months preceding the review, samples were pulled from individuals served up to six months preceding the review; therefore, all CSU providers had 15 individuals selected, for a total of 285 individual records reviewed.
The sample for the interviews was selected and scored similar to the BHQR process: individuals and staff were selected by the provider and Quality Assessors conducting the CSU Review. Results from the interviews were not included in calculating the provider's overall CSU score. Quality Assessors completed a minimum of five individual and five staff interviews per CSU review; however, the actual number fluctuated based on individual and staff availability, their agreement to participate in the interview process, the number of staff, and the number of individuals the provider served at the time of the review. The individuals selected for interviews were currently being served at the CSU, and the staff selected was currently providing services on the CSU. If an individual or staff declined to be interviewed, Assessors selected an additional individual or staff to be interviewed.
CSU Quality Review Overall Scores
Figure 24 shows the average overall score for the 19 providers who received a CSU quality review, and the scores for each category. Table 15 shows the mean, median and standard deviation for each. The overall score mean was 83 percent. Focused outcome areas was the highest-scoring category at 88 percent, followed by compliance with service guidelines (82%).

Page 51

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

The lowest scoring category was the Individual Record Review (IRR), averaging 78 percent. The following are highlights of the overall category scores:
Based on the higher scored questions within the IRR and compliance with service guidelines categories, individuals and staff have needed access to physicians or physician extenders, and individuals have regular contact with nursing staff
Much like the BHQRs, the lower scores in the treatment and discharge planning subcategories within the IRR reflect a lack of documentation supporting personcentered, whole-health and wellness approaches to developing IRPs
Also in line with the BHQRs, the FOAs reflected higher scores for most providers
Figure 24. FY 2016 CSU BHQR Overall Review Scores (N = 19)

Page 52

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 15. CSU Overall Scores (N = 17)

Category

Median

Mean

Overall

83%

82.9%

Individual Record Review

77%

78.3%

Service Guidelines

80%

82.1%

Focused Outcome Areas

88%

87.7%

SD 8.1% 8.5% 13.2% 6.0%

Figure 25. CSU Overall Score Distribution

(N = 19)

5

44

3

2

1

Median 83%

Mean 82.9%

Standard Deviation 8.1%

CSU Individual Record Review Scores
Individual Record Review (IRR) questions were answered once per record reviewed. Each of the six subcategories had a variable amount of questions with a total of 31 scored within the IRR category.
The CSU intake assessment data indicate 99.6 percent of individuals met admission criteria and 99 percent were assessed by a physician or a physician extender within 24 hours of admission. Additional supporting evidence of this was that documentation in 99 percent of

Page 53

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

individuals' records reflected daily status updates completed by a Registered Nurse (RN) and 93 percent of records contained a Medication Administration Record (MAR).
Alternatively, data suggest areas for growth in CSU treatment planning that mirror deficits identified in the BHQR Assessment and Treatment Planning category--such as not incorporating all identified needs (primarily, co-occurring physical health issues) in the IRPs or Nursing Care Plans (NCPs). Inattention to co-occurring physical health issues was also found in progress notes, as they often lacked evidence that physical illnesses were monitored during an individual's course of stay. Additionally, documentation lacked evidence that individuals were offered an opportunity to participate in the treatment team meetings to develop their own plan of care. Discharge planning is also an area of growth for the CSUs, as discharge and aftercare plans were incomplete in 43.7 percent of records reviewed.
Initial bio-psychosocial assessments were present and comprehensive nursing assessments were generally completed upon admission (84% and 78%, respectively). Documentation lacked evidence that individuals met American Society for Addiction Medicine (ASAM) criteria for medically monitored residential withdrawal management when admitted for detoxification (18.8% scored "Yes"). Records lacked evidence of the required assessment of individuals against ASAM criteria for service intensity and placement.

Page 54

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 26. Individual Record Review Category Scores (N = 19)

Assessment/Treatment Planning Admission/Initial Evaluation/Screening for Risk Crisis Stabilization Specific Treatment Planning
Crisis Stabilization Course of Stay Documentation
Transition/Discharge Planning

86% 69% 70%
83% 87%
63%

"When people leave or graduate from this program, they know what recovery is and how to apply it to their life."

Page 55

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 27. CSU Individual Record Review Score Distribution (N = 19) 7
5

22

1

1

1

Median 77%

Mean 78.3%

Standard Deviation 8.5%

CSU Compliance with Service Guidelines Scores
The 15 CSU compliance with service guidelines questions (13 scored and 2 non-scored) were answered once per review to assess the CSU program as a whole.
The CSUs had all required staff including access to a physician. Additionally, the three CSUs serving adolescents met all applicable staff-to-individual ratio requirements.
The lower scoring questions included the following:
The provider is adhering to its current policy for the safe storage of medication (53%) Policies and procedures are present for adherence to required components of crisis
service plans for provision of crisis services to individuals who are deaf, deaf-blind, blind, and hard-of-hearing (53%)

Page 56

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Documentation reflected that CSUs did not have access to specialists such as Addictionologists (67%) or a Pediatric Psychiatrists (67%) when needed. CSU policies often failed to identify a model for substance use treatment (69%), and scoring for providers adhering to their own policies on the notification of medication errors also indicated an area for growth (79%).

Figure 28. CSU Compliance with Service Guidelines (N = 19)

CSU Staffing Requirements Met (n = 19) C&A Minimum Staff Present (n = 3) C&A Staff Ratio Met (n = 3) C&A Nursing Staff Ratio (n = 3)
Adherence to Medication Notification Policy (n = 19) Protocols for Handling Drugs (n = 19) Safe Storage of Medication (n = 19)
Infection Control Plan Adherence (n = 19) Seclusion & Restraint Policy Adherence (n = 19)
Therapeutic Blood Level Monitoring (n = 19) Deaf, Deaf-Blind, Hard of Hearing Policies (n = 19) Model/Curriculum for SU treatment (Non-scored) (n = 16)
Physician Availability (n = 19) Access to Addictionologist (n = 18) C&A Psychiatrist (Non-scored) (n = 3)

100% 100% 100% 100% 79% 95% 53% 84% 95% 89% 53% 69% 100% 67% 67%

Page 57

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 29. CSU Service Guideline Score Distribution (N = 19)

6

5

11

2

1

3

Median 80%

Mean 82.1%

Standard Deviation 13.2%

CSU Focused Outcome Areas Scores
Focused Outcome Area questions were answered once per record reviewed. Each subcategory had a different number of questions for a total of 23 questions assessed in the FOA category. Please refer to the BHQR FOA section for a definition of the six FOA subcategories.
As illustrated below in Figure 30, agencies scored in the 90th percentile in the areas of Choice (99%), Rights (97%), and Community Life (92%). The high score in Choice reflected that documentation supports that the individual's preferences for treatment while at the CSU were honored and, when barriers to treatment were identified, measures were taken to address the barriers. Almost all of the records reviewed contained documentation that individuals' rights and responsibilities for treatment, as well as confidentiality, were reviewed with them upon admission.

Page 58

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

In line with what is noted in the IRR category and within the BHQRs, documentation generally lacked evidence that whole health and wellness was incorporated into treatment. Also, documentation did not support that individuals were active participants in treatment planning. Data suggest this is a significant area for growth as all individuals require highly individualized and tailored plans to meet their needs regardless of length of stay.
Figure 30. FY 2016 CSU Focused Outcome Area Overall Scores (N = 19)

Page 59

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 31. CSU Focused Outcome Areas Overall Distribution (N = 19)

65

33

2

00000

Median 88%

Mean 87.7%

Standard Deviation 6.0%

CSU Staff and Individual Interview Scores
The CSU interview questions were similar to the BHQR interview questions and were divided into the six FOA subcategories. Individual interviews served to assess the individual's quality of life and also perception of care with the provider and services rendered. Staff Interviews helped determine if a person-centered approach was used in providing services and empowering individuals. The data and anecdotal information gathered from individual and staff interviews supplemented the record review and enhanced the review process by providing a tailored perspective to illustrate the quality of care the agency provided.
Data in Figure 32 and Table 16 represent the average scores for the Individual and Staff Interviews. It is notable that all but one FOA scored in the 90th percentile, which seems to indicate a high level of satisfaction for individuals who were served by the providers. The lowest-scoring subcategory for Individual Interviews was Whole-Health (85%) which may

Page 60

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

indicate an increased need for CSUs to attend to and plan for individuals' co-occurring health conditions. This issue was also highlighted in the IRR and FOA categories. Similar to the BHQR staff interviews, CSU staff interview scores seem to indicate knowledge regarding providing services that align with standards assessed by the FOA questions, as well as knowledge of the individuals with whom they work.
Figure 32. CSU Interview Subcategory Data

Choice Person Centered Practices
Whole Health Safety Rights
Community Life Overall
Individual Interviews (N = 94)

96.8% 99.6%
96.0% 98.2%
84.6% 96.8%
95.3% 98.7%
99.6% 97.3%
99.4% 98.7%
94.9% 98.6%
Staff Interviews (N = 96)

Page 61

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 16. CSU Interview Subcategory Data

Individual Interviews (N = 94)

Staff Interviews (N = 96)

FOA

Median Mean

SD Median Mean SD

Choice

100% 96.8%

4.6%

100% 99.6% 1.3%

Person Centered Practices

97%

96.0%

4.0%

100% 98.2% 5.7%

Whole Health

86%

84.6%

6.8%

99% 96.8% 4.6%

Safety

96%

95.3%

4.1%

100% 98.7% 2.1%

Rights

100% 99.6%

1.3%

100% 97.3% 3.7%

Community Life

100% 99.4%

1.9%

100% 98.7% 4.7%

Overall

95%

94.9%

2.0%

99% 98.6% 2.7%

Section 3e: Summary of Findings and Recommendations for Behavioral Health Providers
Summary of Findings
Although many impactful and life-changing things have been captured and recorded by assessors throughout the last fiscal year, we believe that quality improvement is an ongoing activity. The Georgia Collaborative continually reassess our processes to ensure we are capturing results and analyzing outcomes that give us the best information and truest picture of service quality in Georgia.
The table below illustrates the overall scores for the three review types, BHQR, ACT, and CSU. The overall statewide average for all three review types is above 80 percent: BHQR 84 percent, ACT 88 percent and CSU 83 percent. The Assessment and Treatment Planning category (IRR for CSU) was the lowest score of all three review types. The CSU review did not include Billing Validation.

Page 62

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

BHQR ACT CSU

Table 17. BHQR, ACT, CSU Overall Averages

Billing

Assessment/ Service

Validation FOA

Planning Guidelines

81%

85%

79%

90%

92%

88%

85%

85%

NA

88% IRR 78%

82%

Overall 84% 88% 83%

The FOAs represent a new area of review and contains questions on which providers had not previously been assessed. The table below summarizes the three review types highlighted in this report by FOA. Whole Health is the lowest score of the subcategories for both BHQR and ACT reviews and is a trend identified as an area for improvement in CSU reviews, as well. Person Centered Practices is a higher-scoring subcategory for the ACT and BHQRs; however, it is the second lowest subcategory found in the CSU reviews. Initial CSU data reflect individuals do not routinely participate in the treatment planning process throughout the course of stay. Additionally, treatment plans are not tailored to the individual's presenting needs.

Table 18. BHQR FOA Scores

Focused Outcome Areas

BHQR ACT

CSU

Safety

81%

78%

76%

Person-Centered Practices

92%

99%

77%

Community

87%

97%

92%

Choice

92%

97%

99%

Rights

90%

89%

97%

Whole Health

63%

76%

83%

Overall Score

85%

90%

88%

Page 63

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Recommendations
The focus of Billing Validation is compliance with standards for ensuring that claims are justified in accordance with the DBHDD Provider Manual. As this is the second-lowest score of the four categories, we determined that technical assistance to providers was needed. In FY 2016, the Quality Department launched a Quality Training Series with the first module being Billing Validation. We will continue to analyze billing compliance trends to determine whether additional training or technical assistance to providers is indicated.
The three remaining categories of questions representing Assessment and Treatment Planning, Compliance with Service Guidelines, and Focused Outcome Areas will be analyzed throughout FY 2017 for their validity and effectiveness in yielding the data needed to ensure that quality is more accurately captured. As an example, the category of Compliance with Service Guidelines is primarily focused on compliance with measurable requirements of service definitions in the DBHDD Provider Manual. Our initial data analysis indicates a need to expand our focus to encompass a more qualitative look at services provided.
Based on FY 2016 data, the Quality Department and DBHDD partnered to develop the BHQR frequency schedule for providers. The FY 2017 frequency of reviews is based upon providers' FY 2016 overall BHQR score and billing score and is based on providers achieving the benchmarks scores of 80 percent overall and 70 percent for billing validation.
Recommendations:
Based on data analysis, consideration will be given to modifying questions within the remaining categories that may provide additional information about quality.

Page 64

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Additional training opportunities may become available through quality technical assistance consultations (QTAC).
We will continue the Quality Training Series in FY 2017 based on data analysis from FY 2016, for all behavioral health and CSU providers.
As FY 2016 is the baseline year for routine CSU reviews, we will continue to gather and analyze data and processes to improve the reviews to assess the quality within the CSU programs.

Page 65

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

SECTION 4: INTELLECTUAL AND DEVELOPMENTAL DISABILITIES Section 4a. Background
Person Centered reviews (PCR) and Quality Enhancement Provider Reviews (QEPR) are used to assess the extent to which individuals with intellectual and developmental disabilities are satisfied with their services and achieve outcomes important to them, and to monitor provider systems.7 Follow-up review activities provide technical assistance to help providers improve service delivery systems (Quality Technical Assistance Consultation (QTAC)).
The purpose of the QEPR is to monitor providers to ensure they meet requirements set forth by the Medicaid waiver and DBHDD, and to evaluate the effectiveness of their service delivery system. The purpose of the PCR is to assess the person's quality of life as well as the effectiveness of and the satisfaction individuals have with the service delivery system. The Georgia Collaborative quality assessors use interviews, observations and record reviews to compile a well-rounded picture of the individual's circle of supports, how involved the person is in the decisions and plans laid out for that person, and the quality of services provided.
Individuals from both the PCR and QEPR samples participate in the individual interview and Individual Support Plan Quality Assurance Checklist (ISP QA). Both processes also include a Provider Record Review (PRR) for each individual selected in the sample, a Staff Interview (SI) with a sample of direct support providers, and onsite observation of day or residential programs. During the PCR, a Support Coordinator Record Review (SCRR) and Interview (SCI) are completed for the support coordinator working with the individual. During the QEPR, each provider organization receives one administrative review to monitor compliance with

7 Please refer to the following link to access a full description of the review processes and review tools: http://www.georgiacollaborative.com/providers/prv-IDD.html

Page 66

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

required Qualifications and Training (Q&T). The Q&T includes a review of a sample of personnel records to determine if staff has the necessary qualifications, specific to services rendered, and if the training was received within required timeframes.
In this section of the report, results for both the QEPR and PCR are presented by each of these review tools. Results are also presented by Focused Outcome Areas (FOAs). Indicators within each tool are grouped into six FOAs, areas of the person's life that are important to achieve and maintain:
Person Centered Practices--supports and services are being provided based on the person's preferences and direction
Choice--individuals have information they need to make informed choices on life decisions, such as where to live, where to work, and which providers to use
Community Life--individuals are actively participating in their communities as desired Rights--rights are upheld for individuals and they are provided information and
education to ensure they understand and know how to exercise their rights Whole Health--individuals are healthy, aware of their health-related needs, and direct
their own health care regimen Safety--individuals are safe in their home and work environments and in their
communities; they understand or are learning how to self-preserve in all environments

Page 67

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Section 4b: Sample Selection
Person Centered Review (PCR)
During the 2016 fiscal year, 484 individuals were selected for a PCR. A stratified random sample of individuals was selected by region across Georgia so the number of PCRs per region was proportionate to the number of individuals receiving services within that region. The PCR sample was selected from individuals who had not received a PCR during the previous two years, were actively receiving eligible waiver or state funded services, and were 18 years of age or older. Table 19 shows the number of PCRs completed by Region.

Table 19. PCR Sample by Region

Region

N

1

93

2

87

3

122

4

55

5

58

6

68

%
19.2% 18.0% 25.2% 11.4% 12.0% 14.0%

Total

484

100%

Quality Enhancement Provider Review (QEPR)
A stratified random sample of 98 providers was selected from the list of providers rendering services to the 484 individuals selected for a PCR. Based on the number of eligible individuals in their case load, providers were stratified into three categories by size: "Large," "Medium," and "Small." In addition, DBHDD selected one crisis service provider and one support

Page 68

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

coordinator agency to be included in the sample. Table 20 displays the number of providers selected for a QEPR by size.

Table 20. QEPR Provider Sample by Size

Provider Size

N

Small (caseload 30)

47

Medium (30 < caseload < 100)

36

Large (caseload 100)

17

Total

100

Section 4c: Review Processes
The focus of the PCR is on the individual's quality of life and quality of the services received. The focus of the QEPR is on the provider's overall practices, quality of services offered to all individuals served and level of compliance with state requirements. Both the PCR and the QEPR use the individual observation staff assessment (IOSA). This tool includes an interview with individuals, interviews with their staff (one per service), and onsite observation(s) at residential and day programs, as applicable. In addition to the IOSA, the PCR and QEPR include an evaluation of the individual's service plan (ISP QA), a review of the provider's records (PRR), as well as the compliance with service delivery requirements using the Developmental Disability Service Specific (DDSS) review. The number of PRR and DDSS reviews depends on the number of services received by the person.
While the PCR and QEPR share most of the same tools, there are a few exceptions. The PCR also includes an interview with the individual's support coordinator and a review of the record in the Consumer Information System (CIS) maintained by the support coordinator for the

Page 69

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

person. The QEPR has an additional review tool, qualifications and training, which is used to review a sample of records from all staff ensuring they have received proper training and are current on all state requirements (e.g., background screenings, level of education). The total number of records or interviews completed for the PCR and the QEPR is listed in Table 21, for each review tool.

Table 21. Number of Records by Review Tool and Review Type

Review Tool

PCR

QEPR

(N = 484) (N = 100) Total

IOSA Individual interview

484

667

1,150

IOSA Observation

578

409

987

IOSA - Staff interview

833

353

1,186

ISPQA Checklist

484

656

1,140

Support Coordinator Interview (SCI)

484

8

492

Support Coordinator Record Review (SCRR)

484

20

504

Provider Record Review (PRR)

558

1,377

1,935

Staff Qualifications and Training (Q&T)

N/A

1,041

1,041

DDSS - Behavioral Supports Consultation

2

9

11

DDSS - Community Access (Group)

243

489

732

DDSS - Community Access (Individual)

65

209

274

DDSS - Community Living Support

51

183

234

DDSS - Community Residential Alternative

91

267

358

DDSS Crisis

0

8

8

DDSS Prevocational

74

117

191

DDSS Respite

1

7

8

DDSS - Support Coordination

467

20

487

DDSS - Supported Employment

58

80

138

DDSS Transportation

2

8

10

Page 70

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

When a PCR is completed, a report is provided that identifies the strengths of the person's team, providing supports and services as well as recommendations for the person and the support team, including the support coordinator, provider(s), and family. A provider who participates in a QEPR receives a comprehensive report which identifies strengths of the service delivery systems and recommendations for improvement, with a number of performance scores. These scores include the overall score, qualifications and training score, and DDSS score.
The QTAC is an additional review process conducted 90 days after completion of the QEPR. Using findings from the QEPR, technical assistance is provided to support providers and to offer suggestions and guidance to help improve their service delivery systems. This process utilizes a consultative approach with providers to address specific issues and concerns related to an individual receiving services or systems and practices that need improvement.
Providers may also receive a QTAC through a referral based upon other monitoring systems, such as support coordination monitoring, critical incidents or complaints. The QTAC supplements the PCR and QEPR processes by affording DBHDD and contracted providers the opportunity to solicit technical assistance for specific needs within the service delivery milieu.

Section 4d. Overall Review Scores
Data are aggregated into overall provider scores, scores by tool, and scores by focused outcome areas (FOAs). The overall score is the combined score for the IOSA (Individual Interview, Staff Interview and Observation) and the PRR. Each indicator in the overall score is

Page 71

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

categorized into one of six FOAs: Person Centered practices (PCP), Community Life (CL), Choice, Safety, Rights, and Whole Health. The FOAs are constructed to measure the quality of services, as well as individual outcomes related to these six central aspects of a healthy and well-rounded life.
The graphic distributions presented in this report show the number of providers who scored within the specified range of scores as shown on the horizontal/x-axis. With each distribution, we provide the median, mean, and standard deviation (SD). The median represents the score that falls at the middle of a distribution (50th percentile). The mean represents the average score. The standard deviation refers to the amount of variation or dispersion there is in a distribution of scores, or how much scores tend to spread out from the mean. A small standard deviation (relative to the mean score) indicates the majority of scores tend be very close to the mean. In this case, scores may cluster around the mean (or average) score, with only a few scores farther away from the mean (outliers). By contrast, a large standard deviation (relative to the mean score) indicates that scores are more widely spread-out from the average score (mean).8

8 Standard deviations are sometimes used to determine significant differences between scores within a distribution; however, this application requires a distribution to be normally distributed (similar to a bell curve). Most of the distributions presented in this report do not meet the requirements of a normal distribution, a normal bell curve. Therefore, standard deviations may be used to determine the variation of scores around a mean but should not be used to determine significant differences between scores, or between the score and the mean.

Page 72

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Person Centered Review (PCR)
Figure 33 shows the overall score for the 484 PCRs as well as the score for each of the tools completed during the PCR process. The overall score for the PCR was 93.8 percent. Like the QEPR overall score, the PCR score uses results from four tools: the Individual Interview, Staff Interview, Observation, and PRR. The Observation score was highest at just over 98 percent, followed by the Staff (96.4%) and Individual (95.1%) Interviews. The PRR was the lowestscoring area, averaging 79 percent. The Support Coordinator Interview showed a higher average score than the Support Coordinator Record Review, 90.8 percent and 79.9 percent respectively. The distribution in Figure 34 shows that approximately 79 percent of PCR scores (N=382) were 90 percent or greater.

Page 73

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 33. PCR Scores, Overall and by Tool (N = 484)
Individual Interview
95.1%

Observation
98.4%

Overall Score 93.8%

Staff Interview
96.4%

Support Coordinator
Interview
90.8%

PRR 79.0%

Support Coordinator
Record Review
79.9%

Figure 34. PCR Overall Scores

(N = 484)

354

88

10

32

Page 74

Median 95.2%

Mean 93.8%

Standard Deviation 5.2%

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Quality Enhancement Provider Review (QEPR)
Figure 35 shows the average overall score for the 100 providers who participated in a QEPR, and scores for each tool used in the review process. The overall score was 88.4 percent. The Observation score was highest at just over 98 percent, followed by the Staff (96.9%) and Individual (95.1%) Interviews. The lowest-scoring area was the PRR (73.6%). In addition to the overall score, providers also received a score for Staff Qualifications and Training (82.4%) and the DDSS review (99.9%).
Figure 35. QEPR Scores, Overall and by Tool (N = 100)
Individual Interview
95.1 %

Observation
98.2%
Qualifications &
Training 82.4%

Overall Score 88.4%
PRR 73.6%

Staff Interview
96.9%
DD Service Specific 99.9%

Page 75

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 36 shows the distribution of the 100 QEPRs/providers across scores. The mean overall score was 88.4 percent with a standard deviation of 6.1 percent, with 49 providers showing a score of 90 percent or higher.

Figure 36. QEPR Overall Scores

(N = 100)

45 49

Median 89.7%

15

Mean 88.4%

Standard Deviation 6.1%

Page 76

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Section 4e. Person Centered Review
PCR Scores by Focus Outcome Area (FOA)
The overall score for each review is divided into six FOAs. Each FOA score is calculated with the combined results from the Individual Interview, Staff Interview, Observation (IOSA), and PRR.9 Results by FOA are shown in Figure 37.

Figure 37. PCR FOA and Overall Score (N = 484)

Rights 96.6%
Choice 93.3%

Whole Health 93.7%
Overall Score 93.8%
Community Life
83.0%

Safety 97.1%
PersonCentered Practices
90.6%

9 The Support Coordinator Record Review and Interview are also categorized into the six FOAs. However, we exclude these in the PCR FOA Score calculations to more closely resemble the QEPR calculations. Those results are shown separately.

Page 77

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 22 displays the median and mean scores for each FOA, as well as the standard deviation for the PCRs. Mean scores for each FOA ranged from a low of 86.5 percent for Community Life to a high of 98.1 percent for Safety. The community life data suggest that individuals' needs and expectations related to community integration and the development of valued social roles for the individuals may need to be addressed.

Table 22. PCR Scores by FOA

Tool Name
Person Centered Practices Community Life Choice Rights Whole Health Safety

Median
93.4% 86.5% 96.3% 98.1% 95.4% 98.1%

Mean
90.6% 83.0% 93.3% 96.6% 93.7% 97.1%

Standard Deviation
9.0% 16.1% 7.5% 3.8% 7.2% 3.4%

PCR Scores by Tool
Every PCR uses multiple interviews, observations, and record reviews. Table 23 displays, for each tool of the PCR, the number of interviews or record reviews completed, as well as the median and mean scores and standard deviations. Provider documentation (PRR) and the interview with the support coordinator showed the lowest mean scores, 79.0 percent and 79.9 percent respectively. Distributions of the number of PCRs by score for each of these tools are displayed in Figures 38-43.10

10 Of the 484 PCRs completed there were 484 individual interviews, and 412 individuals eligible for one or more observation for a total of 578 observations, 833 staff interviewed, etc. Only individuals receiving residential or day services receive an observation. The N sizes displayed in Table 23 are the number of "tools" completed for

Page 78

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 23. PCR Scores by Tool

Tool Name

N Median Mean

Individual Interview

484

97.1%

95.1%

Observation

578

99.5%

98.4%

Staff Interview

833

97.7%

96.4%

PRR

558

79.1%

79.0%

Support Coordinator Record Review 484

98.4%

90.8%

Support Coordinator Interview

477

84.1%

79.9%

Standard Deviation
5.5% 2.9% 4.3% 14.4% 22.2% 19.2%

each component of the review and may differ from the number of PCRs shown in Figures 38-43, which show the distribution of the number of PCRs by score and by tool.

Page 79

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Page 80

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

PCR Scores by Tool and Focused Outcome Area (FOA)
As described previously, the indicators used in each review tool are divided into six FOAs. In this section, results by FOA for the PCR are presented for each tool used in calculating the overall score for the PCR process--Individual Interview, Observation, Staff Interview and PRR (see Figure 44). The SCRR and SCI are also divided into FOAs. The results for these tools are in Figure 45.
Individual Interview As part of the PCRs completed this year, 484 individuals were interviewed. Scores for the PCR Individual Interview ranged from 83.6 percent for Community Life to 98.7 percent for Rights, with a mean of 95.1 percent. Individuals were least likely to be participating in their communities or provided opportunities to develop meaningful social roles.

Page 81

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Observations Observations were completed at 412 locations (residential or day services), with an average score of 98.2 percent. Nearly 90 percent of the Observations showed scores of above 95 percent. There was very little variation across the different Focused Outcome Areas.
Staff Interviews PCRs included 833 staff interviews, with an average score of 96.4 percent. Ninety percent of the staff scored above 90 percent. Consistent with previous findings, the Staff Interviews reflect high scores for all FOAs, with Community Life the only area scoring under 95 percent (86.6%).
Provider Record Reviews The PCRs included 558 provider record reviews. Results by FOA ranged from a low of 71.6 percent for Choice to a high of 84.3 percent for Rights, with all PRR FOAs showing lower results than for the Interviews or Observations. The lowest scores were in the areas of Person Centered practices, Choice, and Community Life. The lower score for choice suggests that providers may not be consistently documenting efforts of providing individuals with information to make informed choices regarding supports and services, community participation, and daily activities.

Page 82

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 44. PCR Scores by Tool and Focused Outcome Area

Person Centered Practices
Community Life

Choice

Rights

Whole Health

Safety

PCR Sample

0%

25%

Individual Interview (N = 483) Staff Interview (N = 833)

77.7%

92.3% 95.2% 96.1%

83.6% 95.1%
86.6% 73.7%

71.6%

98.0% 96.9% 96.3%

84.3%

98.7% 98.9% 98.3%

80.2%

93.8% 98.5%
95.5%

83.8%

96.6% 99.8% 98.3%

79.0%

95.1% 98.4%
96.4%

50%

75%

100%

Observations (N = 578) Provider Record Review (N = 558)

Page 83

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Support Coordinator Record Review and Interview For the 484 PCRs completed this year, 484 SCRRs and 477 SCIs were completed with a mean score of 79.9 percent and 90.8 percent respectively. Results by FOA are presented in Figure 45. The lowest-scoring areas for the SCRR were in Community Life (67.3%), Rights (73.4%), and Choice (74.1%). The lowest-scoring SCI areas were in Community Life (82.3%) and Rights (86.3%). With the exception of Safety, documentation appears to be consistently lower in providing evidence of supporting individuals, than interviews with the support coordinator might otherwise indicate.

100% 90% 80%

Figure 45. Support Coordinator Record Review and Interview Scores by Focused Outcome Areas

90.8%

91.1%

79.9%

81.1%

92.3% 82.3%

94.1%

92.7%

94.6% 92.6%

86.3%

74.1%

73.4%

70%

67.3%

60%

50%

PCR Sample Person Community

Centered

Life

Practices

Choice

Rights

Whole Health

Record Review (N = 484) Interview (N = 477)

Safety

Page 84

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Section 4f: Quality Enhancement Provider Review (QEPR)
QEPR Scores by Provider Size
Figure 46 displays the distribution of the QEPR Overall Scores and the Qualification and Training (Q&T) scores by size of the organization.11 Scores were generally lower on the Q&T component of the QEPR than for the overall scores but there is little variation across the size categories within any of the components of the QEPR.
Large providers scored below the average for all QEPRs on the Overall Score (86.1% compared to 88.4%) and on Q&T (79.8% compared to 82.4%). Small and medium providers scored at or above the average. Small providers had the highest average Overall Score (90.4%), and medium providers had the highest average score on Q&T (82.2%). All providers, regardless of size, scored above 99 percent on the DDSS tool (not shown in the figure).

11 See Table 20 on page 69 for details regarding provider size categories.

Page 85

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 46. QEPR Scores by Provider Size

Overall Score

88.4% 90.4%
89.0% 86.1%

Qualifications & Training 50%

60%

80.9% 80.0%
82.2% 79.8%

70%

80%

90%

100%

QEPR Sample (N = 100)

Small (N = 47)

Medium (N = 36)

Large (N = 17)

The distributions of QEPRs/providers across scores, both the Overall Score and for Q&T, are shown in Figures 47-52. The standard deviations for providers of all sizes are greater for Q&T than for the overall scores, indicating a broader dispersion across responses.

Page 86

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 47. Overall Score:

Small Providers

(N = 47)

29

16

Median 91.5%

11

Mean 90.4%

Standard Deviation
6.3%

Figure 48. Overall Score:
Medium Providers
(N = 36)
16 17

3

Median 89.6%

Mean 89.0%

Standard Deviation
6.3%

Figure 49. Overall Score: Large Providers (N = 17) 13

Page 87

3 1

Median 86.9%

Mean 86.1%

Standard Deviation
4.0%

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 50. Qualifications and Training

Score: Small Providers

(N = 47)

15

12

6 5
4 3

1

1

Median 87.2%

Mean 80.0%

Standard Deviation
16.1%

Figure 51. Qualifications and Training Score: Medium Providers (N = 36)

8

8

7

5

5

3

Median 86.0%

Mean 82.2%

Standard Deviation
14.2%

Page 88

Figure 52.Qualifications and Training

Score: Large Providers

(N = 17)

6

5

3 2
1

Median 81.4%

Mean 79.8%

Standard Deviation
8.7%

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

QEPR Scores by Focused Outcome Area (FOA)
The Overall Score for each review is divided into six FOAs. Each FOA score is calculated with the combined results from the Individual Interview, Staff Interview, Observation (IOSA), and PRR. Results for the 100 QEPRs are shown by FOA in Figure 53.

Figure 53. QEPR by FOA and Overall Score (N = 100)

Rights 92.4%
Choice 86.2%

Whole Health 89.1%
Overall Score 88.3%
Community Life
78.0%

Safety 93.9%
PersonCentered Practices
82.6%

Table 24 displays the median and mean scores for each FOA as well as the standard deviation. Mean scores ranged from a low of 78.0 percent for Community Life to a high of 93.9 percent for Safety. Community Life data suggest providers may not always meet the needs and

Page 89

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

expectations related to community integration and the development of valued social roles for the individuals being served. Community Life also shows the greatest degree of variation from the mean, with a standard deviation of 14.6 percent.

Table 24. QEPR Scores by FOA

Tool Name
Person-Centered Practices Community Life Choice Rights Whole Health Safety

Median
85.9% 80.3% 88.7% 94.2% 91.4% 95.0%

Mean
82.6% 78.0% 86.2% 92.4% 89.1% 93.9%

Standard Deviation
9.7% 14.6% 9.3% 5.9% 8.7% 4.3%

Page 90

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

QEPR Review Scores by Tool
Every QEPR contains multiple interviews, observations, and record reviews. Table 25 displays the number of interviews or record reviews completed, as well as the median and mean scores and the standard deviation. On average, the mean PRR score (73.6%) based on documentation review, was lower than mean scores reflected for interviews or observations, each over 95 percent. The distributions of providers by score for each of these tools are displayed in Figures 54-59.

Table 25. QEPR Scores by Tool

Tool Name
Individual Interview Observation Staff Interview PRR Qualifications and Training DDSS

N12
667 409 353 1,377 1,044 1,397

Median
96.2% 98.6% 97.9% 77.3% 84.5% 100.0%

Mean
95.1% 98.2% 96.9% 73.6% 80.9% 99.9%

Standard Deviation
4.7% 3.4% 3.6% 13.2% 14.4% 0.4%

12 Each QEPR consists of multiple interviews, observations, and record reviews. Also, if a provider does not offer residential or day services, no observations are completed for that QEPR. Therefore, in our sample, of the 100 QEPRs, 92 providers had one or more observations completed, for a total of 409 observations. The N sizes displayed in Table 25 may differ from the number of reviews shown in Figures 53-58, which are based on the average scores per QEPR to show the distribution of providers across scores.

Page 91

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Distribution of the Number of QEPRS by Score and by Tool

Figure 54. QEPR Individual Interview Scores (N = 100) 86

Figure 55. QEPR Observation Scores (N = 92) 72

11

2

1

Median 96.2%

Mean 95.1%

Standard Deviation
4.7%

Figure 56. QEPR Staff Interview Scores (N = 99) 84

6

9

Median 97.9%

Mean 96.9%

Standard Deviation
3.6%

18 11

Median 98.6%

Mean 98.2%

Standard Deviation
3.4%

Figure 57. QEPR Provider Record Review Scores (N = 99) 34
26
18 11
1144

Median 77.3%

Mean 73.6%

Standard Deviation
13.2%

Page 92

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 58. QEPR Qualifications and Training Scores (N = 100) 25 26
20

11 9 7

1

1

Figure 59. QEPR DDSS Scores (N = 100) 91
9

Median 84.5%

Mean 80.9%

Standard Deviation
14.4%

Median 100.0%

Mean 99.9%

Standard Deviation
N/A

Administrative Qualifications and Training Every QEPR includes a review of staff qualifications and training. A sample of 1,041 staff records was reviewed, ensuring that all services were included. The primary purpose of the record review is to confirm that relevant staff information is accurate and up to date (e.g., driver's license, performance evaluations), and that staff have received all required trainings. Figure 58 shows the distribution of QEPRs/providers across Q&T scores. The mean score was 80.9 percent. This tool does not include results by FOA.

Page 93

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Developmental Disability Service Specific (DDSS) Review The average score on the DDSS tool was close to perfect, with a mean score of 99.9 percent. Only nine of the 100 providers scored below 100 percent in this area (Figure 59).
QEPR Scores by Tool and FOA
As described previously, the indicators used in each review tool are divided into six FOAs. In this section, results by FOA for the QEPR are presented for each tool used in calculating the Overall score for the QEPR process--Individual Interview, Observation, Staff Interview and PRR (see Figure 60).
Individual Interview As part of the 100 QEPRs completed this year, 667 individuals participated in the Individual Interview, with a mean score 95.1 percent. Each of the FOAs scored above 90 percent, with the exception of Community Life (84.1%).
Observations Observations are completed for providers providing residential and day supports (92 of the 100 providers). The objective of the observation is to see how services are being provided and to identify strengths or areas needing improvement regarding quality, health, or safety. For the 92 providers offering at least one of the eligible services, 409 Observations were completed with a mean score of 98.2 percent. All but two QEPRs received scores higher than 90 percent. Data indicate there was minimal variation across FOAs (Figure 60), scores ranging from a low of 95.3 percent for Person-Centered Practices to a high of 99.4 percent for Safety.

Page 94

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Staff Interview As part of the QEPR, 353 Staff Interviews were completed, with an average score of 96.9 percent. Similar to the Individual Interview, Staff Interview results were lowest for Community Life (87.9%), approximately 10 percentage points below the average for all QEPRs. Provider Record Review Every QEPR includes the PRR, a review of the provider's record for the individuals selected as part of the sample, ensuring that each service the provider offers is represented in the sample. Over the course of the year, 1,377 records were reviewed as part of the 100 QEPRs. The scores for each FOA range from a low of 64.2 percent for Choice to a high of 79.5 percent for Safety. Evidence of supporting individuals appears to be consistently lower in provider documentation than from interviews and observations.
"[I'm] doing things that I haven't done in 18 years; things I never thought I would be able to do again, like attending football games".

Page 95

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 60.QEPR Scores by Tool and Focused Outcome Areas

Person Centered Practices
Community Life

Choice

Rights

Whole Health

Safety

QEPR Sample

0%

25%

Individual Interview (N = 667) Staff Interview (n = 353)

71.4%

91.9% 95.3% 96.7%

70.2%

84.1% 95.4%
87.9%

64.2%

97.6% 97.1% 97.3%

79.2%

98.4% 97.9% 98.4%

75.4%

94.1% 98.7%
96.3%

79.5%

96.7% 99.4% 98.4%

73.6%

95.1% 98.2% 96.9%

50%

75%

100%

Observations (N = 409) Provider Record Review (n = 1,377)

Page 96

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Section 4g: ISP QA Checklist
ISP Written to Support
The ISP Quality Assurance (QA) checklist is used to provide an overall rating for each service plan, based upon the degree to which the ISP is written to provide a meaningful life for the individual receiving services. There are three different categories for each ISP.
Service Life: The ISP supports a life with basic paid services and paid supports. Needs that are "important for" the person are addressed, such as health and safety. However, there is not an organized effort to support a person in obtaining other expressed desires that are "important to" the person, such as getting a driver's license, having a home, or acting in a play. The individual is not connected to the community and has not developed social roles, but expresses a desire to do so.
Good but Paid Life: The ISP supports a life with connections to various supports and services (paid and non-paid). Expressed goals that are "important to" the person are present, indicating the person is obtaining goals and desires beyond basic health and safety needs. The person may go out into the community but with only limited integration into community activities. For example, the person may go to church or participate in Special Olympics; however, real community connections are lacking, and the person indicates that he/she wants to achieve more.
Community Life: The ISP supports a life with the desired level of integration in the community and in various settings preferred by the person. The person has friends and support beyond providers and family members. The person has developed social roles that are meaningful to him/her, such as belonging to a Red Hat club or a book club, or having employment in a competitive, rather than segregated, environment. Rather than just going to church, the

Page 97

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

person may be an usher at the church or sing in the choir. Relationships developed in the community are reciprocal. The ISP is written with goals that help support the person in moving toward a community life, as he/she chooses.
Results in Figure 61 indicate that of a majority of ISPs reviewed, 78.6 percent, were written to support a "Good but Paid Life." The smallest percent, 9.2 percent, supported a service life, while 12.2 percent supported a community life. The ISP results support our findings within each of the tools: although most individuals' needs regarding health and safety are being addressed, there is opportunity for improvement to achieve integration in the community or in various settings.

Figure 61. ISP QA Life question (N = 1,137)

100% 75%

78.6%

50%

25%
9.2%

12.2%

0%

Service Life

Good But Paid Life Community Life

Page 98

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

ISP Expectations
Quality assessors reviewed 12 different sections in the ISP. Each section represents an expectation, listed in Table 26, and has four indicators that must be addressed in the plan. The expectation is rated on a scale from zero to four, zero meaning the section is blank or inadequately addresses the requirements for any of the indicators and four meaning that 100 percent of the indicators or requirements in the section are adequately addressed in the ISP.
Data in Table 26 show the percent of ISPs at each rating on the 12 different expectations. For example, results show that 44.5 percent of ISPs reviewed had all four indicators present (addressed) for the expectation regarding the communication chart expectation. Data indicate 54.0 percent of all ISPs reviewed addressed all elements for all of the sections within the ISP. The expectations measuring whether the person's hopes and dreams and goals are person centered and whether goals are person centered were most likely to have zero or one indicator addressed, 17.6 percent and 20.1 percent respectively; however, most ISPs addressed all four requirements for rights, psychotropic medications, behavior supports and the health and safety review section, 93.1 percent and 92.3 percent respectively.

Page 99

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 26. ISP QA Checklist Ratings by Expectation

N = 1,139

Ratings

ISP QA Checklist Description

0 1 2 3

Relationship Map & discussion on ways to develop relationships.

0.3% 4.6% 18.6% 34.7%

Communication chart Person-centered important to/for Hopes and Dreams: if you could be, do or include anything in your life what would it be?

0.6% 0.8% 9.2% 44.9% 0.3% 0.1% 4.2% 24.6%
9.9% 7.7% 13.0% 22.7%

Service summary Rights, psychotropic medications, behavior supports section Meeting minutes

0.9% 4.7% 12.2% 22.2%
0.1% 0.0% 0.8% 6.0% 1.6% 6.9% 15.6% 23.0%

SIS completed and support needs are addressed in the ISP

0.1% 0.1% 3.0% 42.7%

Health and safety review section completed accurately and thoroughly

0.3% 0.0% 0.2% 7.25

Goals are Person Centered Training goal action plan

5.7% 14.4% 20.5% 25.9% 1.1% 6.0% 7.2% 58.7%

Action plans/objectives

0.1% 5.8% 21.3% 42.0%

Total

1.7% 4.3% 10.5% 29.5%

4
41.9% 44.5% 70.8%
46.6% 60.0%
93.1% 52.9%
54.1%
92.3% 33.5% 26.9% 30.7% 54.0%

Section 4h: Quality and Technical Assistance Consultation (QTAC)
There were 238 QTACs completed, of which 99 were completed at the provider level and 139 at the individual level. Other than QEPR follow-ups, the most common referral reason at the provider level was the "request for two-day Person Centered thinking training" (N = 10). At the individual level, the most common referral reasons were: "medication error(s)" (n = 53), "health concern for the person(s)" (n = 21), and "goals not being tracked" (n = 21).

Page 100

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 27. QTAC Referral Reasons

Provider Level:

N

QEPR follow-up

72

Request for two-day Person Centered thinking training

10

Review of person centered documentation

7

Overall score on QEPR

4

Other

4

Person centered training needed

2

Red Alert follow up

2

Individual Level:

Medication error(s)

53

Goals not being tracked

21

Health concern for the person(s)

21

Lack of bowel movement tracking

20

Lack of financial tracking

20

Lack of HRST tracking

18

Safety concern for the person(s)

14

Lack of community exposure

8

Lack of AIMS testing

7

Red alert follow up

7

Doctor's orders not being followed

6

Environmental concern(s)

5

Health concern for the person(s)

5

Lack of behavior support plan

5

Lack of seizure tracking

5

Rights concern for the person(s)

5

Lack of blood pressure tracking

4

Lack of bowel movement protocol

4

Lack of fire drills or other safety drills

4

Follow up to previous QTAC

3

Lack of blood sugar tracking

3

Safety practice concern(s) for the provider

3

Assistance with tracking practices

2

Behavior support plan not followed or supported by staff

2

Page 101

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Human rights practice concern(s) for the provider

2

Lack of behavior support plan tracking

2

Lack of blood pressure protocol

2

Lack of medical follow up after a hospital visit

2

Seizure protocol not followed or supported by staff

2

Lack of assessment(s)

1

Lack of blood sugar protocol

1

Lack of person centered documentation

1

Person centered training needed

1

Request for two-day Person Centered thinking training

1

Review of person centered documentation

1

Table 28 provides information about the QTAC region, the focused outcome area, referral source and referral type, and technical assistance provided.

Table 28. QTACs by Region and Referral Source/Type

QTAC Type

Provider Level

Individual Level

N

%

N

%

Region:

1

12

12.1

9

6.5

2

11

11.1

29

20.9

3

41

41.4

65

46.8

4

6

6.1

4

2.9

5

16

16.2

13

9.4

6

13

13.1

19

13.7

Total

99

100.0

139

100.0

Referral Source:

Georgia Collaborative

ASO

82

82.8

10

7.2

Provider

14

14.1

0

0.0

DBHDD Regional Staff

3

3.0

129

93.5

Page 102

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 28. QTACs by Region and Referral Source/Type

QTAC Type

Provider Level

Individual Level

N

%

N

%

Total

99

100.0

139

100.0

Referral Type:

QEPR Score

78

78.8

0

-

Provider Request

19

19.2

0

-

Red Alert (PCR/QEPR)

3

3.0

8

5.7

Support Coordinator

Monitoring

0

-

128

92.0

Complaint/Grievances

0

-

1

.07

Total13

100

137

Technical assistance is provided at every QTAC. The percent of each type of technical assistance provided is shown in Figure 62 for QTACs completed at the provider versus the individual level. The most common type of technical assistance offered for provider level QTACs was role play (56.6%) and the most common type offered at the individual level was individual discussion (77.9%).

13 Totals do not sum to 100% because some QTACS had multiple referral types while some did not have a referral type recorded.

Page 103

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 62. Technical Assistance Provided for QTAC
100%

77.9% 75%

56.6%

49.5%

50%

39.7%

42.4%

40.4% 36.0%

25%

19.9%

23.2%

14.0%

4.0%

2.2%

7.1%

0%

1:1Training Brainstorming Group

Individual

Group

Role Play

Other

Training Discussion Discussion

Provider Individual

Section 4i: IDD Summary of Findings
Beginning in fiscal year 2016, the Georgia Collaborative implemented revised review tools to evaluate the quality of intellectual and developmental disabilities services and supports--the QEPR and PCR. The tools are based upon the six FOAs identified throughout this report. Expectations, measured with a various number of indicators, are based on how services and systems should be delivered. Indicators address a wide range of requirements and best practices within each FOA.
During the year, 100 QEPRs and 484 PCRs were completed. Overall FOA scores from both review types indicate three areas in which the service delivery system appears to do well:

Page 104

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

safety, rights and whole health. These areas are the foundation of supports provided to people with intellectual and developmental disabilities, and help protect them from potential risk (Table 29). Providers seem to have effective systems and practices in place to support individuals being safe in their environments, exercising rights, and maintaining their health.
On average, QEPR results are lower than PCR results on each FOA. QEPR results for Choice, Person Centered Practices, and Community Life were lower than shown for Safety, Rights, or Health. PCR results were also lowest for Community Life (Table 29).14 These areas seem to be more challenging for providers to meet the expectations of offering informed choice, providing person centered practices, and successfully integrating people into communities as desired.

Table 29. Summary FOA Scores

FOA

QEPR

Person Centered Practices Choice
Community Life Safety Rights Health

82.6% 86.2% 78.0% 93.9% 92.4% 89.1%

PCR
90.6% 93.3% 83.0% 97.1% 96.6% 93.7%

In general, findings from reviews are quite positive, as indicated in the overall results in Table 29; however, there are some specific areas within the FOAs that could benefit from additional attention or analysis to identify trends or areas where quality initiatives may be directed.

14 The FOA scores include results from the Individual and Staff Interviews, Observations, and PRRs.

Page 105

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Community Life
According to recent Center for Medicaid and Medicare Services (CMS) standards, individuals with intellectual and developmental disabilities should be able to participate in their communities in the same manner as people who do not have a disability. Therefore, helping providers develop systems to address this goal is imperative. During both the PCR and QEPR, individuals indicated they are often not:
Getting new experiences in the community Supported to join clubs, social groups, or other community groups, such as the YMCA
or neighborhood association Learning about, identifying or developing new social roles
The lowest-scoring indicators for Community Life in the Observations for both review types indicate individuals are not provided opportunities to develop social roles. Approximately 30 percent of staff interviewed could not describe how the person is:
Provided opportunities to develop social roles Exposed to new community experiences Provided opportunities to develop community employment
In addition, provider documentation lacks evidence to demonstrate how the person is supported to experience the community, develop social roles and natural supports, engage in the community, or have opportunities to seek employment.

Page 106

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Choice
Informed choice is the cornerstone of helping individuals understand and achieve meaningful goals and direct their own supports and services. Without informed choice, person centered practices are not possible. Data indicate the following:
The person (from interview) is not always involved in life's decisions Staff (from interview) is not always providing options of competitive
employment/supported employment Individual Interview results indicate the person's preferences related to goals of
employment are not being actively pursued (not a readiness model) Providers are often not documenting how informed choice is provided to individuals,
particularly regarding competitive employment and community participation/social interaction

Person Centered Practices
Another key area of focus for CMS is to ensure that services and supports are provided using person centered practices and planning. To the extent possible, the person should be at the center of all decisions, plans, and goals. This means that providers need to understand who the person is and what he or she may want, dream and hope for. Data indicate the following:
Individual Interview results suggest the person often is unable to identify something new or experienced, or something learned
Provider documentation often does not reflect the person's dreams and hopes, talents or strengths
Providers do not always adequately describe the person's progress toward goals or the person's response to services or treatment

Page 107

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Approximately 25 percent of staff interviewed were not formally reviewing, with the person, the person's goals and objectives

Provider Documentation
Provider documentation through the PRR is consistently lower when results are compared across the review tools and with the FOAs. While individuals and staff may indicate services are provided to meet a particular need or goal, documentation does not always reflect this.

Section 4j: Recommendations
Based upon a review of data, we provide the following recommendations: Pursue the development of training for direct support professionals and support coordinators to understand how to connect individuals to their communities in meaningful ways, i.e., based on their interests, talents, strengths, desires and what is important to and for them. To enhance competitive employment, The Georgia Collaborative, in partnership with DBHDD and the Georgia Vocational Rehabilitation Services, could develop a training module that targets support coordination, to promote an understanding of employment options and resources available to people with intellectual and developmental disabilities. Training content may include how to present this information to people and their families, as well as success stories from people who are in competitive employment. The Georgia Collaborative could coordinate with the quality councils to develop training focused on problematic documentation areas, to help providers improve their overall documentation efforts.

Page 108

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

SECTION 5: BEHAVIORAL HEALTH AND INTELLECTUAL DEVELOPMENTAL DISABILITIES COMPARISONS
Overall Results by Focused Outcome Area--Behavioral Health and Intellectual and Developmental Disabilities
The Georgia Collaborative uses the focused outcome areas (FOAs), as described throughout this report, in both the BH and IDD review processes. While the concepts are the same, there are some differences in the data collection methods. Data for BH FOAs are taken only from record reviews; while for intellectual and developmental disability providers, indicators in the interviews, observations and provider/support coordinator record reviews are categorized into the six FOAs. Therefore, for comparison to the BH FOA data, we use IDD data from only the provider record reviews. Figure 63 demonstrates the overall scores for the focused outcome areas derived from the QEPR provider record reviews and BHQR record reviews.
Results (Figure 63) indicate that behavioral health providers appear to perform somewhat better than intellectual and developmental disability providers in documenting how FOAs are addressed for individuals receiving services, with average scores of 85 percent and 74 percent respectively.15 Intellectual and developmental disability providers show lower scores on each FOA with the exception of health. The greatest differences are in Choice and Person Centered Practices, on which intellectual and developmental disability provider documentation is 28 and 21 percentage points lower, respectively.

15 As shown in the report, IDD data from the IOSA show higher results for all of the FOAs than the PRR/documentation data indicate.

Page 109

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Figure 63. Intellectual and Developmental Disabilities and Behavioral Health FOA Score Comparisons

QEPR

FOA Overall Score: 74%
Person Centered : 71% Community: 70% Choice: 64% Rights: 79% Whole Health: 75% Safety: 81%

BHQR

FOA Overall Score: 85%
Person Centered: 92% Community: 87% Choice: 92% Rights: 90% Whole Health: 63% Safety: 80%

Page 110

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Record Review Results by Indicator/Question and FOA
Table 30 displays a sample of questions used in the record reviews, from the QEPR and BHQR. These specific questions from each of the six FOAs are identical in both processes (BH and IDD) and are shown to demonstrate areas of congruence and areas for improvement between BH and IDD. Because review protocols between the two programs may differ, as well as the type of information reviewed and procedures for collecting the data, comparative analysis should proceed with caution.

Results between the two programs are similar in terms of developing and reassessing plans

based on the person's need (Table 30). However, intellectual and developmental disability

providers appear to be less likely to document informed choice, how education is provided on

the risk and benefits of medication, how they ensure that rights forms are signed by

individuals, and how community involvement for individuals is achieved. Behavioral health

providers were less likely to document how they ensure that rights and responsibilities are

reviewed annually with individuals receiving services.

Table 30. Comparison Behavioral Health and IDD Record Review

Indicators/Questions

FOA

Record Review Questions (excerpt)

IDD BH

Choice

Informed choice/options of supports and services

75%

94%

Person Centered

Plan is developed and reassessed based on needs

92%

91%

Community

Community involvement is occurring according to the person's preferences

34%

93%

Whole Health

Medication education on risk/benefits Coordination/communication of services and supports

43% 91%

71% 61%

Safety

Safety/crisis plan development and monitoring

89%

85%

Page 111

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 30. Comparison Behavioral Health and IDD Record Review

Indicators/Questions

FOA

Record Review Questions (excerpt)

IDD BH

Rights

Rights/responsibilities reviewed annually with individual

92%

58%

Rights signed by individual or legal guardian

67%

94%

Individual and Staff Interviews Intellectual and Developmental
Disabilities and Behavioral Health Comparison
The Georgia Collaborative conducted individual and staff interviews for both intellectual and developmental disability and behavioral health providers during the onsite review processes. Intellectual and developmental disability provider interviews used in this comparison are from individuals and staff selected as part of a QEPR sample. The IDD interview sample is randomly selected and stratified by service to ensure each service rendered by the provider is represented. The BH interview samples are selected by the provider and assessors during the quality review. Both interviews focus on the quality of the organization's systems and practices. However, due to the differences in sample selection, comparisons across the programs should be made with caution.

Findings indicate individuals receiving intellectual and developmental disability services are less likely to report they are involved in the routine review of progress toward goals or to develop new social roles (Table 31). Individuals receiving behavioral health services appear to be less likely to visit a dentist annually. Individuals appear equally likely to be satisfied with supports and services, feel they are treated with respect and participate in safety/crisis plan development.

Page 112

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 31. Selected Questions Individual Interview: IDD and BH

IDD

BH

(N = (N =

FOA

Individual Interview Questions

667) 737)

Choice

Satisfied with supports and services

99%

98%

Person Centered

Involved in the routine review of progress toward goals

81%

96%

Community Opportunity to develop new social roles

81%

97%

Seen a primary care physician in the last 12 months

99%

91%

Whole Health Seen a dentist in the last 12 months if they so choose

88%

71%

Safety

Individual/person participated in the development of or review the safety/crisis plan

88%

88%

Rights

Feels they are treated with respect and dignity by staff (including physicians)

99.9% 99.5%

Staff Interview results (Table 32) are shown to be similar for behavioral health and intellectual and developmental disability providers on the selected Choice, Whole Health, and Safety indicators. Staff is equally likely: to be aware of desired changes for supports or services; to be able to describe how health is addressed; and to be aware of cultural practices related to rights. Intellectual and developmental disability providers appear to be less likely to do a formal review of progress on goals with the person or to describe how the person is provided opportunities to develop social roles. However, sample methods differ and lacking statistical tests, these results may simply point to areas for further investigation.

Page 113

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

Table 32. Selected Questions Staff Interview: IDD and BH

IDD

FOA

Staff Interview Questions

(N =

351)

Choice

Staff is aware of what to do if the individual/person wants to change a support or service

99.7%

Person Centered

Staff formally reviews progress on goals/objectives with the individual/person

77%

Community

Staff describes how the individual/person is provided opportunities to develop new social roles

65%

Staff is able to describe how the whole health of the Whole Health individual/person is addressed through coordination 99%
of services

Safety

Staff is aware of the crisis/safety plan and can describe how it is reviewed with the individual/person, when appropriate

100%

Rights

Staff is aware of any cultural practices and beliefs related to rights and exercising those rights

100%

BH (N = 753)
99.5% 98% 99%
95%
93%
99.8%

Page 114

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

SECTION 6: CONCLUSION
Information in this report reflects findings from the first year of the ASO quality reviews conducted for the new quality management system, created through extensive collaborative efforts between DBHDD and The Georgia Collaborative, to bring together various service delivery systems, databases, and quality oversight. The review processes incorporated new and revised review tools, new procedures, review of crisis stabilization units, and integrated BH and IDD reviews. Anecdotal information from feedback surveys indicates there are many satisfied individuals served by the system; people feel their lives have been positively affected; and they have experienced high levels of support from Georgia's network of providers. Data from this first year have yielded a baseline of network performance with which to refine and refocus efforts to continuously improve the quality of services in Georgia. As the quality management system moves into the second year of the contract, comparisons to the baseline data will help focus and drive effective and efficient quality improvement initiatives and continuous quality improvement for individuals receiving services across the state.

Page 115

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

APPENDIX A: ABBREVIATIONS AND ACRONYMS

Acronym
ACT AD ASAM ASO BH BHQR CIS CMS CL CST CSU DBHDD DDSS FOA II IDD IFI IOSA IRR IRP ISP QA MAR N NA NCP PCP PCR PRR QEPR Q&T

Definition
Assertive Community Treatment Addictive Diseases American Society for Addiction Medicine Administrative Services Organization Behavioral Health Behavioral Health Quality Review Consumer Information System Center for Medicaid and Medicare Services Community Life Community Support Team Crisis Stabilization Unit Department of Behavioral Health and Developmental Disabilities Developmental Disability Service Specific Review Focused Outcome Area Individual Interview Intellectual and Developmental Disability Intensive Family Intervention Individual Observation Staff Assessment Individual Record Review Individual Recovery / Resiliency Plan Individual Support Plan Quality Assurance Checklist Medication Administration Record Number in sample Not Applicable Nursing Care Plan Person Centered Practices Person Centered Review Provider Record Review Quality Enhancement Provider Review Qualifications and Training

Page 116

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report

QTAC RN SAMHSA SCI SCRR SD SI TA YMCA

Quality Technical Assistance Consultation Registered Nurse Substance Abuse Mental Health Services Administration Support Coordinator Interview Support Coordinator Record Review Standard Deviation Staff Interview Technical Assistance Young Men's Christian Association

Page 117

The Georgia Collaborative ASO | FY 2016 Quality Management Annual Report