Quality management annual report, FY 2017

FY 2017 Quality Management
Annual Report

Quality Management Annual Report FY 2017
SECTION 1: EXECUTIVE SUMMARY ............................................................................................................................... 4
Behavioral Health ..........................................................................................................................................................4 Intellectual and Developmental Disabilities ..................................................................................................................8
SECTION 2: INTRODUCTION ........................................................................................................................................12
SECTION 3: BEHAVIORAL HEALTH QUALITY REVIEWS...................................................................................................14
BACKGROUND......................................................................................................................................................................14 SAMPLING METHOD .............................................................................................................................................................15
Individual Records and Billing Review .........................................................................................................................15 Individual and Staff Interviews ....................................................................................................................................16 BHQR OVERALL REVIEW SCORES............................................................................................................................................17 BHQR Overall Scores....................................................................................................................................................17 BHQR Overall Scores by Tier ........................................................................................................................................19 BHQR Billing Validation ...............................................................................................................................................20 BHQR Assessment and Treatment Planning ................................................................................................................24 BHQR Compliance with Service Guidelines..................................................................................................................26 BHQR Focused Outcome Areas (FOA) ..........................................................................................................................30 BHQR Staff and Individual Interviews..........................................................................................................................33 ASSERTIVE COMMUNITY TREATMENT .......................................................................................................................................35 ACT Quality Review Overall Scores ..............................................................................................................................35 ACT Billing Validation ..................................................................................................................................................36 ACT Assessment and Treatment Planning ...................................................................................................................39 ACT Compliance with Service Guideline Scores ...........................................................................................................40 ACT Focused Outcome Areas .......................................................................................................................................42 CRISIS STABILIZATION UNIT QUALITY REVIEWS ...........................................................................................................................44 Crisis Stabilization Unit Sample Method......................................................................................................................44 CSU Quality Review .....................................................................................................................................................45 CSU Individual Record Review (IRR) .............................................................................................................................46 CSU Compliance with Service Guidelines.....................................................................................................................48 CSU Focused Outcome Area ........................................................................................................................................52 CSU Staff Interview (SI) and Individual Interview (II) ...................................................................................................54 REASSESSMENT FREQUENCY REVIEW........................................................................................................................................57 BHQR Reassessment Review Findings .........................................................................................................................57 CSU Reassessment Review Findings ............................................................................................................................61 TECHNICAL ASSISTANCE/EXIT CONFERENCE ...............................................................................................................................62 SUMMARY OF FINDINGS AND RECOMMENDATIONS FOR BEHAVIORAL HEALTH PROVIDERS ..................................................................63 System Strengths .........................................................................................................................................................63 Opportunities for Improvement...................................................................................................................................66
SECTION 4: INTELLECTUAL AND DEVELOPMENTAL DISABILITIES ..................................................................................70
BACKGROUND......................................................................................................................................................................70 1

Quality Management Annual Report FY 2017
SAMPLING METHOD .............................................................................................................................................................71 Person-Centered Review (PCR) ....................................................................................................................................71 Quality Enhancement Provider Review (QEPR) ...........................................................................................................72
REVIEW PROCESSES ..............................................................................................................................................................73 Onsite Reviews (PCR and QEPR) ..................................................................................................................................73 Quality Technical Assistance Consultation (QTAC) .......................................................................................................75
PERSON-CENTERED REVIEW ...................................................................................................................................................75 PCR Scores by Tool .......................................................................................................................................................75 PCR Scores by Focused Outcome Area (FOA)...............................................................................................................77 PCR Scores by Tool and Focused Outcome Area (FOA) ................................................................................................78 Opportunities for Growth by FOA ................................................................................................................................79 Individual Service Plan Quality Assurance Checklist (ISP QA) ......................................................................................87 ISP Expectations ..........................................................................................................................................................89 PCR Results by Service .................................................................................................................................................91 PCR Strengths and Recommendations ........................................................................................................................93
QUALITY ENHANCEMENT PROVIDER REVIEW (QEPR)..................................................................................................................95 QEPR Scores by Tool ....................................................................................................................................................95 Overall QEPR Score ......................................................................................................................................................96 Overall Crisis Provider Scores ......................................................................................................................................97 Qualifications and Training (Q & T) and Service Specific (DDSS) .................................................................................98 Qualifications & Training Opportunities for Improvement ..........................................................................................99 QEPR Scores by Provider Size.................................................................................................................................... 101 QEPR Scores by Focused Outcome Area (FOA) ......................................................................................................... 102 QEPR Scores by Tool and FOA ................................................................................................................................... 103 QEPR Scores by FOA and Provider Size ..................................................................................................................... 105
QUALITY AND TECHNICAL ASSISTANCE CONSULTATION (QTAC)................................................................................................... 106 INTELLECTUAL AND DEVELOPMENTAL DISABILITY SUMMARY OF FINDINGS AND RECOMMENDATIONS ................................................. 108
System Strengths ...................................................................................................................................................... 108 Opportunities for Improvement................................................................................................................................ 109
SECTION 5: BEHAVIORAL HEALTH AND INTELLECTUAL DEVELOPMENTAL DISABILITIES: OVERALL RESULTS BY FOCUSED OUTCOME AREA ........................................................................................................................................................115
SECTION 6: FEEDBACK SURVEY RESULTS ....................................................................................................................117
SECTION 7: CONCLUSION...........................................................................................................................................120
APPENDIX A: ABBREVIATIONS AND ACRONYMS........................................................................................................122
APPENDIX B: SCORE DISTRIBUTIONS .........................................................................................................................124
BHQR Overall Score Distribution by Year ................................................................................................................ 126 BHQR Tier 1 Overall Score Distribution by Year ...................................................................................................... 126 BHQR Tier 2 Overall Score Distribution by Year ...................................................................................................... 127 BHQR Tier 2+ Overall Score Distribution by Year .................................................................................................... 127
2

Quality Management Annual Report FY 2017
BHQR Tier 3 Overall Score Distribution by Year ...................................................................................................... 128 BHQR Billing Score Distribution by Year.................................................................................................................. 128 BHQR Assessment & Treatment Planning Distribution by Year ............................................................................. 129 BHQR Compliance with Service Guidelines Score Distribution by Year .................................................................. 129 BHQR Focused Outcome Areas Score Distribution by Year..................................................................................... 130 ACT Overall Score Distribution by Year ................................................................................................................... 130 ACT Billing Score Distribution by Year ..................................................................................................................... 131 ACT Assessment & Treatment Planning .................................................................................................................. 131 ACT Compliance with Service Guidelines ................................................................................................................ 132 ACT FOAs Score Distribution by Year ....................................................................................................................... 132 CSU Overall Score Distribution by Year ................................................................................................................... 133 CSU Compliance with Service Guidelines Score Distribution by Year ..................................................................... 133 CSU Focused Outcome Areas Overall Distribution by Year ..................................................................................... 134 CSU Individual Record Review Score Distribution by Year ...................................................................................... 134 BHQR Overall Score Distribution ............................................................................................................................. 135 BHQR Billing Score Distribution............................................................................................................................... 135 BHQR Assessment & Treatment Planning Score Distribution ................................................................................ 136 BHQR Compliance with Service Guidelines Score Distribution ............................................................................... 136 BHQR FOAs Score Distribution................................................................................................................................. 137 PCR Individual Interview Scores by Year ................................................................................................................. 137 PCR Staff Interview Scores by Year.......................................................................................................................... 138 PCR Support Coordinator Review Scores by Year.................................................................................................... 138 PCR Observation Scores by Year .............................................................................................................................. 139 PCR Provider Record Review Scores by Year ........................................................................................................... 139 PCR Support Coordinator Review Scores by Year.................................................................................................... 140 FY 2017 Overall QEPR Score .................................................................................................................................... 140 FY 2017 QEPR Individual Interview Scores .............................................................................................................. 141 FY 2017 QEPR Staff Interview Scores ...................................................................................................................... 141 FY 2017 QEPR Observation Scores .......................................................................................................................... 142 FY 2017 QEPR Observation Scores .......................................................................................................................... 142 FY 2017 QEPR Qualifications and Training Scores .................................................................................................. 143 FY 2017 QEPR Service Discrepancy Scores .............................................................................................................. 143 FY 2017 QEPR Overall Scores by Provider Size ........................................................................................................ 144 FY 2017 QEPR Qualifications and Training Scores by Provider Size ....................................................................... 145
3

Section 1: Executive Summary

Quality Management Annual Report FY 2017

The Georgia Department of Behavioral Health and Developmental Disabilities (DBHDD) contracts with the Georgia Collaborative Administrative Services Organization (the Collaborative) to integrate disparate systems for data, service delivery, finance, and quality oversight for individuals receiving behavioral health or intellectual and developmental disability services. Quality Management within the Collaborative is tasked with incorporating these two distinct systems into one collaborative and integrated program. On June 30, 2017, Quality Management completed the second year of the contract with DBHDD.
There are many similarities between the two systems. Similar tools and processes are used to monitor quality and compliance within each area, including interviews with individuals and providers/staff and review of records maintained by providers for individuals receiving services. Both divisions review an individual plan for people receiving services. Examples include the Individualized Recovery/Resiliency Plan (behavioral health), and the Individual Service Plan (intellectual and developmental disabilities). In both systems, Focused Outcome Areas (FOA) critical to an individual's quality of life are assessed: Person Centered Practices, Choice, Community, Rights, Whole Health, and Safety. However, there are some distinct differences as well. Intellectual and Developmental Disability reviews include onsite observations of licensed residential and day programs, as well as an administrative review of staff qualifications and training requirements. Behavioral Health reviews include extensive claims review to identify billing discrepancies and report the results of the Billing Validation on each review.
Behavioral Health
During the first year of the contract, Quality Management completed one Behavioral Health Quality Review (BHQR) for each of the 141 providers with an additional subset review of 18 Assertive Community Treatment (ACT) programs for a combined 159 reviews. The second year resulted in a total of 167 BHQRs with an additional 18 ACT reviews totaling 185 reviews across a network of 132 behavioral health providers. See the table below. Of the 185 BHQRs completed, 35 were for
4

Quality Management Annual Report FY 2017
providers who had a second review in fiscal year (FY) 2017, based on scoring criteria (<80 percent overall score or <70 percent billing validation). This practice will continue through the upcoming 2018 fiscal year. The table below provides details by review type of the categories surrounding BHQRs as well as Crisis Stabilization Unit Quality Reviews (CSUQR).

FY 2017 Overview of BHQR and CSUQR Review Data

Review Type

Number of Reviews

Records Reviewed

Individuals Interviewed

Staff Interviewed

BHQR

185*

3,816

735

774

CSUQR

23

343

114

115

Total

208

4,159

849

889

*FY17 resulted in 167 BHQR reviews with an additional eighteen ACT specific reviews for a total of 185 reviews.

BHQRs measure provider performance through four different categories: billing validation, FOAs, assessment and treatment planning, and service guidelines. Assertive Community Treatment services are also reviewed when such services are rendered by the provider. Findings are shown in the following table for BHQR and ACT reviews. The BHQR results are similar each year, although two category scores increased in FY 2017 (billing validation and FOA), while the remaining two categories decreased (assessment and treatment planning, and compliance with service guidelines). In FY 2017, both BHQR and ACT results continued to show lower compliance with assessment and treatment planning standards than in other categories. Additionally, FOAs increased for both BHQR and ACT reviews compared to the FY 2016 baseline year.

BHQR FY 2016 BHQR FY 2017 ACT FY 2016 ACT FY 2017

BHQR and ACT Results by Category by Year

Assessment

Billing

and Treatment Service

Validation

FOA

Planning Guidelines

81%

85%

79%

90%

84%

89%

77%

88%

92%

88%

85%

85%

90%

90%

80%

88%

Overall 84% 84% 88% 87%

5

Quality Management Annual Report FY 2017
Detailed findings of the lowest-scoring areas within the broader categories may influence quality improvement initiatives. These findings include the following:
Assessment and treatment planning standards for both BHQR and ACT reviews indicated a need for the IRP to address co-occurring health conditions, whole health and wellness needs, as well as addressing assessed needs. Additionally, discharge plans which clearly define criteria for discharge (i.e., clear, clinical benchmarks) were lacking in both review types.
Although whole health remained the lowest-scoring FOA for behavioral health, it increased for both the behavioral health and ACT reviews conducted in the second year with safety scoring as the lowest FOA for ACT reviews.
The behavioral health whole health indicators demonstrating lower results included the following: o Ongoing assessment to determine the need for external referrals for health services, supports, and treatment were often not documented; o Communication with external referral sources to identify results of testing and treatment were not documented within the record; o Medical conditions were often not assessed, monitored, and recorded; o Safeguards for medications used were often not documented.
Co-occurring health conditions were addressed in 36 percent of the IRPs reviewed in FY 2016 and declined to 34 percent through the reviews of FY 2017.
FY 2017 specific billing issues most often cited remained similar to the FY 2016 results and included the following: o Missing/incomplete service orders o Individuals not meeting admission criteria o Missing progress notes
In January 2016, Quality Management initiated a review of Georgia's 19 crisis stabilization units (CSU). The review continued through FY 2017 and incorporated the same 19 CSUs with four CSUs
6

Quality Management Annual Report FY 2017
requiring a second review within the year due to low initial scores. The focus of CSUQR is to assess compliance with DBHDD and Medicaid requirements, using the same categories as shown for BHQRs except for billing validation and assessment and treatment planning. Billing validation was not reviewed for CSU providers in FY 2017, at the request of DBHDD. In place of assessment and treatment planning, an individual record review (IRR) occurred with similar questions raised such as individual meeting admission criteria, discharge plans documenting necessary information, as well as co-occurring health conditions being addressed. The CSU scores for FY 2016 and FY 2017 are shown in the following table and indicate the following:
Scores increased across all categories The average overall score showed an increase from 83 to 86 percent FOA was the highest scoring category each year

BHQR and CSUQR Results by Category by Year

Assessment

and

Billing

Treatment

Service

Validation FOA

Planning

Guidelines

BHQR FY 2016

81%

85%

79%

90%

BHQR FY 2017

84%

89%

77%

88%

Overall 84% 84%

CSUQR FY 2016

N/A

88% (IRR) 78%

82%

83%

CSUQR FY 2017

N/A

91% (IRR) 80%

87%

86%

Results at the indicator level in FY 2017 were consistent with the initial FY 2016 period, suggesting the following:
Relatively high compliance specific to documentation of admission criteria, timely assessments, daily status updates, and required staffing and physician access
Documentation lacked evidence unique to co-occurring health conditions, as well as proof of meeting the American Society for Addiction Medicine (ASAM) patient placement criteria, when appropriate
7

Quality Management Annual Report FY 2017

Individuals did not consistently participate in treatment planning and their discharge plans were not tailored to their individual needs

Intellectual and Developmental Disabilities
Quality Management completed 481 person-centered reviews (PCR), 100 quality enhancement provider reviews (QEPR) and 208 quality technical assistance consultations (QTAC) as part of the quality reviews for services offered to individuals with intellectual and developmental disabilities. See the table below for FY 2017 details by intellectual and developmental disability review type.

Review Type
PCR QEPR QTAC Total

FY 2017 Overview of IDD Review Data

Number of

Records

Individuals

Reviews

Reviewed

Interviewed

481

894

481

100

1,356

473

208

N/A

N/A

789

2,250

954

Staff
Interviewed
304 398 N/A 702

The focus of the PCR is to assess the individual's quality of life, as well as the effectiveness of and satisfaction with the service delivery system from the individual's perspective. This holistic process included interviews with the individual and direct support staff including the support coordinator (SC) providing services, observations (if applicable to the service setting), record reviews for all services received including support coordination, and a review of the ISP.
The focus of the QEPR is to monitor providers to ensure they meet requirements set forth by the Medicaid waivers and DBHDD to evaluate the effectiveness of their service delivery system. This process includes interviews with individuals receiving services from the provider, interviews with staff, observations for day and residential programs, record reviews including ISP and an administrative review of staff qualifications and training.
The QTACs focus on follow-up review activities and provide technical assistance to help providers improve service delivery systems. Tools of intellectual and developmental disability reviews include the individual interview (II) and staff interview (SI), individual observation staff assessment (IOSA),
8

Quality Management Annual Report FY 2017
provider record review (PRR), support coordinator record review, support coordinator interview, the administrative qualifications and training tool, as well as a tool for developmental disability service specific (DDSS). The table below indicates the results of each tool by review type. Because crisis services are different, the four providers offering crisis services are reviewed separately.

FY 2017 Summary by Tool and Review Type

PCR

QEPR Crisis

Tool

(N = 481) (N = 96) (N = 4)

Individual Interview

91.9%

90.7% 86.0%

IOSA Observation

96.8%

95.7% 92.5%

Staff Interview

94.3%

95.2% 89.1%

Provider Record Review

70.2%

69.6% 84.1%

SC Record Review

73.7%

N/A N/A

SC Interview

83.3%

N/A N/A

Admin Q&T

NA

82.7% 91.8%

DDSS

99.8%

99.8% 100%

The Quality Management team identified a number of strengths this year in the intellectual and developmental disability service delivery system:
Many staff and providers were aware of the unique safety needs and achievements of the individuals they serve.
Individuals were aware of how to self-preserve in unsafe situations. Individuals feel valued in part because most direct service providers have a clear
understanding of each individual's unique communication styles and skills. In many PCRs, it was noted staff promoted independence, as well as the use of person-
centered values and practices. Record review findings indicated more than 95 percent of staff reviewed were in compliance
with background screening requirements. Data indicate relatively high scores for most components of both review processes.

9

Quality Management Annual Report FY 2017
However, several specific findings point to results being used to guide training sessions or quality improvement initiatives:
As in FY 2016, FY 2017 results showed provider record reviews (PRR) and support coordinator record reviews were the lowest scoring components, particularly for providers offering prevocational, community access (group) and community living supports.
Documentation was less evident in some areas of rights, responsibilities, and restrictive interventions, as well as informed choice and aspects of abuse, neglect, and exploitation.
Staff record reviews indicated some key trainings were often missing, including how to work with individuals with co-occurring diagnoses, suicide prevention, and use of the Georgia Crisis System.
Data from the FY 2017 PCRs compared to FY 2016, showed a decrease across the tools and all the FOAs, particularly in the areas of choice, whole health, and community life. Some key findings indicate the following:
Individuals are often not receiving education on their medications, including what they are taking, why, and what the side effects are.
Informed consent forms for psychotropic medications were often not in the record or not signed by the individual or guardian.
For most ISPs, the goals were not person-centered, and most records reviewed for staff indicated a lack of training on person-centered values, principles, and approaches.
Support coordinator and provider record review evidence indicated individuals are often not offered opportunities to seek and find competitive employment, have new community experiences, or participate in community activities.
It has been a goal for DBHDD and the Collaborative to integrate the processes used to measure quality for behavioral health and intellectual and developmental disability services. However, evaluating side-by-side comparisons are difficult due to different types of sampling methods, as well
10

Quality Management Annual Report FY 2017
as differences in the tools and standards used. Quality Management will review the behavioral health and intellectual and developmental disability record review tools to identify modifications which may enhance the ability to make these types of comparisons for the upcoming fiscal year. These and other results are discussed in more detail throughout the report, including a summary of review processes and sampling procedures. The report also includes a summary of findings for each FOA; in-depth analysis of data from the behavioral health, intellectual and developmental disability, and CSU quality reviews; as well as tabular and graphic displays of findings. Throughout the report, abbreviations and acronyms are employed for brevity and efficiency, which can be found in Appendix A.
11

Section 2: Introduction

Quality Management Annual Report FY 2017

The Georgia Collaborative Administrative Services Organization (the Collaborative), contracted by Georgia's Department of Behavioral Health and Developmental Disabilities (DBHDD), employed 30 quality assessors to complete more than 1,400 quality reviews in the past two fiscal years across Georgia. The mission of Quality Management within the Collaborative is to provide DBHDD with valuable and useful data for use in making decisions about the quality of services, funding, development of programs, etc.
This annual report is the result of assessing, gathering, compiling, analyzing, and measuring the quality of the service delivery system. This is accomplished through assessment of and technical assistance to DBHDD's behavioral health, and intellectual and developmental disability providers. Further, this report contains suggestions for modifications in some processes, methods, approaches, and tools to measure the quality and impact of services. Quality Management recognizes that even quality assessment or measurement processes must have a quality review periodically to examine and determine if we are measuring the things most important and meaningful to individuals receiving services within the Georgia.
This second annual report includes behavioral health, intellectual and developmental disability, and crisis stabilization unit (CSU) findings as well as a detailed explanation of the review processes, analysis and comparisons of network performance across providers, and information from individuals about the services received. The difference in proportions test was used to test statistical significance, where applicable.1 Indicators with an N of 90 or greater and a pass/fail rate of more than 10 percent were analyzed to determine change over time, from Year 1 to Year 2. Areas highlighted in blue within tables reflect significance at a p < .05 level unless stated otherwise. While there are similarities in the review processes, there are also distinct differences;

1 Bohrnstedt, George W. & Knoke, David. (1988). Statistics for Social Data Analysis, 2nd Edition. Itasca, Illinois. F.E. Peacock Publishers, Inc., pgs. 198-200.
12

Quality Management Annual Report FY 2017
therefore, behavioral health, intellectual and developmental disability, and CSU results are reported separately. The report is divided into the following sections:
Behavioral Health Quality Reviews o Assertive Community Treatment
Crisis Stabilization Unit Quality Reviews Intellectual and Developmental Disabilities
o Person-Centered Reviews o Quality Enhancement Provider Reviews o Quality Technical Assistance Consultations Behavioral Health, and Intellectual and Developmental Disability Discussion Provider and Individual Satisfaction Surveys Two appendices are attached o Acronym and abbreviation list o Distribution for all tools used in the BHQR, CSUQR, and intellectual and
developmental disability reviews Summaries of findings and recommendations after each section to address areas needing
improvement or training/education programs to help improve the quality of services provided to Georgians
13

Quality Management Annual Report FY 2017
Section 3: Behavioral Health Quality Reviews
Background
The purpose of the Behavioral Health Quality Review (BHQR) is to determine adherence to DBHDD's standards and to assess the quality of the service delivery system through individual record and claims reviews.2 Review questions are based on DBHDD and Medicaid requirements, and are organized into four categories: billing validation, assessment and treatment planning, compliance with service guidelines, and focused outcome areas (FOA). The score for each category represents the percent of relevant questions met or present. The BHQR overall score is calculated by averaging the scores for the four categories.3 Each category accounts for 25 percent of the overall rating. Minor alterations were made before the FY 2017 review process began to adhere to DBHDD and Medicaid requirement changes, as well as language changes specific to tools, providing more clarity for assessors and providers.
When a BHQR is completed, the lead assessor convenes the team to analyze data gathered and plan the exit conference. The exit conference is designed to give immediate, preliminary feedback of the BHQR findings to the provider. A report of these preliminary findings is left with the provider in the form of an exit conference report which outlines the provider's identified strengths and any areas of concern from the four primary categories of the review. Assessors also include any items of concern falling outside the parameters of the review determined to be an area of concern or risk. Technical assistance is also provided during the review and exit conference.
Within 30 days of completion of a BHQR, a final assessment report is posted on the Collaborative's website, and the provider is notified via electronic mail of the posting along with the final scores identified as the Final Assessment. Like the exit conference report, the final assessment identifies
2 Please refer to the following link to access a full description of the review process and review tools. http://georgiacollaborative.com/providers/prv-BH.html 3 The FOA subcategories are individually scored and are not averaged for the final overall FOA result at the review level. The final overall FOA result is calculated by adding all "yes" or "present" responses of all FOA subcategories and dividing by the total "yes" or "no" responses of the combined subcategories for each review.
14

Quality Management Annual Report FY 2017
strengths and opportunities for growth in the four categories of FOA, billing score, compliance with service guidelines, and assessment and treatment planning. The final assessment includes recommendations for improvement and comparisons to a statewide average based on the previous year's results. The final assessment also highlights any areas of concern or risk which fall outside the scope or scoring of the BHQR in the form of additional comments on practices.
Providers are offered an opportunity to appeal the BHQR findings. Appeal information, including timeframes for submission, is provided upon notification of the completion of the final report.
Sampling Method
Individual Records and Billing Review
During the 2017 fiscal year, 132 providers were eligible for review, as determined by DBHDD, and received a BHQR. This is a reduction from the implementation year (FY 2016) of 141 providers. Reduction in providers may have been a direct result of site closure, inability to provide previously reviewed services, limited claims or services to individuals resulting in a lack of adequate volume for review, or at the request of DBHDD to forgo review during the year. A sample of individuals was selected for each of the record reviews, and a sample of those individuals' claims was used for the billing review. To be eligible for the sample, each individual selected must have had at least three claims in the three months (or longer, if necessary) preceding the BHQR.
To select the individuals as part of the record review, an unduplicated list of individuals receiving services from the provider was stratified by service and payer source: Medicaid, state-contract, and fee-for-service claims submissions. The sample was selected proportionate to the providers' ratio of individuals served by payer source, ensuring each service was represented. The sample for each provider consisted of up to 30 individuals for non-intensive and specialty service providers, with an additional 15 individuals for providers who also offered assertive community treatment (ACT). The number of individuals selected per provider was based on the number of individuals served. For FY 2017, 3,816 individuals were sampled for record reviews, averaging approximately 29 records per
15

Quality Management Annual Report FY 2017
provider. However, it is important to note 35 providers had a second review during the year. (See Reassessment "Frequency" Reviews section for more information.)
For each chosen individual record, a random sample of up to 10 paid claims was selected for a billing review. The number of claims reviewed per individual was based on services provided and claim submissions. The total number of claims reviewed for FY 2017 was 29,602, compared to 31,213 in FY 2016. When providers did not have adequate claims submissions in the three months preceding their review, claims selection timeframe was extended, but did not precede the provider's previous BHQR (when applicable).
BHQR billing validation and claim(s) review focused on specific services. The services included are listed in Table 4 beginning on page 26 of this report. All eligible providers were reviewed at least once during the fiscal year and had at least one claim per billed service included in the claims review, ensuring the complete array of services provided and charged were included in the BHQR.
Individual and Staff Interviews
Samples used for the individual and staff interviews were selected by the provider and quality assessors conducting the BHQR; services received or provided were not considered in the selection of interviewees. Interview sampling methods remained the same for FY 2017 as in FY 2016. Quality assessors attempted to complete a minimum of five Individual and five staff interviews per BHQR; however, the actual number interviewed fluctuated based on individual and staff availability, their agreement to participate in the interview process, the number of employees, and the number of individuals the provider served at the time of the review. If an individual or staff declined an interview, assessors selected an additional individual or staff to be interviewed, when possible.
In FY 2017, a total of 735 individual interviews (II) and 774 staff interviews (SI) were completed. This does not include interviews conducted during the crisis stabilization unit (CSU) reviews, which are discussed later in the report. There was an increase in staff interview participation by an additional 21 interviews from the previous year. Results from interviews conducted are not included in the BHQR overall scores yet were used to obtain valuable, qualitative feedback to
16

Quality Management Annual Report FY 2017
promote quality improvement activities. The results of interviews were shared with providers and DBHDD to provide direct communication and perception of individuals receiving services and staff providing services as an additional quality initiative.
BHQR Overall Review Scores
In this report, data are aggregated and presented by overall provider scores as well as by category (billing validation, assessment and treatment planning, compliance with service guidelines, focused outcome areas [FOA], and individual and staff interview). The four main categories, billing validation, assessment and treatment planning, compliance with service guidelines and focused outcome areas each account for 25 percent of the overall score. Each FOA (choice, person-centered practices, whole health, safety, rights, and community life) also has scores displayed and discussed.4
BHQR Overall Scores
Figure 1 shows the average overall score for the 167 reviews and the scores for each category compared to the 141 reviews conducted in FY 2016 graphically presented to demonstrate areas of improvement and decline. The mean of overall scores for FY 2017 was 84 percent. This is the same result obtained from the providers reviewed in FY 2016. Both categories of billing validation (84 percent) and FOAs (89 percent) increased from the FY 2016 rates of 81 percent and 85 percent, respectively. Compliance with service guidelines, the highest category in FY 2016 at 90 percent decreased in FY 2017 by two points to 88 percent. The lowest scoring category remained assessment and treatment planning, previously averaging 79 percent in FY 2016, now 77 percent for FY 2017. Although change was identified across the differing tools and scores from year-to-
4 Unless otherwise stated, category scores were determined by dividing all the "Yes" answers by the sum of the "Yes" and "No" answers in the category. Questions scored as "Not Applicable" (N/A) were not factored into the overall category score. Each subcategory's score was determined similarly based on the questions contained within each subcategory.
17

Quality Management Annual Report FY 2017
year, no statistical significance was found at p < .05. The following are highlights of the overall category scores:
The lower assessment and treatment planning scores driven by the lack of documentation supporting the incorporation of whole health and wellness goals and objectives into individual resiliency plans (IRPs) in FY 2016, appear to be mainly driven by lack of addressing co-occurring health conditions, discharge-planning criteria, and addressing all assessed needs in FY 2017.
Billing validation indicates approximately 16 percent of claims reviewed were unjustified and subject to recoupment; this result declined positively from the previous year result of 24 percent of unjustified claims (See Figure 2).
The FOAs in FY 2016 represented a new area of review for which providers had not previously been assessed; however, it is possible through education in this area or inclusion of such measurement that providers' scores increased from 85 percent in the initial year to 89 percent in FY 2017.
18

84% 81%

Quality Management Annual Report FY 2017

Figure 1. Fiscal Year Results by Category

79% 77%

90% 88%

89% 85%

84% 84%

Billing

Assessment & Service Guidelines Focused Outcome

Treatment

Areas

Planning

FY 2016 (N = 141) FY 2017 (N = 167)

Overall

BHQR Overall Scores by Tier
In July 2014, DBHDD implemented a community behavioral health provider network structure in which providers were classified using a four-tiered structure.5 Tiers are defined as follows:
Tier 1: Comprehensive Community Providers Tier 2: Community Medicaid Providers Tier 2+: Community State Funded Providers Tier 3: Specialty Providers
Table 1 provides a snapshot of the BHQR overall scores by tier and a distribution of scores by each Tier is provided in Appendix B. Consistent with results of FY 2016, scores remained generally lower for Tier 3 providers (83 percent) compared to the other tiers. However, there was an improvement from the previous year, having increased from 78 percent to 83 percent. Both Tier 1- and Tier 2-

5 Policies regarding the implementation and definition of the DBHDD Community Behavioral Health Provider Network Structure can be found at DBHDD's PolicyStat website: https://gadbhdd.policystat.com
19

Quality Management Annual Report FY 2017

level providers exceeded the overall average of 84 percent. Providers identified as a Tier 2+

provider (N = 6) had the highest score which resulted in a mean of 86 percent for FY 2017

(compared to other tiers) yet decreased by two points from FY 2016.

Fiscal Year FY 2016 FY 2017

Table 1. BHQR Overall Scores by Tier

Tier 1 Tier 2 Tier 2+ Tier 3

86% (N = 24)
85% (N = 28)

85% (N = 90)
85% (N = 115)

88% (N = 5) 86% (N = 6)

78% (N = 22)
83% (N = 18)

Overall Average
84% (N = 141)
84% (N = 167)

Tier 2 providers make up the largest number of providers in Georgia. Tier 2 providers may serve adults, children and adolescents, or both, but must have the capacity to provide the entire array of non-intensive services including both mental health and substance use disorder services. Tier 2 providers may also provide specialty services but are not required to do so. Tier 2+ providers (N = 6) scored (on average) 86 percent.
Tier 3 providers demonstrated the most improvement through an increased mean value of 78 percent in FY 2016 to 83 percent in FY 2017. Additionally, there was a five-percentage point increase in the mean comparing both Tier 1 and Tier 2 providers. Both Tier 1 and Tier 2 providers demonstrated consistent overall scores compared with previous year's results. See distributions for tier level Providers in Appendix B.
BHQR Billing Validation
The billing validation score for each BHQR is the percent of justified billed dollars divided by the total paid/charged dollars for the reviewed claims. Billing scores are then averaged across the network of annual reviews to obtain an annual statewide average. Nine providers scored 100 percent, with an additional nine providers scoring 99 percent. See the distribution of BHQR billing scores in Appendix B. In FY 2016, three providers scored zero percent in billing validation, compared to zero providers for the fiscal year 2017. With the second year of reviews
20

Quality Management Annual Report FY 2017
demonstrating increases in billing category results, FY 2018 thresholds for reassessment are increasing from 69 percent to 79 percent. See Reassessment "Frequency" Review section on page 57 for more details. Figure 26 shows the total dollar amount reviewed through paid claims analysis across all providers during the BHQRs for FY 2017 ($2,934,560.52) as compared to FY 2016 ($3,417,902.28) and the dollar amount found to be unjustified, $463,049.93 and $807,050.16, respectively. In FY 2017, fewer funds being reviewed may be due to several factors including but not limited to provider sample size, limited claims availability, as well as the types of service being reviewed. Although the total funds reviewed for FY 2017 were fewer than reviewed in FY 2016, the amounts unjustified and susceptible for recoupment decreased from 24 percent to 16 percent. FY 2017 yielded a report of an additional $2,076,429.98 in Medicaid funds compared to the FY 2017 state funding amount of $429,065.27. In evaluating Medicaid versus state funded claims/encounters, 84 percent of claims reviewed were found justified compared to 85 percent of state-funded claims/encounters.
6 The percent of justified versus unjustified dollar amounts, as depicted within Figure on the next page, are calculations of total dollar amounts reviewed divided by the total number of unjustified/justified dollar amounts. The annual statewide average billing score is not equivocal to the percent justified/unjustified depicted in the figure; statewide billing score is calculated based on averaging the individual provider scores at year-end. Statewide averages specific to the billing score are impacted by providers who receive an extreme low score and who may have greater amounts of dollars reviewed in proportion to the network total.
21

Quality Management Annual Report FY 2017

Figure 2. BHQR Billing Validation Amount Reviewed by by Fiscal Year

$3,500,000.00 $3,000,000.00 $2,500,000.00

$807,050.16 24%

Total Amount Reviewed in FY 2016: $3,417,902.28 Total Amount Reviewed in FY 2017: $2,934,560.52
$463,049.93 16%

$2,000,000.00

$1,500,000.00 $1,000,000.00

$2,610,852.12 76%

$2,471,510.59 84%

$500,000.00

$0.00

FY 2016 Total $ Justified

FY 2017 Total $ Unjustified

When a claim was found to be unjustified, assessors selected all applicable reasons a reviewed claim was identified as a discrepancy; therefore, one claim may have multiple discrepancy reasons identified. The most prevalent billing discrepancy reasons identified in FY 2017 remained consistent with those identified in FY 2016 and are as follows:
Missing/incomplete service orders Individuals not meeting admission criteria Missing progress notes
However, both service orders and individuals meeting criteria improved for the 2017 fiscal year with an evident reduction of 1,500 documented occurrences combined (see Table 2). Furthermore, in FY 2016, discrepancy results indicated there were 779 missing progress notes (2.5 percent) of the 31,213 claims reviewed. This improved slightly by nearly 9 percent in FY 2017 with 676 missing
22

Quality Management Annual Report FY 2017
notes or documentation, or 2.3 percent of claims reviewed. Although improvement has been made for these areas outlined above, they continue to remain the top reasons related to billing discrepancies. Most improved discrepancy reasons from FY 2016 to FY 2017 are printed name missing (92.6 percent improvement) and the record not submitted within required timeframe (90.5 percent fewer instances). Content not being unique to the individual was identified 137 times in FY 2016 compared to 243 in FY 2017, thereby demonstrating a decline by nearly 87 percent. Table 2 provides the number of occurrences for the remainder of all monitored discrepancy reasons and the change in percent from FY 2016 to FY 2017. A negative difference demonstrates improvement from year to year. Overall, 20 (87 percent) of the observed indicators (N = 23) resulted in improvement from the baseline year to FY 2017.

Table 2. BHQR Billing Discrepancy Reasons

FY 2016

FY 2017

Occurrences Occurrences

Claims Reviewed

31,213

29, 602

Missing/incomplete order

1,612 (5.2%) 496 (1.7%)

Does not meet admission criteria

1,132 (3.6%) 748 (2.5%)

Quantitative

Occurrences Occurrences

Progress note is missing

779 (2.5%) 676 (2.28%)

Code is missing/different than code billed

381 (1.2%) 379 (1.28%)

Staff credential missing

299 (1.0%) 295 (1.00%)

Signature missing

275 (0.9%) 194 (0.66%)

Date of entry missing

263 (0.8%) 185 (0.62%)

Units billed exceed time / units documented 236 (0.8%) 112 (0.38%)

Consistency requirements missing

202 (0.7%) 57 (0.19%)

Credential not supported by documentation 199 (0.6%) 276 (0.93%)

Record not submitted within timeframe

197 (0.6%) 19 (0.06%)

Location missing (out-of-clinic)

166 (0.5%) 155 (0.52%)

Time in / time out missing

92 (0.3%)

68 (0.23%)

Date of service incorrect / missing

63 (0.2%)

42 (0.14%)

Printed name missing

42 (0.1%)

2 (0.01%)

Value Change 1,611 -1,116
-384 Change
-103 -2 -4 -81 -78
-124 -145 +77 -178 -11 -24 -21 -40

Percent Change -5.16% -67.5% -30.2% Change -8.6%
4.9% 4.4% -25.1% -26.4% -49.7% -70.6% 45.9% -90.5% -2.2% -22.0% -30.6% -92.6%

23

Quality Management Annual Report FY 2017

Table 2. BHQR Billing Discrepancy Reasons

Performance Standards

FY 2016 Occurrences
FY 2016 Occurrences

FY 2017 Occurrences
FY 2017 Occurrences

Value Change Value Change

Content does not match service definition

735 (2.4%) 489 (1.7%) -246

Content does not support code billed

428 (1.4%) 397 (1.3%)

-31

Content does not support units billed

375 (1.2%) 518 (1.8%) +143

Intervention outside staff's scope/practice

348 (1.1%)

79 (0.3%)

-269

Content is not unique to the individual

137 (0.4%) 243 (0.8%) +106

Multiple services billed at the same time

95 (0.3%)

79 (0.3%)

-16

Non-billable activity Diversionary activities billed

76 (0.2%)

59 (0.2%)

-17

48 (0.2%)

26 (0.1%)

-22

Areas highlighted in blue within table reflect significance at a p < .05 and indicators with n < 90 were not tested due to low volume.

Percent Change Percent Change -29.9% -2.3% 45.7% -75.8% 86.8% -11.3% -17.9% -41.5%

BHQR Assessment and Treatment Planning

Assessment and treatment planning consisted of nine questions answered once per record

reviewed in FY 2016. A change to the question of medical screening occurred for FY 2017 which

was previously combined to include measurement of both a behavioral health assessment and

medical screening to render satisfactory credit, thereby totaling 10 questions for FY 2017. The

questions and percent "yes" on each are presented in Table 3 below, and areas for growth include

the following:

The lowest-scoring question in FY 2016 indicated co-occurring health conditions were often

not included in individuals' plans of care (64 percent) without any explanation. Similar

results were found for FY 2017 review for which 34 percent indicated co-occurring

conditions were addressed in the Individual Recovery / Resiliency Plan (IRP).

Where other needs were identified (housing, employment, childcare, higher education,

etc.), they were only addressed in 59 percent of the plans in FY 2016 and 48 percent in FY

2017.

24

Quality Management Annual Report FY 2017

Fifty-nine percent of individuals had goals, objectives, or interventions in their plans to address wellness in the baseline review year, compare to 56 percent in FY 2017. This area remains an area needing improvement.

Table 3. BHQR Assessment & Treatment Planning Question Scores by Year

Question

FY 2016 FY 2017

All assessed needs are addressed

59%

48%

Co-occurring health conditions addressed in IRP

36%

34%

Current behavioral health assessment

N/A

94%

Current medical screening is present

98%

97%

Discharge plan defines criteria

80%

58%

Goals/objectives honor hopes, choice, preferences, outcomes

91%

89%

Individual meets admission criteria

95%

97%

Interventions/objectives are goal-linked & service-consistent

96%

91%

IRP is individualized in personalized language

78%

87%

Whole health & wellness in IRP

59%

56%

Assessment & Treatment Planning Result

79%

77%

*Areas highlighted in blue within table reflect significance at a p < .05.

Comparisons in scoring reflect a deficit in a comprehensive, whole-person, whole-health approach

to recovery/resiliency planning with a number of the providers in the network. The average score

for this category was 77 percent, compared to the FY 2016 score of 79 percent. Individualized

language, an area identified for needed growth in FY 2016 (78 percent), increased significantly by

nearly 10 points in FY 2017 (87 percent); however, discharge planning previously held a moderate

score of 80 percent in FY 2016, which declined significantly in FY 2017 to 58 percent. This decrease

is likely due to a change in the question scoring criteria in FY 2017 to include specific clinical

benchmarks related to discharge criteria.

Data suggest that although records contained required assessment documentation, such as present medical screenings and current behavioral health assessments, a vast percent of recovery/resiliency plans lacked relevant goals, objectives, and interventions to address individuals' assessed needs (to include co-occurring health conditions) or plan for increased whole health and wellness.

25

Quality Management Annual Report FY 2017

BHQR Compliance with Service Guidelines
The compliance with service guideline questions were answered once per individual record reviewed. The number of questions answered varied, specific to the service scored. The service reviewed for each individual record was based on which services were reflected in the billing claims sample for the respective individual; therefore, multiple services could be reviewed within one individual's record. A total of 18 services were reviewed across all providers in FY 2016, as shown in Table 4. An additional service, opioid maintenance therapy, had three reviews conducted and 69 records reviewed for FY 2017 and is not included in the table below. The (r) size in Table 4 represents the number of providers assessed for the service, based on the claims sample. Additionally, the total number of records reviewed for each service was included immediately following as the (n) size. For FY 2016, each provider was reviewed only once; thus, the n also represents the number of providers reviewed for each service. However, in FY 2017 multiple providers were reviewed more than once; thus, the n representation has been changed to reflect the number of reviews, which allows for consistency in the year-to-year comparisons.

Table 4. BHQR Service Guidelines Scores by Service Type*

Service Type MH Peer Support Individual Addictive Disease Support Services Assertive Community Treatment

FY 2016 87% (r = 7, n = 119) 81% (r = 48, n = 224) 85% (r = 19, n = 314)

FY 2017 95% (r = 6, n = 38) 85% (r = 69, n = 293) 88% (r = 20, n = 334)

Value Percent Change Change
+8 8.4%
+4 4.7%
+3 3.4%

Psychosocial Rehabilitation Program

90% (r = 75, n = 196) 93% (r = 26, n = 179)

+3 3.2%

Intensive Case Management Intensive Family Intervention

95% (r = 10, n = 55) 86% (r = 30, n = 221)

94% (r = 13, n = 53) 85% (r = 37, n = 185)

-1 -1.2% -1 -1.2%

Case Management

85% (r = 69, n = 557) 84% (r = 90, n = 746)

-1 -1.2%

Community Support Community Support Team

84% (r = 84, n = 484) 94% (r = 7, n = 23)

83% (r = 114, n = 637) 92% (r = 7, n = 13)

-1 -1.1% -2 -2.2%

26

Quality Management Annual Report FY 2017

Table 4. BHQR Service Guidelines Scores by Service Type*

Service Type Individual Counseling

FY 2016

FY 2017

Value Percent Change Change

97% (r = 113, n = 1288) 94% (r = 150, n = 1980) -3 -3.3%

Psychiatric Treatment Group Counseling/ Training

95% (r = 91, n = 843) 93% (r = 63, n = 418)

92% (r = 128, n = 950) 90% (r = 78, n = 561)

-3 -3.3% -3 -3.2%

Family Counseling/ Training

96% (r = 92, n = 667) 91% (r = 129, n = 904)

-5 -6.2%

Nursing Assessment & Health MH Peer Support Program

85% (r = 85, n = 884) 95% (r = 33, n = 234)

80% (r = 116, n = 897) 86% (r = 25, n = 199)

-5 -5.5% -9 -10.5%

Psychosocial Rehabilitation - Individual 96% (r = 25, n = 649) 85% (r = 98, n = 904)

-11 -12.9%

Peer Support Whole Health & Wellness 92% (r = 7, n = 45)

68% (r = 12, n = 67)

-24 -35.3%

AD Peer Support - Individual

79% (r = 1, n = 7)

48% (r = 1, n = 13)

-31 -64.6%

*"r" size represents number of reviews conducted for each service and the "n" size equates to the maximum number of records reviewed per service. Areas highlighted in blue within table reflect significance at a p < .05. AD Peer Support Individual service was not statistically tested due to low response volume.

Compliance with service guidelines was the highest-scoring category of the four in FY 2016, with an average of 90 percent but has since declined to an average of 88 percent. Moreover, nearly twothirds (64.1 percent) of reviews resulted in a score above the 88 percent average in this category, with more than half (56.9 percent) exceeding 90 percent.
MH Peer Support Individual, Addictive Disease Support Services, Assertive Community Treatment, and Psychosocial Rehabilitation Program were the top-scoring services demonstrating the highest rate of compliance for FY 2017 compared to all services measured. The most-improved services are outlined in Table 5 and have listed the lowest scoring questions per service, which provide for areas of continued growth and opportunity.

27

Quality Management Annual Report FY 2017

Table 5. FY 2017 Service Type Improvement Results

Service type and FY 2017 lowest scoring question(s):

FY 2016

FY 2017

Mental Health Peer Support Individual

87% (r = 7)

95% (r = 6)

Progress notes contain documentation of the individual's progress (or lack of) toward specific goals/objectives in the treatment plan.
Substance Use Disorder Support Services

77% (n = 119) 71% (n = 199) 81% (r = 48) 85% (r = 69)

Coordination with family and significant others is documented and with [adult] individual's permission. Answer "yes" if attempt is made, but permission is not given. (Review authorization period.)
Contact must be made with the individual receiving services at a minimum of twice each month, at least one face-to-face. (Review authorization period.)
Assertive Community Treatment

43% (n = 224)
64% (n = 224) 85% (r = 19)

44% (n = 292)
66% (n = 292) 88% (r = 20)

There is evidence that the ACT Team is working with informal support systems/collateral contacts at least 2-to-4 times per month with or without the individual present (must be documented) to provide support and skills training to assist the individual in his/her recovery. (Review authorization period.)
Psychosocial Rehabilitation Program

43% (n = 314) 39% (n = 335) 90% (r = 75) 93% (r = 26)

Weekly progress notes must document the individual's progress relative to functioning and skills related to the person-centered

84% (n = 195) 62% (n = 178)

goals identified in his/her Individual Recovery Plan (IRP).

*"r" size represents number of reviews conducted for each service and the "n" size equates to the maximum number of records reviewed per service. Areas highlighted in blue within table reflect significance at a p < .05.

Data also suggests areas of improvement clearly identified through various service types which decreased from FY 2016 to FY 2017. These results are detailed in Table 6, with further analysis and trending included at the question level supplemented by the lowest scoring questions provided by service.

28

Quality Management Annual Report FY 2017

Table 6. FY 2017 Service Type Declined Results*

Service type and FY 2017 lowest scoring question(s):

FY 2016

FY 2017

Addictive Disease (AD) Peer Support - Individual

79% (r = 1)

48% (r = 1)

Progress notes contain documentation of the individual's progress (or lack of) toward specific goals/objectives on the treatment plan. The staff interventions reflected in the progress notes are related to the staff interventions listed on the treatment plan.
Documentation supports that the individual has identified his/her own individual goals for recovery.
Peer Support Whole Health and Wellness

100% (n = 7)
100% (n = 7) 100% (n = 7) 92% (r = 7)

7% (n = 13)
7% (n = 13) 15% (n = 13) 68% (r = 12)

Collaboration with other health care providers to ensure that individual has access to needed services is documented. (One time per authorization.)
There is a minimum of one contact weekly either face-to-face or by phone. (Review authorization period.)
There is evidence in the documentation of an annual physical or at a minimum a discussion and encouragement to have an annual physical exam.
Psychosocial Rehabilitation - Individual

77% (n = 44) 89% (n = 45) 91% (n = 44) 96% (r = 25)

50% (n = 67) 66% (n = 67) 68% (n = 66) 85% (r = 98)

There is a minimum of two contacts each month, and one is face-to-face. (Review authorization period.)
Progress notes contain documentation of the individual's progress (or lack of) toward specific goals/objectives on the treatment plan.
Mental Health Peer Support Program

74% (n = 638) 92% (n = 649) 95% (r = 33)

66% (n = 903) 79% (n = 902) 86% (r = 25)

Progress notes contain documentation of the individual's progress (or lack of) toward specific goals/objectives on the

88% (n = 234)

71% (n = 199)

treatment plan.

*"r" size represents number of reviews conducted for each service and the "n" size equates to the maximum number of records reviewed per service. Areas highlighted in blue within table reflect significance at a p < .05.

The lowest-scoring service remains substance use disorder peer support individual, previously at 79 percent in FY 2016, which was 48 percent for FY 2017; however, only one provider was reviewed as providing this service (who had an overall compliance score of 55 percent) and differed from the

29

Quality Management Annual Report FY 2017
original provider reviewed (overall compliance score of 90 percent) in FY 2016. Additionally, peer support whole health and wellness declined in scoring from FY 2016 (92 percent) to FY 2017 (68 percent) by approximately 24 percentage points with seven reviews conducted in FY 2016 and 12 reviews in FY 2017. The lower scores in both services were driven by a lack of documentation reflecting individuals' progress toward specific goals and objectives in their plans. Additionally, the individual identifying his/her own goals (substance use disorder peer support), collaboration with other care providers, and the requirement of staff making minimal monthly contacts (peer support whole health and wellness) were noted as areas for improvement. One of the 141 providers reviewed in FY 2016 was not reviewed for compliance with service guidelines because the provider only billed a service not reviewed programmatically at that time (opioid maintenance treatment). FY 2017 had two providers of the 167 reviews scored for service as it pertained to opioid maintenance therapy as, during that year, the service became a formally reviewed program. Results of the two providers' reviews, one of which was reviewed twice, scored 98 percent with all questions scoring at least 93 percent.
BHQR Focused Outcome Areas (FOA)
Focused outcome areas questions are answered once per record reviewed. Each FOA has a different number of questions for a total of 24 questions scored. Figure 3 provides the results of FY 2017 compared to FY 2016.
30

Quality Management Annual Report FY 2017
Figure 3. BHQR Focused Outcome Areas Overall Scores by Year

92%

91%

93% 87%

80% 83%

74%

96% 92%

90% 93%

85% 89%

63%

Whole Health

Safety

Person Community Centered Life Practices

Choice

Rights FOA Score

FY 2016 (N= 141)

FY 2017 (N= 167)

Whole Health questions address whether the records reviewed demonstrated individuals were treated holistically with their physical health needs being assessed, documented, and monitored. Of the six FOAs, Whole Health continues to remain the lowest scoring, yet results of FY 2017 reviews reflect the greatest amount of improvement statistically significant at p < .05. Whole Health for FY 2017 was 74 percent compared to 63 percent in FY 2016. Medical conditions being assessed, monitored, and recorded continues to be the area most in need of improvement.
Safety questions address whether providers were attending to certain risk factors for individuals, such as their tendency to experience a crisis, have suicidal or homicidal thoughts or actions, and whether individuals had received information and education about the risks and benefits of prescribed medications. This area represents the second-lowest-scoring area of the FOAs through both the initial fiscal year of implementation as well as the subsequent follow-up year but did demonstrate significant improvement in year-to-year result. For FY 2016, of the three questions in the subcategory, the one most often scored "no" indicated individuals (or their legal guardians)

31

Quality Management Annual Report FY 2017
often had not signed medication consent forms along with the prescriber. This remained the case with slight improvement from FY 2016 (52 percent) to FY 2017 (58 percent).
Person Centered Practice questions are used to measure whether documentation shows that individuals have a voice and participate in creating their care plans and are active participants in modifying them as needed and desired. While the result of person-centered practices declined, it was not of a statistically significant value. All questions scored more than 80 percent for FY 2017, with one of the higher-scoring questions indicating that individuals were active participants in the planning and receiving of services (95 percent).
Community Life questions address how individuals were engaged in their communities of choice and whether they held valued social roles. The five Community Life questions are used to measure whether individuals had been assessed for their need to make changes in their living, learning, working, and/or social environments (99 percent), and whether they had been assisted with establishing goals to address any needed changes (94 percent). Results in FY 2017 improved compared to results of FY 2016, with slight improvement in all questions noted and an overall significant six-point increase from FY 2016 (87 percent) to FY 2017 (93 percent).
Choice questions address how, and if, individuals were provided with options of services and were encouraged to make educated choices concerning supports and services provided. When barriers to services were identified, assessors looked for documentation as to how the individual was engaged in addressing the identified barriers (96 percent). This is a four-point significant increase from the previous year and yielded the highest FOA score in FY 2017.
Rights questions address whether individuals had been apprised of their rights while in treatment, at the point of entry into the system and on an annual basis after, and whether they had been informed of their rights under Federal HIPAA laws. The increase in Rights from FY 2016 to FY 2017 was found to be significant at p < .05. The area of greatest concern identified in FY 2016 (58 percent) continued to remain the highest concern based on data from FY 2017 (65 percent) and addresses whether individuals were apprised of their rights and responsibilities on an annual basis.
32

Quality Management Annual Report FY 2017
The remaining four questions exceeded 94 percent or above for FY 2017 and reference individuals' appraisal of rights at the onset of services, in a language understandable to the individual, as well as individuals having HIPAA Privacy rules and laws reviewed with evidenced signature by an individual (or legal guardian) of acknowledgement for being informed of rights.
BHQR Staff and Individual Interviews
The interview questions were divided into the six FOAs. Individual interviews served to assess the individual's quality of life and the perception of care with the provider and services rendered. Staff interviews helped determine whether a person-centered approach was used in providing services and empowering individuals.
Data in Figure 4 represent the average scores for the individual FY 2017 interviews. Figure 5 represents average scores of staff interviews. It is notable all areas scored in the 90th percentile, demonstrating consistency in results from comparison to the FY 2016 results, which may indicate a high level of satisfaction for individuals who were served by the providers. Results of interviews identified the following:
Personal outcome approaches being used in designing person-centered supports and services as well as the individual feeling free from abuse, neglect, and exploitation (99 percent).
Staff interview responses seem consistent with individuals' general perception as it relates to person-centered and personalized approaches being used (98 percent).
Notably, the most apparent discrepancy between individual and staff interviews for FY 2017 was specific to the question of health-related needs being addressed. Of the individuals interviewed, 89 percent felt their health needs were addressed while staff perceived this to be the case in 97 percent of the interviews conducted. Furthermore, although not directly assessed via the interview questions, provider staff frequently made statements which indicated a high level of employment satisfaction with the reviewed agency.
33

Quality Management Annual Report FY 2017

Figure 4. BHQR Individual Interview Data by Fiscal Year

Whole Health Safety PCP
Community Life Choice Rights Overall

FY 2016 (N = 737)

FY 2017 (N = 735)

92.7% 93.9%
96.2% 96.3% 95.8% 96.7% 95.0% 94.7% 97.4% 97.8% 97.6% 97.8% 95.6% 96.2%

Figure 5. BHQR Staff Interview Data by Fiscal Year
Whole Health

Safety

PCP

Community Life

Choice

Rights

Overall

FY 2016 (N = 753)

FY 2017 (N = 774)

97.7% 97.4%
97.7% 97.6%
98.3% 97.9%
99.2% 99.2%
98.5% 98.5%
99.4% 99.5%
98.5% 98.1%

34

Quality Management Annual Report FY 2017
Assertive Community Treatment
Quality Management conducted reviews of 19 assertive community treatment (ACT) providers, embedded within the routine BHQR. For the 17 providers who offered both non-intensive outpatient and ACT, up to 15 additional individuals receiving ACT services were selected for review. The two providers who offered only ACT services had 30 individuals selected for each review. One ACT provider was reviewed for a second time within the fiscal year due to low initial scores in FY 2016. This provider had fifteen ACT records reviewed at each scheduled visit. Across all ACT providers, 332 records were reviewed to include up to 10 billing claims per record for a total of 3,221 claims. The data below represent findings from reviews of ACT services only.
ACT Quality Review Overall Scores
Figure 6 provides ACT statewide averages by category for FY 2017. The overall score for FY 2017 resulted in 87 percent for all ACT reviews.
Figure 6. FY 2017 ACT Statewide Averages
Billing Validation FY16: 92% FY17: 90%

Service Guidelines
FY16: 85% FY17: 88%

Overall Score
FY16: 88% (N = 19) FY17: 87% (N = 20)

Focused Outcome
Areas
FY16: 88% FY17: 90%

Assess. & Planning
FY16: 85% FY17: 80%

35

Quality Management Annual Report FY 2017
Figure 77 provides the result by category for BHQR reviews and ACT reviews for both FY 2016 and FY 2017. It should be noted three of the four category scores for ACT reviews, except for service guidelines (which equates to the same BHQR result of 88 percent), are higher than the corresponding BHQR scores for non-intensive services for FY 2017. Additionally, in FY 2017 and consistent with BHQR scores, assessment & treatment planning demonstrated a decrease while FOA scores demonstrated an increase.

Figure 7. ACT and BHQR Category Score Results by Fiscal Year

95%

90%

85%

80%

75%

70%

65%
FY 2016 ACT Averages FY 2017 ACT Averages FY 2016 BHQR Averages FY 2017 BHQR Averages

Billing
92% 90% 81% 84%

FOA

Assessment/ Service Planning Guidelines

Overall

88%

85%

85%

88%

90%

80%

88%

87%

85%

79%

90%

84%

89%

77%

88%

84%

ACT Billing Validation
Figure 8 shows the total dollar amount reviewed through claims analysis during the ACT-specific BHQRs for FY 2016 compared to FY 2017, $306,628.32 and $340,428.04 respectively. The total dollar amount found to be unjustified in FY 2017 was two percentage points higher resulting in 9.6 percent of funds being unjustified ($32,707.80) when compared to the previous year.

7 ACT review scores are inclusive of the BHQR review. For the purpose of evaluation, monitoring and analysis providers servicing specific to ACT services are reviewed individually as a group within this section.
36

$350,000.00 $300,000.00 $250,000.00 $200,000.00 $150,000.00 $100,000.00
$50,000.00 $0.00

Quality Management Annual Report FY 2017
Figure 8. ACT Billing Validation by Fiscal Year
Total Amount Reviewed in FY 2016: $306,628.32 Total Amount Reviewed in FY 2017: $340,428.04

$23,422.57 8%

$32,707.80 10%

$283,205.75 92%

$307,720.24 90%

FY 2016 Total Amount Justified

FY 2017 Total Amount Unjustified

Information in Table 7 indicates the specific billing discrepancy reasons found during the ACT reviews with 3,221 claims reviewed for FY 2017 compared to 3,029 claims reviewed in FY 2016. Assessors select each reason a reviewed claim was identified as a discrepancy; therefore, one claim may have multiple discrepancy reasons identified. The most prevalent billing discrepancy reasons for FY 2017 were the same as FY 2016 and were as follows:
Staff credential missing Location was missing out-of-network claims Content did not support units billed
There was demonstrated improvement noted for both missing staff credential (previously 69 instances [2.3 percent] for FY 2016 versus 40 instances [1.2 percent] in FY 2017), as well as missing location for out-of-clinic claims (39 instances [1.3 percent] for FY 2016 compared to 32 instances [1.0 percent] in FY 2017). However, the billing discrepancy "content did not support the units
37

Quality Management Annual Report FY 2017
billed" nearly doubled from 27 instances (0.9 percent) in FY 2016 to 51 in FY 2017 (1.6 percent). Furthermore, while an additional 192 claims were reviewed for FY 2017 than in FY 2016, instances of discrepancy identification also increased overall from 8.3 percent to 9.9 percent. Several discrepancy reasons increased, negatively, from the previous year and are identified as follows:
Signature missing Date of entry missing Date of service incorrect Missing/incomplete service order Time in/time out missing

Table 7. ACT BHQR Billing Discrepancy Reasons by Year

FY 2016

FY 2017

Value

Occurrences Occurrences Change

Claims Reviewed

3029

3221

+192

Not meet admission criteria for service billed

3 (0.1%)

20 (0.6%)

+17

Missing/incomplete order

0 (0.0%)

12 (0.4%)

+12

Quantitative

Occurrences

Change

Staff credential missing

69 (2.3%)

40 (1.2%)

-29

Location missing (out-of-clinic)

39 (1.3%)

32 (1.0%)

-7

Code is missing / different than code billed

25 (0.8%)

19 (0.6%)

-6

Units billed exceed time / units documented

17 0.6%)

24 (0.7%)

+7

Progress note is missing

16 (0.5%)

17 (0.5%)

+1

Date of entry missing

1 (0.03%)

22 (0.7%)

+24

Signature missing

1 (0.03%)

25 (0.8%)

+21

Time in / time out missing

0 (0.0%)

10 (0.3%)

+10

Date of service incorrect/missing Performance Standards

0 (0.0%)

3 (0.1%)

Occurrences

+3 Change

Content does not support units billed

27 (0.9%)

51 (1.6%)

+24

Content does not support code billed

17 (0.6%)

24 (0.8%)

+7

Intervention outside staff's scope of practice

12 (0.4%)

0 (0.0%)

-12

Non-billable activity

9 (0.3%)

8 (0.2%)

-1

Multiple services billed at the same time

7 (0.2%)

2 (0.1%)

-5

38

Percent Change
>100% N/A
Change -45.5% -22.8% -28.5% 32.8% -0.1% >100% >100%
N/A N/A Change -77.6% -32.8% 100% 16.4% 73.1%

Quality Management Annual Report FY 2017

Table 7. ACT BHQR Billing Discrepancy Reasons by Year

Claims Reviewed

FY 2016

FY 2017

Occurrences Occurrences

3029

3221

Value Change
+192

Content does not match service definition

5 (0.2%)

5 (0.2%)

0

Content is not unique to the individual

4 (0.1%)

5 (0.2%)

+1

* Due to the low response volume, statistical difference in proportions testing was not conducted on the above table.

Percent Change
6.0% -17.5%

The billing validation score is the percent of justified billed dollars divided by the total paid/billed dollars for the reviewed claims. The statewide average ACT billing score of 92 percent was higher than the BHQR statewide average of 81 percent in FY 2016. This remained consistent with the FY 2017 result with an ACT billing score of 90 percent versus BHQR score of 84 percent demonstrating ACT providers excel in fewer funds being identified as unjustifiable compared to BHQR reviews overall.

While the majority of scores exceeded 86 percent, more than 50 percent of reviews conducted in FY 2017 resulted in a billing score of 96 percent or greater. Additionally, the one provider who had been reviewed for a second time in FY 2017 scored at or below 80 percent for both reviews and consecutively declined in billing score from the initial review conducted in FY 2016. ACT billing scores decreased for this individual provider from 92 percent (FY 2016) to 80 percent (FY 2017 first review) and further to 74 percent (subsequent FY 2017 review). This is consistent with the steady decline of the BHQR billing scores for this provider by review period of 88, 85, and 76 percent respective to the time of initial review to the time of last review. Targeted technical assistance is conducted in cases such as this where scores continually decline. Ongoing education and frequency of those providers who fall below the 80 percent threshold will continue to be reviewed at two intervals in the upcoming year.

ACT Assessment and Treatment Planning
Figure 9 provides the ACT assessment and treatment planning score results by specific question for both FY 2016 and FY 2017. Questions assessing whether co-occurring health conditions, whole health and wellness, and all assessed needs are being addressed on the IRP were the lower scoring
39

Quality Management Annual Report FY 2017
questions statewide. Additionally, discharge plans defining criteria significantly decreased from 90 percent in FY 2016 to 46 percent in FY 2017 for ACT reviews. Changes to both the ACT and BHQR question specific to discharge plan criteria in which assessors are required to identify additional factors to grant credit for this question may be the most likely reason for the decrease. However, IRPs within the ACT program, again in FY 2017, scored high regarding individualized language (97 percent) and increased significantly by five points from FY 2016 to FY 2017 for this subcategory.

Figure 9. ACT Assessment and Treatment Planning Scores by Year

Individual meets admission criteria
Current medical screening is present
IRP is individualized in personalized language Interventions/objectives are goal-linked & service-
consistent Goals/objectives honor hopes, choice, preferences,
outcomes Current behavioral health assessment N/A
Whole health & wellness in IRP
All assessed needs are addressed
Co-occurring health conditions addressed in IRP
Discharge plan defines criteria

100% 99% 100% 99% 92% 97% 99% 97% 94% 94%

59% 51% 47% 49%
46%

91% 75% 76%
90%

FY 2016 (N = 19)

FY 2017 (N = 20)

ACT Compliance with Service Guideline Scores
The ACT compliance with service guideline score contains 14 indicators. An additional question was added for the 2017 fiscal year and scored a result of 92 percent specific to documentation of the individual's response and a discussion related to the agreement of services identified in the

40

Quality Management Annual Report FY 2017
treatment planning/individual recovery planning. Table 8 below shows the item-level detail to illustrate both the percentage point and percent change across indicators as well as comparing fiscal year results within the ACT compliance with service guidelines. The ACT compliance with service guidelines average increased to 88 percent for FY 2017 compared to 85 percent in FY 2016. Documentation showed 100 percent of individuals received at least one symptom assessment and medication management contact per month. This remains consistent with FY 2016 findings. Important to note is results for all questions specific to compliance with service guidelines exceeded 90 percent, except for the following:
Completion of a treatment plan by ACT staff before reauthorization of services (59 percent) Evidence the ACT team is working with informal support/contacts at least 2-4 times per
month (39 percent)
This is consistent when compared to FY 2016 review findings of 58 percent and 43 percent respectively. Thus, it is evident that, although ACT staff work closely with individuals and informal supports to identify and monitor progress and interventions as well as goals and objectives, continued effort is needed in the above two areas specific to the reauthorization period.

Table 8. ACT Compliance with Service Guidelines Scores

FY 2016 FY 2017

Documentation shows that the individual meets admission or continuing stay criteria.

100%

99%

The ACT team completes a treatment plan review with the staff,

the individual, and his/her family/informal supports prior to the

58%

59%

reauthorization of services.

There is documentation to support when substance use services are needed and are integrated into the treatment plan.

94%

93%

There is evidence the ACT team is working with informal support

systems/collateral contacts at least 2-4 times per month with or

without the individual present (and it is documented) to provide 43%

39%

support and skills training to assist the individual in his/her

recovery. (Review specific to authorization period.)

Value Percent Change Change

-1

-1.0%

+1

1.7%

-1

-1.1%

-4 -10.3%

41

Quality Management Annual Report FY 2017

Table 8. ACT Compliance with Service Guidelines Scores

FY 2016 FY 2017

The ACT team is working with the individual toward educational or vocational needs, interests, per IRP (once per authorization).
Following admission to a psychiatric facility, the ACT team is involved in each individual's discharge planning.
There is documentation of individual's involvement in transition planning.
One of the contacts per month addresses the symptom assessment and management of medications (once a month).
The ACT team has all required staff.
For discharged individuals, there are multiple documented attempts to locate and make contact with the individual prior to discharge (over a 45-day period).
Progress notes contain documentation of the individual's progress (or lack of) toward specific goals/objectives on the treatment plan.
The staff interventions reflected in the progress notes are related to the staff interventions listed on the treatment plan.
The progress notes document individual response to the staff intervention provided. There is documentation of individual's responses and a discussion to the agreement of services identified in the treatment planning/individual recovery planning.
ACT Compliance with Service Guidelines Result

95% 87% 82% 100% 76% 100%
91%
98% 99%
N/A 85% (N = 19)

97% 95% 90% 99.7% 91% 100%
99%
97% 99%
92% 88% (N = 20)

Value Percent Change Change

+2

2.1%

+8

8.4%

+8

8.9%

-.3 -0.3% +15 16.5%

0

0.0%

+8

8.1%

-1

-1.0%

0

0.0%

N/A

N/A

+3

3.4%

ACT Focused Outcome Areas
Focused outcome area (FOA) questions are answered once per record reviewed. Each FOA has a different number of questions for a total of 24 questions scored in this category. The overall score for ACT FOA is consistent from previous year's results equaling 90 percent. While there was improvement noted in the results specific to the lowest scoring indicators of whole health and safety from FY 2016, several indicators decreased slightly, including the categories of rights,
42

Quality Management Annual Report FY 2017
community, and person-centered practices. Figure 10 provides the ACT FOA results for each category by fiscal year. The greatest improvement in the FOA sub scores specific to ACT reviews was whole health, which increased from 76 percent in FY 2016 to 90 percent in FY 2017.

Figure 10. ACT FOA Scores by Category by Year

100% 90% 80% 70%

90% 76%

82% 78%

99% 91%

97% 94%

97% 97%

89% 87%

90% 90%

60%

50% Whole Health

Safety

Person Community Centereed Life Practices

Choice

Rights FOA Score

FY 2016 (N = 19) FY 2017 (N = 20)

43

Quality Management Annual Report FY 2017
Crisis Stabilization Unit Quality Reviews
At DBHDD's request, a review of all crisis stabilization unit (CSU) providers was initiated in January of FY 2016. The purpose of the CSU review was to assess the provider's overall practices and quality of service delivery, and to determine adherence to DBHDD standards through individual record reviews.8 Reviews were conducted in conjunction with a BHQR when the CSU providers also provided BHQR services but resulted in separate CSU scores and final assessment reports.
Review questions are based on the DBHDD Provider Manual and DBHDD Policies, and were organized into three review categories: individual record review (IRR), compliance with service guidelines, and FOAs. The score for each category represents the percent of applicable questions met or present. The CSU overall score is calculated by averaging the three categories, with each category accounting for 33.3 percent of the overall score.
During FY 2017, all 19 eligible CSU providers were reviewed. Like FY 2016, two were freestanding CSUs, and 17 occurred in conjunction with behavioral health quality reviews (BHQRs). Additionally, four CSUs had a second review during FY 2017 due to previous year's results falling below the threshold (79 percent overall score). Results for the reassessment reviews were included within the overall results incorporating results of 23 reviews. Five of the 23 reviews were conducted at CSU locations who provide services to adolescents and 18 reviews were conducted at CSUs providing services to adults only.
Crisis Stabilization Unit Sample Method
A random sample of 15 individuals who had received services within the three months preceding the review was selected for record reviews. When providers did not have an adequate number of individuals served in the three months preceding the review, samples were pulled from individuals served up to six months preceding the review. A total of 343 individual records were reviewed for
8 Please refer to the following link to access a full description of the review process and review tools. http://georgiacollaborative.com/providers/prv-BH.html
44

Quality Management Annual Report FY 2017
CSU providers in FY 2017. Two CSUs had only 14 records reviewed either because of limited claims available or the individual being on the CSU fewer than 24 hours. The sample for the interviews was selected and scored similarly to the BHQR process: individuals and staff were selected by the provider and quality assessors conducting the CSU review. Results from the interviews were not included in calculating the provider's overall CSU score. Quality assessors completed a minimum of five individual and five staff interviews per CSU review; however, the actual number fluctuated based on individual and staff availability, their agreement to participate in the interview process, the number of staff, and the number of individuals the provider served at the time of the review. Individuals selected for interviews were currently being seen at the CSU, and the staff selected was providing services on the CSU. If an individual or staff declined to be interviewed, assessors selected a different individual or staff to be interviewed.
CSU Quality Review
Table 9 shows the overall score for the 23 reviews performed in FY 2017 as well as the results of each category comparing each of the two fiscal years reviewed. The overall score mean was 86 percent for FY 2017 compared to 83 percent in FY 2016, demonstrating a three-point increase. Eighteen reviews (78 percent) scored within the 81-100 percent range compared to the FY 2016 total of thirteen (68 percent). Results of FY 2017 increased for all categories. Focused Outcome Areas remained the highest-scoring category in FY 2017 at 91 percent and again was an increase from FY 2016 result of 88 percent. In addition, consistent with the baseline year compliance with service guidelines followed at 87 percent, a five-point increase from the previous year (82 percent). The lowest scoring category remained as the individual record review (IRR), averaging 80 percent. However, IRR did also demonstrate an increase from the previous year by two percentage points (78 percent). Difference in proportions test conducted did not demonstrate statistical significance in year-to-year review related to CSUQR category scores.
45

Quality Management Annual Report FY 2017

Table 9. CSU Overall Scores by Fiscal Year

(FY 2016 N = 19/FY 2017 N = 23)

Category

FY 2016 FY 2017

Overall

83%

86%

Individual Record Review

78%

80%

Service Guidelines

82%

87%

Focused Outcome Areas

88%

91%

Four CSU providers had been reviewed for a second time during FY 2017 due to low overall score results upon the initial FY 2016 review. The results of these four specific CSUQR reassessment reviews are graphically portrayed within the reassessment review section of this report in Figure 20 on page 61.
CSU Individual Record Review (IRR)
Individual record review (IRR) questions were answered once per record reviewed. Each of the six subcategories had its own unique number of questions with 31 scored within the IRR category. Although no statistical significance was identified in analysis of each individual subcategory as a whole, overall improvement is apparent within the IRR reviews for the 23 reviews completed. The IRR Score in FY 2016 was 80 percent and has since increased to 82 percent for FY 2017. Additionally, it is evident there is room for improvement when reviewing subcategories and specific indicators or questions of the CSU IRR. Figure 11 identifies the subcategories and results for each by fiscal year.

46

Quality Management Annual Report FY 2017

Figure 11. CSU Individual Record Review Category Scores by Year

Assessment/Treatment Planning Admission/Initial Evaluation/Screening for Risk Crisis Stabilization Specific Treatment Planning
Crisis Stabilization Course of Stay Documentation
Transition/Discharge Planning

86% 85%
69% 69%
70% 72%
83% 86%
87% 89%
63% 62%

FY 2016 (N = 19)

FY 2017 (N = 23)

The CSU intake assessment data, similar to FY 2016, resulted in several positive findings such as the following:
Almost all records, in FY 2017, 99 percent (99.6 percent, FY 2016) confirmed that individuals having met admission criteria.
Ninety-eight percent were assessed by a physician or a physician extender within 24 hours of admission (99 percent, FY 2016).
Individual records reflected daily status updates by a registered nurse (RN) in 95 percent of records.
Ninety percent of records contained a correctly documented medication administration record (MAR).
47

Quality Management Annual Report FY 2017
Although the results have decreased for daily RN status updates (significant at p < .01) and a correctly documented MAR (no significance found) from FY 2016 in which results were 99 and 93 percent, respectively, CSU intake assessment data remains well above average. Furthermore, 92 percent of records had a confirmed admission order by the physician/extender to the CSU.
Of the subcategories and individual questions, areas for growth remain in CSU treatment planning. These areas remain similar to areas identified within the BHQR assessment and treatment planning category in FY 2017. These areas include not incorporating all identified needs (primarily, cooccurring physical health issues) in the IRPs or nursing care plans (NCP). Addressing co-occurring physical health issues continued to remain an area of concern for FY 2017 due to 42 percent of records either minimizing or completely lacking evidence of including these co-occurring issues in the IRP or NCP. Moreover, 69 percent of records documented co-occurring disorders are assessed and addressed simultaneously, 74 percent incorporated medical needs within the IRP or NCP, and only 50 percent of records addressed safety issues when applicable. Furthermore, documentation of the American Society for Addiction Medicine (ASAM) patient placement criteria within the record having occurred in only 19 percent of records in FY 2016 remained low at 38 percent in FY 2017. Although transition/discharge planning continues to result in a relatively low score for FY 2017, the question of transition/discharge plans containing the needed documentation has significantly increased from the previous year's result from 56 percent to 62 percent. Moreover, only 58 percent of the records reviewed in FY 2017 had specific step-down service/activity/supports documented to meet the individualized needs. For FY 2017, decreases in initial bio-psychosocial assessments being present (82 percent) and comprehensive nursing assessments completed upon admission (73 percent) were evident compared to FY 2016 (84 percent and 78 percent, respectively).
CSU Compliance with Service Guidelines
The 16 CSU compliance with service guidelines questions (14 scored and two non-scored) were answered once per review to assess the CSU program. CSU staffing requirements had been met for 96 percent of the reviews conducted. The one provider who had not met the requirement was reviewed for a second time and succeeded in meeting the requirement at the second review. Of the
48

Quality Management Annual Report FY 2017
four CSUs serving children and adolescents, all met the staffing requirements and ratios for FY 2017. An additional question was added to the FY 2017 review process specific to a psychiatrist consultation availability in cases in which the CSU physician(s) is/are not specialized in pediatric psychiatry, which scored at 100 percent. Only three service guidelines questions had decreased results when compared to the baseline year and are detailed in Figure 12.

Figure 12. CSU Compliance with Service Guidelines Questions with Greatest Decrease in Score by Fiscal Year

Theraputic Blood Level Monitoring
FY 2016 89%
FY 2017 83%

Seclusion and Restraint Policy
Adherence
FY 2016 95%
FY 2017 91%

Protocols for Handling Drugs
FY 2016 95%
FY 2017 87%

The lower-scoring questions of FY 2017 remained the same as the baseline year review, yet demonstrated some improvement. These questions included the following:
The provider is adhering to current policy for the safe storage of medication, previously 53 percent for FY 2016 increased to 57 percent in FY 2017.
Policies and procedures are present for adherence to required categories of crisis service plans for provision of crisis services to individuals who are deaf, deaf-blind, blind, and hardof-hearing (53 percent in FY 2016) increased to 65 percent for FY 2017.
Documentation in FY 2016 reflected CSUs did not have access to specialists such as addictionologists (67 percent) or pediatric psychiatrists (67 percent) when needed increased in FY 2017 to 96 percent and 83 percent respectively.
49

Quality Management Annual Report FY 2017
CSU policies during the baseline year (FY 2016) often failed to identify a model for substance use treatment (69 percent) improved to 91 percent in FY 2017.
Results for providers adhering to their policies on the notification of medication errors; previously identified as an area for growth, improved from 79 percent to 83 percent; a four-point increase. From FY 2016 to FY 2017, scores increased across the CSU providers specific to service guideline compliance with more than half of reviews scoring above 90 percent. The median score in FY 2017 was 91 percent compared to the median score in FY 2016 at 80 percent. Overall, the statewide average for compliance with service guidelines increased from 82 percent in FY 2016 to 87 percent in FY 2017. See Figure 13 for an annual comparative result of CSU review questions, mentioned above, specific to compliance with service guidelines which did not meet 100 percent for FY 2017.
50

Quality Management Annual Report FY 2017

Figure 13. CSU Compliance with Service Guidelines Results by Question by Year
CSU Staffing Requirements Met (FY16 n = 19, FY17 n = 23)

100% 96%

Access to Addictionologist (FY16 n = 18, FY17 n = 23)

67% 96%

C&A Psychiatrist (Non-scored) (FY16 n = 3, FY17 n = 6)

67% 83%

Model/Curriculum for SU treatment (Non-scored) (FY16 n = 16, FY17 n = 22)

69% 91%

Deaf, Deaf-Blind, Hard of Hearing Policies (FY16 n = 19, FY17 n = 23)

53% 65%

Infection Control Plan Adherence (FY16 n = 19, FY17 n = 23)

84% 87%

Theraputic Blood Level Monitoring (FY16 n = 19, FY17 n = 23)

89% 83%

Seclusion & Restraint Policy Adherence (FY16 n = 19, FY17 n = 23)

95% 91%

Medication Storage Policy Adherence (FY16 n = 19, FY17 n = 23)

53% 57%

Adherence to Medication Notification Policy (FY16 n = 19, FY17 n = 23)

79% 83%

Protocols for Handling Drugs (FY16 n = 19, FY17 n = 23)
FY 2016 (N = 19)

FY 2017 (N = 23)

95% 87%
51

Quality Management Annual Report FY 2017
CSU Focused Outcome Area
Focused outcome area (FOA) questions were answered once per record reviewed. Each FOA had its own unique number of questions for a total of 23 questions assessed overall. Please refer to the BHQR FOA section for a definition of the six FOAs on page 31 and 32.
All agencies reviewed in FY 2017 scored at or above 85 percent for all FOAs and as illustrated below in Figure 14. The overall score for CSU providers increased from 88 percent to 91 percent. Like FY 2016, results were highest in the areas of Choice (98 percent), Rights (92 percent), and Community Life (94 percent). Although both Choice and Rights decreased somewhat from the previous year's review by five and three percentage points, respectively, community life increased from 92 percent to 94 percent. Whole health (89 percent), Safety (85 percent), and Person Centered Practices (88 percent) increased from the baseline year with results of 83 percent (Whole Health), 76 percent (Safety), and 77 percent (Person Centered Practices). All categories except for Choice and Community Life demonstrated statistically significant improvement at the p < .05 level.
The following were findings based on each individual FOA for FY 2017:
Whole Health demonstrated consistent findings when compared with FY 2016 results as well as in line with what is noted in the IRR category and within the BHQRs, documentation lacked evidence whole health and wellness was consistently incorporated into treatment. While there remains improvement as noted when comparing the score results from FY 2016 to FY 2017, 83 percent to 89 percent respectively, data suggests documentation of current medical conditions being assessed, monitored or recorded remains a pronounced area for improvement (79 percent).
Safety documentation demonstrated services were offered in an environment ensuring patient safety (96 percent) and provider's work with the individual to devise a crisis/safety plan as needed (95 percent). However, documentation of the individual (or other) being educated on the risks and benefits of their prescription medications, as evidenced by a
52

Quality Management Annual Report FY 2017
signed consent form, has been identified as an area for improvement due to the average result of 63 percent for FY 2017. Person Centered Practices documentation in FY 2016 did not support that individuals were active participants in CSU treatment planning; however, the result for FY 2017 depicts otherwise with 91 percent of records reviewed indicating the individual was an active participant in the planning of services and 83 percent actively participating in the modification of said plan or services. Community Life documentation supported transition planning throughout service delivery (92 percent), informed choice of the individual specific to housing option (95 percent), discussion of aftercare placement (95 percent) and provider support in assisting individuals with after care placement (95 percent). Choice reflected documentation supports the individual's preferences for treatment while at the CSU were honored (98 percent), and when barriers to treatment were identified, measures were taken to address the barriers (95 percent). Rights supported privacy and confidentiality being reviewed with individuals (95 percent) yet demonstrated lower results with individuals being informed about their rights and responsibilities at the onset of services (88 percent).
53

Quality Management Annual Report FY 2017
Figure 14. FY 2016 and FY 2017 CSU BHQR by FOA

Rights 97%
Choice 99%

Whole Health
83%
88%
(N = 19) FY 2016
Community 92%

Safety 76%
Person Centered Practices
77%

Rights 92%
Choice 98%

Whole Health
89%
91%
(N = 23) FY 2017
Community 94%

Safety 85%
Person Centered Practices
88%

* The FOA subcategories are individually scored and are not averaged for the final overall FOA result at the review level. The final overall FOA result is calculated by adding all "yes" or "present" responses of all FOA subcategories and dividing by the total "yes" or "no" responses combined for each review.
CSU Staff Interview (SI) and Individual Interview (II)
The CSU interview questions were similar to the BHQR interview questions and were divided into the six FOAs. Individual interviews were used to assess the individual's quality of life and the perception of care with the provider and services rendered. Staff interviews helped determine if a person-centered approach was used in providing services and empowering individuals. The data and anecdotal information gathered from individual and staff interviews supplemented the record review and enhanced the review process by providing a tailored perspective to illustrate the quality of care the agency provided using both a quantifiable and qualitative approach.
Data in Figure 15 illustrates the results of staff and individual interviews by fiscal year. Individual interview scores declined slightly from the baseline year to FY 2017 while staff interviews remained steady above 98 percent.

54

Quality Management Annual Report FY 2017

Figure 15. CSU Individual and Staff Interview Results by Fiscal Year

INDIVIDUAL INTERVIEW

FY 2016 94.9% (N = 94)
FY 2017 93.0% (N = 114)

STAFF INTERVIEW

FY 2016 98.6% (N = 96)
FY 2017 98.7% (N = 115)

Figure 16 represents the average FOA scores for the individual interviews while Figure 17 provides results of Staff Interviews for both FY 2016 and FY 2017. Individual interview results remained consistent compared to FY 2016. Again, all but one FOA scored in the 90th percentile, indicating a high level of satisfaction for individuals served by the providers. The lowest-scoring FOA for IIs based on indicator level data remained whole-health (83 percent), which may indicate an increased need for CSUs to attend to and plan for individuals' co-occurring health conditions or health needs such as preventive services, dental services, or primary care. This issue was also highlighted in the IRR and FOA categories.
All SI questions answered for FY 2017 exceeded 95 percent except one safety question in which 79 percent felt adequately staffed related to acuity level of the individuals on the CSU. Additionally, CSU and BHQR SI scores seem to indicate knowledge regarding providing services that align with standards assessed by the FOA questions, as well as knowledge of the individuals they serve.

55

Quality Management Annual Report FY 2017

Figure 16. CSU Individual Interview Subcategory Data by Year

Whole Health Safety
Person-Centered Planning Community Life Choice Rights Overall

FY 2016 (N = 94)

84.6% 82.5%
95.3% 96.2% 96.0% 93.8%
99.4% 95.6% 96.8% 97.6%
99.6% 98.8% 94.9% 93.0%
FY 2017 (N = 114)

Figure 17. CSU Staff Interview Subcategory Data by Year

Whole Health Safety
Person Centered Planning Community Life Choice Rights Overall

96.8% 98.1% 98.7% 98.1% 98.2% 99.5% 98.7% 98.8% 99.6% 99.6% 97.3%
100% 98.6% 98.7%

FY 2016 (N = 96) FY 2017 (N = 115)

56

Quality Management Annual Report FY 2017
Reassessment Frequency Review
BHQR Reassessment Review Findings
Beginning in FY 2017, the frequency of a BHQR and CSUQR was based on minimum scoring thresholds: less than 80 percent overall score or less than 70 percent billing validation. Providers scoring above the minimum threshold receive one BHQR/CSUQR per fiscal year; providers falling below the threshold receive two reviews. A total of 35 providers were evaluated twice during the 2017 fiscal year. However, two of these providers had not been previously reviewed in FY 2016 due to either being a new provider for FY 2017 or having insufficient claims use to justify an on-site review. Providers whose scores fell below the threshold were scheduled for a repeat review at least six months following their initial FY 2016 review allowing ample time for claims submission and documentation to reflect any changes made by the provider based on previous review findings. The average number of days between reviews for these providers was 195 days. The majority of reassessed providers in FY 2017 were Tier 2 (N = 29). Figure 18 displays the distribution of provider scores for the reassessed providers for scores obtained during the first review in FY 2016 and scores obtained during the first and second reviews conducted in FY 2017. Improvement in scores is clearly illustrated when comparing FY 2016 to FY 2017. In FY 2016, more than half of providers scored in the 71 to 80 percent range. The number of providers scoring in this range gradually declined through each subsequent review of FY 2017 while the number of providers scoring in a higher range, i.e., 81 to 90 percent and 91 to 100 percent range, steadily increased.
57

Quality Management Annual Report FY 2017

Figure 18. BHQR FY 2017 Overall Score Distribution

of Reassessed Providers
70%

60%

58%

50%

46% 43% 40%

40% 30%

30%

23%

23%

20%

11%

10%

3% 3% 6% 6%

6% 3%

0%

FY 2016 Review (N = 33, Mean = 80%, Median = 79%) First Review FY 2017 (N = 35; Mean = 79%; Median = 80%) Second Review FY 2017 (N = 35; Mean = 83%; Median = 86%)
Overall Score: The average result for the reassessed providers at first review of FY 2017 was 79 percent compared to 83 percent later in the year. Overall score results demonstrated improvement for the 35 providers as a group based not only on the distribution of scores but also in reviewing those that fell below the established thresholds at either time. Further results show the following:
Sixteen providers (9.6 percent), at the time of first FY 2017 review, fell below the 80 percent threshold for overall score compared to 11 providers or 6.6 percent at time of the subsequent review.
Nine providers (25 percent) decreased in overall scores from initial FY 2017 to subsequent FY 2017 review with five declining by 10 percent or more. Of the providers who declined in overall score, the majority declined in at least three of the four categories with the category of billing validation demonstrating the greatest loss in percentage points.
While 74 percent of all reviews demonstrated an increase in overall score, three providers decreased across all categories from first to second review.
58

Quality Management Annual Report FY 2017
Distributions of reassessed providers based on each category can also be found in Appendix B. Those providers who fell below the established threshold at either of the two reviews during FY 2017 will be required to have an initial and subsequent review scheduled for the 2018 fiscal year regardless of improvement in one or both categories, overall score, and billing score.
Assessment and Treatment Planning was the lowest scoring category for the reassessed providers in FY 2017. The 35 providers initially scored an average result of 70 percent at the time of first review in FY 2017 compared to 78 percent at time of the subsequent FY 2017 review demonstrating improvement. Furthermore, while 25 of the reassessed providers (71 percent) demonstrated improvement in FY 2017, several providers (N = 23) continued to score at or below 80 percent for this category at the time of their second FY 2017 review. However, of the 71 percent of providers demonstrating improvement in this category, more than half exceled by 10 points or more. Improvement is also noted against the statewide average of 77 percent as 21 (60 percent) providers scored at or above this result at the time of their second FY 2017 review, compared to only 10 (17 percent) providers at the time of their first FY 2017 review.
Billing Score: Nearly half (N = 17) of the 35 providers demonstrated improvement from the initial FY 2017 billing score ranging from a one-point to 35-point difference; see Figure 19 below. Eight reviews fell below the threshold of 70 percent initially in FY 2017, yet five (63 percent) increased at the time of their subsequent score with four of the eight exceeding 90 percent. Twenty-seven providers (77 percent) exceeded the 70-percent threshold during the second review in FY 2017. The average billing score for the reassessed providers at both times of review in FY 2017 was 79 percent which is an increase from the average FY 2016 score of 75 percent for these providers; see figure below. As shown in the distribution below, approximately 63 percent of the providers had billing validation scores between 71 and 100 percent in FY 2016; these providers were placed on a reassessment schedule due to a low overall score.
59

Quality Management Annual Report FY 2017

Figure 19. BHQR FY 2017 Billing Score Distribution

45%

of Reassessed Providers

42%

43%

40%

40%

35%

30%

26% 26%

25%

21%

20%

15% 10%

6%

6%6%9%

11%

12% 11%

9%

9%

5%

3%3% 3%3% 3% 3% 3%

3%

0%

FY 2016 Review (N = 33; Mean = 75%; Median = 79%) First Review FY 2017 (N = 35; Mean = 79%; Median = 84%) Second Review FY 2017 (N = 35; Mean = 79%; Median = 89%)
Compliance with Service Guidelines: Although the behavioral health statewide average specific to compliance with service guidelines decreased from FY 2016 (90 percent) to FY 2017 (88 percent), this category demonstrated the most improvement in the number of providers increasing in score from review one to review two compared to any other. Of the reassessed providers, only eight (23 percent) had a decline in score from the time of the first FY2017 review. One provider remained at 100 percent for both reviews specific to compliance with service guidelines, and the remaining 26 (74 percent) demonstrated an increase ranging from one percentage point to 57 percentage points relative to their individual provider score between the two reviews. While 31 percent of providers scored at or below 80 percent at the time of their first review, only nine of the 35 providers (26 percent) fell below 80 percent for their second review score. Comparatively, only 26 percent of the reassessed providers scored above 90 percent at the time of their initial review while more than half (51 percent) reached or exceeded this score at the time of their second review.
60

Quality Management Annual Report FY 2017

Focused Outcome Areas: Providers who were reviewed for a second time in FY 2017 demonstrated improvement overall in FOA scores, 86 percent to 90 percent respectively by first FY 2017 to second FY 2017 review. Twenty-five providers, over 71 percent (n = 25), demonstrated improvement in this area from the time of the first FY 2017 review to the second FY 2017 review while the remaining 29 percent (n = 10) resulted in a decline ranging from three percentage points to 24 points. Personcentered practices showed the greatest impact amongst the providers declining in score and remains an area for growth and improvement.

CSU Reassessment Review Findings
Four of the 19 CSU providers had two reviews during FY 2017. This was a direct result of low overall scores during the review that occurred in FY 2016 (below 80 percent). Figure 20 provides the overall score result by CSU provider at the time of initial FY 2017 review and subsequent FY 2017 review. Of the four CSU providers reviewed for a second time in the fiscal year, the average overall score for these providers at the time of first FY 2017 review was 86 percent. This decreased by two points to a score of 84 percent at the second review. Two CSU providers demonstrated an increase in results at the time of second review in FY 2017, yet one remained below the 80 percent threshold and, thus, is expected to be reviewed for a second time in FY 2018.

Figure 20. CSU Overall Scores of Reassessed Providers

First 2017 Review Score Second 2017 Review Score 2016 Review Score

100% 90% 80% 70% 60% 50%

88% 90%
63% CSU 1

91% 81%
78%

91% 87% 72%

73% 77% 76%

CSU 2

CSU 3

CSU 4

61

Quality Management Annual Report FY 2017

Technical Assistance/Exit Conference
Upon completion of all BHQRs and CSUQRs, the lead assessor completed a formal exit conference. The exit conference supplied providers with tentative scores, provider strengths, and opportunities for growth. Providers received immediate, preliminary feedback of the BHQR and CSUQR findings at the time of the exit conference. Technical assistance was also provided during the exit conference and throughout the review process pertaining to opportunities for growth and areas of risk. The table below (Table 10) provides details on the technical assistance/exit conferences completed during FY 2017.

Table 10. Technical Assistance/Exit Conference Details for FY 2017

Fiscal Year 2017

Quarter 1 Quarter 2 Quarter 3 Quarter 4

Total Year End

Total BHQR Exits

45

40

35

29

149

Total Minutes

2,700

2,448

1,995

1,737

8,880

Total Attendees

356

306

217

207

1,086

Total CSU Exit Total CSU Minutes Total CSU Attendees

2

4

6

11

23

195

235

467

760

1,657

14

40

123

183

360

Total Exits

47

44

41

40

172

A total of 172 exit conferences were completed in FY 2017. The total number of attendees was more than 1,400 with conferences incorporating over 10,500 minutes. Exit conferences and technical assistance will continue to be provided throughout FY 2018.

62

Quality Management Annual Report FY 2017
Summary of Findings and Recommendations for Behavioral Health Providers
Quality Management continually reassesses processes to ensure we are capturing results and analyzing outcomes that give us the best information and truest picture of service quality in Georgia. This includes reviewing of feedback received from both providers and individuals, as well as revision of review tools as needed to ensure accurate and measurable reporting of results.
System Strengths
June 2017 marked the completion of the second year of the Quality Management contract. Quality assessors completed 167 BHQRs, an additional 18 ACT service reviews, and 23 CSUQRs in FY 2017. A total of 172 exit conferences were conducted across all review types. Exit conferences may have occurred jointly across the BHQR, CSUQR, and ACT reviews dependent on whether the provider rendered more than one of those services. Thirty-five providers were reviewed for a second time in FY 2017 due to initial FY 2016 low scores in overall, billing, or both. The overall score for all three review types was greater than 80 percent for the second year: BHQR 84 percent, ACT 87 percent, and CSU 86 percent (Table 11), and FOAs implemented in FY 2016 demonstrated improvement across all three review types in FY 2017. The overall score for BHQRs remained static at 84 percent for both FY 2016 and FY 2017. This is due to the lower scoring categories of FY 2016 increased in FY 2017 while the higher scoring categories of FY 2016 decreased in FY 2017.
63

Quality Management Annual Report FY 2017

Table 11. BHQR, ACT, CSU Overall Averages by Year

Billing

Validation

FOA

BHQR FY 2016

81%

85%

Assessment/ Planning
79%

Service Guidelines
90%

BHQR FY 2017

84%

89%

77%

88%

Overall 84%
84%

ACT FY 2016

92%

88%

85%

85%

88%

ACT FY 2017

90%

90%

80%

88%

87%

CSUQR FY 2016

NA

88%

IRR 78%

82%

83%

CSUQR FY 2017

NA

91%

IRR 80%

87%

86%

The table below, Table 12, summarizes the FOAs by review type and year. The overall score for the FOAs were above 80 percent both years; the CSUs scored close to 100 in FY 2017. Whole health, continually the lowest score for BHQR, has greatly improved from FY 2016 to FY 2017 for both CSU and behavioral health reviews. However, it remains an area for improvement across all providers providing behavioral health routine services. Both safety and person-centered practices, falling below 80 percent for CSU reviews, had a substantial increase to 98 and 99.5 percent, respectively. To summarize, areas suffering the greatest deficit in FY 2016 have all improved based on FY 2017 results across all review types, with CSUs demonstrating the largest impact with an increase in each subcategory to 98 percent or above.

64

Quality Management Annual Report FY 2017

Table 12. BHQR and CSUQR FOA Scores

Focused Outcome Areas

BHQR BHQR CSUQR FY 2016 FY 2017 FY 2016

Whole Health

63%

74%

83%

Safety

81%

83%

76%

Person Centered Practices

92%

91%

77%

Community

87%

93%

92%

Choice

92%

96%

99%

Rights

90%

93%

97%

Overall Score

85%

89%

88%

CSUQR FY 2017
98% 98% 99.5% 99% 99.6% 100%
99%

While reassessments supply lower-scoring providers the opportunity for improvement within the fiscal year, anticipation of increased scores are generally expected. Specific to reassessed providers, Table 13 reflects the number of providers who improved from the first FY 2017 review to the second review within the year. While assessment and treatment planning, as well as compliance with service guidelines, both decreased in statewide BHQR average from FY 2016 to FY 2017, each of these categories rendered results demonstrating the greatest improvement amongst the reassessed providers. Providers who obtained low FY 2016 overall or billing scores were scheduled for review and reassessment in FY 2017. While many of the reassessed providers demonstrated improvements on an individual level from first FY 2017 review to second FY 2017 review, those that continued to fall below the established thresholds will be scheduled for an additional two reviews in FY 2018. Additionally, providers are supplied with technical assistance during reviews to assist with performance improvement.

65

Quality Management Annual Report FY 2017

Table 13. Reassessment Review Results

Number of Providers Who Increased in Score from

FY 2017 Review #1 to FY 2017 Review #2

BHQR CSU

Category

(N = 35) (N = 4)

Billing

18 (51%)

N/A

IRR

N/A

2 (50%)

Compliance w/Service Guidelines 26 (74%) 0 (0%)

Focused Outcome Areas

25 (71%) 2 (50%)

Assessment & Treatment Planning 25 (71%)

N/A

Overall Scores

26 (74%) 2 (50%)

Opportunities for Improvement
The focus of billing validation is on compliance with standards for ensuring claims are justified in accordance with the DBHDD Provider Manual, which remains the second-lowest score of the four categories through FY 2017. In FY 2016, Quality Management launched a Quality Training Series with the first module being billing validation. Although the billing category remains second lowest, a three-point increase resulting in 84 percent for FY 2017 demonstrates improvement across the network. Thus, increasing the threshold to 79 percent from 69 percent for billing thresholds is recommended for FY 2018 specific to reassessment scheduling in anticipation of aligning with the key performance indicators in FY 2019. Billing compliance trends in FY 2018 will be used to determine whether additional training or technical assistance to providers is indicated.
While several providers remained consistent or demonstrated improvements in their results from FY 2016 to FY 2017, the areas of compliance with service guidelines, as well as assessment and treatment planning, both declined overall in results. A specific treatment planning training via in person and available via webinar was introduced in FY 2017 as part of the Quality Training Series. Additionally, Quality Management conducted a specific psychosocial rehabilitation versus case management training to aid in alleviating provider confusion specific to the services and their
66

Quality Management Annual Report FY 2017
requirements. Quality Management shall continue including topics related to lower scoring categories and questions within its annual Quality Training Series.
Recommendation I: Based on data analysis from both FY 2016 and FY 2017, modifications to and the addition of some questions or tools are anticipated to occur may provide additional information about quality as reviews begin in FY 2018. Some of these modifications and the tools affected include the following:
Discharge summary/note documentation includes: 1) Individuals' status at discharge, 2) Specific instructions for ongoing care including appointment date/time, 3) Living situation at the time of discharge, and 4) Date, time, and method of discharge. Must meet all factors for question answered as "yes" (Assessment and Treatment Planning).
Separation of a medical screening and the behavioral health assessment as two separate questions (Assessment and Treatment Planning).
The addition of opioid maintenance therapy as a measurable service (Compliance with Service Guidelines).
Question alteration to include annual diagnosis verification in the individual meeting admission criteria (Assessment and Treatment Planning).
Staff Interview additions and alterations such as the following: 1. Staff can identify the individual's warning signs and triggers prior to a crisis to promote early intervention. 2. Person-Centered Practices: Repeat Staff Interview: Staff can describe any new procedures the agency has adopted to encourage individual involvement in the development and redevelopment of IRPs no matter their age or perceived competency. 3. Community Life: Repeat Interview: Staff can describe how they have assisted the individual in accessing opportunities for community involvement in the past year (per individual preferences).
67

Quality Management Annual Report FY 2017
4. Staff has received training by the agency in providing whole-health informed services.
Recommendation II: Continue the Quality Training Series in FY 2018 based on data analysis from FY 2017, for all behavioral health and CSU agencies to include the following:
CSU Transition/ discharge planning CSU Service Guidelines focusing on medication procedures BHQR Compliance with Service Guidelines DBHDD Provider Manual Safety/Crisis Planning
Recommendation III: CSU review tool(s) should be incorporated into the electronic review system allowing for efficient and effective data gathering used to assess quality within the CSU programs.
Recommendation IV: Access to Addictionologists specific to compliance with service guidelines for CSU providers was scored based on verbal affirmation from the provider in FY 2016 and FY 2017. However, to ensure access to Addictionologists is in fact available to individuals, Quality Management is recommending the provider supply documented evidence to assessors during the review for FY 2018.
Recommendation V: Develop and implement an electronic quality of care process to collect, monitor, and report various issues to DBHDD, specific to provider environmental atmospheres, staffing, or quality of services are outside of the scope of the quality reviews.
Recommendation VI: Sample sizes for FY 2018 BHQRs will be based on provider size and individuals served. See the table below (Table 14) for provider size and sample size information. Providers with ACT and CSU will continue to have 15 individuals sampled and will have no change to the sampling method. Quality Management will oversample individual records for BHQRs and CSUQRs to ensure targeted sample sizes are met in FY 2018.
68

Quality Management Annual Report FY 2017

Table 14. Sample Size Determination for BHQR in FY 2018

Size of Provider
Small Medium
Large

Sample Size
5 - 10 20 30

# of Individuals Served (within six months)
< 50 51-100 > 101

Recommendation VII: Beginning in FY 2018, changes to the quality review procedure specific to voided/adjusted claims and encounters will be altered. During FY 2016 and FY 2017, assessors excluded claim(s)/encounter(s) that had been voided or adjusted prior to the beginning of the review. To prevent providers voiding extensive numbers of claims just prior to scheduled BHQRs, providers may no longer void or adjust any claims/encounters following their notification of BHQR/CSUQR. Providers may resume voiding and adjusting claims/encounters after their review has been completed. Voids/adjustments to claims appearing in the sample will be considered at the time of review only and must include the date adjusted.
Recommendation VIII: Although the billing category remains second lowest for the BHQR, a threepercentage point increase resulting in 84 percent for FY 2017 demonstrates improvement across the network. Thus, increasing the billing threshold for reassessment to 79 percent from 69 percent is recommended for FY 2018.

69

Quality Management Annual Report FY 2017
Section 4: Intellectual and Developmental Disabilities
Background
Person Centered Reviews (PCR) and Quality Enhancement Provider Reviews (QEPR) are used to assess the extent to which individuals with intellectual and developmental disabilities are satisfied with their services and achieve outcomes important to them, and to evaluate provider systems. The purpose of the PCR is to assess the individual's quality of life as well as the effectiveness of and the satisfaction individuals have with the service delivery system. The purpose of the QEPR is to review providers' systems and practices to ensure they meet requirements set forth by the Medicaid waiver and DBHDD, and to evaluate the effectiveness of their service delivery system. Follow-up review activities provide technical assistance to help providers improve service delivery systems through Quality Technical Assistance Consultation (QTAC).
Quality assessors use various tools to collect data from interviews, observations, and record reviews to compile a well-rounded picture of the individual's circle of supports, how involved the individual is in the decisions and plans developed for that individual, as well as the quality of services provided. Individuals sampled for the PCR or QEPR participate in the Individual Interview (II) and Individual Service Plan Quality Assurance Checklist (ISP QA). Both review processes also include a Provider Record Review (PRR) and the Developmental Disability Service Specific (DDSS) requirements for each service received by the individual, a Staff Interview (SI) with a sample of direct support providers, and onsite Observation of day or residential programs.
During the PCR, the Support Coordinator Record Review (SCRR) and Support Coordinator Interview (SCI) tools are completed for the support coordinator working with the individual. During the QEPR, each provider organization receives one administrative review to monitor compliance with requirements through the Qualifications and Training (Q&T) component of the review. The Q&T includes a review of a sample of personnel/staff records to determine if staff has the necessary
70

Quality Management Annual Report FY 2017
qualifications specific to services rendered, and whether required training was received within specified timeframes.
In this section of the report, results for both the QEPR and PCR are presented by each of these review tools. Results are also presented by Focused Outcome Area (FOA), as presented in the BHQR and CSU sections. The overall average scores for intellectual and developmental disability reviews are a weighted average, based on the total number of standards scored [total met / (total met + total not met)]. Except for the Q&T and DDSS tools, indicators within each tool are grouped into six FOAs, areas of the individual's life important to achieve and maintain:
Whole Health--individuals are healthy, aware of their health-related needs, and direct their own health care regimen
Safety--individuals are safe in their home and work environments and in their communities; they understand or are learning how to self-preserve in all environments
Person-Centered Practices--supports and services are being provided based on the individual's preferences and direction
Community Life--individuals are actively participating and developing social roles in their communities as desired
Choice--individuals have information they need to make informed choices on life decisions, such as where to live, where to work, and which supports, services and providers to use
Rights--individual rights are upheld and information and education is provided to ensure understanding of their rights
Sampling Method
Person-Centered Review (PCR)
The PCR used a random sample of 481 individuals receiving services from providers selected for a QEPR. The PCR sample was stratified by region and sampled proportionate to the regions. Therefore, the number of PCRs per region are proportionate to the number of individuals receiving
71

Quality Management Annual Report FY 2017
services within the region. The PCR sample was selected from individuals, age 18 and over, who had not received a PCR during the previous year and were receiving services. Table 15 shows the number and percent of individuals receiving services across the state, as well as the number of PCRs completed within each region.

Table 15. State Population and PCR Sample

Region
1 2 3 4 5 6 Total

Population

N

%

2,750

20.5%

2,252

16.8%

3,436

25.6%

1,415

10.5%

1,726

12.8%

1,853

13.8%

13,342 100.0%

PCR Sample

N

%

96

20.0%

90

18.7%

122

25.3%

51

10.6%

63

13.1%

59

12.3%

481

100.0%

Quality Enhancement Provider Review (QEPR)
During FY 2017, a stratified random sample of 93 providers who did not receive a QEPR in FY 2016 and rendered services to eligible individuals was selected for a QEPR.9 10 Based on the number of eligible individuals, providers were stratified into three categories by size: "Large," "Medium," and "Small." One support coordination agency and one crisis services provider were randomly selected for review as well. In addition, DBHDD selected five additional providers for review, including three crisis providers that resulted in a sample size of 100 providers (Table 16). The number of providers by size is listed in Table 16 for both FY 2016 and FY 2107.

9Providers offering ineligible services at the time of sample selection were excluded from the population. Ineligible services include: environmental accessibility adaptation, financial support services, Georgia Crises Response System (GCRS), hospital residential services, individual directed goods and services, natural support training services, specialized medical equipment, specialized medical supplies, transition services, and vehicle adaptations. 10 Adults who were receiving review-eligible services at the time of sample selection and who were not reviewed as part of a PCR in FY 2016 were considered eligible for review in FY 2017.
72

Quality Management Annual Report FY 2017

Table 16. QEPR Provider Sample by Size

FY 2016

FY 2017

Provider Size

Number

Small (caseload 30)

46

52

Medium (30 < caseload < 100)

36

19

Large (caseload 100)

16

24

Support Coordination Agency

1

1

Crisis Services

1

4

Total

100

100

Review Processes
Onsite Reviews (PCR and QEPR)
The focus of the PCR is on the individual's quality of life and quality of services received. The focus of the QEPR is on the provider's overall practices, quality of services offered to all individuals served, and level of compliance with Medicaid waiver and state requirements. Both the PCR and QEPR use the individual observation staff assessment (IOSA), which includes an interview with individuals, interviews with their staff and onsite observation(s) (OBS) at residential and day programs, as applicable. In addition to the IOSA, the PCR and QEPR include an evaluation of the Individual Service Plan (ISP QA), a review of the provider's records, as well as compliance with service delivery requirements using the DDSS review tool. The number of PRR and DDSS reviews completed depends upon the number of services received by the individual.
While the PCR and QEPR share most of the same tools, there are a few exceptions. The PCR also includes an interview with the individual's Support Coordinator and a review of the record in the consumer information system (CIS) maintained by the support coordinator for the individual. The QEPR has an additional review tool, qualifications and training, which is used to review a sample of records from all staff ensuring required training and other state requirements are current and

73

Quality Management Annual Report FY 2017
documented (e.g., background screenings, level of education). The total number of records or interviews completed this year for the PCR and QEPR is listed in Table 17, for each review tool.11

Table 17. Number of Records by Review Tool and Review Type

Review Tool

PCR N

QEPR N

Total

IOSA - Individual Interview

481

440

921

IOSA - Observation

225

473

698

IOSA - Staff Interview

304

398

702

ISPQA Checklist

481

440

921

Support Coordinator Interview (SCI)

481

3

484

Support Coordinator Record Review (SCRR)

481

20

501

Provider Record Review (PRR)

894

1,356 2,250

Staff Qualifications and Training (Q&T)

N/A

947

947

DDSS - Behavioral Supports Consultation

0

11

11

DDSS - Community Access (Group)

80

610

690

DDSS - Community Access (Individual)

12

193

205

DDSS - Community Living Support

21

129

150

DDSS - Community Residential

60

313

373

DDSS - Crisis

NA

46

46

DDSS - Occupational Therapy

0

12

12

DDSS - Physical Therapy

0

8

8

DDSS - Prevocational

10

138

148

DDSS - Respite

0

11

11

DDSS - Support Coordination

481

36

517

Total Number of PCRs/QEPRs

481

96

577

When a PCR is completed, a report is provided that identifies the strengths of the individual's team. It evaluates the supports and services provided and makes recommendations for the individual and

11 To alleviate administrative burden on providers, PCRs were completed at the same time as the QEPR. Beginning in Year 2, the observations and staff interviews completed as part of the PCR were attached to the QEPR. As a result, the number completed for the PCR appears to be lower; however, these PCR review components are incorporated into the QEPR.
74

Quality Management Annual Report FY 2017
the support team (including the support coordinator, provider, and family). A provider who participates in a QEPR receives a comprehensive report that identifies strengths of the service delivery systems, recommendations for improvement, and several performance scores. These scores include the overall score, qualifications and training score, and DDSS score.
Quality Technical Assistance Consultation (QTAC)
The QTAC is an additional review process that may be conducted 90 days after completion of the QEPR. This additional review is based on any service concerns identified during the PCR or QEPR, or if the provider requests technical assistance. Using findings from the QEPR, technical assistance is provided to support providers and to offer suggestions and guidance to help improve their service delivery systems. The process used a consultative approach to address specific issues and concerns related to an individual receiving services or systems and practices needing improvement. The QTAC supplements the PCR and QEPR processes by affording contracted providers the opportunity to solicit technical assistance for specific needs within the service delivery milieu.
Person-Centered Review
PCR Scores by Tool
Figure 21 shows the average score for each tool used during the PCR, comparing FY 2016 and FY 2017. Findings each year show a similar pattern, with scores for interviews with individuals, staff, and support coordinators higher than scores for provider or support coordinator documentation (record reviews). Scores across all tools have decreased somewhat, with the greatest differences from Year 1 to Year 2 in the provider record review (down 8.8 points), the support coordinator interview (down 7.5 points) and support coordinator record review (down 6.2 points).12
12 All three changes were statistically significant at p<.001. However, the test was completed with large denominators, based on all the indicators scored.
75

Quality Management Annual Report FY 2017

Figure 21. PCR Scores by Tool and Year

100% 90% 80% 70%

95.1% 91.9%

98.4%96.8%

96.4% 94.3%

79.0%

79.9%

70.2%

73.7%

90.8% 83.3%

60%

50%

Individual Observation

Staff

Interview

Interview

FY 2016 (N = 484)

Provider Record Review

Support

Support

Coordinator Coordinator

RR

Interview

FY 2017 (N = 481)

Declines in the PRR, SCRR, and SCI scores were driven by decreases on some indicators within certain FOAs.
The PRR decline was attributed mostly to the FOA choice, for which scores fell 18.9 points, from 71.6 percent in FY 2016 to 53 percent in FY 2017. The greatest differences were related to a lack of documentation demonstrating how individuals were provided choices regarding their living environments and living situations.
The greatest impact for the SCRR was within the rights FOA, for which scores declined 19.9 points, from 73.4 percent to 53.5 percent. Differences were primarily related to documentation regarding the review of: o Individuals' rights and responsibilities o The complaints/grievances policy o The HIPAA privacy and security rules o Informed consent from individuals for taking psychotropic medications
76

Quality Management Annual Report FY 2017
The SCI tool also saw a decline within the rights FOA, for which scores fell 10.2 points, from 86.3 percent to 76.3 percent, and on some safety standards. There was a statistically significant decline in the percent of support coordinators who were aware of:
Individuals' restrictive interventions How to address identified rights restrictions with the individual The individuals' triggers related to behavioral health (i.e., behavioral issues,
decompensation, or relapse) Individualized techniques for implementing behavior, safety and emergency plans, or if the
individual has one of these plans when needed
PCR Scores by Focused Outcome Area (FOA)
All the PCR tools (II, SI, OBS, PRR, SCI, and SCRR) are designed to measure the six basic FOAs, with many indicators within each FOA. Results for each FOA, based on all six tools, are shown in figure 22, by year. Scores have decreased in each area, particularly for Choice (down 12 points), Community Life (down 9.5 points), and Whole Health (down 8.9 points).
77

Quality Management Annual Report FY 2017

100% 90% 80% 70%

Figure 22. PCR Scores by FOA and Year

93.6% 84.7%

96.5% 90.3%

89.1% 82.6% 80.6%

90.7%

95.4% 90.2%

78.7%

71.1%

60%

50% Whole Health

Safety

Person Centered Practices

Community Life

Choice

FY 2016 (N = 484) FY 2017 (N = 481)

Rights

PCR Scores by Tool and Focused Outcome Area (FOA)
In this section, PCR results for FY 2017 are presented by FOA and tool (Figure 23). Findings for the review components varied across each FOA and indicate the following:
Service provider documentation was the lowest scoring component across all FOAs, except for Rights (82.8 percent). Conversely, Support Coordinators' documentation was the lowest scoring component in Rights (53.5 percent), while they performed best documenting standards in Whole Health (93.7 percent) and Safety (92.5 percent).
Both Support Coordinators and service providers were least likely to be in compliance with areas of Community Life and Choice.
Interview findings were at least 75 percent or higher, across all FOAs.13

13 Beginning in FY 2017, staff interviews were only conducted for staff who offer services to the individual but are not connected to the provider who is participating in a QEPR. Therefore, only the individual's "B" providers who were not receiving a QEPR were interviewed. This is a different process than in FY 2016 so comparisons to that year should be made with caution.
78

Quality Management Annual Report FY 2017
Individual and Support Coordinator interview responses were similar regarding Community Life, Whole Health and Person Centered Practices but lower than information obtained from staff or the onsite observations.
Figure 23. FY 2017 PCR Scores by Tool and FOA
Opportunities for Growth by FOA
Each PCR tool is organized around the FOAs and each FOA uses several standards, or indicators, to measure how well the area is being addressed. As indicated in this report, most findings are positive, with relatively high scores across all perspectives of the PCR, i.e., interviews, observations and documentation areas all averaging over 70 percent. However, there are specific areas within
79

Quality Management Annual Report FY 2017
each FOA where indicator level scoring is relatively low or has decreased significantly since FY 2016. These are presented in this section by FOA.
Whole Health The average Whole Health score in FY 2017 was 84.7 percent, down from 93.6 percent in FY 2016. An important aspect of each individual's life is the use of various types of medications, particularly psychotropic and anticonvulsant medication. Indicators across several review components suggest a statistically significant decrease in some areas of medication administration, particularly surrounding aspects of education. Some of the lower scoring indicators also decreased since FY 2016 (Table 18).

Table 18. Low Scoring Whole Health Indicators (PCR)

Indicator
Individual Interview The individual was often not aware of:
What medications he/she is taking Why medications are prescribed The side effects of the medications Staff Interview Staff could not always describe: How to support the individual to learn about medications The side effects of medications taken Provider Record Review Documentation did not demonstrate how: Education is provided to the individual on the risks and side effects of the medication
Individuals and families are provided education on all prescribed medications
* Areas highlighted in blue within table reflect significance at a p < .05.

FY 2016

FY 2017

78.3% (n = 428) 86.0% (n= 351) 74.3% (n = 350)

68.3% (n = 435) 68.5% (n= 410) 52.4% (n = 410)

81.0% (n = 357) 86.3% (n = 388)

67.7% (n = 195) 78.7% (n = 211)

60.5% (n = 332) 33.9% (n = 610) 56.6% (n = 327) 25.7% (n = 723)

In addition to issues surrounding education about medications, data indicated providers often did not have the individual's preventive health care report in the record for the male and female preventive screening(s): mammogram, bone density, hearing evaluation/supports, wheelchair
80

Quality Management Annual Report FY 2017
evaluation, vision evaluation/supports, or dental care - each of these areas showing compliance of less than 50 percent.

Safety Scores were relatively high in areas surrounding Safety, an average of 90.3 percent in FY 2017, down just over six points from 96.5 percent in FY 2016. Some safety indicators showed relatively low scores for FY 2017, and some decreased significantly since FY 2016. Most providers were not documenting how they address abuse, neglect and exploitation or offer education on how individuals can self-preserve. Support Coordinators were not always aware of restrictive interventions, needed behavior or crisis plans, and some triggers related to behavior health.

Table 19. Low Scoring Safety Indicators(PCR)

Indicator Provider Record Review Documentation did not demonstrate:

FY 2016

Ongoing evidence of identifying, addressing, and seeking prevention of abuse, neglect, and exploitation How they offer education to the individual on how to self-preserve or develop effective resiliency skills according to the individual's learning style

53.7% (n = 499) 52.5% (n = 549)

Support Coordinator Interview Support Coordinator was not aware:

Of restrictive interventions/plans in place Of needed crisis plans when the individual had used Georgia Crisis Response services
If the individual had a behavior plan Of the individualized techniques for following the behavior plan
Of specific triggers related to behavior health issues:
Decompensation Relapse
* Areas highlighted in blue within table reflect significance at a p < .05.

87.8% (n= 181) 84.7% (n= 118) 83.2% (n= 143) 79.5% (n = 132)
87.4% (n = 174) 88.9% (n = 180)

FY 2017
22.9% (n = 893) 41.1% (n = 893)
47.8% (n = 92) 53.6% (n= 97) 56.2% (n= 105) 55.8% (n = 104)
58.2% (n = 117) 60.5% (n = 114)

81

Quality Management Annual Report FY 2017
Person Centered Practices Person Centered Practices showed an average score of 82.6 percent, down from 89.1 percent in FY 2016. This is not the lowest scoring FOA, but indicators accounting for much of the change showed a 10 to 20-point decrease since FY 2016 (Table 20). Individuals were not as likely to identify new experiences or something new they had learned, and providers were less likely to document progress toward goals or the unique strengths and talents of people they serve.

Table 20. Low Scoring Person-Centered Practices Indicators(PCR)

Indicator

FY 2016

FY 2017

Individual Interview The individual was often unable to identify something, in the past 6 months:

New or experienced

70.4% (n = 479) 60.3% (n = 471)

That had been learned

78.8% (n = 480) 65.8% (n = 474)

Provider Record Review Documentation did not reflect:

How providers regularly review, with the individual, progress toward and benefit of goals

51.4% (n = 552) 40.4% (n = 891)

The individual's talents

48.9% (n = 552) 38.9% (n = 894)

The individual's hopes and dreams

53.3% (n = 552) 42.0% (n = 893)

The individual's strengths

57.0% (n = 553) 48.9% (n = 894)

Support Coordinator Record Review Documentation did not always demonstrate:

How support coordinators regularly review, with the individual, progress toward and benefit of goals

52.0% (n = 483) 56.1% (n = 472)

Progress notes that describe progress toward goals including the individual's response to the intervention or activity, based on data

63.9% (n = 482) 57.8% (n = 481)

The individual's talents

49.2% (n = 480) 28.8% (n = 479)

The individual's strengths

56.4% (n = 479) 41.0% (n = 480)

* Areas highlighted in blue within table reflect significance at a p < .05.

Community Life Community Life, the degree to which individuals were interacting with and integrated in their surrounding community, showed the lowest average score among the FOAs for both FY 2016 and FY

82

Quality Management Annual Report FY 2017
2017 - 80.6 percent and 71.1 percent respectively. Information from the face-to-face interviews is shown in Table 21. Findings indicate from the three different interview perspectives (individual, support coordinator, staff), individuals were often not developing, or being supported to develop, valued social roles or exploring community employment options.

Table 21. Low Scoring Community Life Interview Indicators (PCR)

Indicator

FY 2016

FY 2017

Individual Interview The individual was often not:

A member of community clubs (i.e., athletic, arts/craft, photography, YMCA), as desired

57.7% (n = 414) 34.0% (n = 400)

A member of neighborhood associations, as desired

65.3% (n = 236) 46.4% (n = 140)

Exposed to new community activities (in the past 6 months)

71.4% (n = 472) 63.2% (n = 473)

Able to describe any new community experiences

77.8% (n = 472 69.1% (n = 475)

Provided opportunities to develop new social roles

75.4% (n = 476) 69.4% (n = 477)

Provided opportunities to learn about social roles in the community

81.1% (n = 476)

71.5% (n = 478)

Actively pursuing preferences related to goals of employment (not a readiness model)

82.4% (n = 335) 70.4% (n = 338)

Support Coordinator Interview The Support Coordinator often was not:

Aware of the individual's new community experiences 73.0% (n = 467) 62.0% (n = 471)

Aware of how the individual is provided opportunities to develop new valued social roles in the community

71.2% (n = 463) 66.9% (n = 475)

Able to define or explain how the individual is supported to uphold valued social roles

79.5% (n = 472) 72.6% (n = 475)

Staff Interview Staff was unable to describe how the individual is provided opportunities to:

Develop new social roles

59.3% (n = 791) 60.4% (n = 298)

Develop community employment

71.4% (n = 795) 58.3% (n = 235)

* Areas highlighted in blue within table reflect significance at a p < .05.

Documentation compliance from Support Coordinator and Provider Record Reviews indicates most providers, both service providers and Support Coordinators, were not documenting how they

83

Quality Management Annual Report FY 2017
support individuals to develop social roles, seek employment, or how individuals participate in the community. Documentation results for Community Life, from record reviews, are shown in Table 22.

Table 22. Low Scoring Community Life Record Review Indicators (PCR)

Indicator

FY 2016

FY 2017

Documentation did not demonstrate: How the individual is supported to learn about, explore and experience the community Support Coordinator Record Review Provider Record Review

48.7% (n = 476) 44.3% (n = 532)

39.0% (n = 480) 39.5% (n = 881)

How the individual is supported to have or has responsibilities in the community as desired
Support Coordinator Record Review
Provider Record Review
How the individual is supported to/able to participate in community activities and employment the same as individuals without disabilities

58.5% (n = 480) 62.8% (n = 530)

59.5% (n = 477) 42.8% (n = 883)

Support Coordinator Record Review Provider Record Review Development of social roles and natural supports that reflect the individual's interests Support Coordinator Record Review Provider Record Review

76.9% (n = 455) 79.1% (n = 460)

62.2% (n = 458) 40.4% (n = 854)

48.5% (n = 480) 42.9% (n = 532)

46.4% (n = 481) 25.4% (n = 881)

Opportunities to seek employment in competitive integrated settings

Support Coordinator Record Review Provider Record Review

48.2% (n = 363) 55.2% (n = 364)

41.4% (n = 394) 24.9% (n = 794)

* Areas highlighted in blue within table reflect significance at a p < .05.

Choice Choice is the second-lowest scoring FOA showing an average score of 78.7 percent for FY 2017. Compared to last year's score of 90.7 percent, this FOA had the greatest decrease a 12-point drop. Scores within this FOA are most negatively affected by 10 to 50-point decreases on standards from the provider and Support Coordinator Record Reviews. Findings showed a lack of documentation
84

Quality Management Annual Report FY 2017
for how individuals were making meaningful and informed choices in different aspects of their lives, including living environments, living situations, competitive employment and community participation.

Table 23. Low Scoring Choice Indicators (PCR)

Indicator

FY 2016

FY 2017

Provider Record Review Providers did not always demonstrate through documentation how individuals:

Make informed choices about competitive or supported employment options

54.6% (n = 379)

30.8% (n = 827)

Make informed choices about or options to change living environments

76.7% (n = 257) 23.0% (n = 708)

Are provided a choice of living situations

74.9% (n = 267) 26.4% (n = 708)

Are provided with information to make informed choices (education, exploration, and experiences)

40.7% (n = 548) 27.4% (n = 891)

Make informed choices about community participation and social interaction

54.8% (n = 540) 42.8% (n= 888)

Support Coordinator Record Review Support Coordinators did not always demonstrate through documentation how individuals:

Are offered employment or educational options

55.4% (n = 401) 49.3% (n = 423)

Are provided choices of living situations

75.0% (n = 324) 47.1% (n = 433)

Make meaningful choices about community participation

56.5% (n = 480) 50.3% (n = 441)

Are provided with education, exploration and experiences
*Areas highlighted in blue within table reflect significance at a p < .05.

41.9% (n = 482) 31.2% (n = 481)

Rights Similar to Safety, scores surrounding Rights were relatively high, showing an average score of 90.2 percent in FY 2017 -- down approximately five points since FY 2016 (95.4 percent met). While the overall score for Rights remains relatively high, up to 40-point decreases were seen on standards within the Support Coordinator Interview and record review tools. Interview results showed a 28point decrease in the percent of Support Coordinators who address identified rights restrictions

85

Quality Management Annual Report FY 2017
(90.6 percent to 62.0 percent met) and a 40-point decrease in the percent of coordinators who are aware of the individual's restrictive interventions/plans (87.8 percent to 47.8 percent met).
Support Coordinator documentation did not always show evidence of a signed informed consent form for psychotropic medications or of a signature on the notification of rights and responsibilities. Documentation was much less likely to show evidence the complaints/grievance policy was being shared with the individual /guardian in a manner accommodating the individual's learning style.

Table 24. Low Scoring Rights Indicators (PCR)

Indicator

FY 2016

Support Coordinator Interview Support Coordinators were often not:

Aware of restrictive interventions/plans in place 87.8% (n = 181)

Addressing identified rights restrictions

90.6% (n = 223)

Support Coordinator Record Review Support Coordinators did not always demonstrate evidence:

The individual or legal guardian's signature on notification that all individuals are informed about their rights and responsibilities at least annually

58.4% (n = 473)

HIPAA privacy and security rules are specifically reviewed with individuals

72.1% (n = 480)

Of an informed consent with the individual/guardian signature for taking psychotropic medications prescribed by a psychiatrist or psychiatric nurse practitioner

49.1% (n = 281)

The complaints/ grievance policy is being shared with the individual/guardian in a manner accommodating the individual's learning style

61.5% (n = 478)

*Areas highlighted in blue within table reflect significance at a p < .05.

FY 2017 47.8% (n = 92) 62.0% (n = 129)
22.7% (n = 476) 55.2% (n = 478) 35.5% (n = 251) 34.7% (n = 479)

86

Quality Management Annual Report FY 2017
Individual Service Plan Quality Assurance Checklist (ISP QA)
During the PCR, assessors review the content of the individual's plan to assess compliance with requirements and the extent to which the plan specifically addresses the individual's needs, goals, and desires.
ISP Written to Support a Meaningful Life The ISP QA checklist provides an overall rating for each service plan, based upon the degree to which the ISP is written to provide a meaningful life for the individual receiving services. There are three different categories for each ISP.
Service Life: The ISP supports a life with basic paid services and paid supports. Needs "important for" the individual are addressed, such as health and safety. However, there is not an organized effort to support an individual in obtaining other expressed desires "important to" the individual, such as getting a driver's license, having a home, or acting in a play. The individual is not connected to the community and has not developed social roles, but expresses a desire to do so.
Good but Paid Life: The ISP supports a life with connections to various supports and services (paid and non-paid). Expressed goals "important to" the individual are present, indicating the individual is obtaining goals and desires beyond basic health and safety needs. The individual may go out into the community but with only limited integration into community activities. For example, the individual may go to church but not have the opportunity to participate in Sunday school or the choir. Community connections are lacking, and the individual indicates a desire to achieve more.
Community Life: The ISP supports a life with the desired level of integration in the community and in various settings preferred by the individual. The individual has friends and support beyond providers and family members. The individual has developed meaningful social roles, such as belonging to a Red Hat club or a book club, or having employment in a competitive rather than segregated environment. Rather than just going to church, the individual may be an usher at the church or sing in the choir. Relationships developed in the community are reciprocal. The ISP is
87

Quality Management Annual Report FY 2017
written with goals that help support the individual in moving toward a community life, as she or he chooses.
Overall findings are presented in Figure 24 and show very similar results for FY 2016 and FY 2017. A majority of ISPs reviewed each year was written to support a "Good but Paid Life." The smallest percent supported a service life, while 16.6 percent (FY 2016) to 17.3 percent (FY 2017) supported a community life. The ISP results support other findings in the report there is opportunity for improvement to achieve more integration in the community or in various settings.

100% 75% 50% 25% 0%

Figure 24. PCR ISP QA Life Question by Year
73.8% 70.8%

9.6% 11.9%

16.6% 17.3%

Service Life

A Good but Paid Life

Community Life

FY 2016 (N = 481) FY 2017 (N = 480)

Findings by region and year are presented in Table 25. While N sizes are relatively small, each region had over 50 ISPs reviewed. Differences between years were relatively small, but those with statistical significance of p <=.05 are shaded in blue in the table below. Compared to FY 2016, in FY 2017 a significantly greater proportion of ISPs in Region 5 supported a Service Life while a significantly greater proportion in Region 2 supported a community life.

88

Quality Management Annual Report FY 2017

Table 25. ISP QA Life Question by Region and Year

Region

Service Life

FY 2016

FY 2017

Good but Paid Life FY 2016 FY 2017

Community Life

FY

2016

FY 2017

1

5.3%

6.3%

58.5% 55.8% 36.2%

37.9%

2

12.6%

10.0%

82.8% 75.6% 4.6%

14.4%

3

9.1%

8.2%

73.6% 74.6% 17.4%

17.2%

4

5.6%

9.8%

83.3% 86.3% 11.1%

3.9%

5

5.3%

20.6%

78.9% 68.3% 15.8%

11.1%

6

19.1%

23.7%

72.1%

*Areas highlighted in blue within table reflect significance at a p < .05.

69.5% 8.8%

6.8%

ISP Expectations
Quality assessors reviewed 12 different sections in the ISP. Each section represents an expectation, listed in Table 26, and has four indicators that must be addressed in the plan. The expectation is rated on a scale from zero to four, zero meaning the section is blank or did not address the requirements for any of the indicators and four meaning 100 percent of the indicators or requirements in the section are addressed in the ISP.
Data in Table 26 show the percent of ISPs at each rating on the 12 different expectations for FY 2017. For example, 52.0 percent of ISPs reviewed in FY 2017 had all four indicators present (addressed) for the expectation regarding the communication chart. Data indicate:
58.7 percent of all ISPs reviewed addressed all elements in each section. On average, over 85 percent of ISPs reviewed had three or four indicators present. Expectations regarding the rights, psychotropic medications, or behavior supports and the
health and safety sections showed the highest rates. All four indicators present in 94.2 percent of ISPs.
89

Quality Management Annual Report FY 2017
The ISP expectations measure how well the individual's hopes and dreams are addressed and whether goals are person-centered. These measures were most likely to have zero or one indicator addressed, 17.9 percent and 15.3 percent respectively.

Table 26. ISP QA Checklist Ratings by Expectation (FY 2017; N = 481)
Ratings

ISP QA Checklist Description

0

1

2

3

4

Relationship Map & discussion on ways to develop relationships
Communication Chart Person-centered Important to/For Hopes and Dreams Service Summary Rights, Psychotropic Medications, Behavior Supports Section Meeting Minutes SIS completed and support needs are addressed in the ISP Health and Safety Review Section completed accurately and thoroughly Goals are Person Centered Training Goal Action Plan Action Plans/Objectives
Total

0.8%
0.8% 0.8% 13.1% 1.9%
0.8%
2.7%
0.8%
0.8%
6.2% 1.5% 0.8% 2.6%

2.3%
1.7% 0.2% 4.8% 4.2%
0.0%
6.0%
0.6%
0.0%
9.1% 1.0% 1.0% 2.6%

12.5%
5.0% 2.9% 8.3% 13.9%
0.2%
16.2%
1.5%

36.0%
40.5% 28.1% 18.5% 25.8%
4.8%
21.2%
32.8%

48.4%
52.0% 68.0% 55.3% 54.3%
94.2%
53.8%
64.2%

0.4%
19.3% 10.8% 22.5% 9.5%

4.6%
21.4% 50.5% 36.2% 26.7%

94.2%
43.9% 36.2% 39.5% 58.7%

Average Rating
3.3
3.4 3.6 3.0 3.3
3.9
3.2
3.6
3.9
2.9 3.2 3.1 3.4

Table 27 shows the average rating (0 4) by expectation for FY 2016 and FY 2017. The average rating for both years was 3.4 and showed very little change by year for each expectation. Expectations measuring how well the individual's hopes and dreams are addressed and whether goals are person centered were the lowest rated expectations both years. Expectations regarding the rights, psychotropic medications, or behavior supports and the health and safety sections showed the highest average ratings, 3.9 for FY 2016 and FY 2017.

90

Quality Management Annual Report FY 2017

Table 27. Average Rating by Expectation (0 - 4)

FY 2016

ISP QA Checklist

(N = 484)

Relationship Map & discussion on ways to develop relationships

3.4

Communication Chart

3.5

Person-centered Important to/For

3.7

Hopes and Dreams

3.0

Service Summary

3.2

Rights, Psychotropic Medications, Behavior Supports Section

3.9

Meeting Minutes

3.2

SIS completed and support needs are addressed in the ISP

3.6

Health and Safety Review Section completed accurately and thoroughly

3.9

Goals are Person Centered

3.0

Training Goal Action Plan

3.2

Action Plans/Objectives

3.2

Overall Average Rating

3.4

FY 2017 (N = 481)
3.3 3.4 3.6 3.0 3.3 3.9 3.2 3.6
3.9
2.9 3.2 3.1 3.4

PCR Results by Service
During the PCR, a record review is conducted with every provider who offers eligible services to the individual. Information in Table 28 shows results for record reviews completed during the PCRs, by service and year. The N is the number of records reviewed for each service and the percent met is based on the total number of indicators reviewed.
The decline in PCR scores, from FY 2016 to FY 2017, was seen across all services as well. Among the services with at least 45 records reviewed:
With the exception of Support Coordination, Supported Employment showed the least amount of change over the two-year period, with the highest score in FY 2017 (82.3 percent) and close to the highest score in FY 2016 (77.8 percent).
Community Living Supports showed the greatest decline, close to 17 points. Prevocational service scores dropped by 11 points and with the exception of Respite (only
one record reviewed) showed the lowest score in FY 2017.
91

Quality Management Annual Report FY 2017

Table 28. Record Review Results by Service and Year

FY 2016

FY 2017

Service

N

% Met

N

% Met

Provider Record Review

558

79.0%

894

70.2%

Behavioral Supports

-

-

2

91.7%

Community Access(Group)

233

77.1%

378

67.9%

Community Access(Individual)

60

79.7%

95

72.9%

Community Living Supports

47

83.8%

72

66.8%

Community Residential

91

80.4%

195

73.4%

Prevocational

70

75.9%

74

64.9%

Respite

1

87.5%

6

61.8%

Supported Employment

54

82.3%

71

77.8%

Transportation

2

84.7%

-

-

Nursing Services

-

-

1

87.8%

Support Coordination Record Review

484

79.9%

481

73.7%

The following table displays scores by FOA for the services reviewed this year during the PCR. Providers of respite (n = 6) scored relatively low in areas of Whole Health, Community Life and Choice, and Prevocational provider records also showed relatively low scores in Community Life and Choice. Support Coordination was the lowest scoring service area relative to Rights.

92

Quality Management Annual Report FY 2017

Table 29. FY 2017 PCR Record Review Results by FOA and Service

Service

Whole Health Safety PCP

Com

Averag

Life Choice Rights

e

Behavioral Supports (N=2) Community Access (Grp) (N=378) Community Access (Ind) (N=95) Community Living (N=72) Community Residential (N=195) Prevocational (N=74) Respite (N=6) Supported Employment (N=71) Nursing Services (N=1) Support Coordination (N=481)

81.3% 59.5% 63.6% 66.0% 83.9% 56.1% 47.8% 53.3% 100.0% 93.7%

96.0% 75.2% 75.9% 71.5% 76.5% 76.1% 73.1% 79.6% 90.9% 92.5%

96.9% 75.1% 80.9% 70.5% 72.6% 73.2% 81.6% 83.4% 76.5% 78.6%

100.0% 61.8% 69.9% 59.9% 62.5% 53.3% 45.5% 86.3% 100.0% 63.5%

93.3% 50.0% 59.8% 52.8% 51.4% 45.3% 46.2% 72.4% 100.0% 65.1%

80.0% 84.2% 85.9% 77.0% 79.9% 84.4% 70.8% 90.2% 69.2% 53.5%

91.7% 67.9% 72.9% 66.8% 73.4% 64.9% 61.8% 77.8% 87.8% 73.7%

PCR Strengths and Recommendations
During each PCR, assessors identify strengths about services offered to the individual and provide recommendations to help improve services and the individual's quality of life. Displayed in Table 30 and 31 display strengths and recommendations identified in at least 40 percent of the PCRs completed in FY 2017. Staff strengths most often cited include an awareness of the individual's unique safety needs, achievements, and how the individual communicates choices. The individual felt valued in approximately 70 percent of PCRs completed. Recommendations include using web resources to assist with rights education, community life, informed choice, and safety profiles.

93

Quality Management Annual Report FY 2017

Table 30. Strengths Most Often Identified During a PCR

Strength

Times Noted

Staff is aware of the unique safety needs of the individual.

383

Staff acknowledges the individual's achievements.

341

The individual feels valued.

336

Staff has a clear understanding how the individual communicates choice making in everyday decisions.

324

Staff is knowledgeable of and provided examples of how an

individual's preferences for exercising rights are actively being

238

supported.

The individual is aware of how to self-preserve in all settings.

205

Staff consistently promotes independence.

197

Staff demonstrates the use of person-centered values and approaches in everyday interactions with the individual.

194

Staff is aware of the addendum process and when a request can be made.

193

Percent of PCRs 79.6% 70.9% 69.9% 67.4%
49.5%
42.6% 41.0% 40.3%
40.1%

Table 31. Recommendations Most Often Identified During a PCR

Recommendation

Times Percent Noted of PCRs

Utilize a video to educate individuals on Rights at http://www.youthforhumanrights.org.

253 52.6%

Help foster learning that shares knowledge relating to a community

life using The International Learning Community website,

239 49.7%

http://www.learningcommunity.us.

Conduct "what if" scenarios to determine the individual's skills in various safety situations.

233 48.4%

Support the individual to explore other potential dreams. This can be done by using the 3 E's, education, exposure, and experience.

219 45.5%

Provide guidance in making informed choices using the following

website:

217 45.1%

http://mn.gov/mnddc/extra/publications/choice/Its_My_Choice.pdf

Creates a safety profile for residential settings for first responders using the http://www.smart911.com website.

213 44.3%

Ensure daily schedules and activities promote exposure to new things ('new places and new faces') and are not stagnant in nature.

208

Identify ways to expose the individual to new experiences in his/her community.

206

43.2% 42.8%

94

Quality Management Annual Report FY 2017
Quality Enhancement Provider Review (QEPR)
QEPR Scores by Tool
Several different scores are calculated and presented to providers at the conclusion of the QEPR. The Overall Score is based on findings from the Individual Interviews, Staff Interviews, Observations and Provider Record Reviews (PRR). Findings are also calculated for the Administrative Qualifications and Training, based on a sample of staff rendering services, and the DDSS reviews. During FY 2017, four crisis providers participated in a QEPR. Because these services are very different, we present results for these four providers separately. In FY 2016, 100 providers participated in a QEPR; 46 Small, 36 Medium, 17 Large providers, and one QEPR for a crisis provider. The sample of providers in FY 2016 was randomly drawn from across the state. For FY 2017, providers were selected from the remaining providers who had not yet received a QEPR; 52 Small, 19 Medium, 25 Large providers and four crisis providers. Consequently, while scores for all the QEPR components appear to be lower in FY 2017, comparisons by year are not an appropriate measure as the FY 2017 sample was not selected to be representative of the state. Moreover, the measure was not selected from the entire population of providers and it is unclear of the underlying sources that may be causing the variation. See Figure 25 for distribution of QEPRs by provider size for FY 2016 and FY 2017.
95

Quality Management Annual Report FY 2017
Figure 25. FY 2016 and FY 2017 QEPR Samples by Provider Size and Type
Overall QEPR Score
Figure 26 shows the average overall scores for providers reviewed in FY 2016 and FY 2017 (excluding crisis providers) and scores for each tool used in the review process. The overall score for FY 2017 was 83.7 percent, with the Observation and Staff Interviews showing the highest scores of just over 95 percent. As in FY 2016, the PRR reflected the lowest-scoring area (69.6 percent). Each tool used to calculate the overall score in the QEPR showed results lower than in FY 2016. While the pattern of scores is similar across the tools each year, it is important to note the QEPRs in FY 2017 used a sample not designed to be representative statewide but rather to guarantee all providers are reviewed within a three to three and a half year period. As a result, comparisons made to FY 2016 data are generally not appropriate and should be made with caution.
96

Quality Management Annual Report FY 2017
Figure 26. Overall QEPR Scores by Tool and Year
Individual Interview FY16: 95.1% FY17: 90.7%

Provider Record Review
FY16: 73.6% FY17: 69.6%

Overall Score
FY16: 88.4% (N = 99)
FY 17: 83.7% (N = 96)

Staff Interview
FY16: 96.9% FY17: 95.2%

Observation FY16: 98.2% FY17: 95.7%
Overall Crisis Provider Scores
Five crisis providers were selected for a QEPR, one in FY 2016 and four in FY 2017. Figure 27 shows the overall scores for these providers and scores by tool and year. Compared to other providers reviewed with the QEPR, crisis providers showed scores somewhat lower for the interviews and Observations but close to 15 points higher on the PRR.

97

Quality Management Annual Report FY 2017
Figure 27. Overall Crisis Provider Scores by Tool and Year
Individual Interview FY16: 83.4% FY17: 86.0%

Provider Record Review
FY16: 88.0% FY17: 84.1%

Overall Score
FY16: 85.1% (N = 1)
FY17: 86.8% (N = 4)

Staff Interview
FY16: 81.3% FY17: 89.1%

Observation FY16: 87.7% FY17: 92.5%
Qualifications and Training (Q & T) and Service Specific (DDSS)
Every QEPR includes a review of staff qualifications and training. In FY 2017, a sample of 947 staff records was reviewed. The sample was stratified by service to ensure all services offered by each provider were included in the review process. The primary purpose of the Q&T record review is to confirm relevant staff information is accurate and up to date (e.g., driver's license, performance evaluations, background screening) and staff has received all required trainings specific to services provided. DDSS reviews are completed to ensure services are provided as specified by DBHDD (Figure 28).

98

Quality Management Annual Report FY 2017
Figure 28. QEPR Qualifications and Training and DDSS Results by Year

QEPR Providers

Crisis Providers

Q & T
FY 2016: 80.6% (N = 1,003) FY 2017: 82.6% (N = 947)

Q & T
FY 2016: 92.8% (N = 11) FY 2016: 91.8% (N = 51)

DDSS
FY 2016: 99.9% (N = 1,389) FY 2017: 99.8% (N = 1,667)

DDSS
FY 2016: 100% (N = 8) FY 2017:100% (N = 48)

Qualifications & Training Opportunities for Improvement
Provider staff, particularly for crisis providers, appears to do relatively well maintaining compliance with required training and qualifications. Most staff (94.8 percent) followed DBHDD's policy 04-104 for Criminal Records Checks; most were properly licensed (97.2 percent); and 100 percent of crisis providers ensured crisis response system staff (mobile team members and intensive support staff) had participated in training and passed an examination demonstrating competence in all crisis protocols and requirements. For providers reviewed this year, several areas present opportunities for improvement in staff training, including the following:
Within the first 60 days of hire, many staff did not have training on: o Use of the Georgia Crisis and Access Line (43.0 percent met) o How to work with individuals with co-occurring diagnoses (47.4 percent met)
99

Quality Management Annual Report FY 2017
o Suicide prevention skills (51.5 percent met) o Holistic care of the individual (66.4 percent met) o Medical, physical, behavioral, and social needs and characteristics of individuals
served (68.9 percent met) Required training for Developmental Disability Professionals did not always include:
o Supports Intensity Scale overview (54.5 percent met) o Individual service planning training (67.9 percent met) The following required topics were not always included in the annual training: o Specific individual medications and their side effects (54.9 percent met) o Emergency and disaster plan procedures (68.8 percent met)
The Q&T review component uses a sample of employees to determine compliance with standards. The number of employees per provider varies based upon number of individuals served and services rendered. Therefore, while documentation for a majority of staff may support a "met" on any given standard, fewer providers may have all employees in compliance with the standard. For example:
While 73.2 percent of staff had documented evidence of training on person centered values, principles, and approaches, only 55.3 percent of providers had documentation all employees had received the training, i.e., at least one staff was not in compliance with the standard.
Approximately 95 percent of staff records reviewed were in compliance with background screening requirements standards. However, only about 75 percent of providers reviewed this year had all sampled staff in compliance with the standards, i.e., at least one staff was not compliant with a background screening standard for approximately 25 percent of the providers.
73.4 percent of employees had accurate and up to date annual work performance evaluations, while only 38.3 percent of providers were compliant across all staff reviewed.
100

Quality Management Annual Report FY 2017
82.6 percent of staff were current on annual tuberculous testing, but only 46.3 percent of providers were compliant across all staff reviewed.
QEPR Scores by Provider Size14
Figure 29 displays the distribution of the QEPR Overall and Q&T scores by size of the organization and year.15 There is very little difference across provider size on the overall scores, ranging from 82.7 percent to 86.1 percent in FY 2017. However, the 19 Medium-sized providers reviewed in FY 2017 appear to have scored somewhat lower on the Q&T component (76.0 percent) compared to Small (81.5 percent) or Large (86.4 percent) providers. All providers, regardless of size, scored above 98 percent on the DDSS tool (not shown in the figure).
14 Crisis provider scores were excluded. There were one large, one medium and two small crisis providers reviewed in FY 2017 and one small crisis provider reviewed in FY 2016. 15 See Figure 25 on page 96 for details regarding provider size categories.
101

Quality Management Annual Report FY 2017
Figure 29. Overall and Qualifications and Training Score by Provider Size and Review Year
QEPR Scores by Focused Outcome Area (FOA)
The Overall score for each review is divided into six FOAs. Each FOA score is calculated with the combined results from the Individual Interview, Staff Interview, Observation (IOSA), and Provider Record Review.16 Results for the non-crisis QEPRs are shown by FOA and year in Figure 30. Scores within each FOA declined from FY 2016. The lowest scoring areas each year were Community Life, Choice and Person Centered Practices. Measures surrounding Safety and Rights remained relatively high, showing scores of approximately 91 percent each year.
16 FOAs calculated for the PCR also included the Support Coordinator Interview and SCRR, which are not part of the QEPR process. Comparisons between the PCR and QPER are not appropriate.
102

Quality Management Annual Report FY 2017
Figure 30. Overall QEPR Scores by FOA and Year

Rights FY16: 92.4% FY17: 91.2%
Choice FY16: 86.2% FY17: 75.9%

Whole Health FY16: 89.1% FY17: 83.2%
Overall Score FY16: 88.4%
(N = 99) FY 17: 83.7%
(N = 96)
Community Life
FY16: 78.0% FY17: 70.4%

Safety FY16: 93.9% FY17: 90.9%
Person Centered Practices FY16: 82.6% FY17: 81.6%

QEPR Scores by Tool and FOA
In this section, results by FOA for the QEPR are presented for each tool used in calculating the Ooverall score for the QEPR process--Individual Interview, Observation, Staff Interview and PRR (see Figure 31).17 Findings for the providers reviewed this year indicate the following:

Provider documentation (PRR) is the lowest scoring tool across all the FOAs, ranging from a low of 51.3 percent for measures surrounding Choice to 82.3 percent for Rights.
Observations and staff interviews reflected relatively high Community Life scores, 94.7 percent and 85.2 percent respectively, but the individual's perspective in this FOA was somewhat lower (75.9 percent), and providers did not document this as well (63.9 percent).
Community Life was the lowest-scoring FOA from the individual's perspective.

17 See Table 16 in the Onsite Review Section for the number of interviews and records completed for each QEPR component. 103

Quality Management Annual Report FY 2017
Individual and staff interviews and observations all showed scores over 90 percent in the FOAs of Safety, Choice, and Rights.
Observation scores were approximately 95 percent or higher in all the FOAs with the exception of Person Centered Practices (88.8 percent).

Figure 31. FY 2017 QEPR Scores by Tool and Focus Outcome Area

Whole Health Safety
Person Centered Practices Community Life Choice Rights 0%

66.5%

87.4% 96.5%
94.4%

74.8%

93.0% 98.4% 97.3%

86.6% 88.8% 94.4%
75.4%

75.9% 94.7%
85.2% 63.9%

51.3%

94.2% 95.0% 93.1%

96.8% 92.7%
97.5% 82.3%

25%

50%

75%

100%

Individual Interview (N = 440) Staff Interview (N = 398)

Observations (N = 473) Provider Record Review (N = 1,356)

104

Quality Management Annual Report FY 2017
QEPR Scores by FOA and Provider Size
There is little variation by FOA across the size of the provider organization (See Figure 32). One exception is small providers appear to have performed better in Whole Health than other providers reviewed in FY 2017.
Figure 32. FY 2017 QEPR Provider Scores by FOA and Size

Whole Health Safety
Person Centered Practices Community Life Choice Rights 50%

81.1% 80.5%

89.2%

91.4% 91.6% 90.2%

81.0% 81.4% 82.0%

70.2% 70.8% 70.2%

78.8% 75.9% 74.5%

91.6% 90.5%
91.4%

60%

70%

80%

90%

100%

Small (N = 52)

Medium (N = 19)

Large (N = 25)

105

Quality Management Annual Report FY 2017
Quality and Technical Assistance Consultation (QTAC)
A total of 208 QTACs were completed, of which 147 were at the provider level and 61 at the individual level. Due to the ability to check more than one referral type, totals do not sum to 147. Most QTACs were requested at the provider level (70.7 percent) and most had an internal referral source (75.5 percent). The majority of QTACs (64.6 percent) were generated from the QEPR.

Table 32. QTACs by Referral Source and Type

Provider (N = 147) Individual (N = 61)

Referral Source

Number Percent Number Percent

Health Quality Manager

1

0.7%

9

15.0%

Internal

109

73.8%

51

83.3%

Provider

37

25.5%

1

1.7%

Referral Type:

New Provider

1

0.7%

-

-

Provider Request

40

27.2%

1

2%

Support Coordinator Monitoring

1

0.7%

10

16%

Alert (PCR/QEPR)

13

8.8%

50

82%

QEPR QTAC

97

64.6%

-

-

The following table provides a list of detailed reasons for the QTACs completed in FY 2017. The most frequently cited reasons were related to alert follow-ups and the need/request for Person Centered Thinking training.

Table 33. QTAC Referral Reasons: FY 2017

Reason

N %

Alert follow-up

60 27.0%

Person-Centered Thinking training needed or requested

27 12.2%

Other

19 8.6%

Review of person-centered documentation

14 6.3%

Tracking health related issues

7 3.2%

Goals not being tracked

4 1.8%

Follow up to previous QTAC

4 1.8%

Medication errors

3 1.4%

Safety concern for the person

2 0.9%

106

Quality Management Annual Report FY 2017

Reason

Table 33. QTAC Referral Reasons: FY 2017

Assistance needed for KPI reporting and data collection Lack of Behavior Support Plan Lack of financial tracking Assistance with criminal background checks Rights training

Total

N %
2 0.9% 1 0.5% 1 0.5% 1 0.5% 1 0.5%
146 100.0%

Technical assistance is provided at every QTAC. The most common type of technical assistance offered for provider level QTACs was group discussion with the provider and staff (61.9 percent) and the most common type offered at the individual level was individual discussion (59.0 percent) with staff. The dispostion of most of the QTACs was "closed," indicating no additional follow-up was necessary.

107

Quality Management Annual Report FY 2017

Intellectual and Developmental Disability Summary of Findings and Recommendations
Beginning in FY 2016, the Collaborative implemented revised review tools to evaluate the quality of intellectual and developmental disability services and supports--the QEPR and PCR. Most of the tools are based upon the six FOAs identified throughout this report and indicators to address a wide range of requirements and best practices within each FOA. June 2017 marked the completion of the second year of the Collaborative contract. Quality assessors completed 100 QEPRs, 481 PCRs, and 208 QTACs.

System Strengths
In general, findings from reviews were quite positive. Strengths in the system included providers and staff who are aware of individuals' unique safety needs and achievements. Individuals are aware of how to self-preserve, and they feel valued, in part because most direct service providers have a clear understanding of each individual's unique communication styles and skills. In many PCRs, staff promotes independence as well as the use of person centered values and practices.

Findings on many of the tools for both the PCR and QEPR showed scores of over 90 percent (see Table 34). Similar to FY 2016, compliance surrounding documentation is the lowest scoring area, for both service providers and Support Coordinators.

Table 34. FY 2017 Summary by Tool and Review Type

Tool

PCR

QEPR Crisis

(N = 481) (N = 96) (N = 4)

IOSA - Individual Interview

91.9% 90.7% 86.0%

IOSA - Observation

96.8% 95.7% 92.5%

IOSA - Staff Interview

94.3% 95.2% 89.1%

Provider Record Review

70.2% 69.6% 84.1%

SC Record Review

73.7%

NA

NA

SC Interview

83.3%

NA

NA

Admin Q&T

NA

82.7% 91.8%

DDSS

99.8% 99.8% 100%

108

Quality Management Annual Report FY 2017
FOA scores from both review types suggest the service delivery system appears to do fairly well across the six critical areas, showing scores of at least 70 percent. Scores from both review types were over 90 percent for Safety and Rights. Providers seem to have effective systems and practices in place to help individuals be safe in their environments and exercise their rights, and individuals have indicated they do feel safe and their rights are being upheld.
Whole Health scores for both individuals who participated in a PCR and providers reviewed this year were approximately 85 percent. Support Coordinators did very well on the health-related sections of the ISP in accurately completing all components of the rights, psychotropic medications and behavior supports section and the health and safety review section.
Opportunities for Improvement
Despite the overall positive findings for this fiscal year, scores from the PCR showed decreases across most review indicators when compared to FY 2016, including the average score on all the tools and all the FOAs. While overall FOA scores remained relatively high, drill down to the standards/indicators measuring each FOA provides insight into opportunities for developing quality improvement initiatives or training programs.
Whole Health The overall PCR FOA score for Whole Health was 84.7 percent, down from 93.6 percent in FY 2016. Issues surrounding medication, particularly education, were the lowest scoring areas. Many individuals were not aware of the reasons they were prescribed medications or the potential side effects. Providers often did not demonstrate through documentation how they provide education on the risks and side effects of medications. Staff was often unaware of the side effects of medications prescribed to individuals they supported and was often unable to describe how individuals learned about their medications. In addition, providers were not always receiving training required on the specific medications individuals are prescribed and their side effects. Just over 31 percent of staff reviewed did not have this training in place for providers reviewed in FY 2017. In FY 2016, approximately 45 percent of
109

Quality Management Annual Report FY 2017
staff records indicated this training was missing. Individuals receiving Respite and Supported Employment were least likely to have Whole Health standards present.
Recommendation IX: Quality Management recommends adding to the DBHDD provider policy a requirement that individuals and families be provided education on their medications, to promote independence and an ability to help self-direct health care. Techniques on how to help educate individuals/families on medications could be developed by the ASO, and these techniques can be offered to providers to assist in implementing the training programs.
Safety Information from the individuals' interviews indicates people feel safe, are free from abuse, neglect, and exploitation, and most (75 percent or more) feel they receive education in these areas. It is critical for providers to help individuals identify, address, and seek prevention from abuse, neglect, and exploitation (ANE). While providers may be helping most individuals understand safety issues surrounding ANE, only 22.9 percent of providers have this documented. Fewer than half of staff records included how education is provided to help the individual with self-preservation. In addition, only 43.4 percent of staff records reviewed this year, and 32.6 percent reviewed in FY 2016, documented staff/provider training on the Georgia Crisis System.
Information from Support Coordinator interviews appears to point to a need for a better understanding of some issues surrounding behavioral health. Advocating for the health and safety of individuals is critical and requires an understanding of each individual's health and safety needs. However, 50 percent to 60 percent of SCs were unaware of needed crisis plans or behavior support plans, the individualized techniques needed to follow a plan for the individual, or specific triggers that may be the catalyst to behaviors.
Recommendation X: Some providers have already implemented practices using various techniques to focus on a specific area for training each month, such as "The Right of the Month." Training by Quality Management could be developed and implemented to support providers to initiate
110

Quality Management Annual Report FY 2017
techniques specific to training on abuse, neglect, and exploitation and education on selfpreservation.
Person Centered Practices A key area of focus for the Centers for Medicare and Medicaid Services (CMS) and DBHDD is to ensure services and supports are provided using person centered practices and planning. To the extent possible, the individual should be at the center of all decisions, plans, and goals. Providers need to understand who the individual is and what he or she may want, hope for, and dream. Unfortunately, many ISPs did not ensure all the individual's goals were person centered or their hopes and dreams were appropriately addressed throughout the plan, approximately 44 percent and 55 percent respectively.
In addition, 44.7 percent of providers have staff who had not received training on person centered values, principles, and approaches. Most individuals receiving services seem to understand their own talents, but fewer than 40 percent of providers (38.9 percent) and Support Coordinators (28.8 percent) have documentation that reflects the individual's talents.
Recommendation XI: DBHDD will be providing training on the changes to the ISP being implemented in FY 2018. It is recommended this training include a section regarding writing person centered goals.
Community Life According to CMS standards and DBHDD's vision, individuals with intellectual and developmental disabilities should be able to participate in their communities in the same manner as individuals who do not have a disability. Community Life was the lowest scoring area in the PCR data for both FY 2016 and FY 2017 and for providers reviewed both years through the QEPR. During FY 2017, information from interviews with the individual suggests about 30 percent of individuals receiving services had not been given the opportunity to develop new social roles in the community. Findings across all the interview data (individuals and staff/providers), as well as from documentation reviews, indicate individuals are often not participating in their communities as desired, seeking and
111

Quality Management Annual Report FY 2017
finding competitive employment, having new community experiences, or participating in community activities and employment.
The ISP should be written to help ensure the individual is able to integrate into the community in various settings, as desired. Approximately 17 percent of ISPs met this level. Individuals receiving Prevocational and Respite were least likely to have Community Life standards present.
Recommendation XII: Develop training that includes practical, hands-on assignments to help staff and Support Coordinators understand how to connect an individual to the community, such as searching the web or calling organizations who would support individuals with intellectual and developmental disability. The practical experiences and actual application of the session's philosophy, and steps on how to connect people in the community, will better equip staff and Support Coordinators to actively apply lessons learned for people being supported.
Recommendation XIII: Provide a training on person centered goals and approaches to assist Support Coordinators in developing ISPs consistent with a Community Life.
Recommendation XIV: As part of the DBHDD training on the revised ISP, provide training that includes specific approaches to assist Support Coordinators in developing ISPs consistent with an integrated life in the community.
Choice Informed choice is the cornerstone of helping individuals understand and achieve meaningful goals and direct their own supports and services. Results for Choice have decreased more since FY 2016 than for any other FOA. This is evident in the PCR data, and for providers reviewed in FY 2016 compared to providers reviewed in FY 2017. Information from documentation indicates records are often lacking evidence informed choice is provided to the individual for competitive or supported employment, living situations and environments, community participation or social interactions, educational opportunities, or the manner in which services are provided.
112

Quality Management Annual Report FY 2017
Recommendation XV: Quality Management could develop a webinar to specifically address how direct support staff and Support Coordinators can document how individuals are offered and making informed choices in various areas of their lives.
Rights The proper use of restrictive interventions is strictly monitored and enforced by DBHDD. Individuals have not indicated any violations regarding the improper use of any type of restrictive intervention or any unauthorized restrictions in their home, community, work, or day program. However, records maintained by providers and support coordinators, as well as interviews with the Support Coordinators, may indicate some issues regarding rights restrictions.
There has been a decrease in the percent of Support Coordinators who address identified rights restrictions or who are even aware of a plan for the restrictive intervention. Individuals are not always informed about their rights and responsibilities on an annual basis, and consent for psychotropic medication is often not present and signed by the individual.
Recommendation XVI: Ensure education about rights and rights restrictions is a focus of the Support Coordinator Training noted in Recommendation XIV.
Recommendation XVII: Support Coordinators should help provide education to individuals and families on the importance of giving and receiving copies of signed consent forms for psychotropic medications. Through this, Support Coordinators could ensure consent forms are being completed and subsequently document this in their own support notes.
Provider Documentation Throughout the findings in this report, documentation by providers and Support Coordinators has shown consistently lower scores when compared across review tools or within the FOAs. Support Coordinators seem to do well documenting areas of health and safety, but not as well regarding rights, the only area in which providers scored higher. Provider documentation for providers reviewed this year was lowest in the area of choice (51.3 percent).
113

Quality Management Annual Report FY 2017
Recommendation XVIII: Quality Management should develop a webinar that specifically addresses how direct support staff and Support Coordinators can improve documentation specific to areas identified throughout the intellectual and developmental disability section of this report.
114

Quality Management Annual Report FY 2017
Section 5: Behavioral Health and Intellectual Developmental Disabilities: Overall Results by Focused Outcome Area

Quality Management analyzed the Focused Outcome Areas (FOA), as described throughout this report, in both the behavioral health and intellectual and developmental disability review processes. While the FOAs are the same, there are some differences in the data collection methods. Data for behavioral health FOAs are taken only from record reviews; while for intellectual and developmental disabilities, indicators from the interviews, observations and record reviews are grouped into the six FOAs. Therefore, for intellectual and developmental disabilities, we use data from only the QEPR Provider Record Reviews to show a comparison of FOA scores, i.e., QEPR PRR and BHQR record reviews.
Figure 33. Focused Outcome Areas by Review Type

BHQR
FOA Overall: 89%
Whole Health 74% Safety 83%
Person Centered 91% Community 93% Choice 96% Rights 93%

QEPR
FOA Overall: 69.6%
Whole Health 56.5% Safety 74.8%
Person Centered 75.4% Community 63.9% Choice 51.3% Rights 82.3%

Results (Figure 33) indicate behavioral health providers appear to perform better than providers of intellectual and developmental disability services in documenting how FOAs are addressed for individuals receiving services, with average scores of 89 percent and 70 percent, respectively. This
115

Quality Management Annual Report FY 2017
is a continued trend from FY 2016 to FY 2017 with providers of intellectual and developmental disability services showing lower scores on each FOA. The greatest differences are in Choice and Person Centered Practices, on which providers of intellectual and developmental disability services documentation is 28 and 21 percentage points lower, respectively. Choice is the highest behavioral FOA in behavioral health (96 percent) but the lowest for intellectual and developmental disability (51.3 percent). Rights appears to be a higher scoring FOA for both. In review of specific indicators in both the behavioral health and intellectual and developmental disability reviews, direct comparisons could not be identified. Although both reviews measure the same areas, behavioral health and intellectual and developmental disability questions were not initially designed to be compared. Furthermore, behavioral health and intellectual and developmental disability providers are required to record information using different provider manuals, requirements by waiver program and national standards. Recommendation XIX: Quality Management should review the behavioral health and intellectual and developmental disability record review tools to determine if any modifications can be made in order to draw future comparisons.
116

Section 6: Feedback Survey Results

Quality Management Annual Report FY 2017

Following completion of on-site reviews for both behavioral health and intellectual and developmental disabilities, providers are offered the opportunity to complete a feedback survey. For intellectual and developmental disability services, individuals who participate in the interview are also offered the opportunity to provide feedback about the process. Surveys are optional and may be completed by the same agency more than once in a fiscal year. In addition, it is not known who completes the survey, as individuals and staff are not required to submit their name or provider agency. A five-point Likert scale is used: strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, or strongly disagree.

The following table (Table 35) provides information from providers for data entered into the system during the year, July 2016 through June 2017. Response rates were generally low, but findings were overwhelmingly positive across all the review types and from both providers and individuals. The percent is calculated as follows: (Strongly Agree + Somewhat Agree) / (Strongly Agree + Somewhat Agree + Disagree + Strongly Disagree). Additionally, two surveys were linked to the joint review process through the provider submission of name with the survey responses. The results of the two surveys, using the calculation mentioned previously, was 100 percent.

Table 35. The Collaborative Provider Feedback Surveys

Percent: Strongly Agree + Somewhat Agree/ Total Responses

IDD

Survey Questions

(N=74)

Overall, you are satisfied with the review/consultation process.

98.6%

The Collaborative staff interacted with you and your staff in a professional manner.

100.0%

The Collaborative staff interacted with the individuals you support in a professional manner.

100.0%

The Collaborative staff answered your questions and concerns clearly and

consistent with DBHDD manual. (If you disagree, please explain at the end of 98.6%

the survey.)

The Collaborative staff facilitated an environment which was collaborative and positive.

98.6%

BHQR (N=43) 88.4% 97.7% 97.7%
90.7%
90.7%

117

Quality Management Annual Report FY 2017

Table 35. The Collaborative Provider Feedback Surveys

Percent: Strongly Agree + Somewhat Agree/ Total Responses

IDD

Survey Questions

(N=74)

You would contact the Collaborative staff for technical assistance, training, and resource support, if needed.

100.0%

The process provided constructive feedback.

100.0%

The process helped identify the strengths of your supports and services.

100.0%

The feedback you received will help provide supports and services that meet the desired outcomes of the individuals you support.

98.6%

The recommendations generated from this process can be used to make a positive contribution to the individuals served.

98.6%

The recommendations generated from this process can be used to make a positive contribution to your organization overall.

98.6%

The feedback provided will assist your organization with making quality improvements to systems and practices.

98.6%

BHQR (N=43) 95.4% 97.7% 97.7% 100.0%
95.4%
100.0%
100.0%

Individuals are provided a survey after they complete the interview during either the PCR or QEPR. Only 17 surveys were received during FY 2017. All responses (100 percent) to the following questions were either Strongly Agree or Agree:
1. Overall, I am pleased with how the interview went. 2. The purpose of the interview was explained to me. 3. I was treated with respect. 4. The person who interviewed me seemed interested in what I said. 5. The person who interviewed me was pleasant. 6. My questions were answered. 7. The length of the interview was good.

Quality Management will continue to regularly review its processes, including information obtained from these surveys, to ensure providers are equipped with the necessary tools and opportunities to
118

Quality Management Annual Report FY 2017
best serve individuals across the state. Quality Management shall also tailor its quality training both to its internal staff as well as for Georgia providers to promote a cooperative and constructive partnership in the quality of services provided in the state of Georgia specific to the individual's needs.
119

Quality Management Annual Report FY 2017
Section 7: Conclusion
FY 2017 marked the second complete year of the Collaborative Quality Management review process. The tools and standards used in the behavioral health quality reviews (BHQR) differ from the intellectual and developmental disability quality reviews, but Quality Management assessors have worked collaboratively to best serve the needs of all individuals in the services they receive.
During FY 2017, 789 reviews were completed for both behavioral health and intellectual and developmental disability reviews. The overall score for BHQRs for FY 2017 was 84 percent, consistent with FY 2016. Billing validation increased from baseline year to 84 percent. Focused outcome areas also increased from FY 2016 to FY 2017. Assessment and treatment planning and compliance with service guidelines both minimally decreased by approximately two points. Each of the four categories are equally weighted and comprise 25 percent of the overall score, thus decrease in two categories counter the increase of the other two categories when comparing FY 2016 to FY 2017. While there is demonstrated success noted in the different BHQRs categories, there are opportunities for growth specifically related to addressing all assessed needs, co-occurring issues being assessed and addressed, as well as appropriate discharge planning criteria documentation.
Additionally, FY 2017 marked the first year reassessments were completed for providers who met the scoring criteria. Thirty-five providers were reviewed for a second time during the fiscal year with many having individual success and improvement in scores. In reviewing scores for these specific providers at the time of their first FY 2017 review compared to their second FY 2017 review, providers increased their scores in all areas. For example, overall scores went from 79 percent to 83 percent. Furthermore, while it is anticipated the reassessment of providers will lead to elevated results from year-to-year, declines in two categories and increases in the remaining two categories counter balance within the overall score.
Intellectual and developmental disability assessors completed 100 QEPRs, 481 PCRs, and 208 QTACs. Findings on many of the tools for both the PCR and QEPR showed scores of over 90
120

Quality Management Annual Report FY 2017
percent. Providers appear to have effective systems and practices in place to support individuals to be safe in their environments and to exercise their rights, and at the same time individuals have indicated they do feel safe and their rights are being upheld. Similar to FY 2016, compliance surrounding documentation within the record review is the lowest scoring area, for both service providers and Support Coordinators.
Where possible, Quality Management conducted joint BHQR and QEPRs to reduce administrative burden to providers. Using such a method promotes not only additional education to the network but also amongst assessors in the field, further strengthening expertise and collaboration. Each joint review lends valuable information and feedback through joint discussions and exit conferences, for providers and assessors alike. Additionally, providers are supplied an additional opportunity to complete a formalized survey following the process. Of the two identified surveys obtained regarding the joint review processes, both received 100 percent in terms of positive responses related to collaborative staff, its policies, and processes related to the review.
Feedback obtained from the satisfaction surveys provides a qualitative approach to individual and staff perceptions related to review processes and yielded positive results in FY 2017. Quality Management will continue to use feedback from the surveys to review processes and provide professional, effective, and constructive approaches to ensure providers are equipped with the necessary tools and opportunities they need to best support individuals.
Provider trainings in FY 2018 are to occur both in person as well as via webinar through the formalized Quality Training Program based on collaboration and partnership with DBHDD. Quality Management shall use findings from the behavioral health, CSU, and intellectual and developmental disability reviews to guide topics for trainings. Such trainings will be prioritized based on BHQR, CSUQR, and intellectual and developmental disability reviews for areas indicating low compliance and minimal documentation. Additionally, as the collaboration and partnership continues, changes to measurement tools to further align indicators is both recommended and required for ongoing smooth review processes.
121

Appendix A: Abbreviations and acronyms

Quality Management Annual Report FY 2017

Acronym
ACT AD ASAM ASO BH BHQR C&A CIS CL CMS CST CSU CSUQR DBHDD DDSS FOA FY II IDD IFI IND IOSA IRR IRP ISP QA ISP GRP MAR MH N NA NCP OBS PCP

Definition
Assertive Community Treatment Addictive Diseases American Society for Addiction Medicine Administrative Services Organization Behavioral Health Behavioral Health Quality Review Child and Adolescent Consumer Information System Community Life Centers for Medicaid and Medicare Services Community Support Team Crisis Stabilization Unit Crisis Stabilization Unit Quality Review Department of Behavioral Health and Developmental Disabilities Developmental Disability Service Specific Review Focused Outcome Area(s) Fiscal Year Individual Interview Intellectual and Developmental Disability Intensive Family Intervention Individual Individual Observation Staff Assessment Individual Record Review Individual Recovery / Resiliency Plan Individual Service Plan Quality Assurance Checklist Individual Service Plan Group Medication Administration Record Mental Health Number in sample Not Applicable Nursing Care Plan Onsite Observations Person Centered Practices

122

PCR PRR QEPR Q&T QTAC RN SC SCI SCRR SI SP SU TA

Person Centered Review Provider Record Review Quality Enhancement Provider Review Qualifications and Training Quality Technical Assistance Consultation Registered Nurse Support Coordinator Support Coordinator Interview Support Coordinator Record Review Staff Interview Service Provider Substance Use Technical Assistance

Quality Management Annual Report FY 2017

123

Appendix B: Score Distributions

Quality Management Annual Report FY 2017

Graphical distributions for each category show the percent of providers who scored within the specified range of scores, as indicated on the horizontal/x-axis. With each distribution, the median, mean, and mode may be provided. The median represents the score that falls in the middle of distribution (50th percentile) and the mean represents the average score. Standard deviation refers to the amount of variation or dispersion there is in a distribution of scores, or how much scores tend to spread-out from the mean. Standard deviations are sometimes used to determine significant differences between scores within a distribution; however, this application requires a distribution to be normally distributed (similar to a bell curve). Most of the distributions presented in this report do not meet the requirements of a normal distribution, meaning they do not look like a normal bell curve; therefore, standard deviations have been excluded from the report at this time.

Table of Contents (Links available by clicking on Figure Name)
Behavioral Health Quality Review Distributions
BHQR Overall Score Distribution BHQR Tier 1 Overall Score Distribution BHQR Tier 2 Overall Score Distribution BHQR Tier 2+ Overall Score Distribution BHQR Tier 3 Overall Score Distribution BHQR Billing Score Distribution BHQR Assessment & Treatment Planning Distribution BHQR Compliance with Service Guidelines Score Distribution BHQR Focused Outcome Areas Score Distribution
Assertive Community Treatment (ACT) Quality Review Distributions
ACT Overall Score Distribution ACT Billing Score Distribution ACT Assessment & Treatment Planning Distribution ACT Compliance with Service Guidelines Score Distribution ACT Focused Outcome Areas Score Distribution
CSU Quality Review Distributions
CSU Overall Score Distribution
124

Quality Management Annual Report FY 2017
CSU Compliance with Service Guidelines Score Distribution CSU Focused Outcome Areas Score Distribution CSU Individual Record Review Score Distribution
Reassessment BHQR Distributions
BHQR Overall Score Distribution of Reassessed Providers BHQR Billing Score Distribution of Reassessed Providers BHQR Assessment & Treatment Planning Distribution of Reassessed Providers BHQR Compliance with Service Guidelines Score Distribution of Reassessed Providers BHQR Focused Outcome Areas Score Distribution of Reassessed Providers
PCR Distributions: Scores by Tool and Year
PCR Individual Interview Scores by Year PCR Staff Interview Scores by Year PCR Support Coordinator Interview Scores by Year PCR Observation Scores by Year PCR Provider Record Review Scores by Year PCR Support Coordinator Review Scores by Year
QEPR Distributions: Scores by Tool and Year
FY 2017 Overall QEPR Score FY 2017 QEPR Individual Interview Scores FY 2017 QEPR Staff Interview Scores FY 2017 QEPR Observation Scores FY 2017 QEPR Provider Record Review Scores FY 2017 QEPR Qualifications and Training Scores FY 2017 QEPR Service Discrepancy Scores FY 2017 QEPR Overall Scores by Provider Size FY 2017 QEPR Qualifications and Training Scores by Provider Size
125

Quality Management Annual Report FY 2017
BHQR Overall Score Distribution by Year

60%

51%49%

50%

40%

30%

26%25%

20%

16%19%

10%

1% 0% 1% 1% 2% 2% 3% 5%

0%

FY 2016 (N = 141; Mean = 84%; Median = 87%) FY 2017 (N = 167; Mean = 84%; Median = 86%)

BHQR Tier 1 Overall Score Distribution by Year

80%

70%

68%

60%

58%

50%

40%

30% 20%

17%21%

25% 11%

10%

0%

FY 2016 (N = 24; Mean = 86%; Median = 88% ) FY 2017 (N = 28; Mean = 85%; Median = 85%)

126

Quality Management Annual Report FY 2017
BHQR Tier 2 Overall Score Distribution by Year

60%

51%50%

50%

40%

30%

24%24%

19% 19% 20%

10%

1%

2% 4% 5%

0%

FY 2016 (N = 90; Mean = 85%; Median = 87%) FY 2017 (N = 115; Mean = 85%; Median = 86%)

BHQR Tier 2+ Overall Score Distribution by Year

90%

83%

80%

80%

70%

60%

50%

40%

30%

20%

20%

17%

10%

0%

FY 2016 (N = 5; Mean = 88%; Median = 87% ) FY 2017 (N = 6; Mean = 86%; Median = 85%)

127

Quality Management Annual Report FY 2017
BHQR Tier 3 Overall Score Distribution by Year

60%

56%

50%

41%

40%

30%

27%

20% 10%

5%

14% 5% 6% 6%

11% 9% 11%

11%

0%

FY 2016 (N = 22; Mean = 78%; Median = 83%) FY 2017 (N = 18; Mean = 83%; Median = 91%)

BHQR Billing Score Distribution by Year

60%

50%

48% 42%

40%

33%

30%

28%

20% 10% 4% 1% 1% 2% 1% 1% 1% 1% 2% 2% 1% 4% 5% 5% 10%8%
0%

FY 2016 (N = 141; Mean = 81%; Median = 88%) FY 2017 (N = 167; Mean = 84%; Median = 90%)

128

Quality Management Annual Report FY 2017
BHQR Assessment & Treatment Planning Distribution by Year

35%

31% 31%31%

30%

30%

25%

23%

20%

16%

17%

15% 10%

9% 4% 5%

5%

1%1% 1%1%

0%

FY 2016 (N = 141; Mean = 79%; Median = 79%) FY 2017 (N = 167; Mean = 77%; Median = 78%)

BHQR Compliance with Service Guidelines Score Distribution by Year

70%

60%

58% 53%

50%

40%

34%31%

30%

20%

10%

7% 6% 7%

1%

1% 1% 1% 1%

0%

FY 2016 (N = 140; Mean = 90%; Median = 92%) FY 2017 (N = 167; Mean = 88%; Median = 91%)

129

Quality Management Annual Report FY 2017

BHQR Focused Outcome Areas Score Distribution by Year

60% 50%

47%

50%

41%

40%

30%

30%

20%

17%

10%

2%

4% 2%

8%

0%

FY 2016 (N = 141; Mean = 85%; Median = 87%) FY 2017 (N = 167; Mean = 89%; Median = 90%)

ACT Overall Score Distribution by Year

60%

50%

47%

50%

42%

40%

30%

26% 25%

20% 11%
10%

0%

FY 2016 (N = 19; Mean = 88%; Median = 91%) FY 2017 (N = 20; Mean = 87%; Median = 91%)

130

Quality Management Annual Report FY 2017

ACT Billing Score Distribution by Year

70% 60%

58% 60%

50%

40%

37%

30% 20%

15% 20%

10%

5% 5%

0%

FY 2016 (N = 19; Mean = 92%; Median = 96%) FY 2017 (N = 20; Mean = 90%; Median = 96%)

ACT Assessment & Treatment Planning Score Distribution by Year

70% 60%

58%

50% 40%

35% 40%

30% 20%

26%

20%

16%

10%

5%

0%

FY 2016 (N = 19; Mean = 85%; Median = 80%) FY 2017 (N = 20; Mean = 80%; Median = 80%)

131

60% 50% 40% 30% 20% 10%
0%
60% 50% 40% 30% 20% 10%
0%

Quality Management Annual Report FY 2017
ACT Compliance with Service Guidelines Score Distribution by Year

50%

32%

21%

11% 5%

10%

37% 35%

FY 2016 (N = 19; Mean = 85%; Median = 89%) FY 2017 (N = 20; Mean = 88%; Median = 89%)

ACT FOAs Score Distribution by Year

55% 47% 37% 35%

16% 10%

FY 2016 (N = 19; Mean = 88%; Median = 91%) FY 2017 (N = 20; Mean = 90%; Median = 91%)
132

Quality Management Annual Report FY 2017

CSU Overall Score Distribution by Year

50%

47%

45%

43%

40%

35%

35%

30%

26%

25%

21%

20%

17%

15%

10% 5%

5% 4%

0%

FY 2016 (N = 19; Mean = 83%; Median = 83%) FY 2017 (N = 23; Mean = 86%; Median = 88%)

CSU Compliance with Service Guidelines Score Distribution by Year

70%

61%

60%

50%

40%

32% 32%

30%

20% 10%

4%

11%9%

11% 4%

17%16% 4%

0%

FY 2016 (N = 19; Mean = 82%; Median = 80%) FY 2017 (N = 23; Mean = 87%; Median = 91%)
133

Quality Management Annual Report FY 2017

CSU Focused Outcome Areas Overall Distribution by Year

60%

57%

50%

47%

40%

37% 35%

30%

20%

16%

10%

9%

0%

FY 2016 (N = 19; Mean = 88%; Median = 88%) FY 2017 (N = 23; Mean = 91%; Median = 92%)

CSU Individual Record Review Score Distribution by Year

70%

65%

60%

50%

42%

40%

37%

30%

26%

20%

11%

10%

5% 4% 5% 4%

0%

FY 2016 (N = 19; Mean = 79%; Median = 77%) FY 2017 (N = 23; Mean = 82%; Median = 82%)

134

Quality Management Annual Report FY 2017

BHQR Overall Score Distribution

of Reassessed Providers

50%

43%

46%

40%

40%

30%

23%

23%

20%

11%

10%

3% 3%

6%

3%

0%

First Review (N = 35; Mean = 79%; Median = 80%) Second Review (N = 35, Mean = 83%, Median = 86%)

BHQR Billing Score Distribution

of Reassessed Providers

45%

40%

43%

40%

35%

30%

26% 26%

25%

20%

15%

11% 11%

10% 5%

9% 6% 3% 3% 3% 3% 3% 3%

9% 3%

0%

First Review (N = 35; Mean = 79%; Median = 84%) Second Review (N = 35; Mean = 79%; Median = 89%)

135

Quality Management Annual Report FY 2017

BHQR Assessment & Treatment Planning Score Distribution of Reassessed Providers

50%

40%

37%

40%

30%

29%

20%

20%

11%

17%

11%

14%

10%

9% 3% 3%

6%

0%

First Review (N = 35; Mean = 70%; Median = 68%) Second Review (N = 35; Mean = 78%; Median = 78%)

BHQR Compliance with Service Guidelines Score Distribution

of Reassessed Providers

50%

46%

43%

40%

30%

29% 26%

20%

14% 14% 14%

10%

9%

3%

3%

0%

First Review (N = 35; Mean = 83%; Median = 86%) Second Review (N = 35; Mean = 85%; Median = 90%)

136

Quality Management Annual Report FY 2017
BHQR FOAs Score Distribution of Reassessed Providers

70%

66%

60%

51%

50%

40%

30%

29% 23%

20%

14%

10%

6% 3% 9%

0%

First Review (N = 35; Mean = 86%; Median = 87%) Second Review (N = 35; Mean = 90%; Median = 91%)

PCR Individual Interview Scores by Year

80%

70%

67.8% 66.1%

60%

50%

40%

30%

21.6%

20%

11.0%

18.2%

10%

0.8%3.1%5.8%

5.4%

0%

FY 2016 (N = 484; Mean = 95.1% ; Median = 97.1%) FY 2017 (N = 481; Mean = 91.9%; Median = 93.7%)

137

Quality Management Annual Report FY 2017
PCR Staff Interview Scores by Year

80%

70%

67.8%69.3%

60%

50%

40%

30%

24.5%

20%

11.1%

16.0%

10% 0%

6.8%

0.0%0.4%

0.9% 2.2% 0.0% 0.8%

FY 2016 (N = 482; Mean = 96.4%; Median = 97.7%) FY 2017 (N = 224; Mean = 94.3%; Median = 96.0%)

PCR Support Coordinator Review Scores by Year

60%

50%

45.5% 45.1%

39.6%

40%

33.1%

30%

20%

10.6%

10% 5.0% 0%

0.4%0.0%

0.4% 0.0%

7.3% 7.1%
0.2%0.2%0.2%0.0%0.2%1.0%1.5%2.5%

FY 2016 (N = 477; Mean = 90.8%; Median = 98.4%) FY 2017 (N = 479; Mean = 83.3%; Median = 96.7%)

138

Quality Management Annual Report FY 2017
PCR Observation Scores by Year

60%

51.9%

50%

48.5% 48.8%

40%

38.1%

30%

20%

10%

3.9%2.7%6.1%

0%

0.0%

FY 2016 (N = 412; Mean = 98.4%; Median = 99.5%) FY 2017 (N = 181; Mean = 96.8%; Median = 98.7%)

PCR Provider Record Review Scores by Year

35%

30%

29.3% 26.6%

25%

23.9% 22.5%

20%

18.3%

17.7%18.3%

15%

14.1%

10% 5% 0%

0.0%0.2%

3.4%5.0% 6.2% 0.3%1.2%

5.6%7.3% 0.2%

FY 2016 (N = 356; Mean = 79.0%; Median = 79.1%) FY 2017 (N = 478; Mean = 70.2%; Median = 70.5%)

139

Quality Management Annual Report FY 2017

PCR Support Coordinator Review Scores by Year

30%

25%

24.3% 22.9%

20.2%21.1%

20%

17.0%

15%

15.8%15.3%

14.5%

11.4%

10%

9.6%

6.8%

5.0%5.4%

5% 0.8%0.2%0.4%0.8%1.7%1.5%0.2%2.7%

2.5%

0%

FY 2016 (N = 484; Mean = 79.9%; Median = 84.1%) FY 2017 (N = 481; Mean = 73.7%; Median = 76.4%)

FY 2017 Overall QEPR Score

60%

52.6%

50%

40%

30%

20.6%

23.7%

20%

10%

1.0% 2.1%

0%

(N = 96; Mean = 83.7%; Median = 85.7%)

140

70% 60% 50% 40% 30% 20% 10%
0%
100% 80% 60% 40% 20% 0%

Quality Management Annual Report FY 2017
FY 2017 QEPR Individual Interview Scores
57.7% 35.1%

1.0%

1.0% 4.1%

1.0%

(N = 96; Mean = 90.7%; Median = 85.8%)
FY 2017 QEPR Staff Interview Scores

83.3%

2.1% 8.3%

6.3%

(N = 95; Mean = 95.2%; Median = 96.0%)

141

Quality Management Annual Report FY 2017

FY 2017 QEPR Observation Scores

80%

72.2%

70%

60%

50%

40%

30%

20%

12.2%

11.1%

10%

1.1% 3.3%

0%

(N = 89; Mean = 95.7%; Median = 96.5%)

FY 2017 QEPR Observation Scores
40%

30.9% 30%
24.7%

20%

18.6%

12.4%

10%

6.2%

1.0% 1.0% 0%

5.2%

(N = 96; Mean = 69.9%; Median = 71.4%)

142

40% 30% 20% 10%
0%
100% 80% 60% 40% 20% 0%

Quality Management Annual Report FY 2017
FY 2017 QEPR Qualifications and Training Scores

33.0% 25.8%

15.5%

1.0%

2.1%

5.2%

2.1%

7.2%

8.2%

(N = 96; Mean = 82.6%; Median = 86.8%)
FY 2017 QEPR Service Discrepancy Scores

84.5%

15.5% (N = 96; Mean = 99.8%; Median = 100.0%)

143

Quality Management Annual Report FY 2017
FY 2017 QEPR Overall Scores by Provider Size

80%

70%

68.0%

60%

50%

48.1%

42.1%

40%

36.8%

36.5%

30%

28.0%

20% 15.8%

11.5%

10%

5.3%

4.0%

1.9% 1.9%

0%

Small (N = 52; Mean = 89.2%; Median = 87.2%) Medium (N = 19; Mean = 81.1%; Median = 84.3%) Large (N = 25; Mean = 82.7%; Median = 82.8%)

144

Quality Management Annual Report FY 2017
FY 2017 QEPR Qualifications and Training Scores by Provider Size

60%

50%

48.0%

40%

36.8%

30.8% 30%

23.1%

24.0%

20%

19.2%

15.8%

15.8%

11.5%

10.152%.0%

9.6%

10%

8.0%

5.3% 5.3%

5.3%

4.0%

1.9% 1.9% 1.9%

4.0%

5.3%

0%

Small (N = 52; Mean = 81.5%; Median = 87.6%) Medium (N = 19; Mean = 76.0%; Median = 81.8%) Large (N = 25; Mean = 86.4%; Median = 87.3%)

145