SHINES : requested information on Georgia's child welfare information system

Special Examination Report No. 12-24

December 2012

Georgia Department of Audits and Accounts
Performance Audit Division
Greg S. Griffin, State Auditor Leslie McGuire, Director

Why we did this review
This review was conducted at the request of the House Appropriations Committee. Specifically, the Committee is interested in the cost of the SHINES system and whether the system is effective, in terms of the Division of Family and Children Services' (DFCS) personnel being able to access it in real-time and input information critical to child safety.
About SHINES
SHINES is the name given to Georgia's Statewide Automated Child Welfare Information System used by DFCS. SHINES is used to manage the state's child protective, foster care, and adoptions services programs. It is a web-based application accessed by DFCS personnel through a secure Virtual Private Network.
The system was designed and implemented by Accenture, LLP after a competitive bid process. The contract began in fiscal year 2006 and the system was deployed statewide in fiscal year 2008.
The federal government provides a 50% reimbursement for all costs associated with SHINES provided that the state meets certain reporting and management requirements. The initial contract for design, development, implementation, and help desk and maintenance was $49.8 million, of which $23.6 million was state funds. The estimated annual average cost for SHINES operation is approximately $13.6 million.

SHINES
Requested Information on Georgia's Child Welfare Information System
What we found Overall we found that the SHINES system was built as designed and without significant cost overruns. It was designed and implemented by Accenture, LLP according to Georgia's 2005 Request for Proposal and is completely web-based. The goals of the implementation contract were to improve effectiveness of the management of child welfare cases and to ensure the system performed the functions that would make managing cases more efficient.
Currently, DFCS case managers have laptops or tablet computers (e.g., a laptop with a swivel screen, not an iPad or other similar tablet device). Staff can access SHINES remotely, through a secure Virtual Private Network as long as they have an internet connection. However, the case managers do not have real-time access to the internet in the field, such as through air cards that could be plugged into their laptops and connected to a 3G/4G cell tower network, smart phones, or tablet devices, such as iPads.
According to reviews conducted by the federal Administration for Children and Families, reporting has improved with the implementation of SHINES. SHINES has the capacity to capture all information related to child welfare outcomes, including information related to child safety. It should be noted that the federal reviews focus largely on the ability of SHINES to capture all required information and on validating a small sample of SHINES data to scanned documents also included in the system.
While the system functions as designed, its effectiveness is dependent on the timeliness, quality, and reliability of the information entered by the end users. We conducted a limited review of timeliness by reviewing entry dates for allegations of abuse and neglect, initial visits and foster care visits. During fiscal

270 Washington Street, SW, Suite 1-156

Atlanta, Georgia 30334

Phone: (404)657-5220

www.audits.ga.gov

year 2011, 31% (18,833 of 60,207) of the reported allegations of abuse or neglect were entered into SHINES five or more days after they were actually received. DFCS policy requires that these allegations be entered immediately. We also found that 45% of the initial visits (20,870 of 46,396) occurred prior to the date the reported allegation was entered into SHINES.
Additionally, SHINES does not have edit controls over date fields to prevent errors or changes. Determining the frequency with which dates are being entered in error or overwritten, and the impact of such, would require visiting a statistically significant sample of DFCS offices and verifying entered dates with source paper documents (if available). This fieldwork was not possible within the time frame of this review.
What we recommend We recommend consideration be given to further review of the issues related to controls over data entry and how the system is being operationalized in the individual DFCS offices. Due to the time constraints of this project, we were unable to fully explore the implications and underlying causes.
This report is intended to provide answers to questions posed by the House Appropriations Committee. We hope that this report provides pertinent information to inform policy decisions.

SHINES

i

Table of Contents

Purpose of the Special Examination

1

Background

1

SHINES Development and Implementation

1

Federal Law & Regulations

2

Organizational Structure

3

SHINES Technology

4

Training

5

Federal Monitoring

6

Financial Information

8

Requested Information

9

The SHINES system functions as designed, but its effectiveness is dependent upon the timeliness, quality, and reliability of the information entered by the end users. 9

Has SHINES improved the accuracy of data reported to the federal level?

14

Is SHINES capturing information deemed critical to child safety?

15

Was SHINES implemented as planned; were there any major delays or

modifications?

16

What was the total cost of the system (by implementation phase

and fund source)?

18

Is SHINES accessible and usable according to end users?

21

Appendices

23

Appendix A: Objectives, Scope, and Methodology

23

Appendix B: SHINES Development and Federal Reviews

26

Appendix C: Intake Entry by County

27

Appendix D: AFCARS Critical Data Elements

31

Appendix E: SHINES User Survey Results

33

SHINES

ii

SHINES

1

Purpose of the Special Examination
This review of the Department of Human Services' Division of Family and Children Services' SHINES data system was conducted at the request of the House Appropriations Committee. Specifically, the Committee was interested in the cost of the SHINES system and whether the system was effective, in terms of users being able to access it real-time and input information critical to the safety of the child. We addressed the following objectives:
Has SHINES improved the accuracy of data reported to the federal level?
Is SHINES capturing information deemed critical to children's safety?
Was SHINES implemented as planned; were there any major delays or modifications?
What was the total cost of the system (by implementation phase and fund source)?
Is SHINES accessible and usable according to end users?
A description of the objectives, scope, and methodology used in this review is included in Appendix A. A draft of the report was provided to the Department of Human Services' Division of Family and Children Services (DFCS) for its review, and pertinent responses were incorporated into the report.
Background
SHINES Development and Implementation Georgia's child welfare services programs, including child protective services, foster care, and adoptions, are state-supervised and administered through 159 local offices in 15 service regions. Prior to SHINES, Georgia used a system known as IDS to manage child welfare cases. According to DFCS staff, IDS was an amalgamation of several different systems and not a complete child welfare information system. IDS had little top-down control and few statewide reporting features. As a result, DFCS relied heavily on each of the local offices to submit clean, properly formatted data to meet federal reporting requirements.
Since 1994, the state has attempted to develop a federally-approved Statewide Automated Child Welfare Information System (SACWIS) (Appendix B contains a timeline of SHINES development). In 2005, these efforts culminated in a Request for Proposal for a system that would achieve the goals of better outcomes for children, greater efficiency for their staff, and improved accountability. The contract for design and implementation was awarded to Accenture, LLP after a competitive bid process. The system design was based on Texas' SACWIS solution and was named SHINES.
SHINES is used by DFCS to manage all state child welfare programs, including child protective services, foster care, and adoptions. It was deployed statewide in June 2008 after a successful pilot in Douglas County. All division employees located in the counties use SHINES to manage their cases. Each county and regional office has its own technological infrastructure allowing their employees to access SHINES. This infrastructure is managed by Georgia Enterprise Technology Services under a contract with the Department of Human Services.

SHINES

2
Federal Law & Regulations Congress addressed concerns regarding the lack of data on children receiving foster care and adoption services in the Omnibus Budget Reconciliation Act of 1986. The Act required the U.S. Department of Health and Human Services' Administration for Children and Families (ACF) to create a national foster care and adoptions data system (the Adoption and Foster Care Analysis and Reporting System (AFCARS)) to track outcomes.
Congress later passed the Omnibus Budget Reconciliation Act of 1993, which further addressed the need for more uniform state data for the new AFCARS system. Under Title IV-E of the Social Security Act, ACF offered reimbursement to states for up to 75% of the costs of developing an ACF-approved SACWIS. Requirements were added for planning, approval, and data field requirements and a system of monitoring and compliance was also created. In fiscal year 1997, reimbursement rates were reduced to 50%.
Federal funding can be used for development, implementation, operation, and training costs. In order to be eligible for funding, the SACWIS must function as the sole child welfare case management tool in the state and contain all case management histories for children served under Title IV-E or IV-B of the Social Security Act.1 The SACWIS must also report all required data elements to ACF for inclusion in the AFCARS database. Implementing a SACWIS is optional; however, whether operating a SACWIS or not, all states must report the required foster care and adoptions data elements to ACF for the AFCARS database (a list of the required elements is included in Appendix D).
In order to ensure compliance with the funding requirements, ACF conducts several different reviews of the system and the state's reporting of data and child welfare outcomes. These reviews are shown in Appendix B and discussed in more detail on page 6.
SACWIS Requirements As noted above, in order to qualify for the 50% reimbursement, states must meet certain reporting and management requirements outlined in the federal law and regulations. States must submit Advanced Planning Documents (APD), including a business plan, supporting the federal expenditures for the system. This information includes a list and description of the state planning team, system deliverables, project schedule, needs assessment, feasibility study, alternatives analysis, cost benefit analysis, and the resources used in development of a Request for Proposals (RFP). The APD must also include a budget showing planned costs by federal and state fund sources.

1 Under Title IV-E of the Social Security Act, the federal government provides entitlement funds that support state foster care programs. Title IV-E funds support payments to foster care providers, recruitment of foster care parents, and training of staff. Federal Title IV-B grant funds support preventive intervention programs to keep children from being removed from their homes, including family preservation and reunification programs. These funds are also used for evaluation and research.

SHINES

3
Georgia is currently one of 36 states (including the District of Columbia) with an operational SACWIS solution. Three additional states are in the development stage and 13 states operate non-SACWIS child welfare systems (including Puerto Rico).
Exhibit 1 Division of Family and Children Services SHINES Organizational Chart

Organizational Structure The SHINES application is managed by a director and supported by a team of DFCS personnel (Exhibit 1). The SHINES Director and team are part of the DFCS Federal Regulation and Data section. SHINES staff is responsible for training, maintenance, and improvements to the application. DFCS also contracts with Accenture, LLP to provide ongoing maintenance and help desk services for SHINES. Accenture staffs the help desk, fielding calls from SHINES end users. Accenture also works with DFCS personnel to make changes and technical improvements to the system. Hardware and IT support services are provided through a contract with Georgia Enterprise Technology Services.

SHINES

4

SHINES Technology
SHINES is the sole case management system for all DFCS child welfare activities and is used by all DFCS staff, by external private foster care providers, and by other Georgia child welfare stakeholders such as the Georgia Office of the Child Advocate and Administrative Office of the Courts. SHINES also interfaces with the Case Plan Reporting System used by juvenile court judges and Court Appointed Special Advocates. In addition to SHINES' case tracking capabilities, the system also produces all required federal reports and all data used for federal child welfare entitlement and grant programs. This includes Title IV-B and Title IV-E foster care and adoptions funds and grant funds from the Promoting Safe and Stable Families program and Child Abuse Prevention and Treatment Act. The child welfare phases and processes tracked through SHINES are described in Exhibit 2.

Exhibit 2 Child Welfare Processes and Descriptions Supported by SHINES

Child Welfare Process

Description

Intake

Record initial contact regarding allegations of abuse or neglect; includes a search of records for any prior history

Case Screening

Evaluate intake information to determine whether it is necessary to establish a case; establish case records, and assign to case manager

Investigation

Collect all investigation information, the final investigation decision, and all related records and contacts

Assessment

Automated risk assessment and re-assessment of risk factors affecting the case; collect and record special needs/services, referrals to other agencies

Eligibility Determination Case Management
Resource Management Court Processing

Determine eligibility for Title IV-E, Medicaid, and other related programs
Prepare and document case plans, match client with available placements, generate automated alerts to complete required case activities
Record provider information based on the types of services, level of care, and cost
Prepare state court documents including petitions, letters, and supervisory approvals, and notifications of pending court action

Financial Management Administration Reporting
Source: SHINES RFP

Facilitate accounts payable and accounts receivable
Record and track case assignments and workload; online user help guides and manuals
Produce all required federal and state reports

SHINES

5
Users enter information into a series of tabs and screens; access to individual screens is driven by the user's job role (Exhibit 3.) For example, an investigator will not have access to the same screens as a foster care case manager, and vice versa. There are 170 application screens in SHINES.
Exhibit 3 SHINES Portal: Screen Shot with Tabs for Data Entry

Source: SHINES data system
All DFCS child welfare personnel who work in the field (investigators, foster care case managers, etc.) are provided with a laptop or tablet computer.2 Approximately 3,500 laptops were first distributed to child welfare case managers statewide in 2004. They were provided to all DFCS offices for use by child welfare staff as part of a Field Productivity Enhancement Project. These laptops have since been updated and are currently leased through the Department of Human Services' contract with Georgia Enterprise Technology Services (GETS). The current laptops have swivel screens, internal Wi-Fi capability, and Ethernet connections. Case managers can connect to the internet, and thus to SHINES, using the Ethernet connection in their DFCS county or regional office. Additionally, they can access SHINES through the secure Virtual Private Network (VPN) using a Wi-Fi connection at home, or at a Wi-Fi `hotspot' such as a coffee shop, fast food restaurant, or public library.3
Training All DFCS case management personnel were trained to use SHINES during the statewide system rollout in 2008. The department's Education and Training Section currently conducts a week-long training session for all new staff, prior to beginning
2 Tablet refers to a laptop with a swivel screen, not an iPad or other similar tablet device. 3 DFCS does not fund home broadband services for case managers.

SHINES

6
their assignments. The session includes SHINES training specific to the individual's job role. For example, new foster care case managers are trained on the relevant foster care case management screens. The Data Integrity Specialists Unit also conducts on-going and refresher trainings on an as-needed basis. Finally, SHINES management also distributes materials when the application is updated or changed; the distribution is either through email or a web-based training course.
Federal Monitoring
There are several federal monitoring reviews that assess the state's compliance with federal foster care and adoptions laws and regulations. (Appendix B shows the federal reviews Georgia has undergone since 2001.) Because Georgia has a SACWIS, it is also subject to a SACWIS Assessment Review, which measures compliance with regulations and requires changes be made as necessary based on the review's findings. The Assessment Review is conducted by ACF. The AFCARS Assessment Review, the Child and Family Services Review, and Title IV-E Reviews either directly or indirectly evaluate elements of SHINES. Each review is discussed in more detail below:
SACWIS Assessment Review: ACF seeks to ensure that the SACWIS implementation is conducted in accordance with federal regulations and policies and that the state is making progress towards becoming fully SACWIS compliant. It also ensures that the system is created as described in the state's submitted planning documents. These reviews may be conducted at any time and consist of a state self-assessment and an on-site system assessment by ACF examiners. The review concludes, for each of the 90 requirements, whether the system conforms, conditionally conforms, or does not conform. States can be deemed "out of conformance" because of missing financial components, existing paper processes, redundant data entry processes, faulty alerts and notifications, or Title IVE eligibility determination issues. Georgia's last review was conducted in 2009.
AFCARS Assessment Review All states operating programs under Title IV-E of the Social Security Act must submit data to the national AFCARS database. AFCARS Assessment Reviews focus on SHINES' capability to collect, extract, and transmit data accurately. The review uses a ratings scale of 0 to 4 to quantitatively assess the data sample submitted.
For example, one data element the state must report is the number of children in foster care who have been diagnosed with a disability. The AFCARS team reviews this data field and compares it to related fields and potentially to hardcopy records, to assess whether the data is complete, accurate, and verifiable to other documents. The review focuses on ensuring the reported data accurately reflects SHINES data.
The review covers 22 general requirements and 103 specific data elements. If problems are identified, the review team develops an improvement plan that addresses the individual problems specific to the data elements reviewed. The plan may include corrective action such as revising program coding, addressing incomplete fields, and adding needed fields. The state is required to address the plan. The last review was completed in 2011.

SHINES

7
Child and Family Services Review (CFSR) This review focuses on the state's outcomes for child safety, permanency, and well-being related to children in state custody and is conducted by a joint federal and state team. The review includes two parts: a comparison of the outcomes for Georgia's foster care population to national averages and a review of 65 sample cases at three sites.
Through the sample review of 65 cases, the CFSR seeks to determine whether, based on source documents, the state is operating in compliance with its own policies. The team compares information in the data system to source documents to confirm accuracy. For example, the state reports on the "timeliness initiating investigations of reports of child maltreatment." In 2007, the CFSR team reviewed the 25 cases in which this measure was applicable (of the 65 cases selected) and determined that the state initiated investigations in accordance with its policies in 19 cases (76% of the time). The state received a rating of needs improvement in this area.
Georgia had a CFSR in 2001 and in 2007; both covered information contained in IDS. SHINES was being developed during the second review. The next review is scheduled to occur in 2013.
Title IV-E Reviews These reviews are conducted by ACF to determine whether federal Title IV-E reimbursements, made for eligible children, were accurate. This review is primarily financial. It is designed to determine whether federal funds were used for eligible children in eligible placements and, if not, to recoup any overpayments. The reviewer pulls a statistical sample of child welfare cases from the AFCARS database, as well as corresponding documentation, and reviews the cases for Title IV-E eligibility. Georgia was reviewed in 2009 and August of 2012.

SHINES

8

Financial Information
SHINES related expenditures are discussed in detail beginning on page 18. Since the contract began in 2006, the state has spent $50 million on the design, development, implementation, and maintenance of the SHINES system. An additional $51 million has been provided in federal funds, for total expenditures of $101 million. SHINES is not a separate line item in the Department of Human Services' (DHS) budget; rather, these expenses are imbedded in its administrative budget and DFCS child welfare programs.

According to DFCS staff, SHINES primarily supports four child welfare budget programs: Adoption Services, Child Welfare Services, Special ProjectChild Welfare Services, and Out of Home Care. SHINES costs are included in these program budgets and expenditures as well as in the Department of Human Services' administrative budget. Since fiscal year 2011, expenditures in these four programs have stayed relatively stable, increasing 0.5% over the period. However, overall, DFCS has seen a 28% reduction from 2011 levels. Approximately $180 million (46%) of this reduction was because the Child Care Program was moved from DFCS to the Department of Early Care and Learning in fiscal year 2013.

Exhibit 4

Fund Sources and Expenditures for DFCS for Fiscal Years 2011-2013

FY 2011

FY2012

FY2013

(Actual)

(Actual)

(budgeted)

FUND SOURCES

State Federal1
Other

$ 333,414,835 $ 1,032,793,512 23,709,471

358,262,910 $ 873,912,414 25,200,266

309,652,872 668,436,181 19,823,541

Total Fund Sources

$ 1,389,917,817 $ 1,257,375,590 $ 997,912,594

PROGRAM EXPENDITURES Adoption Services Child Welfare Services Special Project - Child Welfare Services Out of Home Care
Subtotal: Child Welfare Programs

$ 87,585,188 $ 253,293,012 500,000 192,128,029
533,506,228

96,479,314 $ 250,069,747
250,000 194,159,433
540,958,494

88,607,444 252,686,570
250,000 194,511,601
536,055,615

After School Care Child Care Services2 Energy Assistance Family Violence Services Refugee Assistance Support for Needy Families - Basic Assistance Support for Needy Families - Work Assistance Federal Fund Transfers to Other Agencies Community Services
Federal Eligibility Benefit Services

13,671,000 274,644,284 83,864,221 12,992,140
9,951,040 51,482,360 56,317,252 102,989,789 19,148,200 231,351,302

Subtotal: All Other DFCS Expenditures

856,411,589

Total Expenditures

$ 1,389,917,817 $

1 Federal Funds for 2011 include federal stimulus funds 2 Responsibility transferred to the Department of Early Care and Learning

Source: People Soft and HB 742

15,897,972 181,559,853 65,693,309 14,172,124
8,130,438 49,143,077 21,441,150 97,465,895 17,926,957 244,986,320
716,417,096
1,257,375,590 $

15,650,000 -
25,171,849 14,085,964 8,749,006 51,482,361 21,725,483 86,561,488 17,189,183 221,241,645
461,856,979
997,912,594

SHINES

9

Requested Information

The SHINES system functions as designed, but its effectiveness is dependent upon the timeliness, quality, and reliability of the information entered by the end users.
According to DFCS, SHINES was intended to be a comprehensive child welfare case management tool that would help the state maximize federal funding for child welfare programs and meet federal data reporting requirements. SHINES was also intended to replace IDS, the state's previous child welfare reporting system. IDS was decentralized and managed at the local level. In its RFP for the design and development of the SHINES system, DFCS identified goals including the following:
Improve case managers' efficiency and effectiveness and eliminate duplicate data entry.
Provide timely, accurate information and alerts so that case managers and supervisors can make timely, quality case decisions and actions.
Improve the capability to provide access to information and tools that will consistently support policy and practice standards throughout the state.
Provide current and accurate data for mandatory reporting, performance management, and accountability.
Provide consistent and accurate data management and reporting to maximize federal funding.
Provide an internal and external audit trail of all cases sufficient to meet state and federal requirements.
While SHINES was designed to meet these goals, successful implementation depends on staff using it as designed. Additionally, as the system contains policydriven rules, controls are necessary to prohibit the system from being circumvented. DFCS recognized there were inherent challenges to successful implementation and, during the planning process, identified specific areas of concern. These areas were documented in the state's planning documents submitted to ACF and included: concern about the individual offices' interpretation of the state's policies on child welfare and how SHINES addressed these policies; acceptance and use of the system by case managers and supervisors; and "culture shock" in the move to automation. DFCS took steps to address each of these issues, including training of local directors on policy interpretation and launch of a SHINES pilot program in a single county prior to statewide implementation.
As discussed in later sections, overall we found that the system was built as designed without significant cost overruns. However, our review found that stronger controls are needed to ensure data integrity. Currently, line and supervisory staff can change dates (e.g., intake report dates or dates investigations and child visits occurred) after they have been initially entered. Additionally, edit controls are not present to prevent or limit incorrect entries. For example, we saw January 12, 1949 as an entry date for a fiscal year 2011 report of abuse. To determine the frequency with which dates are being overwritten or entered in error, and the impact of such, would require site visits to multiple local DFCS offices and verification of entered dates

SHINES

10
with source documents. This fieldwork was not possible within the timeframe of this review.
Additionally, we found that staff are not entering information into SHINES in a timely manner relative to when the actual events occurred.4 The timeframe in which DFCS must respond, and determine whether to remove a child, is driven by the date a report of alleged abuse or neglect is received. According to statute and regulations, DFCS investigations, assessments, court hearings, case plans, and site visits, must be completed within established timeframes, which are based on the removal date. Inaccurate dates in the system could affect child safety, federal compliance, and, potentially, federal funding. We analyzed the difference between the dates events occurred and the dates the information was entered according to a system date stamp that cannot be changed. As discussed in the following section, there were indications that the data entry fields are not being used as designed and that information is not being entered in a timely manner.5 Our analysis regarding timeliness is discussed on the following pages. These areas are noted for further review and consideration.
Recording Critical Dates in SHINES
Individuals report allegations of child abuse or neglect to DFCS county intake personnel who are responsible for entering the information into SHINES (these reports are referred to as intake reports). According to DFCS policy, all intake reports must be recorded in SHINES immediately upon receipt and SHINES is constructed to allow for real-time data entry.
Cases are assigned a priority of "immediate" or "5 work days," which indicates the timeframe in which a child must be seen by a case manager. Case managers are supposed to record their findings in SHINES within three business days of their initial visit.
Based on the investigation findings, the child may be placed in foster care. Foster Care case managers are required to visit each child on their caseload once a month (in DeKalb and Fulton counties case managers are required to visit each child twice a month). Case managers are supposed to enter information on these visits into SHINES within three business days.
When any of these events are entered in SHINES, two date fields are generated:
SHINES Entry Date: This is the date and time the allegation (intake report) was reported to DFCS, the initial visit was conducted or the site visit occurred. For intake reports, if the user is entering information in real-time, SHINES automatically generates the date and time. However, the date and time can also be entered manually if the intake is being entered after the fact. DFCS staff indicated this capability is necessary because some intake reports are faxed and the actual date and time of the receipt of the fax must be entered into the SHINES system. For initial visits and foster care, the employee enters the date and time for the event. This date can be changed by a supervisor after it has been submitted.
SHINES Intake Entry Date Stamp: This date is automatically generated by SHINES upon entry of the event and is unchangeable.
4 Events, for purposes of this report, include reports of abuse and neglect, initial visits, and foster care visits. 5 SHINES allows the employee to enter the date an event actually occurred; however, it also timestamps the entry. As a result, each event includes two dates, the date the event occurred and the date the event was entered into SHINES. The two dates may or may not be the same.

SHINES

11
Timely entry of information DFCS policy states that all allegations of abuse or neglect (referred to as intake reports) must be entered upon receipt. In support of this policy, and as designed, SHINES allows users to enter information in real-time, while taking the information over the phone. However, during fiscal year 2011, 77% (46,070 of 60,207) of all intake reports were not entered into SHINES the same day they were received (see Exhibit 5 below and Appendix C for a breakdown of intake entry by county). Additionally, 31% (18,833 of 60,207) of reports were entered into SHINES five or more days after the intake report was received.
Exhibit 5 77% of the Intakes Were Not Entered on the Day They Were Received in Fiscal Year 20111

23%

21%

10% 8% 6%

31%

0%

20%

40%

60%

80%

100%

Time Between Intake Report and Intake Entry2

Same Day

Next Day

2 days later

3 days later

4 days later

5+ days later

1Graph includes 60,207 intakes from fiscal year 2011for which there was both an intake date and a timestamp for entry into SHINES. 2Percentages will not add to 100% due to rounding.
Source: SHINES Database

According to DFCS policies, once an intake report is entered into SHINES, it is automatically assigned a priority. The priority for visitation is "immediate," "up to 24 hours" or "5 work days" based on the data entered from the initial call, and dictates how soon a case manager must make initial contact with the child.6 However, according to our analysis, 45% of initial visits began prior to intake information being entered into SHINES (see Exhibit 6). As a result, case managers would have been directed to conduct an initial visit by a process other than that supported by SHINES.

6 These categories were used in calendar year 2011. In 2012, the "immediate" and "up to 24 hours" categories were combined.

SHINES

12
Exhibit 6 45% of Initial Visits Occurred Prior to Intake Entry in Fiscal Year 20111

21%

16%

18%

45%

0%

10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Date Contact Occurred Compared to Intake Entry
3+ Days After 1-2 Days After Same Day Prior
1Includes 46,396 initial visits from a total of 60,207 intakes from fiscal year 2011. Source: SHINES

We also analyzed data on foster care visits made during calendar year 2011. DFCS
policies state that case managers must visit children in foster care once a month and
the date and details of all visits must be entered into SHINES within three business days of the visit.7 According to the data, 54% (121,129 of 225,297) of all foster care
visits were entered within this time frame. The remaining 45% (102,500 of 225,297) were recorded later.8 Of the 225,297 total entries, 67,888 (30%) occurred between
six and 30 business days after the visit (See Exhibit 7).

7 Fulton and DeKalb county case managers must visit children in foster care twice per month. Per DFCS policy, this includes business days. Weekends are excluded from the 72 hour calculation.
8 1,643 records were missing visit information. According to DFCS, this occurred because of system errors related to the merging of cases in SHINES or because the initial investigator was unable to locate the family based on the information provided by the reporter. An additional 25 records had visit dates which occurred prior to the date of entry of the visit.

SHINES

13

Exhibit 7
45% of Foster Care Visits Were Entered More Than 3 Business Days After the Site Visit Occurred in Calendar Year 20111

54%
within 72 hours2

45%
after 72 hours2

10%

20%

23%

12%

30%

3%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Date of Visit Information Entry into SHINES Compared to Actual Date of Visit

Same day 4 - 5 business days after

Day after 6 - 30 business days after

2 - 3 business days after 30 - 358 business days after

1For 225,297 statewide foster care case visit contacts occurring in calendar year 2011. This includes 7,905 cases and 13,348 persons. Each child in foster care must be visited at least once per month by a DFCS case manager (Fulton and DeKalb county case managers must visit twice per month). 2Percentages will not add up to 100% due to rounding and due to cases with missing visit information (1,643, 1%).
Source: SHINES
Agency Response: In its response, the agency noted that it agrees that "enhanced system controls could improve data accuracy." The agency indicated that there are "legitimate instances in which overwrites or modifications are reasonable...The purpose of this flexibility is to allow supervisory staff, after reviewing the intake report (including history, allegations, determination factors, etc.), the ability to record the appropriate Response Time." The agency also indicated that it is developing a procedure to monitor frequent overwrites to determine if any trends or anomalies can be detected. If trends are detected, the agency stated that "they will be addressed appropriately."
In response to the analysis that indicated 45% of initial investigation visits occurred prior to intake information being entered into SHINES (indicating workers were employing a process other than that supported by SHINES), the agency noted that: "The safety of children is our greatest concern, therefore, if we receive a phone call indicating that a child may be unsafe, we would opt to make the home visit as soon as possible, even if it means that the report would have to be entered into SHINES after the visit to assess the child's safety had been made."

SHINES

14

Has SHINES improved the accuracy of data reported to the federal level?
For purposes of federal reporting, accuracy is assessed in terms of whether the submitted data conforms to established guidelines and regulations and whether that data is complete. Recent federal monitoring reviews, discussed previously on pages 6 and 7, suggest that DFCS' ability to conform to federal reporting requirements has improved as compared to its ability under IDS. For example:
Georgia is currently a "SACWIS compliance assessment initiated" state according to the Administration for Children and Families (ACF). As the state attempts to achieve full SACWIS compliance, the ACF conducts SACWIS Assessment Reviews (SAR). The SAR is focused largely on the utility of the data system rather than the accuracy of the data. Georgia's most recent SAR occurred in 2009. Georgia SHINES failed to comply with 18 of 88 SACWIS requirements.9 Additionally, SHINES was conditionally compliant with 31 of the elements reviewed. However, the state did meet the requirements for 34, or 41%, of the SACWIS elements reviewed.
According to the state's latest AFCARS review in 2011, Georgia SHINES was generally compliant with standards. The technical changes required were characterized as minor.
DFCS cites the state's increased Title IV-E penetration rate (this is the rate of eligible children in foster care qualifying for federal IV-E funds reimbursements) from 35% in 2010 to 58.1% in 2011 as evidence of SHINES ability to capture data for improved federal reporting performance.
Georgia has undergone three Title IV-E reviews. The first and second Title IV-E reviews evaluated child welfare cases that were managed under IDS. The 2009 review used case information data from SHINES. The state's performance improved from the 2006 review to the 2009 review. The 2006 review found that the state was not in substantial compliance with federal requirements, resulting in a Program Improvement Plan (PIP). Since that time, the state accomplished all of the required improvements identified on the PIP. The 2009 review found Georgia to be in substantial compliance with federal eligibility requirements. The 2012 review was completed during the course of this special examination but the details of the results are not yet available.
Both the 2001 and 2007 CFSRs evaluated IDS. In response to the PIP developed to address findings from 2007, DFCS pointed to SHINES as a strategy for improvement. The state has since met all of the requirements of the PIP, thus avoiding federal financial penalties.
All of these reviews are focused on the data contained in SHINES (or the system that was in use at the time of the review). While improvement plans may address specific business processes, or data field edit checks, these recommendations are

9 5 Five of the 88 requirements were not applicable to SHINES. Therefore the total evaluated requirements for this review was 83.

SHINES

15

limited to the specific data elements reviewed. Hence, problems originating from data being altered or fabricated would not be identified unless hardcopy files were retained that were in conflict with entered data.
Child welfare outcomes have also improved since SHINES implementation in 2008; however, we could not verify that these outcomes had improved directly as a result of SHINES implementation. In addition, DFCS estimates that the time required to enter an intake was reduced after SHINES implementation (from 101 minutes to 48 minutes) and that this results in an estimated cumulative savings of $10.6 million from 2008-2013. DFCS calculated these as direct funds saved as a result of SHINES implementation compared to IDS. In addition, DFCS notes that the state avoided $8 million in federal penalties as a result of SHINES implementation. While the reviews noted above monitor certain aspects of the SHINES system, they are not designed to detect the types of problems discussed in the previous section such as issues related to data entry.
Is SHINES capturing information deemed critical to child safety?
For our purposes, "critical" information was determined to be the data Georgia is required to submit to ACF for the AFCARS database. In addition, we considered information on investigations and site visits by case managers to be information critical to the safety of the child.
The state is required to submit information on 66 foster care and 37 adoptions data elements to ACF annually to be entered in the AFCARS database. All of this information is collected in SHINES. (See Appendix D for a description of these data elements). As described previously, ACF reviews the state's information system to assess the quality, accuracy, and integrity of the information submitted as well as the system's ability to collect, extract and transmit the data accurately. ACF also conducts a SACWIS assessment review of SHINES. A SACWIS review can be initiated by ACF or by the state. There is no required schedule for these reviews.
Georgia's latest AFCARS review occurred in 2011 and the state was found to be "in conformance with most of the AFCARS standards" and to be "user-friendly and robust in the type of information that it has the capacity to collect". The state received a rating of full compliance for 14 of 66 foster care elements (21%) and 19 of 37 adoption elements (51%). The review concluded that the technical changes required were minor but that the quality of the data needed to be monitored and that the "[s]tate will need to develop and implement a method to ensure accurate and timely entry of data into SHINES."
SHINES contains additional fields related to child safety, permanency, and wellbeing, such as records of site visits by investigators and case managers. SHINES includes the date the visit occurred, who conducted the visit, and comments on the content of the visit. The combination of data in these fields is used to support measures regarding child safety. For example, while these particular date elements may not be reported to ACF, the elements would be reviewed during CFSR reviews that focus on outcomes. As noted earlier, however, these reviews are not designed to address overall business practices and usage of the system (ensuring it is in line with the intent). The reviews are designed to ensure compliance with the state's policies, not to assess the appropriateness or completeness of the information.

SHINES

16

Agency Response: In its response to the report, the agency noted that it is "continuously exploring ways to improve the utilization and integrity of SHINES in order to capture and report [the] most accurate and timely information about child welfare in Georgia." It indicated that, in response to the AFCARS review described above, its Data Quality Unit's focus for the past year has been on reviews of cases "in which a child is moving from foster care to an adoptive placement...Once [it] is satisfied that [the] adoption data is as accurate as possible, the [Unit] will focus on another area." It also noted that the Program Evaluation and Analysis Section, which is part of the Office of Quality Management, randomly selects cases for review based on AFCARS reporting elements.

Was SHINES implemented as planned; were there any major delays or modifications?

State IT projects, valued at or above $1 million, must have a third party Independent Verification and Validation (IV&V) Vendor regularly evaluate the project schedule, budget, objectives, risks, issues, and organizational readiness.

According to a review of the following documents, all project deliverables were noted as completed: SHINES RFP, Accenture proposal, Accenture contract for Design, Development, and Implementation, contracts for the Help Desk and Maintenance Services, project change control forms, and the Independent Verification & Validation (IV&V) reviews.
In 1994, DFCS attempted to initiate a SACWIS system; while a SACWIS system was not fully developed, DFCS did receive federal financial and technical assistance during this time and was able to make infrastructure and technology upgrades. According to staff, these efforts helped prepare DFCS to make a second attempt at developing a SACWIS in 2002. Over time, the project was delayed for two primary reasons:

The state contracted for planning assistance; however, the contract was terminated due to performance issues resulting in project delays. These delays added $1.1 million to the initial planning costs, increasing the total preplanning costs to $4.6 million. In addition, this same vendor was slated to serve as the Independent Verification and Validation (IV&V) vendor; therefore, DFCS had to select a new one.

Data from the IDS system had to be converted to SHINES. The state contracted with a vendor to perform the conversion yet delays occurred and added costs were incurred during implementation for Accenture to provide extra conversion support.
As a result, the initial RFP for the SACWIS was delayed by twenty-one months over the original plan, according to the earliest available planning document. Statewide rollout of the SACWIS was delayed by nine months.
During the project implementation phase (February 2006 to July 2008), the state implemented two processes, using external reviewers, to ensure the system was being implemented per the agreed contract. The control processes involved change orders and verification of deliverables as described below:

A committee of child welfare stakeholders and DFCS personnel reviewed all change orders, and, if the change order resulted in a cost increase, it was also submitted to ACF for review. There were four change orders to the Accenture design, development, and implementation contract between February 2006 and October 2008, resulting in increased costs of $309,000

SHINES

17
(half of which were covered by federal funds). This represented a 1.2% increase over the initial contracted agreement. The current contract for maintenance and help desk services (with Accenture), which was established in December 2007 and has been renewed annually, includes 13 amendments over the period resulting in additional costs of $280,850, of which, 91% were paid for with federal funds.
The SHINES IV&V vendor's final project review found the project to be within budget and that the majority of identified risks had been mitigated. Remaining issues included those related to the inclusion of adoptions data in the SHINES system and payments to providers through the DFCS provider payments system.
Planned System Enhancements
Several enhancements are planned for the SHINES system in the future, including functionality related to provider selection and payment tracking. These include:
SHINES Hardware and Technical Upgrades The addition of 12 new servers and supporting software to house the SHINES system. These servers will increase speed and provide data storage sufficient to house all information for 5 years at an average cost of $333,000 per year. There was anecdotal evidence that access issues were being experienced at the local DFCS offices. Problems, according to DFCS, were attributed to outdated equipment and wide area network limitations in the county offices. According to DFCS, these issues are being addressed in a statewide LAN/WAN upgrade currently in progress.
Safety Framework DFCS is in the planning phase of redefining the state's policy and practices pertaining to child safety, called Springboard Georgia. Changes will require added processes in SHINES with additional assessments and more documentation. It is estimated that these changes and enhancements will cost $2.5 million.
Fostering Connections Title IV-E eligibility determinations for children in foster care are being separated from Title IV-E determinations for eligibility for federal adoption assistance funds. This process was recently completed and included several additional SHINES screens and functions.
Family Preservation Resource Management Currently, case managers needing to access services related to family preservation (in-home services, assessments, etc.) must use a paper form for approval. Future enhancements to SHINES will add automated support for this process. DFCS personnel have noted that the difficulty of automating this data is due to the large number of providers across the state and the small dollar amount of their contracts (average annual cost per provider is $70,000). No estimate of the cost of these changes is currently available.
Adoption Assistance Application and Agreement This change will result in automated processing and payment of adoptions funds to providers. No estimate of the cost of these changes is currently available.

SHINES

18

What was the total cost of the system (by implementation phase and fund source)?
The complete estimated cost of SACWIS/SHINES implementation and operation from 1994 through 2012 is approximately $142 million. This includes $41 million for technology and infrastructure upgrades expended from 1994 to 2004 (including $8.8 million in 2004 for laptop computers for all case managers), $25.3 million for design and development of the system, and $24.5 million for ongoing help desk and maintenance costs (see Exhibit 8). Additionally, projected costs for operations, maintenance, and upgrades average approximately $13.6 million per year for federal fiscal years 2013 to 2016 (for a total cost of $54 million over the next four years). These figures do not include the leasing of laptops for all case managers or technology infrastructure costs to deliver internet service to the state's 159 local DFCS offices.10
Exhibit 8 Estimated SHINES Costs for Federal Fiscal Years 2005-2012

2005-2012 Federal Fiscal Years (FFY) SHINES Costs

Millions of Dollars

$-

$10

$20

$30

2005 $1 $2

2006 $2 $3

2007

$6

$5

2008 2

$13

$13

2009

$8

2010

$8

$8 $7

Federal State

2011

$6

$6

2012

$7

$7

1Figures are unaudited. Above costs include planning, design and development, operations, personnel, and maintenance costs. 22008 costs include the majority of Accenture contract expenses for design, development, and implementation of the SHINES system (approximately $25 million, half federal funds, half state funds).
Source: DFCS SACWIS Advanced Planning Documents

10 The Department of Human Services contracts with Georgia Enterprise Technology Services for all technology infrastructure used by the DFCS field offices.

SHINES

19

In 2004, the state also provided all DFCS case managers with tablet PCs for use in the field.11 The initial purchase of 3,500 tablet PCs cost approximately $8.8 million
in state funds. These computers have since been updated and are now leased from
Georgia Enterprise Technology Services. Per DFCS, the average monthly cost is $124 per laptop.12 The monthly costs include replacement and repair costs as well as
technical support.

In 2005, DFCS issued a Request for Proposal (RFP) for design and implementation of a SACWIS solution. Bids were received from six vendors and Accenture, LLP was selected. Implementation began in February 2006 and was completed in October 2008. The total contract cost for Accenture to design, test, and deploy SHINES statewide was $25.3 million (See Exhibit 9)

Exhibit 9

SHINES Design, Development, and Implementation Costs

(Calendar Years 2006-2008)

Vendor

SHINES Implementation Phase

Total Cost1

Federal

Planning

$2,499,786 $1,249,893

Accenture Design,
Development, and
Implementation Contract

Design, Build, Test

$10,124,134

Pilot

$5,374,542

Training

$2,749,765

Statewide Deployment

$4,249,637

Contract Adjustments / Changes

$308,850

$5,062,067 $2,687,271 $1,374,883 $2,124,819
$154,425

Total

$25,306,714 $12,653,357

1Figures are unaudited Source: DFCS Accenture Project Contracts

State
$1,249,893 $5,062,067 $2,687,271 $1,374,883 $2,124,819
$154,425 $12,653,357

In addition to design and implementation of the SHINES system, the state also contracts with Accenture to provide maintenance and help desk support. The contract is due to expire in December 2012 and a new RFP for these services has been issued. At the time of release of this report, a new contract has not yet been awarded.
The current Accenture maintenance contract averages $4 million per year and these costs are expected to continue through 2016 (see Exhibit 10 for a breakdown of maintenance and help desk costs paid to Accenture from 2008 to 2012). The contract also included further enhancements to the SHINES system and additional funding from the Child Abuse Prevention and Treatment Act and the Promoting Safe and Stable Families Program. These grants supported planned enhancements and have historically been used to produce training materials and add additional capabilities to the SHINES system.

11 Tablet refers to a laptop with a swivel screen, stylus, and handwriting recognition technology, not an
iPad or other similar device. 12 Currently, there are 3,900 laptops under contract; however not all of these laptops are used solely by
the child welfare staff. This includes all statewide DHS field service personnel in the county offices.

SHINES

20

The total paid to Accenture for the maintenance, help desk operations, and enhancements of the SHINES system is $24.5 million. Combined with the contract cost for implementation, the total Accenture SHINES contract cost is $49.8 million.

Exhibit 10 SHINES Maintenance and Help Desk Costs (Fiscal Years 2008-2013)

Fiscal Year
2008

Description
Initial contract for maintenance and help desk support for SHINES from Dec 1, 2007 through June 30, 2008.

Total Cost1 $2,680,662

Federal

State

$1,340,331 $1,340,331

2009 Contract cost through June 30, 2009.

2009

Decrease of $230,000 (half federal and half state) due to an adjustment based on the actual hours provided by the Accenture staff.

2010 Contract cost through June 30, 2010.

2010

Use of federal grant funds from the Child and Adolescent Protection Act (CAPTA) and Every Child Every Month (ECEM) to enhance federal reporting systems in SHINES.

2011 Contract cost through June 30, 2011.

2011

Further enhancements to comply with federal reporting requirements related to CAPTA and ECEM.

2012 Contract cost through June 30, 2012.

2013

Contract extension to December 31, 2012.
Total

1Figures are unaudited Source: DFCS - Accenture Project Contracts

$5,688,738 $2,844,369 $2,844,369

-$230,000

-$115,000 -$115,000

$4,010,000 $2,005,000 $2,005,000

$1,659,663 $1,659,663

$0

$3,800,000 $1,900,000 $1,900,000

$1,659,687 $1,278,559

$381,128

$3,648,000 $1,596,800 $24,513,550

$1,824,000 $1,824,000

$798,400

$798,400

$13,535,322 $10,978,228

Currently, DHS leases all laptops from Georgia Enterprise Technology Services at an average cost of $124 per month per laptop. DHS notes that there are approximately 3,500 laptops in use by the child welfare staff for a total annual estimated cost of $5.2 million. According to DFCS, the annual estimated cost to operate SHINES will be $13.6 million per year for the next four federal fiscal years (2013-2016).13 Combined these figures total $18.8 million and represent the annual average cost for delivering SHINES access to the DFCS field staff for the next four years.

DFCS staff also note that, currently, there are no major budgeted technology upgrades for field personnel, such as real-time connectivity solutions to the SHINES system, either in the form of air cards or true tablet devices (such as iPads). According to staff, it would cost an estimated $1.44 million per year to provide air cards to all case managers for full-time wireless access to SHINES. DHS has also

13 These figures include all maintenance and operations costs, personnel, and contract services for SHINES. They do not include costs for the Department of Human Services' statewide technology infrastructure paid to Georgia Enterprise Technology Services.

SHINES

21

noted other possible solutions, such as replacing all field laptops with `netbook' devices which have built-in air cards to access the internet. However, these plans are only in the beginning stages and no concrete estimates for statewide implementation are available. Based on interviews with ACF personnel, while a few states have distributed real-time solutions on a local or regional basis, none have done so statewide.
Is SHINES accessible and usable according to end users?
Based on initial interviews with the requestor and DFCS staff, there were concerns that field staff may be experiencing problems accessing and using SHINES, either due to hardware or software issues. To determine whether such problems exist, we surveyed all DFCS county social services staff and supervisors (see Appendix E for survey questions and results). The survey included questions regarding the usability of SHINES and access to the system for data entry. Of the 3,043 employees surveyed, 509 (17%) in 120 offices responded. Due to the small response, we did not extrapolate the responses to the whole population. However, we have included the results in Appendix E and selected responses are included below.
Overall, responses indicated that data on child welfare events are not being entered into the system on the date the events occur but rather on the following day or later. Regarding accessibility issues, the survey asked employees to indicate their agreement, on a scale from strongly disagree to strongly agree, with the statement "I can usually access SHINES in the field when I need to." While 24% (72 of 294) responded they agreed or strongly agreed, 42% (215 of 509) indicated the question was not applicable to them. This response appears to agree with responses to a question regarding use of laptops in the field in which 63% (321 of 509) indicated that they leave their laptop in their home or office rather than take it into the field. Of the 509 respondents, 67% (339) stated that they spend at least one day a week in the field and 92% (470) use a laptop or tablet computer.
In response to: "I can usually access SHINES in the office when I need to" 80% (405 of 508) of respondents agreed or strongly agreed.
When asked to rate SHINES on a scale of 1 to 5 (1=poor to 5=excellent) regarding speed, connectivity, saving work, and submitting work, responses averaged 3.5.
When asked how often they experience issues with accessing SHINES, 127 (25%) said weekly, 149 (29%) said monthly and 105 (21%) said once every few months.
When asked "How much time passes between an event and entering it into SHINES," 4% (16 of 360) replied that they enter the information on the same day.14 59% (212 of 360) replied that they enter the information within one to three days.

14 "Events" may include court dates, home visits, intakes, assessments, etc.

SHINES

22
When asked whether they agree that they are able to enter case related information into SHINES within 72 hours of receipt, collection, or event, 41% (171 of 415) agreed or strongly agreed.
192 respondents (42%) agreed or strongly agreed that they would like more training on SHINES.

SHINES

23
Appendix A: Objectives, Scope, and Methodology
Objectives This report examines the Department of Human Services' Division of Family and Children Services information system called SHINES. Specifically, our examination set out to determine the following:
Has SHINES improved the accuracy of data reported to the federal level? Is SHINES capturing information deemed critical to children's safety? Was SHINES implemented as planned; were there any major delays or
modifications? What was the total cost of the system (by implementation phase and fund
source)? Is SHINES accessible and usable according to end users?
Scope This audit generally covers activity related to SHINES that occurred from its creation in 2005 to June 2012 with consideration of earlier or later planning periods when relevant. Information used in this report was obtained by reviewing relevant laws, rules, and regulations, interviewing agency officials and staff from DFCS, analyzing data, planning documents, and reports provided by DFCS and the federal government, as well as by conducting a survey of SHINES users. Three data sets were used to inform the first and third objectives, respectively. The first two data sets were from the SHINES system and included intakes and foster care visits for fiscal year 2011. The reports were provided by the DFCS data unit. The third data set was the compilation of survey responses collected by the audit team.
Given the compressed time frame for report submission and the limited nature of the review this project was not conducted according to GAGAS. We addressed internal controls both directly and indirectly as part of our work in all three objectives. Information related to the scope of our internal control work is described by objective in the methodology section below.
Methodology In order to determine if SHINES improved the accuracy of data reported to the federal level we reviewed four major federally-conducted evaluations of child welfare systems. The SACWIS Assessment Review appraises the functionality of a state's SACWIS system. Information regarding Georgia's most recent performance on a SACWIS Review can be found in the background and in the second finding of this examination. Additionally, the team studied the results of three major child welfare reviews that include an evaluation of information system elements. Those three were the Title IV-E Review, the Adoption and Foster Care Analysis and Reporting System Review, and the Child and Family Services Review. The team collected all of the most recent and available versions of these documents both from ACF and from DFCS. The results of these reviews are discussed in the second section of this examination.
In order to determine if SHINES is capturing information deemed critical to children's safety we used the results from Georgia's most recent AFCARS Review.

SHINES

24
The ACF named 66 foster care and 37 adoptions data elements `critical'. These critical elements are required to be reported to AFCARS. The AFCARS Review therefore reports the pervasiveness and accuracy with which these elements are being supplied to ACF by the state's information system. Results regarding SHINES' ability to report all ACF-defined critical fields can be found in the third section of this examination.
As part of determining whether critical information was being collected in the SHINES system, we analyzed the timeliness of data entry for intake and foster care visits. These analyses resulted in information that is presented in the first section of this examination. This intake report data was analyzed to determine the timeliness of data entered into the SHINES system based on records of intake calls and the SHINES system timestamp of the data entry of those records. DFCS provided intake records from fiscal year 2011 which included over 60,000 entries. This information included the report date, report entry date, and first visit date among other fields. The results from the analysis of this data are in the first finding of this report. In Exhibit 7 on page 13, analyst calculated the time between foster care visit dates and the dates of entry into SHINES using only "business days". DFCS policy states that records of foster care visit entries should be entered into SHINES within 72 hours allowing for weekends. Analyst used Excel to remove weekends from the calculations.
To determine whether SHINES was implemented as planned and whether there were any major delays or modifications, we reviewed the Annual Planning Documents DFCS developed and submitted every year since the inception of SHINES, as well as all project milestones. We also reviewed the: SHINES RFP, Accenture proposal, the Accenture contract for Design, Development, and Implementation, contracts for the Help Desk and Maintenance Services, project change control forms, and the Independent Verification & Validation (IV&V) reviews.
To determine the total cost of the system (by implementation phase and fund source) we collected and compared a variety of financial documents. We reviewed the following to compile information on expenditures: DFCS SACWIS Annual Planning Documents, which contain historical and projected expenditures for SHINES development and operations, from federal fiscal year 2004 to 2011; and contracts with Accenture, LLP for implementation, maintenance, and help desk services from calendar years 2006 to 2012. The audit team also reviewed Budget Comparison Reports, the Governor's Budget Report, and the Appropriations Bills from fiscal years 2004 to 2013. However, with the exception of the contract costs, all costs associated with SHINES (e.g., personnel, technology and infrastructure upgrades, and training costs) could not be separately identified in these sources because SHINES costs are not budgeted as a separate line item; however, they are imbedded in various administrative and subprogram budget categories.
To determine if SHINES is accessible and usable according to end users, we designed, distributed, and analyzed an email-based survey for all case managers and regional staff. The objective of the survey was to determine:
What are the challenges and shortcomings of SHINES? Are the problems technological, equipment, or user error?

SHINES

25
Have all end users been trained? If not, why not? How is training conducted? Does the system meet user's needs? If not, what is missing?
The audit team wrote a series of questions regarding accessibility, data entry, and training. The questions were either multiple choice or designed using Likert scaling. The final survey can be seen in Appendix E. It was distributed by email to 3,043 DFCS employees statewide. The list of recipients was provided by DFCS and included every county and regional staff member that uses SHINES.
Results were received by the audit team electronically, then compiled and analyzed. A summary of the responses to each question is available in Appendix E. The response rate was 16.8% (509 of 3,043), with respondents from 120 local offices. Though the response rate was not high, we found the results reliable because the survey was conducted on all users rather than a sample of SHINES users. A more detailed discussion of the results of the survey begins on page 21 of this report.
Non-GAGAS Project This special examination was not conducted in accordance with generally accepted government auditing standards (GAGAS) given the timeframe in which the report was needed. However, it was conducted in accordance with Performance Audit Division policies and procedures for non-GAGAS engagements. These policies and procedures require that we plan and perform the engagement to obtain sufficient, appropriate evidence to provide a reasonable basis for the information reported and that data limitations be identified for the reader.

SHINES

26

Appendix B: SHINES Development and Federal Reviews

SHINES

27

Appendix C: Intake Entry by County

Graph shows the time needed by each county DFCS office to enter 50% or more of their intake reports into SHINES (an intake report is a call, fax, email, letter, or personal report to a DFCS office alleging abuse and/or neglect of a child has occurred). DFCS policy states that all intake reports are to be entered into SHINES immediately upon receipt. Some counties did not achieve 50% of intake reports entered within 4 days.

Total Intakes Fiscal Year 2011
948 831 1,189 1,336 1,255 832 1,272 3,276 2,001 871 1,099 1,220 3,839 2,984 1,047 1,514 1,279 956 1,370 903

20 Counties with the Highest Number of Intakes

County

Same Day

Next Day

2 days later 3 days later 4 days later

Barrow Bartow Bibb Chatham Cherokee Clarke Clayton Cobb DeKalb Dougherty Douglas Floyd Fulton Gwinnett Hall Henry Muscogee Paulding Richmond Whitfield

56% 55%
56% 57%
59% 53%
59% 62%
44% 54% 51% 73% 72% 52% 70% 55% 37% 30% 45% 44%

Total Intakes Fiscal Year 2011
209 107 124
22 416 107 106 305 140 273 124
77 132 180 171
26 426
76 647 478 152
52 Source: SHINES

County
Appling Atkinson Bacon Baker Baldwin Banks Ben Hill Berrien Bleckley Brantley Brooks Bryan Bulloch Burke Butts Calhoun Camden Candler Carroll Catoosa Charlton Chattahoochee

All Other Counties

Same Day

Next Day

2 days later 3 days later 4 days later

51% 54% 81%
41% 37% 53% 69% 61% 51% 65% 56% 53% 50% 54% 44% 42% 57% 59% 35% 81% 72% 37%

SHINES

Total Intakes Fiscal Year 2011
296 30
104 374 481 381 254 528 116 188
90 193 269 133
65 74 23 246 136 146 78 233 480 818 152 418 22 510 423 184 74 278 45 386 69 172 214 451 79 608 92 137 125 86 99 184 143 157 463 204 Source: SHINES

County
Chattooga Clay Clinch Coffee Colquitt Columbia Cook Coweta Crawford Crisp Dade Dawson Decatur Dodge Dooly Early Echols Effingham Elbert Emanuel Evans Fannin Fayette Forsyth Franklin Gilmer Glascock Glynn Gordon Grady Greene Habersham Hancock Haralson Harris Hart Heard Houston Irwin Jackson Jasper Jeff Davis Jefferson Jenkins Johnson Jones Lamar Lanier Laurens Lee

28

Appendix C (continued)

Same Day 58% 59% 61% 52%
61%

Next Day

2 days later 3 days later 4 days later

61% 63%
57%

58% 31%

57% 42%
50% 62%

33% 56% 58%
12%

57% 57%
53% 56% 61% 65% 51% 58%
52% 50% 54%
38% 52% 53% 69% 53% 67%
54% 55% 53%
59%

54% 54% 59% 51% 70% 63% 54% 57% 69% 60% 38%

SHINES

29

Total Intakes Fiscal Year 2011
527 47 63
742 196 166 222
48 238
80 141
63 158 219 104 169 369 778 201
96 125 372 232 114 627
69 171
13 183
52 560
46 48 67 233 321 40 318 28 10 114 45 69 79 298 549 325 98 79 527 Source: SHINES

County
Liberty Lincoln Long Lowndes Lumpkin Macon Madison Marion McDuffie McIntosh Meriwether Miller Mitchell Monroe Montgomery Morgan Murray Newton Oconee Oglethorpe Peach Pickens Pierce Pike Polk Pulaski Putnam Quitman Rabun Randolph Rockdale Schley Screven Seminole Spalding Stephens Stewart Sumter Talbot Taliaferro Tattnall Taylor Telfair Terrell Thomas Tift Toombs Towns Treutlen Troup

Appendix C (continued)

Same Day 64% 52%
52%

Next Day

2 days later 3 days later

50% 79%
54%

4 days later

60%

56% 29%
56% 56% 50%
25% 62% 58% 54% 56% 61% 21% 55% 52% 44% 55% 71% 50% 52% 46% 60% 54% 57% 52% 58%

60% 34% 25% 49%
68% 35%
50% 70%
61% 62% 65%
53% 48%
63% 33%
62% 57% 50%

SHINES

30

Appendix C (continued)

Total Intakes Fiscal Year 2011

County

Same Day

Next Day

2 days later 3 days later 4 days later

46 Turner

63%

39 Twiggs

56%

153 Union

54%

253 Upson

54%

585 Walker

52%

710 Walton

23%

507 Ware

60%

49 Warren

61%

193 Washington

66%

249 Wayne

51%

17 Webster

59%

30 Wheeler

63%

227 White

58%

87 Wilcox

57%

67 Wilkes

51%

56 Wilkinson

63%

261 Worth

52%

286 Out of State1

53%

70 None2

53%

60,207 Statewide

55%

1Out of State refers to inquiries from other states regarding children placed in Georgia.

2Intakes for which there is no county of residence or for which the county was not entered by the intake worker.

Source: SHINES

SHINES

31

Appendix D: AFCARS Critical Data Elements

AFCARS Critical Data Elements - Foster Care

# 1 State

# 34 Child's Behavior Problem

# 2 Report Date

# 35 Death of Parent

# 3 Local Agency (County)

# 36 Incarceration of Parent

# 4 Record Number

# 37 Caretaker Inability to Cope Due to Illness or Other

# 5 Date of Most Recent Periodic Review (if applicable) # 38 Abandonment

# 6 Child Birth Date

# 39 Relinquishment

# 7 Child Sex

# 40 Inadequate Housing

# 8 Child's Race

# 41 Current Placement Setting

# 9 Hispanic/Latino Origin

# 42 Is Current Placement Out-of-State?

# 10 Disability

# 43 Most recent case plan goal

# 11 Mental Retardation

# 44 Caretaker Family Structure

# 12 Visually/Hearing Impaired

# 45 1st Primary Caretaker's Birth Year

# 13 Physically Disabled

# 46 2nd Primary Caretaker's Birth Year (if applicable)

# 14 Emotionally Disturbed

# 47 Mother's Date of TPR

# 15 Other Medically

# 48 Father's TPR (Termination of Parental Rights)

# 16 Has this child ever been adopted?

# 49 Foster Family Structure

# 17 If yes, how old was the child?

# 50 1st Foster Caretaker's Birth Year

# 18 Date of First Removal from Home

# 51 2nd Foster Caretaker's Birth Year

# 19 Total Number of Removals from Home

# 52 1st Foster Caretaker's Race

# 20 Date Child was Discharged from foster care

# 53 1st Foster Caretaker's Hispanic or Latino Origin

# 21 Date of Latest Removal

# 54 2nd Foster Caretaker's Race (if applicable)

# 22 Date of Latest Removal Transaction Date

# 55 2nd Foster Caretaker's Hispanic Origin

# 23 Date of Placement in Foster Care

# 56 Date of Discharge from foster care

# 24 Number of Previous Placement Settings

# 57 Date of Discharge Transaction Date

# 25 Manner of Removal From Home

# 58 Reason for Discharge

# 26 Physical Abuse

# 59 Title IV-E (Foster Care)

# 27 Sexual Abuse

# 60 Title IV-E (Adoption Subsidy)

# 28 Neglect

# 61 Title IV-A (Aid to Families with Dependent Children)

# 29 Parent Alcohol Abuse

# 62 Title IV-D (Child Support)

# 30 Parent Drug Abuse

# 63 Title XIX (Medicaid)

# 31 Child Alcohol Abuse

# 64 SSI or other Social Security Act Benefits

# 32 Child Drug Abuse

# 65 None of the Above

# 33 Child Disability

# 66 Amount Of Monthly Foster Care Payment

SHINES

32

Appendix D - continued

AFCARS Critical Data Elements - Adoptions

# 1 # 2 # 3 # 4 # 5 # 6

State Report Period End Date Record Number State Agency Involvement Child Date of Birth Child Sex

# 20 # 21 # 22 # 23 # 24 # 25

Date of Father's TPR Date Adoption Legalized Adoptive Family Structure Adoptive Mother's Year of Birth Adoptive Father's Year of Birth Adoptive Mother's Race

# 7 Child's Race # 8 Child Hispanic Origin # 9 Has Agency Determined Special Needs # 10 Primary Basis for Determining Special Needs
# 11 Mental Retardation
# 12 Visually/Hearing Impaired
# 13 Physically Disabled
# 14 Emotionally Disturbed # 15 Other Diagnosed Condition # 16 Mother's Birth Year # 17 Father's Birth Year # 18 Mother Married at Time of Birth # 19 Date of Mother's TPR

# 26 # 27 # 28 # 29
# 30
# 31
# 32
# 33 # 34 # 35 # 36 # 37

Adoptive Mother's Hispanic Origin Adoptive Father's Race Adoptive Father's Hispanic Origin Relationship of Adoptive Parent to Child - Stepparent Relationship of Adoptive Parent to Child - Other Relative Relationship of Adoptive Parent to Child - Foster Parent Relationship of Adoptive Parent to Child - Other NonRelative Child Was Placed from
Child Was Placed by Receiving Monthly Subsidy Monthly Amount Adoption Assistance - title IV-E

Source: Administration for Children and Families (ACF)

SHINES

33

Appendix E: SHINES User Survey Results
The audit team surveyed 3,043 Division of Family and Children Services employees across the state. The survey was intended to measure SHINES users' experience with the speed, connectivity, accessibility, and performance of the program. 509 unique responses were received for a 16.8% response rate.

County 2 Peachtree, Atl. Appling Banks Barrow Bartow Ben Hill Berrien Bibb Bleckley Brantley Brooks Bryan Bulloch Burke Butts Camden Candler Carroll Catoosa Chatham Chattooga Cherokee Clarke Clayton Cobb Coffee Colquitt Columbia Cook Coweta Crawford

# of Responses
4 2 3 5 8 2 4 16 1 2 1 1 2 4 3 4 1 6 3 9 3 8 4 5 14 2 10 1 3 1 4

Geographic Information: Office Location

County

# of Responses

County

# of Responses

Crisp

1

Jackson

6

Dade

1

Jefferson

1

Decatur

2

Jones

3

DeKalb

25

Lamar

3

Dodge

1

Lanier

4

Dooly

2

Laurens

1

Dougherty

4

Lee

1

Douglas

6

Liberty

3

Effingham

3

Long

1

Elbert

2

Lowndes

10

Emanuel

2

Lumpkin

1

Evans

2

Macon

1

Fannin

5

Marion

1

Fayette

2

McDuffie

1

Floyd

9

Meriwether

3

Forsyth

4

Miller

1

Franklin

2

Mitchell

4

Fulton

22

Monroe

2

Gilmer

7

Montgomery

1

Glynn

6

Murray

6

Gordon

4

Muscogee

9

Grady

4

Newton

4

Greene

1

Oconee

2

Gwinnett

16

Oglethorpe

1

Habersham

4

Other Location

6

Hall

8

Paulding

7

Hancock

1

Peach

1

Hart

5

Pickens

1

Heard

2

Polk

4

Henry

6

Pulaski

1

Houston

4

Putnam

4

County Rabun Randolph Richmond Screven Seminole Spalding Stephens Sumter Talbot Tattnall Taylor Telfair Terrell Thomas Tift Toombs Troup Twiggs Union Upson Walker Walton Ware Wayne Webster Wheeler White Whitfield Worth County Not Identified Total

# of Responses
1 1 7 2 4 6 3 5 1 1 1 3 3 3 4 1 4 1 3 1 5 4 6 1 1 1 3 10 1 29 509

SHINES

34

General Information (For Questions 1-8, N=509) 1. How long have you been in your current position?

1 to 5 years 240

6 to 10 years 117

11 to 19 years 60

2. On average, how many days a week are you in the field?

20 or more years 21

1

2

3

4

5 Not Applicable

54

46

106

86

47

170

11%

9%

21%

17%

9%

33%

3. Which of these following best describes your current PRIMARY role?

Administrative Staff Adoptions County/ Regional Director Family Preservation Family Support/Diversion Intake Investigation Other Placement Resource Manager Supervisor

32

6%

19

4%

31

6%

45

9%

11

2%

30

6%

59

12%

88

17%

91

18%

12

2%

91

18%

4. Which of these SHINES functions do you currently use (mark all that apply)?

Intake Case Screening Investigation Assessment Eligibility Determination Foster Care Adoption Resources Management Legal Financial Management Reports Approvals LENSES Other Not Applicable

214 42%

311

61%

212 42%

221 43%

44

9%

244 48%

146 29%

84

17%

235 46%

79

16%

334 66%

189

37%

151 30%

105

21%

2

0%

SHINES

35

5. On average, how many cases are you responsible for at any given time?

1 to 10 10 to 15 15 to 20 20 to 30 30 to 40 More than 40 Not Applicable

54

11%

64

13%

77

15%

57

11%

11

2%

103 20%

143

28%

6. What type of state-issued hardware do you use to access SHINES (mark all that apply)?

Tablet/Laptop

470 92%

Desktop

89

17%

Mobile Phone

86

17%

Not Applicable

1

0%

7. If applicable, do you GENERALLY take your laptop/tablet into the field or leave it in the office/home?

Into Field Into field but leave in car Leave in office or home Not Applicable

79

16%

17

3%

321

63%

92

18%

8. What was the primary method of training used during your INTIAL SHINES training?

Classroom Online Other Peer Training Self-taught Text Box Webinar

386

76%

13

3%

10

2%

50

10%

39

8%

5

1%

6

1%

SHINES

36

9. Please rate your response to the following statements regarding SHINES usability:

Strongly Disagree

Strongly

Not

Agree Applicable

1

2

3

4

5

I like to use SHINES frequently. N=502

5%

13% 29% 27%

27%

24

64 144 134

136

7

I find the system unnecessarily

complex. N= 505

12% 23% 26% 20%

19%

63

118 129 99

96

4

I think the system is easy to use. N= 506

9%

18% 31% 29%

13%

44

93 158 147

64

3

I need the support of a technical

person to use this system. N= 490

48% 27% 18% 6%

2%

233

131

90

27

9

19

I think the various functions of

this system are well integrated. N=503

11% 24% 37% 20%

7%

56

120 187 103

37

6

I think there is too much inconsistency in this system.
N=491

10% 20% 32% 20%

17%

51

99 158 98

85

18

I would imagine that most people

would learn to use this system very quickly.

16% 27% 31% 16%

9%

83

138 157 82

45

4

N=505

I think the system is very cumbersome to use.
N=494

11% 24% 28% 20%

17%

56

118 136 100

84

15

I feel very confident using the system.
N=498

2%

6% 29% 33%

30%

10

32 142 165

149

11

I needed to learn a lot of things

before I could get going with this

17% 22% 23% 21%

16%

system.

85

108 114 101

79

22

N=487

SHINES

37

10. Please rate your response to the following statements regarding SHINES connectivity and training:

Strongly Disagree

Strongly

Not

Agree Applicable

1

2

3

4

5

I can usually access SHINES

remotely using my VPN when I

15%

12% 17% 27% 29%

need to.

67

50

73 118

125

76

N= 433

I can usually access SHINES in

the office when I need to. N=508

2%

5% 14% 40% 40%

8

23

72 203

202

1

I can usually access SHINES in

the field when I need to. N= 294

43%

16% 16% 13%

12%

127

47

48

37

35

215

I am adequately trained to use all

of the SHINES screens/modules I

4%

need to fill my responsibilities.

21

14% 20% 34% 27%

69 102 172

138

7

N= 502

I would like more SHINES training.
N=490

18%

19% 24% 21%

18%

90

92 116 102

90

19

The training I received allows me

to perform all of my job functions

7%

in SHINES.

34

13% 26% 33% 22%

64 127 162

107

15

N=494

I am aware of all of the mandatory fields that I must enter case information for (AFCARS, etc.).

7%

18% 24% 32% 19%

30

77 105 139

83

75

N=434

I would like more ongoing training.
N=493

15%

15% 28% 24%

17%

72

76 140 119

86

16

I am able enter case related

information into SHINES within

72 hours of receipt, collection, or

11%

20% 28% 28% 13%

event.

45

81

118 115

56

94

N=415

When assigned a NEW case, I am

able to immediately view all

relevant information in SHINES at the time of assignment.

11%

20% 28% 27%

15%

40

74 103 100

57

135

N=374

SHINES

38

11. Please rate your experience with the following elements/functions of SHINES:

Poor

Excellent

Not Applicable

1

2

3

4

5

Speed (the loading of web

information on screen, etc.) 13% 19% 37% 23%

7%

N= 490

66

95

182

112

35

19

Connectivity (accessing

SHINES, logging on, etc.) N= 507

11% 21% 34% 28%

6%

56

107 174 140

30

2

Saving work N=477

6% 14% 36% 32%

12%

29

65

173 154

56

32

Submitting work N=467

5% 10% 35% 36%

13%

25

49 164 170

59

42

12. How often do you have issues accessing SHINES for any reason, if ever? N=509

Daily Multiple times a week Weekly Monthly Once every few months Never Not Applicable

42 8% 70 14% 127 25% 149 29% 105 21% 9 2%
7 1%

13. On average, how much time passes between an event (home visit, court hearing, assessment, investigation, medical, etc.) and that information being entered into SHINES? N=360

Same day 1 to 2 days 3 days 4 days 5 days More than 5 days Not applicable/ I do not go into the field

16

4%

102

28%

110

31%

43

12%

37

10%

52

14%

149

14. Which statement best describes your opinion of SHINES? N=509
o SHINES supports ALL of my required child welfare activities. o SHINES supports MOST of my required child welfare activities. o SHINES supports SOME of my required child welfare activities. o SHINES supports LITTLE of my required child welfare activities. o SHINES supports NONE of my required child welfare activities. o Not applicable / I do not use or access SHINES in my work.

76 (15%) 291 (57%) 116 (23%) 15 (3%)
1 (0%) 10 (2%)

SHINES

39

For Supervisors, Directors only:

1. If you supervise case managers, on average how many TOTAL CASES are you responsible for at any given time? N=271

0

2

1%

1 to 25

10

4%

26 to 50

38

14%

51 to 75

28

10%

More than 75

45

17%

Not Applicable

148

55%

2. Please rate your response to the following statements:

Strongly Disagree

Strongly Not

#

Agree Applicable Responding

1

2

3

4

5

I am able to track all of my case managers' cases in a timely manner.

1% 10% 23% 41% 24%

2

14 32 57

33

115

138

I can see my case

managers' outstanding

4%

6% 19% 44% 26%

tasks and required work

5

9 27 62

37

111

140

in SHINES.

I rely on SHINES

automated alerts, errors, and to-dos to track work that needs to be

8% 12

23% 23% 27% 36 35 41

19% 30

98

154

completed.

I am able to respond to SHINES automated alerts, errors and to-dos.

3% 4

13% 25% 36% 24%

19 37 53

36

103

149

The Performance Audit Division was established in 1971 to conduct in-depth reviews of state-funded programs. Our reviews determine if programs are meeting goals and objectives; measure program results and effectiveness; identify alternate methods to meet goals; evaluate efficiency of resource allocation; assess compliance with laws and regulations; and provide credible management information to decision-makers. For more information, contact
us at (404)657-5220 or visit our website at www.audits.ga.gov.