Special Examination 11-35

December 2011

Georgia Department of Audits and Accounts
Performance Audit Operations
Russell Hinton, State Auditor Leslie McGuire, Director

Why we did this review
This special examination was conducted at the request of the Senate Appropriations Committee. The Committee requested that we review a sample of agency performance measurement data submitted to the Office of Planning and Budget (OPB) to determine the (a) reliability of submitted data, (b) the relevancy of the measures given the purpose and operations of the agency, and (c) how best practice states use performance measures in decision-making and how data is collected and reported.

Governor's Office of Planning and Budget (OPB)
Requested Information on Performance Measures
What we found Performance measures can be used to inform budgetary decisions and to hold agencies accountable for achieving their goals. According to our review of best practices and a review of other states, there are certain aspects of effective performance systems which Georgia lacks. The states we reviewed have established structures that require specific information be provided to decision-makers, have assigned roles and responsibilities for implementing the process, and have established processes to ensure that performance information is reliable.

Who we are
The Performance Audit Operations Division was established in 1971 to conduct in-depth reviews of state programs. The purpose of these reviews is to determine if programs are meeting their goals and objectives; provide measurements of program results and effectiveness; identify other means of meeting goals; evaluate the efficiency of resource allocation; and assess compliance with laws and regulations.
Website: www.audits.ga.gov Phone: 404-657-5220 Fax: 404-656-7535

In Georgia, performance management information is provided to the General Assembly primarily through the Governor's Budget Report (GBR) and the state's Strategic Plan; however, these tools were not designed to inform legislative budgetary decisionmaking. Because these are the only formal tools available to the General Assembly, they were also the only tools we could compare with best practices in other states. Compared to the frameworks established in other states, cited as best practice states in the literature, Georgia's current structure does not provide for the type and amount of information, legislative participation, or controls over data reliability that are necessary to ensure that performance measures are useful and used.1
We assessed performance measures for six sample agencies in the GBR in terms of their relevance and reliability.2 Our review found that programs included in the GBR for three of the sample
______________________ 1As noted in the report, legislators may also request performance information directly from OPB or agencies. However, this is not a formalized process. 2The following Georgia agencies were selected for review: the Department of Community Health, the Department of Economic Development, the Department of Education, the Department of Transportation, the State Board of Pardons and Paroles, and the State Soil and Water Conservation Commission.

agencies did not represent priority programs given the proportion of agency funding dedicated to them. The remaining three did include measures for the programs receiving the largest portion of agency funds.
Additionally, the GBR does not include a comprehensive set of measures that would allow legislators to determine overall program or agency performance. Research on best practices indicates that a comprehensive set of measures, of various types, is most effective for legislative use. A comprehensive set of input, output and outcome measures is needed to hold programs accountable for achieving their purpose and cost-per-unit, workload, and efficiency measures assist decision-makers in making funding decisions and determining the state's return-on-investment.
The strategic plans of six sample agency were also reviewed. As implemented, the state's strategic planning process was not developed to assist the General Assembly in budgetary decision-making. It is an executive branch tool used by the Governor and his staff to hold agency heads accountable for performance and by the Office of Planning and Budget (OPB) to inform the budget development process. Currently, the agency plans do not include results related to the measureable objectives the agencies have established.
Finally, there is not a systematic process to ensure the validity and accuracy of the performance information presented in the GBR. Prior to 2007, agencies were required to submit disclosure statements detailing how data was collected and any limitations to that data for each performance measure submitted to OPB for inclusion in the GBR. However, since then, these disclosures have not been required or provided. Our review found that only one of six agencies, the Department of Transportation, has processes in place to ensure the accuracy of performance measurement data reported. It should be noted that, according to OPB personnel, agencies were instructed to maintain records for the measures reported and disclose "important information" about the measures in the description field in BudgetTool (the budget formulation and submission portal). Best practices describe valid and accurate data as having been succinctly defined by the agency with described limitations and benefits and which can be replicated by an outside entity using this information. Currently, Georgia's performance management data, as provided in the GBR and the state Strategic Plan, do not meet these criteria.
Although information in the GBR is not sufficient for budgetary decision-making, it is possible that legislators can and do request additional information from agencies as they deem necessary. It is also reasonable to expect, and we did find, that agencies have collected performance information on measures other than those reported in the GBR that could be useful for making decisions. However, currently, the GBR is the only established method of providing performance information to decision-makers. If changes are made to provide for a more robust performance management system, additional review would be beneficial to determine how other states with effective performance management systems implemented their systems and what obstacles they encountered. We did not include an analysis of actual implementation or the challenges encountered to implement an effective system as part of our review.
In its response to the report, OPB noted that the report "provides helpful information to the state in the area of best practices in performance measures. The summary of best practices will inform the improvements being made by the Office of Planning and Budget (OPB) and its partners in the Legislative Branch to monitor and improve program performance. We believe any desired increase in legislative oversight can be achieved through the existing committee structure, without statutory revisions. Additionally, OPB will continue to work with the legislative budget offices as part of our Zero Based Budget (ZBB) analysis process to review agency performance measures." OPB also noted that the Governor's Budget Report is "not intended to be the sole source of performance measures." The response noted that it is "common practice for OPB to share its working papers including performance information and research with the House and Senate budget offices to explain the Governor's Budget recommendations." Additional information from OPB's response is included in the body of the report following pertinent sections.

Requested Information on Performance Measures

Table of Contents

Purpose of the Special Examination

Background

State Performance Management Processes

Georgia's Performance Management Framework

Program Performance Reporting

Strategic Planning

Requested Information

Provide an analysis of how best practice states use performance measures in

decision-making and how data is collected and reported. What structures do

these states have in place to ensure the data is reliable?

What are the qualities of "effective" performance measurement and what types

of measures are most suited for use in budgeting decisions (i.e., most relevant

to legislators)?

Provide an analysis of the relevancy of the measures given the purpose and operations

of the agency. Do the indicators effectively measure the outcomes and efficiency

of the program as defined in the purpose statement of the appropriations bill

or in statute?

Provide a survey of the reliability of data as submitted. How do agencies collect and

confirm the data and rate the quality and accuracy of the data utilized.

Appendices

Appendix A: Objectives, Scope and Methodology

Appendix B: Sample Performance Questions for Legislators' Use

Appendix C: Program Reporting as Related to Program Funding

Appendix D: Analysis of Performance Measures for Six Sample State

Agencies

Appendix E: Comparison of Georgia Agency Performance Measures to

Other States

Requested Information on Performance Measures

Purpose of the Special Examination
Our review of state agency performance measures was conducted at the request of the Senate Appropriations Committee. The Committee asked that we review a sample of agency performance measurement data submitted to OPB to determine the (a) reliability of submitted data, (b) the relevancy of the measures given the purpose and operations of the agencies, and (c) how best practice states use performance measures in decision-making and how data is collected, verified, and reported. We did not evaluate the implementation of an effective performance system as part of our review.
A description of the objectives, scope, and methodology used in this review is included in Appendix A on page 25. A draft of the report was provided to the Governor's Office of Planning and Budget (OPB) for its review, and pertinent responses have been incorporated in the report as appropriate.
Background
State Performance Management Processes An effective state performance management process can be used to hold agencies accountable for achieving their goals and to inform budgetary decision-making. Performance management includes the following core processes: planning, budgeting, managing, evaluating, and measurement & reporting. These activities comprise an ongoing cycle (as shown in Exhibit 1) and occur at all levels of government, from the Governor's Office to the program management level.
Exhibit 1
The Performance Management Process

Source: The National Performance Management Advisory Commission

Requested Information on Performance Measures

Effective performance reporting relies on collecting reliable performance data from state activities and evaluating the results based on targeted goals established by stakeholders with an interest in the outcome. This information can then be used to coordinate and further direct state activities towards accomplishing goals. It also allows stakeholders to determine how to allocate limited resources and assess "return-on-investment" for future funding and funds already expended.

While a performance management framework can be used internally by

To instill accountability in government...states should create a performance-driven tracking and
measurement system that gives citizens a clear view of how
government programs are working

an organization to ensure it is operating effectively, we focused our review on how this information is used to inform the budgetary process. Benefits to using performance information in managing government programs include the ability to determine the effect of budget decisions and whether state activities are actually contributing to the state's goals.

and whether they are achieving

results.

While there are benefits to utilizing performance management

Source: Pew Center for the States

information, it cannot completely replace the traditional budget process and the "ad hoc" factors that go into it, such as the many political factors,

financial concerns, and socio-economic conditions that affect funding decisions at the

state level. As such, performance measures do not always provide a mechanistic or

formulaic link to a funding decision, especially within the special nature of government

service management as some programs and agencies cannot be held accountable

through the budget process and may require funding regardless of perceived program

performance or management's efficiency or productivity (for example, state Medicaid

programs).

Additionally, there may be significant obstacles, including costs, to producing reliable performance information at the state level. To overcome these obstacles, states need the personnel to collect and evaluate the information; technological capacity to produce reliable information; statutory, regulatory, and operational authority to require performance measurement; and acceptance by the legislature, budget offices and agencies, of the value of performance information in decision-making. Establishing this infrastructure may be expensive and the overall value and impact of performance information should be weighed with the cost of producing it. However, the Pew Center on the States notes that reliable and useful state program performance information is necessary for stakeholders (the Governor, legislators, budget offices, and the public) to hold agencies and programs accountable for achieving their statutory purpose and goals and to determine funding decisions with limited state resources. 1

1 The Pew Charitable Trusts is an independent non-profit organization; its mission is to improve public policy, inform the public and stimulate civic life. As a Division of the Charitable Trusts, the Pew Center on the States identifies and advances state policy solutions.

Requested Information on Performance Measures

Georgia's Performance Management Framework Performance management information is primarily available to the General Assembly through two avenues: the reporting of program performance measurement information in the Governor's Budget Report (GBR) and the state strategic planning framework.2 These processes were largely established in 1993 with passage of the Budget Accountability and Planning Act (SB335), which addressed both program performance reporting and strategic planning. The Act made OPB responsible for managing these processes and acting as a hub for agency and program performance information. Since 1993, the General Assembly has made changes to the statute and to these processes which have weakened the requirements for collecting and reporting this type of information (See Exhibit 2 on the next page). Currently, OPB is required to:

Develop and implement a program budgeting system that relates funding to achievement of established goals and objectives, measures agency performance against attainment of planned outcomes, and provides for program evaluations for policy and funding determinations.

Develop and implement a process of strategic planning to establish and periodically update an overall plan for state government and require all state agencies to develop a strategic plan that is consistent with that overall state plan.

Consult with the General Assembly on a regular basis concerning the development and implementation of the strategic planning process, and the development of outcome measures for program evaluation.
The state's strategic planning process requires that all state agencies submit agency strategic plans to OPB with goals and measurable objectives for their activities.3 According to OPB, this information is used by the Governor and his staff to hold agency heads accountable for performance and to inform the budget development process. In addition, OPB requests, but does not require, agencies to report on a select number of program-level performance measures for inclusion in the annual Governor's Budget Report.

2 The creation of agency budgets is the third method by which performance information is provided to OPB and used to inform budgetary decision-making. Budget analysts working with agency program personnel may use, view, and/or collect agency performance data when formulating the individual agency budgets; however, this is not a formalized process. 3 Statute requires that each department, board, bureau, commission, institution, authority, and other agency of state
government submit a strategic plan to OPB.

Requested Information on Performance Measures

Exhibit 2 Timeline of Georgia Agency Performance Management

1993

Budget Accountability and Planning Act of 1993: o Created the Budgetary Responsibility Oversight Committee (BROC): composed of an equal
number of representatives appointed by the Speaker and senators appointed by the President of the Senate. The Committee consulted with OPB and the Governor regarding the development and implementation of the strategic planning process, including reviewing and evaluating the strategic plans for the state and for individual agencies, as submitted by OPB. o Required annual program evaluations be conducted, and submitted to the Committee, by OPB and the Department of Audits and Accounts. The evaluations described the program, assessed its performance, compared the performance with other entities, and recommended improvements. o Reaffirmed the role of OPB in budgeting and strategic planning. o Required OPB to institute a process of outcome-based budgeting (formulated as "Results Based Budgeting" (RBB) with measures to be included in the GBR).

1998 First year of Results Based Budgeting (RBB) performance measures included in the GBR by agency.

2004 Last year of RBB. Additional narrative information was also provided by each agency regarding performance and accomplishments. Governor instituted Prioritized Program Based Budgeting.

2005 Agency performance measures were included in the GBR by program. HB 509 replaced outcomebased budgeting with program budgeting (effective in fiscal year 2006). The fiscal year 2006 Appropriations Act was reformatted to appropriate funds by program and fund source rather than by agency and object class.

2006 No performance measures were included in the GBR.

2008 HB 529 eliminated BROC and formally established the Senate and House Budget Offices. BROC activities were effectively transferred to these offices but specific statutory activities were not realigned. No performance measures were included in the GBR. OPB reorganized strategic planning processes, merging five separate planning requirements into a single process. Agency performance measures provided in the fiscal year 2008 GBR by program.

2009 OPB eliminated its Performance Research and Evaluation division due to staff reductions. The division's responsibilities included managing the state performance measurement and strategic planning processes and conducting program analyses and management reviews of state entities.

2010 OPB's Strategic Planning Director position, responsible for strategic planning and performance reporting, is vacant as of November 2010.

2011 OPB filled the Strategic Planning Coordinator position in August 2011.

Source: OPB Documents and statutes

Requested Information on Performance Measures

Program Performance Reporting Program performance reporting is one component of Georgia's performance management framework. As the state's budgeting authority, OPB collects program-level
performance information from all state budget units for two purposes: to assist OPB analysts in agency budget preparation and for publication in the annual GBR.4 According to OPB, the annual operating budget results from collaboration and communication between OPB staff and agency budget analysts. The process includes reviews of performance information from multiple sources as well as information on the Governor's priorities, industry trends, current events, current topics of interest, national events, etc. Agencies submit program-level performance information for inclusion in the annual GBR directly to OPB and also enter this information into OPB's budget formulation and submission portal, BudgetTool.

While OPB has elected to include program performance measurement information in the GBR, there is currently no statutory requirement to do so. OPB personnel also note that the program-level performance measures included in the GBR are not selected for inclusion based on specified criteria.5 Rather, measures are selected from the list of measures the agencies submit. OPB prefers to present measures with information available for multiple years or those with a trend of tracked performance. The audit team interviewed staff from a sample of six state agencies who said that the measures currently included for their agencies are updated each year with little variation and are requested for update by OPB and inclusion in the GBR primarily due to their having been reported the year before. The current cadre of program measures was initially selected in fiscal year 2008 for inclusion in the GBR and has since been repeated with little variation although the number of measures listed has decreased for most agencies. According to OPB staff, the budget instructions to agencies for fiscal year 2012 and 2013 required agencies to update the measures and work with OPB to revise or add measures. The six sample agencies provided the audit team with lists of additional measures that they noted were also important for stakeholders and which may provide additional information for budgetary decision-makers. The measures in the GBR are not intended to be exhaustive or complete for each program or agency.

In 2003 and 2004, under Results-Based Budgeting, agencies were required to submit disclosure documents describing how the data was verified and a statement of its accuracy with each performance measure. Since 2007, agencies have not been required to submit these disclosures. According to OPB personnel, agencies were instructed to maintain records for the measures reported and disclose "important information" about
the measure in the description field in BudgetTool. Staff also noted that, beginning with the fiscal year 2013 GBR, agencies will be requested to "sign off" on their performance
measure submissions as accurate in BudgetTool. However, OPB does not plan to conduct evaluations to confirm the performance measures' accuracy. OPB staff did indicate that the measures included in the GBR are sometimes limited because agencies do not provide complete information, the information provided is poor, or there is insufficient trend data for informing decisions.

4 Defined as a department, institution, agency, or other unit of organization for which separate appropriations are made. 5 According to OPB, it is using criteria to select measures for inclusion in the 2013 GBR.

Requested Information on Performance Measures

Strategic Planning OPB also manages the state's Strategic Plan. According to statute, strategic planning is defined as the process through which a preferred future direction and organizational mission are established and periodically updated in light of changing trends and issues and goals, objectives, and strategies are adopted and implemented to guide an organization toward that preferred future direction. (See Exhibit 3.) All agencies submit their strategic plans, goals and measurable objectives to OPB through its web-based HORIZON system. There are three components to the state's strategic planning framework: the State Strategic Plan, Agency Strategic Plans, and Annual Agency Implementation Plans.

Exhibit 3
Georgia's State Strategic Planning Process Hierarchy

Source: OPB State Strategic Planning Guidelines
The State Strategic Plan is created by the Governor and OPB and includes the overall state vision and mission. It represents a four to five year statement of the vision, direction, and priorities of the state and outlines the state's policy areas for each of five statewide goals: Educated, Healthy, Safe, Growing, and Managed.6 The State Strategic Plan should direct the creation of all agency strategic plans and objectives and these are required to align with the goals of the overall plan. Within each of the five statewide goals are multiple sub-objectives with specific indicators, set by OPB, to determine progress.
6 These goals, sub-goals, and indicators were set under the prior administration and OPB personnel have noted that these will be updated in the near future. As of December 2011 this has not occurred.

Requested Information on Performance Measures

The Agency Strategic Plans are statutorily required and each department, board, bureau, commission, institution, authority, and other agency of state government...submit[s] its own strategic plan to [OPB]... contain[ing]:

a description of its future direction; a statement of its organizational mission; a description of the current and anticipated future needs being addressed by its
preferred future direction and organizational mission; a description of planned actions designed to address these needs; a description of the goals for the program or services to be improved; a course of action for achieving the planned improvements, including an
implementation timetable; a description of the evaluation system to be used to determine if the goals are
being attained; an estimated annual cost for each planned improvement of a program or service.
Agency Strategic Plans are three to four year plans and provide a description of the agency's mission, goals, a set of measurable objectives to determine progress toward the goals, and the strategies (activities) that will be conducted to achieve those goals (see Exhibit 4). All Agency Strategic Plans must align to the overall State Strategic Plan and agencies must describe how their activities will contribute to the overall state goals. The strategies should include the expected impact on the measurable objective; cost of implementation and operation; and start and completion dates. OPB recommends that the Agency Strategic Plan focus on three to five activities the agency is doing to improve services and customer experiences. OPB guidelines state that the primary user of the Agency Strategic Plan is the agency.

Exhibit 4 Agency Strategic Plan Components
Mission: The purpose of the organization; why the agency exists. Statutory and regulatory requirements provide the foundation for the mission statement.
Goals: A result or outcome one is trying to achieve. Goals should be focused on outcomes and should address the most critical issues facing the agency and its customers.
Measurable Objectives: Specific, measurable results to be achieved. Objectives determine if the goal is being met and OPB recommends they be stated as "X to Y by date".
Strategies: A coherent set of actions that has a reasoned chance of working to improve results. Strategies are the activities the agency conducts to achieve its objectives.
Source: State Strategic Planning Guidelines

Requested Information on Performance Measures

The Agency Annual Implementation Plans (AAIP)7 provide details on the specific strategies and activities the agency will be working on for the next fiscal year and fulfill the requirement for a "periodic work program" as outlined in statute. According to statute, The Governor, through [OPB], shall require each budget unit,8 other than those of the legislative branch and the judicial branch, to file periodic work programs with [OPB]. As provided in Code Section 45-12-83, no allotment of funds shall be approved for any budget unit until such budget unit has filed a periodic work program with [OPB] and the periodic work program has been approved by the Governor.
AAIPs include updated strategic plan information describing the activities conducted to achieve strategies in the near-term, such as performance improvements the agency is seeking to achieve over the next year, measurable objectives stated to reflect a 12-month or less time frame, project milestones, and costs. OPB recommends that each goal, measure, and strategy in the implementation plan have a designated "owner" in the agency and that targets be identified and tracked at least quarterly. OPB also recommends that agencies employ internal progress reports to track goals.

Mid-year and year-end progress reports are provided in addition to the Implementation Plan as updates. Agencies provide these reports in January and at the end of the fiscal year, respectively. OPB uses these reports to determine if what was planned [in the AAIP] was accomplished. Agencies use the HORIZON system to note the progress of the strategies being implemented. Agencies do not, and are not required to, report on their progress towards or accomplishment of the objectives or goals.
Throughout the strategic planning framework, performance results and information are intended to flow in both directions; agency measurable objectives affect the success of agency goals which affect the success of state goals and vice versa, so long as those goals and measures properly reflect one another. OPB guidelines also state that the Governor and OPB evaluate whether the various agencies' budget requests sufficiently align with the agencies' strategic plan information. Further, the agency strategic plan is intended to be used in all phases of the budget cycle and the Governor and OPB reportedly use the plan to evaluate the agencies' budget requests and to prioritize state spending. In addition, the Governor and the Governor's staff participate in and direct the overall strategic planning process; this includes the state's Chief Financial Officer and the Chief Operating Officer. Agency heads report to these officers and are accountable for their agency's strategic performance in contributing to the state's overall goals as part of their review process.
Agency Response: OPB noted that Georgia's model for strategic planning is intentionally different than strategic planning in some best practice states. Georgia's plan "focuses on the new and different things the agency will focus on over a 3 year time horizon. The measurable objectives... are intended to measure agency progress towards their 3-5 goals. As such, the measurable objectives (formatted as x to y by date) are short term and focused on measuring the impact of the strategies identified in the agency strategic plan."

7 The Agency Annual Implementation Plan format and requirements are currently being revised by OPB and renamed as an Annual Performance Plan but the details have not been established. The Annual Operating Budget and the information collected in BudgetNet are also components of the periodic work program. 8 Per O.C.G.A. 45-12-71, "Budget unit" means a department, institution, agency, or other unit of organization for which
separate appropriations are made.

Requested Information on Performance Measures

Requested Information
Provide an analysis of how best practice states use performance measures in decision-making and how data is collected and reported. What structures do these states have in place to ensure the data is reliable?
States that effectively use performance measures to inform decision-making have established structures that: require specific information be provided, identify the participants in the processes, and outline processes to ensure the information is reliable. According to research by the Pew Center on the States, and other best practices literature, the following states represent best practices in the use of performance measures in decision-making: Iowa, Maryland, New Mexico, Oregon, Texas, Utah, Virginia, and Washington. As shown in Exhibit 5, while the details of the structures may differ across the states, the common structures that they have in place can be categorized as: statutory provisions, legislative involvement and oversight, and processes to ensure reliability of information. Each of these categories and examples from the states are discussed in more detail on the following pages.

Exhibit 5 Performance Processes: State to State Comparison
Performance-Supporting Processes

Washington Virginia Utah Texas Oregon
New Mexico Maryland Iowa Georgia

Budget Process: Annual Budget (A), Biennial Budget (B)

ABAABBABB

Statutory Provisions

Performance Measurement Use in the Budget Process Required Types of Measures are Required (Outcomes, Outputs, Efficiency, Productivity, Workload, Cost-per-Unit, etc.) Evaluation or Audit of Performance Measures Required
Legislative Involvement and Oversight

Legislative Hearings Held for Agencies to Present Performance Results

Legislative Branch Establishes/Approves Performance Measures

Performance Measures Presented in the Budget Document

Processes to Ensure Reliability of Performance Information

State Auditor Conducts Evaluations

Executive Branch Conducts Evaluations

Legislative Branch Conducts Evaluations

Internal Agency Requirements

Agency Performance Measures/Results Posted to a Statewide Website
Source: State agency w ebsites (executive budget offices, legislative offices, and state auditors)

Requested Information on Performance Measures

Statutory Provisions Although having a performance budgeting framework in statute does not guarantee that it will be effectively implemented, research indicates that clear policies and statutes do have a positive impact on the process. Correlations were found between states identified as having strong performance budgeting systems and more comprehensive performance laws; this correlation is not present in states identified as having weaker performance budgeting systems.

The Pew Government Performance Project's (GPP) report, Measuring Performance: The State Management Report Card for 2008,9 assessed states' access to and use of appropriate
data for decision-making. Pew assessed whether the states had the following practices:

valid cost and performance information was produced (including information on past performance)
the governor, budget office, state agencies, and state legislature use the information during budget deliberations
state agencies, the budget office, and the governor routinely use the information when developing, reviewing, and making budget decisions
the state legislature routinely uses the information to allocate resources and design/redesign programs.

Pew graded all 50 states on the above criteria and noted each as having Strength, Mid-
Level, or Weakness in Budgeting for Performance. Georgia was rated as Mid-Level. Additional studies were built off this research to identify common attributes across performance budgeting statutes and determine if correlations existed between the presence of these
attributes and a rating of Strength in Pew's study. The researchers found that the strong states Delaware, Iowa, Louisiana, Missouri, New Mexico, Oregon, Texas, Utah, Virginia and Washington had, on average, more performance budgeting components in statute and that they were more likely to be used in practice. Researchers also found that the more comprehensive the law was in terms of specifying responsible parties and measurement requirements, the stronger the implementation of performance budgeting.
Budgeting staff from these strong states also reported that decision-making was improved by the use of the information.

A 2007 survey of state legislative fiscal offices found that Georgia had
"almost no review of performance measures in the legislative process".
National Conference of State Legislators (NCSL)

Legislative Involvement and Oversight Legislative involvement in the performance process is necessary if states want to use performance measures to hold agencies accountable and to inform state budgeting. Overall, legislators are in a position to support a culture that values the use of the measures in the budget process. According to best practices, legislators should be involved with the agencies and the executive budget staff in setting and agreeing upon agency and program performance measures, questioning agencies about their performance in legislative hearings, and using performance information for budgeting and accountability purposes.

9 Pew's Government Performance Project was designed to improve service to the public by strengthening government policy and performance. The Project evaluated how well states manage employees, budgets and finance, information and infrastructure.

Requested Information on Performance Measures

In order to facilitate legislative involvement, legislators must believe the information is not only reliable but useful to them. According to best practices,10 the following five
actions can be taken to improve the usefulness of the performance information delivered
to legislators.

Executive branch budget staff should ensure that performance information emphasizes policy results rather than administrative measures.
Legislative and executive branch staff should identify and jointly agree on key results measures for use in budget documents and performance reports.
Executive agencies should provide regular performance reports. Executive branch staff should collaborate with legislative staff to ensure that
performance reports are useful, accurate, brief, clear, and timely. Executive branch agencies should publish performance information online to
make it easily accessible to legislators and the public.

Legislative Oversight at the Federal Level Performance management at the federal level was established with passage of the Government Performance and Results Act (GPRA) of 1993.11 It was the beginning of federal strategic planning and was specifically created to assist Congress in holding federal agencies accountable for achieving their purpose and goals. The passage of GPRA was seen as essential in providing an additional level of accountability for federal agencies, especially those with programs and activities that cannot be easily held accountable through the budget (this includes programs seen as essential public services such as Medicaid: cutting the budget to a poorly performing Medicaid program may not be likely to increase performance or vice versa).

Therefore, this type of legislative oversight based on performance is particularly

important to hold agencies accountable for their statutory purpose and mission.

According to the Act, the federal Office of Management and Budget (OMB) must

develop the overall federal performance plan. The Act also required the OMB issue

guidelines for agency strategic plans and required that agencies consult with Congress

regarding their performance results and expectations. Congressional input

In some states, a joint legislative committee participates in the selection

was seen as a key factor in the original GPRA legislation and Congress noted that more information on program effectiveness was needed in order to determine what it is getting in the way of results from each dollar spent, and how those

of measures as well as the questioning of agencies

results would change with an increase or decrease in funding.

relative to achieving the set

goals. In this manner, the state achieves legislative oversight of the process.

Legislative Oversight in Other States A review of eight best practice states found that each has engaged the legislature in various aspects of the performance process. Legislators are

Other states have established policies and procedures in agencies that require information be

involved in reviewing, and holding agencies accountable for their strategic plans, goals, and objectives as well as reviewing performance information to assist in appropriations decisions. Generally, the performance information is

provided to legislators. included in the budget process and submitted to the legislature concurrently

with agency budget requests. The legislatures are often involved in setting

the performance measures and questioning agencies as to the success or failure of their

10 Five Actions to Enhance State Legislative Use of Performance Information, Managing for Performance and Results
Series, Judy Zelio, IBM Center for the Business of Government, 2008. 11 The Act was updated as the GPRA Modernization Act of 2010.

Requested Information on Performance Measures

goals during legislative hearings. States manage their legislative review and oversight in various ways. Examples of how states manage this review are included below. Some have established joint legislative committees to facilitate these processes; others have established processes, without the framework of a committee, to provide information to the legislatures.

Iowa's Fiscal Services Division of the Legislative Services Agency conducts program evaluations and tracks agency performance measures to provide information to appropriations committees to assist in budget deliberations.

Maryland's Department of Budget and Management reports annually to the state appropriation and budget committees on agency performance measures and includes performance results spanning a four year period.

New Mexico's Legislative Finance Committee collaborates with the executive budget office in managing the state strategic planning process and setting state strategic priorities. The Committee and the executive budget office work
together to create a list of key agencies and key measures. Quarterly, the agencies report to the Committee and the executive budget office on their progress and results relative to these measures.

Oregon's Legislative Fiscal Office reviews agency strategic plans and agencysubmitted key performance measures and reports on this information to joint legislative committees. Staff from the Office, and the executive budget office, cooperate on combined core teams to review agency performance proposals and recommend changes to agency performance measures. All agency performance documents are submitted annually to the state appropriations committees.
Texas' Legislative Budget Board12 produces the general appropriations document, reviews agency performance reports, and actively participates in the creation of agency strategic plans, including setting agency goals and measurable objectives. All agencies report their performance in their biannual budget submissions and the Budget Board holds special hearings to discuss each agency's prior-year performance results.

To assist legislators in questioning agencies about their performance measures, the National Conference of State Legislatures developed a questionnaire that can be used when agencies present program performance information (see Appendix B). The questionnaire is designed to help legislators determine how the information was collected and reported. It should be noted that OPB personnel have stated that they are open to legislative cooperation and recommendations regarding the types of measures to be included in the GBR document and they intend to consult the Senate and House budget offices regarding the types of measures to be included in the fiscal year 2013 GBR.

12The Legislative Budget Board is a joint committee of legislators; they produce the appropriations bill.

Requested Information on Performance Measures

Oversight of Performance Information Reliability and Accuracy All eight of the sample states reviewed identified a specific entity (or multiple entities) as responsible for regular evaluation or audit of agency performance measures and reports to ensure reliability and accuracy of the information provided. (See Exhibit 5 for the chart of states.) Two of our sample states (Iowa and New Mexico), and another identified by Pew (Missouri), statutorily require multiple evaluators to conduct these reviews (state auditor, legislative branch, executive branch, and/or outside entity). Legislators in these states also participate in the selection and approval of the agency
and/or program performance measures. Only two of the states Pew rated as weak in budgeting for performance specified any evaluation or audit requirements. The best practice states we reviewed ensure the performance information reported is accurate by requiring, for example, that: agencies have internal verification processes; outside consultants review the information; and the state auditor or other governmental agency conduct routine certification audits of a select number of performance measures.

Maryland's Office of Legislative Audit Criteria for Performance Measure Review

Maryland provides an example of how performance information is verified for accuracy and reliability. Its Office of

Certified: Reported performance was reasonably accurate.

Legislative Audit (OLA) is responsible for

Certified with Qualification: Reported performance was reasonably accurate even though minor deficiencies were noted with the supporting documentation, controls were not sufficient, or the methodology used to calculate reported performance was not consistent with the measure definition.
Inaccurate: Reported performance differed significantly from actual performance; the calculation process was wrong (e.g., data relevant to the calculation was excluded); or, as reported, the measure was misleading (e.g., the agency failed to disclose the measure as a rate when applicable.)

reviewing the measures included in the state's comprehensive plan. Because of the number of measures included in the plan, not all are reviewed each year. The Office selects a number for review and state agencies are required to submit the performance information, which is then audited and rated according to specified criteria (shown in the
box). Each measure is rated as certified, certified

Factors Prevented Certification: Reported performance could not be verified, as documentation was unavailable, internal agency controls were not adequate to ensure the accuracy of the results, or results were not presented in a manner consistent with the performance measure description.

with qualification, inaccurate or noted that factors
prevented certification. As a part of the measurement certification process, the OLA may recommend that agencies establish

procedures to include all relevant data, retain

documents, or consider other forms of data for calculating performance. Maryland's

Department of Budget and Management then uses this information to produce an

annual report, which is delivered to the House and Senate Appropriation Committees

prior to the legislative session.

Maryland's performance measurement information feeds into a larger performance management framework known as StateStat, which is used to make budget recommendations. The StateStat team (made up of a Director and a team of analysts) works under the Governor's Office and provides the Governor and his staff information used to hold agencies accountable for outcome-based performance. The Governor and his staff meet bi-weekly with agency leaders to question reported results. The combination of data-driven performance reporting and oversight has been credited for StateStat's success.

Texas and Oregon verify performance measure reporting in a manner similar to Maryland. Auditors in both states routinely review the measures and rate them on a
similar scale: accurate; accurate with exceptions (i.e., controls are inadequate to ensure accuracy, however performance appears accurate; or controls are strong, but source

Requested Information on Performance Measures

documentation is unavailable for testing); unable to determine; or inaccurate. In Virginia, auditors review the measures as well as the accuracy of the management systems used to collect and report the results. In New Mexico, the Legislative Finance Committee evaluates agency performance measures and has established guidelines which state it will seek to reproduce the reported results to determine accuracy.

All the states examined encountered challenges in presenting credible and reliable performance information; in some cases there was an overload of information making it difficult to identify what was actually relevant. To address issues with accuracy, some states moved to external verification of the measures. Some states addressed reliability issues by involving budget staff in the selection of measures to ensure that the measures will be most helpful to decision-makers.

Georgia as compared to Best Practice States In Georgia, the General Assembly receives performance management information primarily through the Governor's Budget Report (GBR) and, potentially, the State Strategic Plan. While not specifically designed to inform legislative budgetary decisionmaking, as the only two reporting tools available, they lack fundamental components necessary to be effective for this purpose. As noted earlier, legislators may also request performance information directly from agencies or OPB. We also found that agencies in our sample collected additional performance information that could be used for these purposes.

According to a 2005 survey of Georgia House and Senate Appropriations members, when asked to rank the sources of information they trust and use most, the majority of respondents noted, on average, that they relied on their own advisors most heavily while agency staff was ranked the lowest.13 This may be due to the lack of verified information and legislative involvement in the format and selection of the measures. As noted previously, legislators are more likely to participate in the performance process when information is verifiable and formatted for their needs. Georgia's framework currently provides that agencies self-report all performance information to OPB. The agency is not required to present proof of verification, and OPB does not verify the information, nor is any other external body required to do so.

Additionally, performance management information for legislative use is limited. From fiscal years 1998-2004, under Results-Based Budgeting (RBB), the GBR included extensive performance measurement information reported by the agencies. We selected a sample of state agencies for review to determine if and how inclusion of measures had changed over time. We selected the following six agencies based on the number of employees, service sector, strategic plan support area, and the number of measurable objectives included in the overall state strategic plan: the Department of Community Health, the Department of Transportation, the Department of Education, the State Soil and Water Conservation Commission, the Department of Economic Development, and the State Board of Pardons and Paroles. (See Appendix A for more information on how the sample was selected.) Since 2004, the overall number of measures reported by these six agencies has decreased by 53%. This overall decrease includes a significant decrease in the number of outcome-oriented measures. As noted earlier, outcomes describe the progress toward the program's mission and objectives and whether it is accomplishing

13 Legislative Influences on Performance-Based Budgeting Reform, Fiscal Research Center, Georgia State University, May 2006.

Requested Information on Performance Measures

the goals for which it exists. Outcomes are, therefore, key measures of program achievement. The ratio of outcome to output measures reported for the six sample agencies in the fiscal year 2004 GBR was over 9 to 1. The ratio in fiscal year 2012 was only 1:1 with far fewer overall measures, especially for the larger agencies reviewed. Not only are outcomes important for determining the accountability of agencies and programs, a mix of measures is recommended (including outcomes, outputs, inputs, efficiency and productivity measures, and cost-per-unit measures) for reporting to determine accountability and provide sufficient information to support budgetary decision-making.

We also reviewed the Agency Strategic Plans for the six state agencies we selected. These plans do not provide information legislators can use in either budgetary decisionmaking or accountability processes either. None of the plans included results relative to the agencies' stated measurable objectives or goals; plans were incomplete (e.g., some did not include requested cost information). Additionally, while all agencies are required to submit and update their plan, there was not a strategic plan on file for every agency and the plans were not systematically updated. For the six sample agencies, we reviewed the fiscal year 2011 agency strategic plans and annual implementation plans; each included indicators of the status of activities (on time, behind schedule, or stopped) but not the results of measurable objectives or goals. Therefore, while the status of the agency's noted activities may be indicated as "on time," it cannot be determined whether the measurable objectives (or the agency's goals which rely on those objectives) have been achieved. Without actual results data, it is not possible to determine whether the agency is accomplishing its goals, whether the strategies chosen are effective, and whether the agency is contributing to the overall goals and indicators for the state.14 Overall, the information in both of Georgia's processes is not collected, reported, or verified in a manner that ensures it is effective for legislative use in holding agencies accountable or budgetary decision-making. According to 2013 budget guidelines, agencies are required to report at least three measures for each program: a workload measure, an efficiency measure, and an outcome measure.

Agency Response: OPB noted that, while the information on best practices is helpful, additional information regarding execution is needed. They noted "[P]articularly in times of tight budgets, the state would need to evaluate the value add of implementing additional processes and oversight in both the Executive and Legislative branch to the level described of best practice states."

Regarding the performance measurement information included in the GBR, OPBs response noted that the GBR "is not intended to be the sole source of performance measures... [It] contains a selection of measures for each agency. Performance information for decision making flows through a variety of channels. It is common practice for OPB to share its working papers including performance information and research with the House and Senate budget offices to explain the Governor's Budget recommendations. Legislators and their budget offices request additional information from the Governor, OPB and agencies as they deem necessary. Most state agencies collect performance information in much greater detail and volume than is published in the GBR that is used in decisionmaking and day to day management of the agency and its programs. Many agencies also publish statistical and performance information on their respective websites and as part of presentations to their boards." OPB also noted that it is updating its website to include "a more extensive list of agency performance measures and links to agency websites that have performance dashboards or other

14 All agency strategic plans are submitted to OPB's HORIZON web-based database.

Requested Information on Performance Measures

performance data."

Regarding legislative oversight, OPBs response stated that "[t]he legislative committee process is the appropriate vehicle for legislative oversight of agency budgets and performance. Improved oversight can be addressed through the committee process if the legislature so desires. Budget staff communication around performance measures has increased." The response also noted that, during the fall of 2011, "OPB, the House Budget Office, and the Senate Budget and Evaluation Office met to review the performance measures for 78 programs. The purpose of the joint review was to develop a common understanding of the program's purpose and identify and agree on a common set of measures. This joint review process will continue next spring as part of OPB's ZBB analysis."

What are the qualities of `effective' performance measurement and what types of measures are most suited for use in budgeting decisions (i.e., most relevant to legislators)?

There are two components to effective performance measurement reporting, and the measures themselves. In order to be effective for the users, as discussed earlier, the information must be relevant to them. Additionally it should be provided in an agreed upon format. These components are discussed below.

Effective Performance Reporting The performance information included in Georgia's GBR is limited to a statement of the program name and the performance measures; supporting narrative and explanatory information is not provided (as it once was under RBB). The Governmental Accounting Standards Board (GASB) recently released guidelines for government performance reporting that provide a framework for effectively communicating performance information to those using the information in making budget decisions. GASB recommends that the number of measures included be limited to essential information and be measured against planned expectations.

The essential components of an effective performance report include a

Goals: Statements of what a program is striving to achieve in the long term

statement of purpose and scope, a list of major goals and objectives, key measures of performance, and a discussion and analysis of results and challenges. The purpose and scope address the reasons for the

Objectives: Quantifiable and measurable statements of what a program expects to achieve in a shorter, defined period;

report and what programs are included, as well as why these programs are important. The goals and objectives section addresses what the programs are intended to accomplish and links the goals to measurable

achievement directly affects achievement of goals.

objectives. GASB notes that a limited number of essential measures, or key measures, should be selected so the user is not overwhelmed. By

comparing these measures to internal (i.e., within the organization) or

external (i.e., with other similar organizations in other states) performance, the user can

better determine the degree of success of the measures. The three broad types of

measures include: measures of service efforts (inputs); measures of service

accomplishments (outputs and outcomes); and measures that connect the first two

types (efficiency and cost-outcome). (See Exhibit 6 on the next page.) A descriptive

narrative should be included explaining why the results differ from expected results,

external factors affecting performance, and the effect of performance results.

Requested Information on Performance Measures

Effective Performance Measures Effective measures are generally described as relevant to the program, understandable, comparable to internal or external criteria, timely, consistent over time, reliable and replicable, and communicated effectively to the users. Additionally, measures should be linked to the agency's overall mission and goals, focused on the agency's core activities, and free from bias. Collectively, agency performance measures should allow legislators to understand the operations and outcomes of the agency's activities, how those outcomes relate to government-wide priorities, and the return-on-investment for public funds. There are a variety of types of measures as shown in Exhibit 6.

Type of Measure Inputs

Exhibit 6

Types of Performance Measures

Definition

Examples

Resources actually used to produce Amount of funds

outputs and outcomes

Number of employees

Process (workload)

Amount of work that comes into a program but not yet completed

Number of miles of road needing repair; Number of customers who come in for service

Outputs

Amount of product or services delivered

Miles of road paved Number of students served

Outcomes
Efficiency and Productivity

Consequences of what the program did; something the program wants to maximize or minimize
Ratio of input to amount of output or outcome and the ratio of output to input or outcome (often in relation to costs)

Impacts

Data that estimates the extent to which the program actually caused the outcomes. Seldom used.

Source: H. Hatry, Performance Measurement, 2nd Ed.

Percentage of patients discharged from a mental hospital who are capable of living independently
Number of people gaining employment after completing a training program per dollar of program cost
Number of expectant teenage mothers who, because of the program, had healthy babies (i.e., without the program, they would have lost their babies or had babies with substantial health problems)

The U.S. Government Accountability Office (GAO) examined the use of performance
information in budgetary decision-making in Arizona, Maryland, Texas, Virginia, and Washington.15 It reported that workload, output, and cost-per-unit measures (also
known as measures of efficiency and productivity) were most frequently used by
legislators. Examples of the types of measures include:

Number of people gaining employment after completing an agency's training program per dollar of program cost
Number of customers who reported that the service received had significantly helped them per dollar cost of that service
Number of clients, who 12 months after completing the service, had stopped the risk behaviors targeted by the program per dollar cost of that service
State vehicle cost as a percent of contracted rental rates

15 Performance Budgeting: States' Experiences Can Inform Federal Efforts, United States Government Accountability Office (GAO), February 2005.

Requested Information on Performance Measures

Expenditures per job created/retained by a program Daily cost per adult inmate, average daily cost per bed

Additionally, when members of the Georgia Senate and House Appropriations Committees were surveyed in 2005, they responded similarly with regard to the types of performance information they most wanted to see: program results, outcomes, and costper-unit information.16 This combination of performance metrics (outputs and/or outcomes combined with per unit or per employee-hour of cost) allows legislators to match funding levels with a certain level of service. Performance measures also allow legislators to determine how levels of service will be affected by a change in funding level. Performance information can be helpful in making budget trade-offs during times of fiscal stress and to show how programs contribute to state goals. However, as noted
earlier, performance information does not provide a mechanistic link to a budget decision, but is one among many factors.

Provide an analysis of the relevancy of the measures given the purpose and operations of the agency. Do the indicators effectively measure the outcomes and efficiency of the program as defined in the purpose statement of the appropriations bill or in statute?

We assessed relevance on two levels: 1) whether the programs included in the GBR were relevant based on the percent of agency funds they represent (priority programs) and 2) whether the measures included for these programs were aligned with the purpose of the program. Overall, we found that for three of the six agencies reviewed, the programs included in the GBR were relevant based on the percent of agency funds they represented. We reviewed 25 programs for these six agencies listed in the fiscal year 2012 GBR for alignment with program purpose.17 For four agencies, the majority of measures included were aligned with the purpose of the programs. However, only three of the 25 programs had measures that addressed all activities noted in the program's purpose statement. Although information in the GBR is not sufficient for budgetary decision-making, it is possible that legislators can and do request additional information from agencies as they deem necessary. It is also reasonable to expect, and we did find, that agencies have collected performance information on measures other than those reported in the GBR that could be useful for decision-making. However, currently, the GBR is the only established method of providing performance information to decision-makers.

Do the programs included represent Priority Programs? According to GASB, priority programs are defined as those representing the majority of budgeted or expended funds. However, GASB also notes that obtaining stakeholder input and assessing their needs is equally important. Therefore, the programs selected should also: address results for a major state goal or objective; address an issue receiving considerable public discussion; include information considered important by elected officials, citizens, or experts in the field; and/or include information about a program or

16 Legislative Influences on Performance-Based Budgeting Reform, Fiscal Research Center, Georgia State University, May 2006. 17 A total of 30 programs were listed for these six agencies in the 2012 GBR. We excluded five programs under this part of the review. Four were excluded from the Department of Community Health because three moved to the new Department of Public Health and one because legislation regarding its purpose could not be located. One was also excluded from the Department of Economic Development because legislation regarding its purpose could not be located.

Requested Information on Performance Measures

service that affects a large proportion of the population or has potentially significant economic, social, or environmental effects. According to the sample agencies and OPB, current programs are not included in the GBR based on any particular priority (there is no requirement that they be selected based on any criteria or that they be included in the GBR at all). Therefore, the percentage of agency funds represented by the sample agencies' programs is the only criteria by which to assess priority.

Based on the percentage of agency funds a program represents, we found that there were priority programs in three agencies that were not reported on in the fiscal year 2012 GBR. Based on our analysis there are programs in the Department of Economic Development, the Department of Transportation, and the Department of Education that were not reported in the GBR but make up a significant percentage of the agencies' budgets. (See Appendix C for more detail.) For example, the Department of Education's Pupil Transportation, Severely Emotionally Disturbed, and Preschool Handicapped programs did not have performance measures included in the GBR although they represented significantly more funds than other programs which were reported. It should be noted that the measures not being included in the GBR does not mean that measures do not exist for these programs. In interviews with the sample agencies, all provided examples of additional measures for their programs that were not included in the GBR. OPB selects which measures will be included in the GBR.

The programs included in the GBR for the Department of Community Health, the State Board of Pardons and Paroles, and the Soil and Water Conservation Commission did represent priority programs within their agencies. Pardons and Paroles and Soil and Water have a limited number of programs, are relatively small agencies, and have defined missions covering a targeted area of state services and activities. While Community Health is much larger with a more diverse mission, the programs representing the largest amount of agency funds were represented in the GBR.

Characteristics of Best Practice Performance Measures
Measure Aligns with Purpose: Do the measures align with the purpose statement of the program?
Comprehensive: Do the measures sufficiently convey the information needed to hold the program accountable for its performance based on its statutory purpose?
Describe Return-on-Investment (ROI): Do the measures included convey whether the program is providing a sufficient return-on-investment for state funds?
Outcomes-Oriented: Do the measures included describe the outcomes of the program's activities?
Source: GASB, Pew Center on the States, and GAO

Do the Measures Align with Program Purpose? Overall, we found that the majority of performance measures included in the fiscal year 2012 GBR for our six sample agencies were relevant and linked to the stated program purpose. However, the measures were not comprehensive enough to determine accountability for program activities or sufficient for making informed budget decisions. Additionally, the included measures did not provide the type of information noted by legislators as being most useful, such as efficiency, productivity, and cost-per-unit information (see Appendix D for analysis and results). This may be because the measures are not collected or reported for any particular stakeholder's use. As a result, the information is not formatted to assist in legislative budgetary decision-making or to inform the public regarding whether agencies are accomplishing their mission based on their stated purpose. During interviews with agency staff, we obtained additional measures and explanatory information that was not included in the GBR, this included additional outcome information and efficiency and productivity measures; this information could have further informed the decision-making processes As noted earlier, each of the sample agencies noted that there were additional measures collected but not reported in the GBR. Below

Requested Information on Performance Measures

are examples where additional information was collected by the agency (DCH) that could be used to make the measures more comprehensive and where additional measures were used in other states that provided a more comprehensive picture of performance.

Department of Community Health (DCH) The measures presented in Exhibit 7 do not by themselves allow legislators to assess accountability of the Medicaid program or provide information for making budgetary decisions. However, as discussed below, when paired with additional information available from the agency, a more comprehensive picture of performance develops.

The operative purpose of the first two programs is access. The measures do not directly or comprehensively address access; the measures are not efficiency or productivity measures; the measures are duplicated; and the measures are incomplete because they do not address access for segments of the population served. As presented, the measures do not include information legislators can use to make budgetary decisions.

Exhibit 7

DCH's State Medicaid Programs

Measures as Presented in the Fiscal Year 2012 GBR

Program and Purpose

Program Measures

Aged, Blind and Disabled Medicaid: "Provide healthcare access primarily to elderly and disabled individuals."

The percentage of members with a diagnosis of diabetes who had at least one HgA1c performed during the year.

Low Income Medicaid: "Provide healthcare access primarily to low-income individuals."

Percentage of members in Georgia Families receiving recommended immunizations by their 3rd birthday.
Percentage of members in Georgia Families 12 mos. - 19 years who had a visit with a PCP.

PeachCare: "Provide access to health insurance coverage for qualified low-income Georgia children."

Percentage of members in Georgia Families receiving recommended immunizations by their 3rd birthday.
Percentage of members in Georgia Families 12 mos. - 19 years who had a visit with a PCP.

In our interviews with DCH staff and reviews of DCH reports on their website, we found that they do collect information on a number of other measures that, combined, do measure access, such as:

o Adults' Access to Preventative/Ambulatory Health Services: the percentage of Medicaid members who received access to preventative or ambulatory health services during the calendar year.
o Oral Health (Annual Dental Visit Rate): the percentage of members receiving an annual dental visit during the calendar year.

Requested Information on Performance Measures

In addition, DCH has established the following cadre of six performance measures, with targets, for the Medicaid programs. These measures are designed to encourage Care Management Organizations (CMOs) to achieve better quality outcomes for Medicaid members.

o Well-Child Visits in the First 15 Months of Life Six or More Visits o Adults' Access to Preventive/Ambulatory Health Services 2044 years
of age and 4564 years of age o Use of Appropriate Medications for People With Asthma o Childhood Immunization Status Combination 2 o Lead Screening in Children o Comprehensive Diabetes Care HbA1c Testing

Georgia Department of Education (GaDOE) As shown in Exhibit 8, the
Virtual School measures represent student achievement measures rather than how the program achieves its stated purpose of assisting students in recovering credits, accessing supplementary resources, or earning additional credits. Additionally, there are no efficiency or productivity measures included to determine the value of the Virtual School program (for example, costs on a per student basis for successful outcomes).

Similarly, the measures for State Schools include only explanatory inputs and
do not measure whether students are prepared to become productive citizens as stated in the purpose. Sufficient measures might include the percentages of students successfully living independently after graduation, or the percentage employed or attending postsecondary schools.

Exhibit 8

Georgia Department of Education

Measures as Presented in the Fiscal Year 2012 GBR

Program and Purpose

Program Measures

Georgia Virtual School : Expand the accessibility and breadth of course offerings so that Georgia students can recover credits, access supplementary resources, enhance their studies, or earn additional credits in a manner not involving on-site interaction with teacher.

Number of Advanced Placement (AP) courses offered
Percentage of students completing courses
Percentage of students passing the appropriate End of Course Test for courses that require such a test

State Schools: Prepare sensory-impaired and multidisabled students to become productive citizens by providing a learning environment addressing their academic, vocational, and social development

Number of students enrolled at Atlanta Area School for the Deaf
Number of students enrolled at Georgia School for the Deaf
Number of students enrolled at Georgia Academy for the Blind

Requested Information on Performance Measures

Provide a survey of the reliability of data as submitted. How do agencies collect and confirm the data and rate the quality and accuracy of the data utilized?

Georgia's current system for collecting and reporting performance measurement information is lacking in terms of its utility for budgetary decision-making. The measures reported in the GBR are not necessarily related to priority programs, they are not comprehensive enough to be used to hold agencies accountable, nor do they include the types of measures needed for budgetary decisions. A comprehensive set of input, output and outcome measures is needed to hold programs accountable for achieving their purpose and cost-per-unit, workload, and efficiency/productivity measures are needed to assist decision-makers in making funding decisions and determining the state's return-on-investment.

As noted earlier, performance measurement information must be viewed as reliable and accurate by its users in order to be effective. Other states we reviewed have instituted internal and external methods for assessing the reliability and accuracy of the data, such as evaluations and reviews of the measures to test the accuracy and replicate the data. However, the measures must also be relevant to the purpose of the program they are measuring and targeted towards the achievement of a goal. Otherwise, the measurement is not meaningful for budgetary decision-makers because it is not useable to determine whether the program's actions are impacting the agency's or state's movement towards achievement of a goal.

Our review found that, in addition to lacking the breadth and depth of measures necessary to be effective for budgetary decision-making, Georgia also lacks formal verification and validation processes necessary to ensure the accuracy of the information reported. Currently, state agencies are not required to have verification or validation processes in place to ensure measures are accurate and can be replicated. Unless controls are in place to ensure the data is collected and reported in the same manner each time, and that these methods are sound, there is no assurance that information collected at a later date would yield the same or similar results. The Georgia Department of Transportation was an exception in our review. The Department has established an internal reporting process for determining the reliability of all performance information prior to reporting. The Department requires a specified staff member be responsible for reporting a performance measure and ensuring its validity. This person must "sign off" on the validity of measures and complete forms detailing the measure's definition, intended purpose, and limitations. We also found that the Department of Community Health contracts with an outside entity to conduct performance measure reliability tests for its Medicaid program, and for additional program performance measures. Federal guidelines require that these reviews address the reliability and accuracy of performance information reported by the Care Management Organizations (CMOs).

Historically, OPB required agencies to submit data limitation disclosures which served to highlight issues that impacted the data collection or reporting processes. Using this information, decision-makers had a better idea of the factors that may impact the accuracy or reliability of the information presented. However, this requirement was eliminated in 2007. According to OPB, beginning in 2012, agencies will again be required to provide information on the limitations of the data.

Requested Information on Performance Measures

Agency Response: OPB noted that its current "performance measure guidelines emphasize that agencies are responsible for validating the data reported and auditing data quality. OPB will strengthen the validation processes in the next cycle of performance measure reporting and incorporate the validation process into the ZBB review process. OPB will also clarify its guidelines and provide training to agencies in the area of data collection and validity. OPB will also implement an agency certification process as part of the measure submission process next year."

OPBs also noted that, through the ZBB process, it plans to review approximately 10% of programs - in terms of program services, cost of services provided, program purpose and goals and performance measures - each year. OPB states its intention to "continue to work with the House and Senate budget offices to review the performance measures and assess the reliability and relevancy of the measures as part of the ZBB review. We believe this will address the gap identified in the report for more legislative involvement in the improvement, selection and validation of the measures."

Requested Information on Performance Measures

Examples of Measures from Other States
We compared our sample agencies' performance measures to those used in Arizona, Iowa, Kentucky, Oregon, Texas, Utah, and Virginia, which are known for performance budgeting.1
(See Appendix E for complete table of sample agency programs and other states' measures.):
Medicaid Program measures Texas, Virginia, and Iowa offer a variety of measure types that provide information for holding programs accountable and determining the value of public funds. Each state has a mix of measures to assess access, cost, service, and outcomes, such as:
number of participants, number of participants receiving services, percent of eligible population served, average cost per recipient, pharmacy costs per member, cost savings from Medicaid utilization and care management strategies, percent of program savings from discovered overpayments and cost avoidance, percent of members receiving hospital / inpatient services, percent of members receiving cancer screens, percent of members receiving emergency room visits
State Health Benefit Plan measures: Common measures across other states included: program cost and efficiency measures, outcomes and effectiveness measures; and administrative measures. The combination of these measures would be useful in determining the extent that Georgia's SHBP is accomplishing its stated purpose. Examples include:
average costs per enrollee, average monthly state-paid contribution per enrollee, per member annual costs compared to national averages, percent change in enrollee medical/dental premiums as compared to industry average, prescription drug program costs as a percentage of total costs, percentage of enrollees receiving recommended preventative care, percent of enrollees utilizing annual check-ups and screenings, percent of claims processed by a certain period
Schools for the Deaf and Blind measures: Other states focused on outcome measures, effectiveness measures, and cost measures such as:
percent of students prepared to enter neighborhood schools on competitive grade level and functioning to non-disabled peers,
percent of students transitioning to post-secondary programs, percent employed, percent of graduates, percent of students demonstrating improvement in math/reading, cost per student served, student to staff ratios
Virtual School measures: Kentucky has a relatively comprehensive set of measures that are aligned with Georgia's program purpose statement. They measure population and access, achievement, and outcomes as shown below:
number of students participating and completing, number of schools participating, achievement averages, exam/assessment performance, number of students successfully completing retaking courses for credit recovery, number of drop-out recoveries who re-enroll in virtual school program and successfully complete
coursework
1Texas' and Virginia's measures make up the majority of those we selected because they were available from a single statewide website. Source: Review of other states' websites, interviews with other states' officials

Requested Information on Performance Measures

Appendix A Objectives, Scope, and Methodology

We conducted this special examination in response to a request from the Senate Appropriations Committee. The Committee asked that we review the performance measures of state agencies as submitted to the Governor's Office of Planning and Budget. Specifically, the Committee asked:

Based on a sampling of agency performance measure data as submitted to the Office of Planning and Budget, please provide:

1. A survey of the reliability of data as submitted. How do agencies collect and confirm the data and rate the quality and accuracy of the data utilized?
2. An analysis of the relevancy of the measures given the purpose and operations of the agency. Do the indicators effectively measure the outcomes and efficiency of the program as defined in the purpose statement of the appropriations bill or in statute?
3. An analysis of how best practice states use performance measures in decisionmaking and how data is collected and reported. What structures do these states have in place to ensure the data is reliable?

We reversed the order of the questions, in consultation with the Senate Budget and Evaluation Office, to address the best-practice components first. As noted in the questions and answers section of the report, we found that Georgia lacks the comprehensive and statutorily based system other states have implemented to ensure information is appropriate for budgetary decision-making. As a result, we did not analyze individual agency performance measures relative to accuracy because the internal processes for establishing reliability are not present. In addition, we did not evaluate the implementation of an effective performance system as part of our review.

State Agency Sample Selection We selected six Georgia executive agencies for a review of performance measures and strategic plans. Agencies were selected based on four criteria:

Responsibility for the most measurable objectives within the five sectors of the state strategic plan. The Plan sectors are: Healthy, Educated, Growing, Safe, and Managed.

Sector of Government Service: Natural Resources/Agriculture/Forestry; Commerce and Trade/Finance; Education; Health and Human resources; Public Safety; Transportation.

Agency size by number of employees (using figures from the State Personnel Administration's workforce development listing). Categorized as Large (1000+ employees); Medium (500 1000 employees); or Small (1 500 employees

Requested Information on Performance Measures

Agency

Sample Agencies Selected for Review

Government Sector of Service

Agency Size Total # of employees

State Strategic Plan Area

Department of Community Health and Human

Health

Resources

Department of Transportation

Transportation

5367 Large HEALTHY

4613

Large

SAFE, MANAGED

Department of Education Education

792 Medium EDUCATED

State Soil and Water Conservation Commission
Department of Economic Development
State Board of Pardons and Paroles

Natural Resources / Ag and Forestry
Commerce and Trade / Finance
Public Safety

29 Small GROWING 157 Small GROWING 677 Medium SAFE

# of Measurable Objectives in State Strategic Plan 16
72
48
25
3
13

We determined that, in order to be relevant, measures had to be: tied to priority programs for the agency; limited, but focused on "key" activities; and aligned with the purpose of the agency/program. For example, if an agency reports performance measures for one of its smallest, least critical programs, those measures may not be relevant for budgetary decision-making and holding agencies accountable for overall performance. Likewise, if the measures reported do not relate to the purpose of the program, they are not considered relevant.
We determined whether the performance measures reported for these agencies in the GBR were relevant based on whether they were: related to the agencies' priority programs; whether the measures were focused on key activities; and, whether the included performance measures reflected the agency's and/or program's purpose, as stated in the appropriations document. The results of our review are discussed in more detail in the following sections.
For the six sample agencies, we compared the performance measures included in the GBR to the purpose statements listed in the fiscal year 2012 Appropriations Act (see Appendix D for list of measures for the sample agencies and the analysis results). We determined whether the measures were: aligned with the stated purpose of the program; comprehensive; described the efficiency, productivity, or cost of the program; and outcome-oriented. These characteristics are noted as best practices by GASB, Pew, GAO, etc. and were applied using the following definitions:
1. Relevance/Linkage to program purpose statement As defined by the Governmental Accounting Standards Board (GASB), "Relevance" of performance measures is assessed according to the following criteria: A close logical relationship to the purpose for which it is intended to be used, and that the information should be sufficient to allow users of the information to assess the condition of the program/agency and make decisions. GASB tests of relevance include the following: Does the information present the major goals and objectives of the programs and services being reported? Does the information provide a basis for assessing the level of

Requested Information on Performance Measures

accomplishment of the goals and objectives? Do the goals and objectives relate to concerns that
are important to users? In addition, the Government Accountability Office (GAO) defines "linkage" as whether performance measures align with division and agency goals. The above definitions were applied and interpreted in this instance as to whether the measures directly support achieving the purpose statement of the program. If the purpose statement listed specific activities to be conducted then those activities should be reflected in the measure/s and an observable link should be seen between them. This is also a conclusion as to whether the included measure/s obviously tie to the purpose with no further explanation needed (as none is provided in the current reporting format).

2. Comprehensive As defined by GASB, comprehensive means that data is not omitted that would make a difference in a user's assessments or decision-making and that all major aspects of performance for a major goal or objective are reported. In addition, GAO notes that performance measures should cover the "core program activities" of the program and be "balanced" in that no one activity is overemphasized by the measures. As in the above determination of Relevance/Linkage this also reflects the specificity of the purpose statement and whether there are measures included that reflect all activities noted.

3. Efficiency, Productivity, and /or Cost measures included Per Performance Measurement, 2nd Ed. (Hatry), "efficiency" and "productivity" are defined respectively as the relationship between the amount of input (usually dollars) and the amount of output or outcome of an activity or program, and the ratio of the amount of output or outcome to the amount of input. Per the GAO, the above types of measures, as well as workload measures and other cost-perunit measures, are the measurement types most useful for budgeting decisions. Georgia legislators on the House and Senate Appropriations Committees, in a 2005 survey, also noted that these types of measures would be most useful to
them for their purposes. The Pew Charitable Trusts' Policy Framework to
Strengthen State Government Planning, Budgeting and Accountability also emphasizes "return-on-investment analysis" in a performance framework. This means developing a cost-benefit analysis that includes identifying program benefits, converting that benefit into a monetary value, tabulating the full costs of the program, identifying intangible benefits, and comparing the monetary benefits to the costs.

4. "Outcome" measures included The IBM Center for the Business of Government notes that a focus on policy outcome measures is most useful for legislative purposes. The Mercatus Center of George Mason University publishes an annual performance report scorecard on the sufficiency of federal agency performance reports and states that performance reporting should be focused on outcome-oriented performance measurement, most or all strategic goals should be outcome oriented, and that most goals should be stated as clear, measurable outcomes, meaningful to the public.

Requested Information on Performance Measures

Other States Review We selected a sample of other states for review primarily based on designation by the Pew Center on the States as having strong performance-based budgeting systems. We reviewed the following states: Texas, Virginia, Maryland, Iowa, Washington, Oregon, Utah, and New Mexico. The criteria Pew used to determine the strength of performance budgeting were:
Evidence of the production of valid cost and performance information (including information on past performance)
Evidence that the governor, budget office, state agencies, and state legislature use cost and performance information during deliberations on the budget
Evidence that state agencies, the budget office, and the governor routinely use cost and performance information when developing, reviewing, and making decisions on the budget
Evidence that the state legislature routinely uses cost and performance information to allocate resources and as a basis for program design and redesign

In addition, analysts also received recommendations from academic experts, OPB, and GAO reports. We reviewed the states' executive budget office and legislative websites as well as state performance dashboards (if available). We also used information regarding performance measures for program comparisons from various other states.

Resources Used We used information from the following resources for our internal agency review:
HORIZON - All sample Agency Strategic Plans, all sections of the State Goal Report, and all sample Agency Annual Implementation Plans were downloaded from OPB's HORIZON system. HORIZON is Georgia's online Strategic Planning Support Tool. All state agencies use it to submit their strategic plans and update their annual implementation plans as required by OPB. HORIZON is accessible via OPB's BudgetNet website and is accessible to anyone.
Annual Governor's Budget Reports (GBR) Analysts reviewed the performance measures for the six sample agencies from the FY2008-FY2012 budget reports, as well as the FY2004 Budget Report.
BudgetNet BudgetNet is OPB's budget submission website.
Additional Documents from OPB Other documents provided by OPB included agency performance measure submissions for FY2012 as well as reviews of FY2010 agency performance measures.

Additionally, it should be noted that there may be other avenues of performance reporting within Georgia state government that we did not review. This includes, but is not limited to, performance reporting and evaluations conducted by the House and Senate budget offices, information on agency/program performance collected by individual legislators' staff, etc. There are various direct channels by which legislators or the executive may collect and use performance information other than GBR and strategic planning documents. We did not review these other sources. Additionally, we did not review internal OPB documents regarding individual agency budget creation. Per OPB management, budget personnel at OPB working with agencies during the creation of their annual operating budget may collect various forms of performance information from different sources. Per OPB this information is used when formulating the Governor's budget recommendation. Although OPB noted that these documents are

Requested Information on Performance Measures

available for review, including to any legislator who requests them, they also noted that they are not formatted in a performance report and represent working papers of raw data. Due to the unformatted nature of this data, the likely variation in data collected from agency to agency as a result, and its specific use as background for creating agency budgets, we did not review these documents.

This project was not conducted in accordance with generally accepted government auditing standards (GAGAS) given the timeframe in which the report was completed. However, it was conducted in accordance with Performance Audit Division policies and procedures for non-GAGAS engagements. These policies and procedures require that we plan and perform the engagement to obtain sufficient, appropriate evidence to provide a reasonable basis for the information reported and that data limitations be identified for the reader.

Requested Information on Performance Measures

Appendix B

Sample Performance Questions for Legislators' Use
For Continuation Funding: What are the outcomes for each program's major outcome indicators in the most recent years for which data are available? How do these compare to the outcome values proposed for the new budget period? How do these recent outcome values compare to the targets set for those years? How do the targets that the programs propose for the new budget period compare to recent targets? What do you estimate the outcome values would be if the funds are decreased / increased by X percent? What do you estimate the added cost would be to achieve a "Y" increase in your outcomes? If the agency has substantially missed its targets on key outcomes in recent years: Why were those targets missed? What actions are the programs planning in order to meet the program's new budgetyear target? If a substantial increase in outcomes is projected without a request for added resources, how does the program expect to meet the new targets? What are the major uncertainties that might affect the likelihood that those targets will be met? Who are the program's significant client groups? Are there major citizen subgroups that the program affects, such as groupings by location within the state, gender, race/ethnicity, age group, types of disability, for example? What have been the outcomes for these subgroups? For which of these clients have the outcomes been disappointing and for which are they encouraging? What actions are planned to improve the outcomes for those subgroups for which the outcomes have been disappointing? What have been the unintended positive effects of the program, if any? What have been the unintended negative effects of the program, if any? How does the budget request support the agency's strategic plan?
For Initial Funding for New Programs: What is the mission of the program? What outcome indicators will be used to track results? For outcome indicators for which data are already available, what are the current levels for each of those outcome indicators (the baselines)? What do you expect will be the numerical values for each outcome indicator for the budget period and for each of the following two/three years? Which citizen groups will benefit from the program? To what extent? Which citizen groups might lose because of the program? To what extent? When are those benefits expected to occur? In which years?
Source: National Conference of State Legislatures (NCSL)

Requested Information on Performance Measures

Appendix C

Program Reporting as Related to Program Funding
(Based on Programs Included/Not Included in the Fiscal Year 2012 Governor's Budget Report for Six Sample Agencies)

Agency Programs

FY2012 State

% of State

FY2012 Total % of Total Funds

General Funds General Funds

Funds

Department of Community Health (DCH)

Aged, Blind and Disabled Medicaid

$1,182,182,833

59% $4,312,561,737

38%

Low Income Medicaid

$640,361,746

32% $2,782,522,996

25%

State Health Benefit Plan

0% $3,085,096,671

27%

PeachCare

$53,812,229

$231,777,738

Health Care Access and Improvement Adolescent and Adult Health Promotion 1 Adult Essential Health Treatment Services1 Immunization 1

$6,104,116 $3,822,917
$618,560 $2,684,539

< 1% N/A N/A N/A

$10,778,937 $40,643,553
$7,457,006 $10,234,255

< 1% N/A N/A N/A

Department of Economic Development (GDEcD)

Tourism

$9,590,993

30%

$9,590,993

23%

Business Recruitment and Expansion

$7,708,241

24%

$8,463,675

20%

International Relations and Trade

$2,101,547

Film, Video, and Music

$1,010,892

Small and Minority Business Development

$896,415

Not Included:

Innovation and Technology

$5,965,859

19%

$13,634,805

32%

Department of Transportation (GDOT)

Routine Maintenance

$196,954,077

10%

Construction Administration

$92,724,600

Local Road Assistance Administration

$76,236,577

Traffic Management and Control

$73,339,218

Transit

$3,376,525

50%

$40,351,224

Airport Aid

$2,454,358

37%

$37,997,710

Not Included:

Capital Construction Projects

$873,892,377

44%

Payments to State Road and Tollway Auth.

$234,902,012

12%

Capital Maintenance Projects

$154,372,981

Local Maintenance and Improvement Grants

$100,642,250

State Board of Pardons & Paroles (SBPP)

Parole Supervision

$39,232,439

76%

$41,866,067

76%

Clemency Decisions

$7,207,791

14%

$7,273,580

13%

Victim Services

$440,453

< 1%

State Soil & Water Commission (GSWCC)

Conservation of Agricultural Water Supplies

$233,535

$2,556,314

26%

Conservation of Soil and Water Resources

$1,344,676

52.01%

$2,429,733

25%

U.S.D.A. Flood Control Watershed Structures

$123,242

$3,989,063

40%

Department of Education (GaDOE) (percentages listed for state and total funds including QBE / not including QBE)2

Quality Basic Education Program

$6,119,150,453

88% $6,119,150,453

75%

State Schools

$23,379,119

< 1% / 3%

$24,802,241

< 1% / 1%

Georgia Virtual School

$4,792,820

< 1% / 1%

$5,811,034 < 1% / < 1%

School Improvement

$5,161,681

< 1% / 1%

$5,161,681 < 1% / < 1%

Communities in Schools

$933,100 < 1% / < 1%

Not Included:

Federal Programs

0% / 0% $1,025,153,992

13% / 50%

Quality Basic Education Equalization

$436,158,587

6% / 51%

$436,158,587

5% / 21%

Pupil Transportation

$130,310,693

2% / 15%

$130,310,693

2% / 6%

Severely Emotional Disturbed (SED)

$64,275,760

< 1% / 8%

$67,248,655

< 1% / 3%

Preschool Handicapped

$27,891,099

< 1% / 3%

$27,891,099

< 1% / 1%

School Nurses

$26,399,520

< 1% / 3%

$26,399,520

< 1% / 1%

Nutrition

$23,119,188

< 1% / 3%

$47,756,489

< 1% / 2%

Testing

$13,573,504

< 1% / 2%

$30,255,306

< 1% / 1%

Technology/Career Education

$14,201,164

< 1% / 2%

$30,214,087

< 1% / 1%

Non Quality Basic Education Formula Grants

$16,867,421

< 1% / 2%

$16,867,421 < 1% / < 1%

Regional Education Service Agencies

$8,571,299

< 1% / 1%

$8,571,299 < 1% / < 1%

1These program s m oved to the new Departm ent of Public Health in July 2011 (at the start of fiscal year 2012).

2Included for GaDOE program s are "percentages of total agency funds" / "percentages of total agency funds m inus QBE" (as QBE m akes

up 75% of total GaDOE funds).

Source: FY2012 Budget Report (OPB BudgetNet)

Requested Information on Performance Measures

Analysis of Program Performance Measures for Six Sample State Agencies1

Agency / Programs2

Program Purpose3

Department of Community Health (DCH)

Program Performance Measure/s4:

Fiscal Year 2012 GBR Performance Measure(s)

Relevant/Linked to Program Purpose

Comprehensive

Efficiency / Productivity / Cost
Measures

Outcome Measures

Aged, Blind and Disabled Provide healthcare access primarily to elderly and disabled

Medicaid

individuals.

1. The percentage of members with a diagnosis of diabetes who had at least one HgA1c performed during the year. GBHC only population for FY11.

1. Percentage of members in Georgia Families receiving

Low Income Medicaid

recommended immunizations by their 3rd birthday (Medicaid and Provide healthcare access primarily to low-income individuals. Peachcare).

2. Percentage of members in Georgia Families 12 mos - 19 years

who had a visit with a PCP (Medicaid and Peachcare).

PeachCare

1. Percentage of members in Georgia Families receiving

recommended immunizations by their 3rd birthday (Medicaid and

Provide access to health insurance coverage for qualified lowincome Georgia children.

Peachcare).5 2. Percentage of members in Georgia Families 12 mos - 19 years

who had a visit with a PCP (Medicaid and Peachcare).5

Provide a healthcare benefit for teachers and state employees

State Health Benefit Plan

that is competitive with other commercial benefit plans in quality of care and access to providers; and to provide for the efficient

1. Percent of SHBP enrollees in consumer directed plan options.

management of provider fees and utilization rates.

Department of Economic Development (GDEcD)

T ourism

Provide information to visitors about tourism opportunities throughout the state, operate and maintain state welcome centers, fund the Georgia Historical Society and Georgia Humanities Council, and work with communities to develop and market tourism products in order to attract more tourism to the state.

1. Number of jobs generated by tourism activities. 2. Amount of tourism expenditures (billions).

Develop international markets for Georgia products and to attract

International Relations and international companies to the state through business and trade 1. Number of leads developed to create business opportunities for

missions, foreign advertising, a network of overseas offices and Georgia companies.

Trade

representatives, and by providing technical and educational

2. Number of companies served.

assistance to businesses.

Business Recruitment and Recruit, retain, and expand businesses in Georgia through a

statewide network of regional project managers, foreign and

Expansion

domestic marketing, and participation in Georgia Allies.

1. Number of jobs created. 2. Amount of new direct capital investment into the state (billions). 3. Number of new projects initiated.

Appendix D

Increase industry awareness of Georgia business opportunities,

Film, Video, and Music

financial incentives, infrastructure resources, and natural resources in order to attract film, video, music, and electronic

1. Amount of new direct capital investment into the state (millions).5 2. Number of new projects initiated.5

gaming industry projects and businesses to the state.

1Agencies selected as a representative sample for review (for selection methodology and criteria see pg. 25-26 of the report).
2Some programs not included in this part of the review: Three programs which moved from the Department of Community Health to the Department of Public Health as of July 2011: Adolescent and Adult Health Promotion, Immunization, and Adult Essential Health Treatment Services, and two programs, one from the Department of Community Health: Health Care Access and Improvement and one from the Department of Economic Development: Small and Minority Business, for which statutory provisions regarding their creation and authorization could not be located.
3From OPB BudgetNet. 4For definitions and descriptions of the criteria used, see pg. 26-28 of the report. 5Duplicated measures from other programs. Sources: OPB BudgetNet, Fiscal Year 2012 Governor's Budget Report (GBR)

Requested Information on Performance Measures

Appendix D (continued)

Analysis of Performance Measures for Six Sample State Agencies1

Agency / Programs

Program Purpose2

Department of Transportation (GDOT)

Fiscal Year 2012 GBR Performance Measure(s)

Relevant/Linked to Program Purpose

Program Performance Measures3:

Comprehensive

Efficiency / Productivity / Cost Outcome Measures
Measures

Routine Maintenance

Ensure a safe and adequately maintained state transportation system 1. Percentage of state highways with pavements that meet or

by inspecting roads and bridges, cataloguing road and bridge

exceed minimum standards (PACES - rating equal to or greater

conditions and maintenance needs, and providing routine maintenance for state roads and bridges. The purpose of this appropriation is also to maintain landscaping on road easements and rights-of-way through planting, litter control, vegetation removal, and grants to local

than 70). 2. Percentage of on-system bridges with a sufficiency rating less than or equal to 50.

governments, to provide for emergency operations on state routes, 3. Customer Satisfaction with welcome centers (% rating A or B).

and to maintain state rest areas and welcome centers

4. Number of shoulder miles of litter and debris removed.

Construction Administration

Improve and expand the state's transportation infrastructure by planning for and selecting road and bridge projects, acquiring rights-of-way, completing engineering and project impact analyses, procuring and monitoring construction contracts, and certifying completed projects.

1. Percentage of on-system projects in the STIP advanced/let to construction in the year programmed. 2. Percentage of construction projects completed within original contract time.

1. Percent of off-system projects in the statewide transportation

Local Road Assistance Administration

Provide technical and financial assistance to local governments improvement program (STIP) advanced/let to construction in the

for construction, maintenance, and resurfacing of local roads year programmed.

and bridges.

2. Percentage of let local road and bridge construction projects

completed within the original contract time.

Traffic Management and Control

Ensure a safe and efficient transportation system statewide by conducting traffic engineering studies for traffic safety planning, permitting for activity on or adjacent to state roads, providing motorist assistance and traffic information through the Highway Emergency Response Operators (HERO) program and Intelligent Transportation System, and conducting inspections, repairs, and installations of traffic signals.

1. Customer Service: Average HERO incident response time in minutes. 2. Customer Service: Average roadway clearance time for autos in minutes. 3. Customer Service: Average roadway clearance time for trucks in minutes.

T ransit

Preserve and enhance the state's urban and rural public transit programs by providing financial and technical assistance to Georgia's transit systems.

1. Number of transit riders.

XXX

Airport Aid

Support safe and accessible air transportation infrastructure by inspecting and licensing public airports, providing planning assistance to local airports, maintaining the Statewide Aviation System Plan, and awarding grants to local airports for maintenance and improvement projects.

1. Percentage of airports meeting state airport licensing standards.

Requested Information on Performance Measures

Appendix D (continued)

Analysis of Performance Measures for Six Sample State Agencies1

Agency / Programs

Program Purpose2

State Board of Pardons and Paroles (SBPP)

Fiscal Year 2012 GBR Performance Measure(s)

Relevant/Linked to Program Purpose

Program Performance Measures3:

Comprehensive

Efficiency / Productivity / Cost Outcome Measures
Measures

Parole Supervision

The purpose of this appropriation is to transition offenders from prison back into the community as law abiding citizens by providing drug testing, electronic monitoring, parole supervision, and substance abuse treatment, and collecting supervision fees, victims' compensation, and restitution.

1. Percent of parolees acceptably completing parole supervision as compared to the national average (45% in 2007), based on methodology used in the Bureau of Justice Statistics (BJS) annual report. 2. Average monthly rate of parolees employed.

Clemency Decisions

1. Percent of parolees successfully completing parole supervision as

compared to the national average (44% in 2006) based on the

The purpose is to collect data on offenders within the correctional system, make determinations regarding offender eligibility for parole, investigate allegations of employee misconduct, manage the agency's

methodology used in the Bureau of Justice Statistics (BJS) annual report. 2. Number of Board preconditions of parole imposed on inmates before

public relations efforts, and administer the Re-Entry Partnership Housing Program.

release from prison. 3. Customer Service: Percent of Board Visitors' Day participants who

rate their entire experience as good or excellent.

Victim Services

The purpose is to provide notification to victims of changes in offender 1. Customer service: Percent of Victims' Visitor Day participants who

status or placement through the Victim Information Program, to

rate their overall experience as good or excellent.

conduct outreach and information gathering from victims during

2. Number of people registered in the Georgia Victim Information

clemency proceedings, to host victim and visitor days, and act as a Program (VIP) system.

liaison to victims to the state corrections system.

3. Number of correspondence items sent to victims.

State Soil and Water Commission (GSWCC)

Conservation of Agricultural Water Supplies

Conserve ground and surface water by increasing the uniformity and efficiency of agricultural water irrigation systems, installing meters on sites with permits for agricultural use to obtain data on agricultural

1. Number of agricultural water meters installed in high-priority watersheds. 2. Number of agricultural irrigation systems audited for uniformity and

water usage, and by administering the use of federal funds to construct and renovate agricultural water catchments.

efficiency.

X X

Conservation of Soil and Water Resources

Conserve Georgia's rural and urban natural resources by providing grants to encourage the reduction of erosion and other non-point source pollution from agricultural lands, by providing technical assistance teaching best management practices on erosion and sedimentation control to landowners and local governments, by certifying erosion and sedimentation control personnel, and by reviewing and approving erosion and sedimentation control plans for soil and water conservation districts.

1. Number of agricultural acres protected and benefited by conservation plans. 2. Number of individuals certified or recertified in erosion and sedimentation control.

U.S.D.A. Flood Control Watershed Structures

Inspect, maintain and provide assistance to owners of USDA flood control structures so that they comply with the state Safe Dams Act.

1. Number of watershed dam assessments completed in conjunction with USDA to determine if dams could serve as water supply reservoirs. 2. Number of commission/USDA watershed dams maintained.

Requested Information on Performance Measures

Agency / Programs Department of Education (GaDOE)

Analysis of Performance Measures for Six Sample State Agencies1

Program Purpose2

Fiscal Year 2012 GBR Performance Measure(s)

Relevant/Linked to Program Purpose

Program Performance Measures3:

Comprehensive

Efficiency / Productivity / Cost Outcome Measures
Measures

Quality Basic Education Program

1. Number of students.

Provide formula funds to school systems based on full time equivalent 2. Number of schools making Adequate Yearly Progress (AYP).

students for the instruction of students in grades K-12 as outlined in 3. Percentage of schools making AYP.

O.C.G.A. 20-2-161.

4. Statewide high school graduation rate.

5. Statewide high school dropout rate.

State Schools

Prepare sensory-impaired and multi-disabled students to become productive citizens by providing a learning environment addressing their academic, vocational, and social development.

1. Number of students enrolled at Atlanta Area School for the Deaf. 2. Number of students enrolled at Georgia School for the Deaf. 3. Number of students enrolled at Georgia Academy for the Blind.

Appendix D (continued)

Georgia Virtual School

Expand the accessibility and breadth of course offerings so that Georgia students can recover credits, access supplementary resources, enhance their studies, or earn additional credits in a manner not involving on-site interaction with a teacher.

1. Number of Advanced Placement (AP) courses offered. 2. Percentage of students completing courses. 3. Percentage of students passing the appropriate End of Course Test (EOCT) for courses that require such a test.

School Improvement

Provide research, technical assistance, resources, teacher

professional learning, and leadership training for low-performing schools and local educational agencies to help them design and

1. Number of schools on the needs improvement list. 2. Number of schools removed from the needs improvement list.

implement school improvement strategies to improve graduation rates

and overall student achievement.

Communities in Schools

Support Performance Learning Centers and maintain a network of local affiliate organizations across the state, and to partner with other state 1. Number of Performance Learning Centers (PLCs). and national organizations to support student success in school and 2. Graduation rate for PLC students. beyond.

Requested Information on Performance Measures

Comparison of Georgia Agency Performance Measures to Other States

Georgia Performance Measures

Other States' Measures

Department of Community Health (DCH)

Aged, Blind and Disabled Medicaid: Provide healthcare access primarily to elderly and disabled individuals.
Low Income Medicaid: Provide healthcare access primarily to lowincome individuals.
PeachCare: Provide access to health insurance coverage for qualified low-income Georgia children.

Texas Health and Human Services Commission:

Average Medicaid Acute Care Cost per Recipient Month (a

1. The percentage of members with a

measure of the monthly average number served)

diagnosis of diabetes who had at least one Percent Eligible Clients Receiving Acute Care Services

HgA1c performed during the year. GBHC only Percent 100% Poverty Population Covered by Acute Services

population for FY11.

Average Medicaid Acute Care Child Recipient Months per Month

Iowa Department of Human Services: Percent long-term care resources to home/community care Percent members aware of available preventive resources

Percent EPSDT Eligible Population Screened-Medical

State funds saved through the Medicaid Preferred Drug List

Percent EPSDT Eligible Population Served-Dental

Rate growth in pharmacy costs per member per month

Percent Medicaid Eligible Population Served

Increase in savings from Medicaid pharmacy cost saving strategies

1. Percentage of members in Georgia Families receiving recommended immunizations by their 3rd birthday (Medicaid and Peachcare).

Emergency Room Visits per 1,000 Average Member Months/Year Percent children and adults with access to managed care

Percent CHIP-eligible Children Enrolled

Proportion children on Medicaid with six wellchild visits

Average CHIP Programs Benefit Cost without Prescription Benefit Proportion children on Medicaid with a dental visit

Average CHIP Programs Benefit Cost with Prescription Benefit Proportion women receiving prenatal care from 1st trimester

Savings from utilization and care management strategies

Virginia Department of Medical Assistance:

Savings from surveillance and utilization review

2. Percentage of members in Georgia

Percent 3-6 year-old children enrolled in Medicaid who received Increase over prior year in revenue collections from third parties

Families 12 mos - 19 years who had a visit the recommended number of well-child screenings

Increase in State collections of Medicaid overpayments

with a PCP (Medicaid and Peachcare).

Percent two year olds who are fully immunized

Percent increase in member satisfaction with administration of

Percent enrolled children who utilize dental services

Medicaid Program over prior year

1. Percentage of members in Georgia Families receiving recommended immunizations by their 3rd birthday (Medicaid and Peachcare).

Percent covered births which are normal birth weight Number Medicaid enrolled physicians actively submitting claims Number enrolled dentists in the network Percent clean claims paid in 30 days Number providers, recipients, and record reviews each year

Percent increase in satisfaction with Provider Services over prior year
Percent clean Medicaid claims accurately paid or denied on time Number children who are enrolled Number children who are enrolled in Supplemental Dental

Percent eligibles are participating in Disease Management

2. Percentage of members in Georgia Families 12 mos - 19 years who had a visit with a PCP (Medicaid and Peachcare).

Percent two year olds who are fully immunized Percent enrolled children who utilize dental services Number eligible children enrolled

Appendix E

State Health Benefit Plan: Provide a healthcare benefit for teachers and state employees that is competitive with other 1. Percent of SHBP enrollees in consumer commercial benefit plans in quality of care directed plan options. and access to providers; and to provide for the efficient management of provider fees and utilization rates.

Employees Retirement System of Texas:

Percent Medical Claims Processed within Thirty Days

Percent All Electronic Pharmacy Claims Paid Within 21 Days Total Cost Paid Per Member for Admin and Claims Processing Total Cost Paid/ Member/Pharmacy Admin and Claims Processed In-Network Services as a Percentage of Total Services Mental Health/Substance Abuse Costs as a % of Total Costs Prescription Drug Program Costs as a % of Total Costs Number Employees Covered by Health Care Plans Percent Participants in HMOs Average Monthly State Contribution Per Enrollee

Additional measures1: State employees' preferred provider organization plan per member annual cost vs. national costs
Administrative cost per insurance enrollee Percent customers satisfied Number enrollees Percent change in medical premium compared to industry average Percent change in dental premium compared to industry average Average days to resolve claims, inquiries, and appeals

Virginia Department of Human Resource Management:

Percent participants receiving recommended preventive care

Percent customers rating services received as good or better

Number participants covered by health plans by type

Percent enrolled participants in the disease management program Percent increase in utilization for the state wellness program

Percent employees expressing satisfaction with group health benefits

Percent increase of employees and family members utilizing

annual checkups and screenings

1Measures compiled by the Utah Office of the Legislative Fiscal Analyst from Alaska, Arkansas, Florida, Louisiana, New Mexico, Oregon, Texas, and Virginia Sources: Fiscal Year 2012 Governor's Budget Report, OPB BudgetNet, Texas Legislative Budget Board website, Virginia Performs website, Results Iowa website, etc.

Comparison of Georgia Agency Performance Measures to Other States

Requested Information on Performance Measures

Georgia Performance Measures

Other States' Measures

Appendix E (continued)

Department of Education (GaDOE)

Texas Department of Education:

1. Number of students.

Percent All Students Passing All Tests Taken Percent Econ. Dis. Students Completing High School

Percent Students Completing High School

Percent Students Completing High School by Race/Ethnicity

2. Number of schools making

Percent Students Retained in Grade 5 and 8

Adequate Yearly Progress (AYP). Percent Students with Disabilities Who Graduate High School

Quality Basic Education Program:

Percent Eligibles Taking Advanced Placement Exams

Provide formula funds to school systems

Annual Statewide Dropout Rate for All Students

based on full time equivalent students for the instruction of students in grades K-12 as

3. Percentage of schools making AYP. Percent Meeting Adequate Yearly Progress

outlined in O.C.G.A. 20-2-161.

Iowa Department of Education:

4. Statewide high school graduation rate.

Percent state/federal funds distributed according to formula Percent state/federal funds distributed using correct data Percent districts reporting data required under EDFacts Percent state/federal reporting requirements met by the Dept. for

accountability and information purposes

5. Statewide high school dropout rate.

Percent teachers in core areas appropriately licensed and assigned Percent teachers meeting the def. of highly qualified teachers Percent school districts visited by a school improvement team Percent school districts maintaining accreditation standards Amount federal funds lost due to noncompliance with program requirements

Virginia Department of Education: Percent students successfully completing Algebra I by 8th grade Percent high school students exiting with a diploma Percent high school students earning the Advanced Studies Diploma Percent 3rd graders passing the Standards of Learning test Percent students enrolled in one or more AP/dual enrollment courses Central office staff costs per public school Percent total federal allocation spent within the allowable period Percent compliance with federal program monitoring report for NCLB Percent special ed. students who pass statewide assessment tests Percent youth with disabilities graduating from high school with an Advanced/Standard diploma Number Standards of Learning tests taken by students using a webbased delivery system
Percent school divisions that increased their reimbursements from federal/state funding sources
Number teachers who become nationally board certified each year

1. Number of Advanced Placement Kentucky Virtual High School:

Georgia Virtual School:

(AP) courses offered.

Number students registered/completing/participating

Expand the accessibility and breadth of course offerings so that Georgia students can recover credits, access supplementary

Performance on AP exams

2. Percentage of students completing Performance on End-of-Course-Assessments

courses.

Number schools expanding offerings via Virtual School

resources, enhance their studies, or earn 3. Percentage of students passing the Number students successfully retaking courses (credit recovery)

additional credits in a manner not involving appropriate End of Course Test

Number drop-out recoveries who re-enroll in virtual academy and

on-site interaction with a teacher.

(EOCT) for courses that require such successfully complete coursework (first graduates in spring 2005)

a test.

Number repeating students and families

Texas Schools for the Deaf and Blind and Visually Impaired:

State Schools: Prepare sensory-impaired and multidisabled students to become productive citizens by providing a learning environment addressing their academic, vocational, and social development.

1. Number of students enrolled at Atlanta Area School for the Deaf.
2. Number of students enrolled at Georgia School for the Deaf.
3. Number of students enrolled at Georgia Academy for the Blind.

Percent Student Learning Indicator Attained

Percent Short-term Program Students Demonstrating Progress

Percent Grad Students Currently Employed Percent Instructional Areas Rated as Satisfactory or Above Percent Students Achieve Prog in Core & Expanded Curricular Percent Agreeing Positive Change Due to School Consultation Percent Academic Courses Passed Percent Graduates Accepted in Postsecondary Education Percent Students Grades 3-12 Performing Equal or Better than Deaf Peers on SAT-10

Utah Schools for the Deaf and Blind (USDB): Number students prepared to enter their neighborhood schools on competitive grade level and functioning comparable to their nondisabled hearing or sighted peers Percent children deaf or hard of hearing transitioned out of intensive services before 3 years of age Percent students with vision loss transitioned out of special education services before age 3

Virginia School for the Deaf and Blind: Percent deaf/blind students in grades 1-8 demonstrating improvement

Number students that are able to transition from intensive services provided in the USDB Parent Infant Program and classroom services (campus and magnet) to itinerant services

in their instructional reading/math level from fall to spring. Percent students passing End of Course Learning assessments Percent staff that meet the definition of highly qualified

Percent of deaf students remaining in USD classes through high school Percent students with only vision disabilities mainstreamed by 3rd grade

Number improper accommodations reported to the Dept. of Education

Percent students completing pre-vocational survey assessments

Number citations issued during annual inspections

Number reportable incidents that occur when transporting students

1Measures compiled by the Utah Office of the Legislative Fiscal Analyst from Alaska, Arkansas, Florida, Louisiana, New Mexico, Oregon, Texas, and Virginia

Sources: Fiscal Year 2012 Governor's Budget Report, OPB BudgetNet, Texas Legislative Budget Board website, Virginia Performs website, Results Iowa website, etc.

Comparison of Georgia Agency Performance Measures to Other States

Requested Information on Performance Measures

Georgia Performance Measures

Other States' Measures

Appendix E (continued)

Department of Transportation (GDOT)

Airport Aid: Support safe and accessible air transportation infrastructure by inspecting and licensing public airports, providing planning assistance to local airports, maintaining the Statewide Aviation System Plan, and awarding grants to local airports for maintenance and improvement projects.
Construction Administration: Improve and expand the state's transportation infrastructure by planning for and selecting road and bridge projects, acquiring rights-of-way, completing engineering and project impact analyses, procuring and monitoring construction contracts, and certifying completed projects.
Local Road Assistance Administration: Provide technical and financial assistance to local governments for construction, maintenance, and resurfacing of local roads and bridges.
Routine Maintenance: Ensure a safe and adequately maintained state transportation system by inspecting roads and bridges, cataloguing road and bridge conditions and maintenance needs, and providing routine maintenance for state roads and bridges. The purpose of this appropriation is also to maintain landscaping on road easements and rights-ofway through planting, litter control, vegetation removal, and grants to local governments, to provide for emergency operations on state routes, and to maintain state rest areas and welcome centers
Traffic Management and Control: Ensure a safe and efficient transportation system statewide by conducting traffic engineering studies for traffic safety planning, permitting for activity on or adjacent to state roads, providing motorist assistance and traffic information through the Highway Emergency Response Operators (HERO) program and Intelligent Transportation System, and conducting inspections, repairs, and installations of traffic signals.
Transit: Preserve and enhance the state's urban and rural public transit programs by providing financial and technical assistance to Georgia's transit systems.

Texas Department of Transportation:

Number Fatalities Per 100,000,000 Miles Traveled

1. Percentage of airports meeting state airport licensing Percent Change in the Number of Small Urban and Rural

standards

Transit Trips

Percent Bridges Rated in Good Condition or Higher

Percent Construction Projects Completed on Budget

Percent General Aviation Pavement in Good or Excellent

Condition

1. Percentage of on-system projects in the STIP advanced/let to construction in the year programmed.

Percent Railroad Crossings with Signalization Percent Two-lane Highways with Improved Shoulders Statewide Maintenance Assessment Program Condition

2. Percentage of construction projects completed within Score

original contract time.

Statewide Traffic Assessment Program Condition Score

Iowa Department of Transportation: Percent highway miles that meet or exceed a sufficiency

Virginia Department of Transportation:

rating of tolerable or above

1. Percent of off-system projects in the statewide transportation improvement program (STIP) advanced/let to construction in the year programmed.

Number traffic crash related deaths on Virginia roadways Ratio of annual highway program cost awarded versus

Percent Congestion Free Travel on Interstate roadways annual program cost estimate

Number Annual Hours of Delay per Traveler due to

Shoulder miles of new paved shoulders awarded for

congestion on state highways (by traffic area)

construction on the primary highway system

2. Percentage of let local road and bridge construction Percent Construction and Maintenance projects delivered Percent total dollars paid to the total awarded amount for

projects completed within the original contract time. both by their Originally Scheduled Completion Date and all contracts dollars

within Budget

Percent Structure Inventory and Appraisal (SI&A) values

Percent total construction-related expenditures that were for our bridge system that meets last year's values

1. Percentage of state highways with pavements that meet or exceed minimum standards (pavement

expended on Construction Engineering and Inspection (CEI) Number new transportation research dollars secured

activities.

Percent non-committed right of way parcels returned to

conditions evaluation system - PACES - rating equal to or Percent total expenditures classified as administrative private, commercial, or public uses

greater than 70).

expense

Percent originally programmed projects let for

2. Percentage of on-system bridges with a sufficiency Percent environmental compliance (cumulative statewide construction in the current fiscal year versus programmed

rating less than or equal to 50.

average compliance score)

projects

3. Customer Service: Customer Satisfaction with welcome Percent Planned Expenditures used (Planned vs. Actual Percent airports that meet at least 75 percent of the

centers (percent rating A or B).

expenditure)

4. Number of shoulder miles of litter and debris removed. Dollar value cost reduction/avoidance generated by

facility and service objectives for their functional roles Number transit passengers (ridership)

research, resource expertise and availability, and

Total transit revenue mileage

implementation of knowledge management strategies.

Number aviation cargo / waterway freight / rail freight

Percent Pavement Lane Miles constructed (Plan to Actual tons originated and terminated in Iowa

1. Customer Service: Average HERO incident response time in minutes.

comparison) Percent Bridge structures constructed (Plan to Actual comparison)

2. Customer Service: Average roadway clearance time for Percent Construction Projects Delivered within Budget

autos in minutes.

Percent Construction Projects Delivered by their

Originally Scheduled Completion Date

3. Customer Service: Average roadway clearance time for Percent structures rated in Fair or better condition

trucks in minutes.

Percent interstate and primary roadway pavement lane

miles rated in Fair or better condition

Percent Maintenance Projects Delivered within Budget

Percent Maintenance Projects Delivered by their

1. Number of transit riders.

Originally Scheduled Completion Date

1Measures compiled by the Utah Office of the Legislative Fiscal Analyst from Alaska, Arkansas, Florida, Louisiana, New Mexico, Oregon, Texas, and Virginia

Sources: Fiscal Year 2012 Governor's Budget Report, OPB BudgetNet, Texas Legislative Budget Board website, Virginia Performs website, Results Iowa website, etc.

Requested Information on Performance Measures

Comparison of Georgia Agency Performance Measures to Other States

Georgia Performance Measures

Other States' Measures

State Board of Pardons and Paroles (SBPP)

1. Percent of parolees successfully completing parole

Clemency Decisions:

supervision as compared to the national average (44% in

The purpose of this appropriation is to collect data on offenders within the correctional system, make determinations regarding offender eligibility for parole, investigate allegations of employee misconduct, manage the agency's public relations efforts, and administer the ReEntry Partnership Housing Program.
Parole Supervision: The purpose of this appropriation is to transition offenders from prison back into the community as law abiding citizens by providing drug testing, electronic monitoring, parole supervision, and substance abuse treatment, and collecting supervision fees, victims' compensation, and restitution.

2006) based on the methodology used in the Bureau of Justice Statistics (BJS) annual report. 2. Number of Board preconditions of parole imposed on inmates before release from prison. 3. Customer Service: Percent of Board Visitors' Day participants who rate their entire experience as good or excellent.
1. Percent of parolees acceptably completing parole supervision as compared to the national average (45% in 2007), based on methodology used in the Bureau of Justice Statistics (BJS) annual report.
2. Average monthly rate of parolees employed.

Texas Board of Pardons and Paroles: Number Parole Cases Considered / Processed Average Percentage of Sentence Served by Inmates Released from Prison Average Time (Months) Served by Inmates Released from Prison
Percent Cases for Which Favorable Parole-release Decision is Made
Number Offenders Released/Parole Percent Technical Violators Whose Charges Were Disposed within 40 Days Percent Releasees Successfully Discharging Parole/Mandatory Supervision
Percent Releasees Receiving New Convictions Releasee Annual Revocation Rate

Iowa Board of Parole: Percent Parole deliberations resulting in Parole Percent Work Release Requested Granted Percent Parole Revoked Percent Work Releases Revoked Number Parole Release Deliberations Number Paroles Granted Number Parole Revocation Hearings Conducted Number Paroles revoked Number Individuals on Paroles Number Work Releases Granted Number Work Releases Revoked Number Risk Assessments Conducted Percent Offender Risk Assessments calculated within Timeframes Percent Victims Notified as Designated

Victim Services: The purpose of this appropriation is to provide notification to victims of changes in offender status or placement through the Victim Information Program, to conduct outreach and information gathering from victims during clemency proceedings, to host victim and visitor days, and act as a liaison to victims to the state corrections system.

1. Customer service: Percent of Victims' Visitor Day participants who rate their overall experience as good or excellent. 2. Number of people registered in the Georgia Victim Information Program (VIP) system. 3. Number of correspondence items sent to victims.

Virginia Parole Board: Average number of days between the date of hearing and decision Cost per Virginia Parole Board decision Number Meetings (appointments) with victims

Percent Executive Clemency Applications Reviewed/Processed Timely Number Executive Clemency Applications Reviewed Number Executive Clemency Applications Recommended Number Pardons Reviewed Number Pardons Recommended

Appendix E (continued)

State Soil and Water Commission (GSWCC)

Conservation of Agricultural Water Supplies: Conserve ground and surface water in Georgia by increasing the uniformity and efficiency of agricultural water irrigation systems, by installing meters on sites with permits for agricultural use to obtain data on agricultural water usage, and by administering the use of federal funds to construct and renovate agricultural water catchments.

1. Number of agricultural water meters installed in highpriority watersheds.
2. Number of agricultural irrigation systems audited for uniformity and efficiency.

Conservation of Soil and Water Resources:

Conserve Georgia's rural and urban natural resources by

providing grants to encourage the reduction of erosion and

other non-point source pollution from agricultural lands, 1. Number of agricultural acres protected and benefited

by providing technical assistance teaching best

by conservation plans.

management practices on erosion and sedimentation

control to landowners and local governments, by certifying 2. Number of individuals certified or recertified in

erosion and sedimentation control personnel, and by

erosion and sedimentation control.

reviewing and approving erosion and sedimentation

control plans for soil and water conservation districts.

U.S.D.A. Flood Control Watershed Structures: Inspect, maintain and provide assistance to owners of

1. Number of watershed dam assessments completed in conjunction with USDA to determine if dams could serve as water supply reservoirs.

USDA flood control structures so that they comply with the

state Safe Dams Act.

2. Number of commission/USDA watershed dams

maintained.

Virginia Department of Environmental Quality:

Texas Soil and Water Conservation Board:

Percent facilities in compliance with water permit

Percent District Financial Needs Met by Conservation

requirements

Board Grants

Percent Groundwater Technical Evaluations completed Percent Projects Addressing List of Impaired Water Bodies

within the 60 day timeframe required by procedures manual Percent Problem Areas with Certified Plans

Number waters fully or partially removed from the EPA's Percent Eligible Acres in Brush Control Areas Treated and

Impaired Waters List

Cleared

Number watersheds for which an assessment of surface

water has been conducted

Iowa Department of Natural Resources:

Number Impaired Waters in Iowa

Virginia Department of Conservation & Recreation:

Percentage Iowa Lakes with Good Water Quality

Number acres managed under at least 1 of 5 agricultural Water quality index for Iowa streams (Iowa Water Quality

priority conservation practices

Index)

Number acres preserved for land conservation purposes

towards achieving 100,000 acres statewide annually

Oregon Department of Agriculture, Soil and Water

Number local erosion and sediment control programs

Conservation Commission:

reviewed annually deemed consistent

Percent monitored stream sites associated with

Percent known dams regulated with regular operation and predominantly agriculture use with significantly increasing

maintenance certificates

trends in water quality.

Percent funds budgeted for Soil & Water Conservation

Percent monitored stream sites associated with

Districts dispersed according to State Budget and State

predominantly agriculture use with water quality in good to

Board policies

excellent condition.

Number localities reviewed for compliance with the land Percent monitored stream sites associated with

use and water quality protection provisions of the

predominantly agriculture use with decreasing trends in

Chesapeake Bay Preservation Act

water quality.

Requested Information on Performance Measures

Comparison of Georgia Agency Performance Measures to Other States

Georgia Performance Measures

Other States' Measures

Department of Economic Development (GDEcD)

Appendix E (continued)

The Texas Office of the Governor:

Additional Measures1:

Number Domestic Leisure Travelers to Texas

Number state website hits

Tourism:

Number Jobs Announced by Companies Receiving Enterprise Fund

Number visitors per year

Provide information to visitors about tourism

Grants

Number visitors per year from selected states/countries

opportunities throughout the state, operate and 1. Number of jobs generated by tourism activities. Number New Jobs Announced by Businesses Receiving Assistance

Number visitors during off-season

maintain state welcome centers, fund the Georgia

Capital Investment by Projects Receiving Assistance

Number visitors by (travel mode)

Historical Society and Georgia Humanities Council, 2. Amount of tourism expenditures (billions).

Number Defense Related Economic Development Projects

Number in-state tourists

and work with communities to develop and market

In-state Film/TV/Commercial/Video Game Production Expenditures Percent state tourists participating in nature or heritage based

tourism products in order to attract more tourism to

Number Jobs Created by the Moving Image Industry Incentive Program activities.

the state.

Number inquiring about nature or heritage activities while visiting web

Iowa Department of Economic Development:

site.

Number firms maintained on active prospect list

Percent market share of domestic tourism

Increase in number persons recruited to fill skilled positions

Percent awareness of state as a visitor destination

Percent increase in jobs in industries in Biosciences, Advanced

Total jobs created or retained

Manufacturing, and Information Solutions

Jobs created or retained per FTE

Business Recruitment and Expansion:

1. Number of jobs created.

Number new biotech companies created over 5 years

Percent jobs created or retained at above county average wage

Number jobs per year created or retained through the efforts of multi- Number of (category) jobs created or retained by agency funds

Recruit, retain, and expand businesses in Georgia through a statewide network of regional project managers, foreign and domestic marketing, and

2. Amount of new direct capital investment into the county regional economic development organizations

state (billions).

New investment per year in business deals through the efforts of

regional economic development organizations

Number jobs created through business expansions Number jobs created through business start-ups Number jobs created through business relocations

participation in Georgia Allies.

3. Number of new projects initiated.

Number businesses attracted per year, per region

Number rural jobs created Number export-related jobs created from non-Mexico trade

The Virginia Tourism Authority:

Percent satisfaction with state economic development efforts

Cost per unique user of VTC websites

Number funded projects and initiatives

Number annual unique Website users

Number new companies enabled

Amount leveraged for every dollar invested - The total dollar value of Number companies assisted

partner funds and like-kind values contributed to match the Authority's Number/Amount contracts and grants administered

partnership advertising and marketing program dollars of investment Amount economic development brought into state through competitive

Film, Video, and Music: Increase industry awareness of Georgia business opportunities, financial incentives, infrastructure resources, and natural resources in order to attract film, video, music, and electronic gaming industry projects and businesses to the state.

expressed as a ratio

Number film industry contacts - The total number of companies

1. Amount of new direct capital investment into the contacted or making contact with the Film Office for filming

state (millions).

opportunities in Virginia

2. Number of new projects initiated.

The Virginia Economic Development Partnership: Dollars invested by new and existing companies

grants
Amount expenditures per job created/retained by X program Percent customers rating informational assistance as excellent Percent customers rating analysis and forecasting assistance as excellent Percent department investment in rural areas ROI - return to General Fund

Number Jobs created by new and existing companies

ROI - public dollars saved through X program

Companies counseled or engaged in a Trade event

ROI - partner matching investment

Dollar Unit Cost of Direct Jobs Announced

Amount invested by new or existing companies assisted by the agency

For additional information or for copies of this report call 404-657-5220. Or see our website:
http://www.audits.ga.gov/rsaAudits/