Follow-Up Review Report No. 19-06 June 2019 Georgia Department of Audits and Accounts Performance Audit Division Greg S. Griffin State Auditor Leslie McGuire Director Why we did this review This follow-up review was conducted to determine the extent to which the Georgia Department of Transportation (GDOT) and the General Assembly addressed recommendations presented in our December 2016 special examination (Report 16-17). The 2016 special examination was conducted at the request of the Senate Appropriations Committee. We were asked to review how GDOT determines which highway projects it will fund and to what extent GDOT follows industry standards or best practices for setting priorities and selecting highway projects. The Committee also asked us to identify opportunities for making the project selection process more transparent. About Capacity Projects The statewide transportation planning process for capacity projects is overseen by GDOT s Planning Director who is appointed by and reports to the Governor. Capacity projects (e.g. widening new roadways managed lanes etc.) are intended to reduce congestion. The Planning Division selects and prioritizes capacity projects in coordination with Metropolitan Planning Organizations and nonurban local officials. Follow-Up Review Transportation Project Selection and Prioritization Initial progress made in selecting and prioritizing capacity projects but work remains to address findings What we found Since our 2016 review the Georgia Department of Transportation s (GDOT) Planning Division has improved the project selection and scoring process by revising the standard criteria refining the scoring methodology and better ensuring that all projects are scored. However work remains to fully address findings related to formalizing initial project selection incorporating benefit-cost analysis and tracking the impact of congressional balancing requirements. Findings related to process improvements and communication with stakeholders also remain mostly unaddressed. Project Selection Criteria and Scoring Methods The original review found there was no standard criteria or formal evaluation process for project selection. Once projects were selected they were scored to determine funding order but the criteria and methodology used were problematic. The Division has still not established standard criteria or a formal evaluation process for selecting potential projects which would introduce more objectivity and transparency to the project selection process. While the Division continues to review relevant data as needed (e.g. crash history) this process remains informal rather than a documented study. The Division has revised the scoring criteria and methodology that is used to prioritize projects after selection but additional improvements are needed. The revised criteria are directly tied to 270 Washington Street SW Suite 1-156 Atlanta Georgia 30334 Phone (404)656-2180 www.audits.ga.gov strategic goals and are focused on three categories congestion & mobility safety and economic development. The Division has also enhanced the scoring methodology by reassigning point values scoring on a continuum rather than an "all or nothing" basis and capping the point total. Additionally the Division is doing a better job of ensuring that all projects are scored rather than allowing certain projects to bypass the process. While the scoring process has been improved since the release of the original report several weaknesses continue to exist. The project scoring criteria are focused on existing conditions rather than outcomes and the process does not include a benefit-cost analysis to compare the relative value of projects. Furthermore the Division still does not track or report on the impact of congressional balancing requirements which can allow lower priority projects to be advanced ahead of projects of greater need or benefit. Process Improvements The original review found that the Division lacked 1) a comprehensive system for tracking projects through each step of the selection and prioritization process and documenting decisions and 2) detailed policies and procedures to guide decision-making. Since then the Division has taken an initial step toward implementing a new tracking system but has not developed more detailed policies and procedures. In 2018 Division management began working with a software developer to help streamline and semi-automate the project selection and prioritization process. According to Division management the software will have the capability to centrally compile and maintain results automate the input of performance data graphically display data and improve reporting capabilities. However these discussions are still in the early stages and there is no timeline for actual implementation. Also the Division still does not have specific policies and procedures for guiding key steps including proposing projects compiling tracking and screening potential projects programming projects or selecting projects to include in the STIP. Consequently there is less assurance that the process is consistent and objective. Communication with Stakeholders (Including Metropolitan Planning Organizations) The original review recommended that the Division improve its communication regarding its overall project selection process and scoring methodology as well as the rationale for selecting or not selecting specific projects. These recommendations remain unaddressed. The Division does not provide a description of the project selection process or scoring methodology on its website. The General Assembly has not established any additional requirements regarding public disclosure of the process or project selection decisions. The Division has not developed any additional guidance for coordinating with metropolitan planning organizations. According to Division management federal code provides sufficient guidance as to the planning process and roles and responsibilities. GDOT s Response GDOT indicated that it "has made several revisions that have resulted in improvements to the selection and prioritization of capacity projects" and "has implemented the recommendations determined to enhance the existing process." However GDOT continues to disagree with the findings marked "not addressed" or "partially addressed." GDOT stated its intent to better document its processes and evaluate its prioritization criteria. The following table summarizes the findings and recommendations in our 2016 report and actions taken to address them. A copy of the 2016 performance audit report 16-17 may be accessed at http //www.audits.ga.gov/rsaAudits. Transportation Project Selection and Prioritization 3 Transportation Project Selection and Prioritization Follow-Up Review June 2019 Original Findings/Recommendations The Planning Division should revise its process to ensure projects are formally evaluated against a set of standard criteria before they are selected and programmed. We recommended the Planning Division formalize its initial project review process and ensure that studies are conducted as prescribed in the Planning Manual. Further the Planning Division should develop a process for screening project proposals prior to programming. Current Status Not Addressed The Planning Division has not revised or formalized the initial project selection process which would increase objectivity and transparency. There are no standard criteria against which all potential projects are evaluated. As noted in the original report Division staff conduct project reviews that may involve the Georgia Statewide Travel Demand Model crash data and traffic data analysis subject matter expertise from GDOT district offices and local officials and financial considerations. However this review is a more informal process rather than a study with documented results. The Division also has a project scoring process but this process does not occur until after the project has been programmed and funding has been allocated. As such the results can help prioritize scheduled projects but cannot inform the initial programming decision. The Planning Division should establish controls to ensure projects are consistently evaluated against scoring criteria. The Planning Division should also prioritize projects according to their scores to help inform decisions about which projects to select and program. We recommended that the Planning Division ensure that all capacity and economic development projects are objectively evaluated establish data controls to ensure accuracy and completeness of project scoring data and develop policies to ensure that decision-making is based on the results of the prioritization process. Partially Addressed Once projects are selected the Planning Division is better ensuring that the projects are scored and that the scores are complete and accurate. In the original review we found that 25 of 280 (9%) capacity projects programmed in the upcoming 10 years were not scored. Of the 255 projects with scores 23 (9%) were incomplete due to missing data. Since then the percentage of unscored projects has dropped to approximately 3%. In addition there is no longer missing data for those projects that are scored. While the Division has improved the scoring data it has not implemented any new controls to ensure that the scores are driving the prioritization process. In response Division management continues to emphasize that programming decisions are influenced by numerous factors including deliverability funding scheduling constraints congressional balancing requirements and other statutory mandates. Management also noted that the process has been made even more complex by the most recent federal transportation funding bill which establishes targets in various transportation goal areas. While we recognize the complexity and the numerous factors that must be considered we still recommend implementing strategies to ensure decision-making is as objective as possible. For example the Division could track the consistency between the project priority list and the programming list and document decisions to implement lowscoring projects ahead of higher-scoring projects. The Planning Division should utilize project scoring criteria that most effectively assess a project s need and potential impact. Specifically we recommended project scoring criteria that are aligned with longterm goals outcome focused and nonduplicative. Partially Addressed The Planning Division revised its project scoring criteria to focus on three goal areas mobility & congestion safety and economic development. Each of these criteria is directly tied to a GDOT strategic goal(s). For example the economic development criteria is tied to the state s goals of expanding Georgia s role as a major logistics hub for global commerce and improved access to jobs. Additionally the revised criteria are simplified and no longer include redundant measures. However all the criteria used are based on existing conditions rather than projected outcomes. Planning Division management noted that outcomes can be difficult to predict and states are just beginning to move Transportation Project Selection and Prioritization 4 Transportation Project Selection and Prioritization Follow-Up Review June 2019 Original Findings/Recommendations Current Status towards outcome-focused criteria. The Planning Division should incorporate benefit-cost analysis as part of its project selection process. We recommended the incorporation of benefit-cost analysis as a criterion by the Planning Division but also that the General Assembly may consider requiring such an analysis in statute. Not Addressed The Planning Division has not incorporated benefit-cost analysis into its project selection process and the General Assembly has not made any statutory changes that would require benefit-cost analysis. Benefit-cost analysis is considered a best practice because it can help determine if a project is a justified investment and it can be used to compare the relative value of projects for ranking/priority purposes. The Planning Division has not incorporated benefit-cost analysis due to concerns that it would not be representative of true project need and the high cost of right of way in urban areas would skew the benefit/cost lower. The Planning Division should revise its scoring methodology to better ensure that project scores accurately reflect the need for the project and the potential benefit. We recommended the Planning Division modify its designation of key criteria and assign potential point values that reflect the relative importance of each criterion cap the number of safety points and establish a total point range score each criterion on a continuum weight criteria according to project location and scale project scores to account for potential statistical issues. Fully Addressed The Planning Division has revised its scoring methodology. The scoring methodology no longer designates "key criteria " which lacked rationale at the time of the original review. Further the new scoring method reassigned point values to better reflect each criterion s relative importance. For example the congestion & mobility category which is the primary purpose of capacity projects accounts for half of the potential point total. The Division also capped the maximum score at 100 to prevent a single measure from exponentially driving up the overall score. All new criteria are scored on a continuum rather than the previous "all or nothing" point application used for several criteria. The point allocations also account for differences between rural and urban areas. GDOT the Planning Division and the General Assembly should explore alternative methods for considering regional needs. We recommended that GDOT and the Planning Division begin regularly tracking the impact of congressional balancing on project selection and prioritization decisions and report on such impacts to the GDOT Board and General Assembly. Not Addressed The Planning Division has not initiated any additional tracking of congressional balancing requirements that could be used to explore alternatives for considering regional needs. While GDOT continues to track congressional balancing in a spreadsheet and provides a congressional balancing report that outlines investments in each district there is no documentation or analysis of the balancing requirement s impact on programming decisions. For example the Division does not document which projects are re-prioritized due to congressional balancing constraints. Additionally the General Assembly has not made any changes to the congressional balancing requirements stipulated in state law. Without a change in state law Planning Division management noted that the recommendation could not be fully implemented. The Planning Division should streamline and automate the project selection and prioritization process and ensure that all relevant information is tracked and accessible. As part of this effort the Planning Division should consider the need for decisionmaking software or tools with greater functionality. Partially Addressed The Planning Division has taken an initial step towards streamlining and automating its project selection and prioritization process. In 2018 the Planning Division began working with a Georgia-based software developer to implement a new application. Division staff indicated that the new software will have the capability to centrally compile and maintain results automate the input of performance data graphically display data and improve reporting. However the development of the Transportation Project Selection and Prioritization 5 Transportation Project Selection and Prioritization Follow-Up Review June 2019 Original Findings/Recommendations We recommended the Planning Division aim for greater automation in the selection and prioritization of capacity projects. Specifically we suggested that the agency evaluate the extent to which additional or new analytical tools would be required for the greater degrees of automation. Current Status software remains in the very early stages without any timeline for implementation. The Planning Division should establish more specific policies and procedures to guide the project selection and prioritization process. We recommended the Planning Division develop and/or update policies and procedures to guide all aspects of the project selection and prioritization process. Revised policies and procedures should address the processes for proposing projects compiling tracking and screening potential projects programming projects scoring projects and selecting projects to include in the Statewide Transportation Improvement Program (STIP). Not Addressed The Planning Division has not established more specific policies and procedures to guide the project selection and prioritization process. While the Division has updated its manual since the original report the manual still does not adequately address processes for proposing projects compiling tracking and screening potential projects programming projects or selecting projects to include in the STIP. The Division has also compiled white papers to serve as a resource but the white papers provide primarily a high-level overview. The Planning Division should work with MPOs to clarify its level of input and assistance in the development of the Transportation Improvement Programs (TIPs). We recommended the Planning Division work with Metropolitan Planning Organizations (MPOs) to understand their concerns about the level of input and identify solutions. The Planning Division could conduct additional planning meetings with those MPOs that lack staff. The Planning Division could also develop and distribute guidelines for coordination with MPOs to ensure that roles and responsibilities are clearly communicated. Not Addressed The Planning Division has not developed any additional guidance documents for coordinating with the MPOs or changed any practices. In the original report we found that the MPO s role varied and several MPOs were concerned with their level of influence in prioritizing and selecting projects. Division management disagreed with these findings and recommendations contending that federal code defines the scope of the metropolitan transportation planning process. Division management also noted that MPOs identify their planning priorities and activities in the Unified Planning Work Program and that MPOs vote on which projects are included in the TIP. During the follow-up Division management has re- stated this position as well as their commitment to continuous and cooperative participation with the MPOs. The Planning Division should better communicate its overall project selection process and its criteria and scoring methodology. In addition the Planning Division should improve its communication on project selection decisions. We recommended the Planning Division provide additional information on its website. We also recommended that the Division communicate information regarding its rationale for selecting or not selecting specific projects or adding and removing projects from the STIP. In addition we Not Addressed Recommendations regarding communication improvements remain unaddressed. GDOT does not provide a description of the project selection process and scoring methodology on its website. Further the Planning Division did not indicate any changes to policy or practice regarding the communication of scoring criteria and methodology to MPOs. The Planning Division s SSTP still does not include a listing of projects with 100% state funding although management did note that most of the projects that are now funded with state funds were projects that were already programmed with federal funds. Finally the General Assembly has not established any new statutory requirements regarding the public disclosure of the Planning Division s process and project selection decisions although management did present the capacity project Transportation Project Selection and Prioritization 6 Transportation Project Selection and Prioritization Follow-Up Review June 2019 Original Findings/Recommendations recommended including a listing of 100% state-funded projects as part of the Statewide Strategic Transportation Plan (SSTP). Finally we recommended that the General Assembly consider requiring GDOT to publicly disclose more information regarding its process and project selection decisions. Current Status selection criteria goal areas to legislators last year. While no significant changes have been made Planning Division management recapped the planning activities that the Division has continued to conduct. Specifically management noted that projects are listed in the STIP with a priority ranking as demonstrated by the year they show funding. Each STIP update cycle the Division conducts public meetings to discuss projects. The Division also meets with rural local elected officials and state transportation board members and assigns a GDOT planner to each MPO for communication purposes. 10 Findings 1 Fully Addressed 3 Partially Addressed 6 Not Addressed The Performance Audit Division was established in 1971 to conduct in-depth reviews of state-funded programs. Our reviews determine if programs are meeting goals and objectives measure program results and effectiveness identify alternate methods to meet goals evaluate efficiency of resource allocation assess compliance with laws and regulations and provide credible management information to decision makers. For more information contact us at (404)656-2180 or visit our website at www.audits.ga.gov.