Evidence-based decision making: developing a knowledge base for successful program outcomes in transportation asset management [Dec. 15, 2015]

GEORGIA DOT RESEARCH PROJECT 12-18 FINAL REPORT
Evidence-Based Decision Making: Developing a Knowledge Base for Successful Program
Outcomes in Transportation Asset Management
OFFICE OF RESEARCH 15 Kennedy Drive
Forest Park, GA 30297-2534

TECHNICAL REPORT STANDARD TITLE PAGE

1.Report No.: FHWAGDOT 12-18

2. Government Accession No.:

3. Recipient's Catalog No.: N/A

4. Title and Subtitle:
Evidence-Based Decision Making: Developing a Knowledge Base for Successful Program Outcomes in Transportation Asset Management
7. Author(s):
Adjo Amekudzi-Kennedy, Ph.D., Janille Smith-Colin, Jamie M. Fischer, Richard Boadi, Margaret-Avis Akofio-Sowah, Stephanie Amoaning-Yankson, Stefanie Brodie and Amy Ingles

5. Report Date: December 15, 2015 6. Performing Organization Code:
8. Performing Organ. Report No.:

9. Performing Organization Name and Address:
Georgia Transportation Institute/UTC Georgia Institute of Technology 790 Atlantic Drive Atlanta, GA 30332-0355

10. Work Unit No.: N/A
11. Contract or Grant No.:

12. Sponsoring Agency Name and Address:
Georgia Department of Transportation Office of Research 15 Kennedy Drive Forest Park, GA 30297-2534

13. Type of Report and Period Covered:
Final; January 2012 December 2015
14. Sponsoring Agency Code:
N/A

15. Supplementary Notes: Prepared in cooperation with the U.S. Department of Transportation, Federal Highway Administration.
16. Abstract:
MAP-21 and AASHTO's framework for transportation asset management (TAM) offer opportunities to use more rigorous approaches to collect and apply evidence within a TAM context. This report documents the results of a study funded by the Georgia Department of Transportation, conducted by the Infrastructure Research Group (IRG) at Georgia Institute of Technology, to link TAM investments to outcomes. The study methodology includes a review of evidencebased approaches from various fields including healthcare, education and organizational management, an "evidence exchange" among transportation practitioners, and several case studies demonstrating how evidence can be gathered and used in TAM. The study identifies attributes of an evidence-based approach that can enhance AASHTO's existing framework for TAM program advancement and decision making. These attributes are used to develop (1) a hierarchy of evidence that can inform evidence-based decision making (EBDM) for TAM, (2) a documentation framework for TAM that can be applied at the project and program levels to support evidence-based decisions, and (3) a prototype evidence database that can be used to accumulate the knowledge needed for evidence-based decision making in a TAM context. This study also applies evidence-based decision-making processes to provide guidance on how to develop higher quality evidence for TAM decision making. The documentation framework is used to synthesize and gather data on evidence related to TAM decision making, and this information was used to populate the TAM evidence database. The primary outcome of this work is a prototype TAM evidence database and resource for knowledge sharing that can assist transportation agencies in meeting the performance-based planning requirements of MAP-21.

17. Key Words: evidence-based, decision making, transportation asset
management

18. Distribution Statement:

19. Security Classification (of this report):
Unclassified
Form DOT 1700.7 (8-69)

20. Security classification (of 21. Number

this page): Unclassified

of Pages: 32

22. Price:
N/A

Evidence-Based Decision Making: Developing a Knowledge Base for Successful Program Outcomes
in Transportation Asset Management
Prepared for: Georgia Department of Transportation
Point of Contact: Angela T. Alexander
Prepared by: Georgia Institute of Technology
Principal Investigator: Adjo Amekudzi -Kennedy, Ph.D.
Graduate Research Assistants: Janille Smith-Colin Jamie M. Fischer Richard Boadi
Margaret-Avis Akofio-Sowah Stephanie Amoaning-Yankson
Stefanie Brodie Amy Ingles
December 15, 2015

Evidence-Based Decision Making for Transportation Asset Management
EXECUTIVE SUMMARY
MAP-21 and AASHTO's framework for transportation asset management (TAM) offer opportunities to use more rigorous approaches to collect and apply "evidence" within a TAM context. This report documents the results of a study funded by the Georgia Department of Transportation, conducted by the Infrastructure Research Group (IRG) at Georgia Institute of Technology, to link TAM investments to outcomes. The study methodology includes a review of evidence-based approaches from various fields including healthcare, education and organizational management, an "evidence exchange" among transportation practitioners, and several case studies demonstrating how evidence can be gathered and used in TAM. The study identifies attributes of an evidence-based approach that can enhance AASHTO's existing framework for TAM program advancement and decision making. These attributes are used to develop (1) a hierarchy of evidence that can inform evidence-based decision making (EBDM) for TAM, (2) a documentation framework for TAM that can be applied at the project and program levels to support evidencebased decisions, and (3) a prototype evidence database that can be used to accumulate the knowledge needed for evidence-based decision making in a TAM context. This study also applies evidence-based decision-making processes to provide guidance on how to develop higher quality evidence for TAM decision making. The documentation framework is used to synthesize and gather data on evidence related to TAM decision making, and this information was used to populate the TAM evidence database. The primary outcome of this work is a prototype TAM evidence database and resource for knowledge sharing that can assist transportation agencies in meeting the performance-based planning requirements of MAP-21.
The key findings of this study are as follows:
Evidence quality is strengthened through the accumulation of high-quality studies and systematic documentation.
The quality and amount of information available to decision makers ultimately shape the quality of decision-support information and, potentially, the outcomes of a decision.
Considerations such as data collection, data availability, and data quality are essential features in documenting useful evidence for decision making.
A hierarchy of evidence can be used by transportation agencies to augment TAM decisions at both the program-level and project-level.
Agencies are at different levels along several stages of maturing TAM practice. All agencies have made investments in TAM interventions. Some have made investments that have sometimes not provided desirable results, and still others have made highly effective investments with clear evidence of desired performance outcomes.
Accumulated experience about TAM, so long as it is well documented, can become an evidence base for more effective asset management decision making in the future,
In the resource-constrained environment of transportation decision making, it is critical for decision makers to justify actions that use public funds. Evidence is necessary to improve decision making. Better evidence can contribute to improved credibility of an agency. The ultimate questions decision makers want to address are: (1) What did we do? (2) How did we perform? and (3) How can we do better? An evidence-based database can assist agencies by providing higher quality evidence with which to answer these questions.
Page i

Evidence-Based Decision Making for Transportation Asset Management Applying the prototype database and planning tool involves developing a formal process to document TAM interventions and outcomes within the agency. This will involve establishing a process for systematic review of asset management projects and programs in a cost-effective manner. In addition to applying the PICMO (problem, intervention, context, mechanism, outcome) documentation process to organize information about interventions-and-outcomes; inputting PICMO information into databases will be done together with information on the agency's confidence that the documented interventions were responsible for the documented outcomes. With regard to implementation, periodic (e.g., quarterly) evidence exchange meetings can be held where best practices are shared and the information from these is documented. In the longer term, the evidence-based TAM database will work better as a planning tool if a critical mass of agencies adopt and use it, making broadly available their interventions and outcomes over time, together with clear documentation of the respective levels of confidence they associate with these data records. Interested agencies can work though a pooled-fund study to develop a more comprehensive database and the protocols for inputting and maintaining data.
Page ii

Evidence-Based Decision Making for Transportation Asset Management
TABLE OF CONTENTS
CHAPTER 1: INTRODUCTION ................................................................................................... 1 Background and Objectives ........................................................................................................ 1 Evidence and Performance-Based Approaches .......................................................................... 2
CHAPTER 2: APPLICATIONS OF EVIDENCE-BASED THINKING....................................... 3 Lessons from Healthcare............................................................................................................. 3 Lessons from Social Policy and Programs.................................................................................. 4 Lessons from Education.............................................................................................................. 4 Lessons from Organizational Management ................................................................................ 5 Evidence-based Approaches in Transportation .......................................................................... 6
CHAPTER 3: TAM EVIDENCE QUALITY AND DOCUMENTAION ..................................... 7 Case Study Development ............................................................................................................ 7 Documenting Evidence in TAM ................................................................................................. 8 Defining Quality ......................................................................................................................... 9 Study Design ........................................................................................................................... 9 Study Quality .......................................................................................................................... 9 Consistency ........................................................................................................................... 10 Applicability ......................................................................................................................... 10 TAM Maturity Hierarchy.......................................................................................................... 10
CHAPTER 4: THE TAM EVIDENCE EXCHANGE.................................................................. 13 Overview of the TAM Evidence Exchange .............................................................................. 13 Webinar Results and Case Studies............................................................................................ 14 System Integration: General Observations ........................................................................... 14 Data, Evidence Quality and Value: General Observations ................................................... 15 Evidence Generation and Knowledge Exchange: General Observations ............................. 16 Risk Management and Tradeoffs: General Observations ..................................................... 17
CHAPTER 5: EVIDENCE-BASED DATABASE FOR TRANSPORTATION ASSET MANAGEMENT.......................................................................................................................... 19
Asset Management Evidence Databases in Practice................................................................. 19 NCHRP Asset Management Knowledge Portal ................................................................... 19 Netlipse Knowledge Networking....................................................................................... 20 Water Infrastructure Knowledge Database - WaterID.......................................................... 20
Evidence-based Database for TAM Development ................................................................... 20 Database Completeness, Level of Confidence and Value Added............................................. 25
Page iii

Evidence-Based Decision Making for Transportation Asset Management SUMMARY OF FINDINGS ........................................................................................................ 26 REFERENCES ............................................................................................................................. 27 ACKNOWLEDGEMENTS .......................................................................................................... 32 LIST OF TABLES Table 1: Evidence Based Studies in Transportation ....................................................................... 6 Table 2: PICMO Documentation Framework................................................................................. 8 Table 3: Role of Evidence in TAM Maturation............................................................................ 11 Table 4: Oregon DOT PICMO ..................................................................................................... 14 Table 5: Ohio DOT PICMO ......................................................................................................... 15 Table 6: Tillamook County PICMO ............................................................................................. 16 Table 7: North Carolina PICMO................................................................................................... 16 Table 8: AMOTIA Consortium PICMO....................................................................................... 17 Table 9: Portland Water Bureau PICMO...................................................................................... 17 LIST OF FIGURES Figure 1: Welcome and Information Page .................................................................................... 21 Figure 2: Contact Information Form............................................................................................. 22 Figure 3: Record Entry Form ........................................................................................................ 22 Figure 4: Main Database Screenshot 1 ......................................................................................... 23 Figure 5: Main Database Screenshot 2 ......................................................................................... 23 Figure 6: Search Page ................................................................................................................... 24 Figure 7: Results Page .................................................................................................................. 24
Page iv

Evidence-Based Decision Making for Transportation Asset Management
CHAPTER 1: INTRODUCTION
Background and Objectives
The 2012 national surface transportation legislation, Moving Ahead for Progress in the 21st Century (MAP-21) has introduced requirements for performance-based decision making and performance management in Transportation Asset Management (TAM). MAP-21 defines Asset Management as a strategic and systematic process of operating, maintaining, and improving physical assets, with a focus on engineering and economic analysis based upon quality information, to identify a structured sequence of maintenance, preservation, repair, rehabilitation, and replacement actions that will achieve and sustain a desired state of good repair over the lifecycle of the assets at minimum practicable cost. This definition derives from various sources issued prior to the passage of MAP-21. For example, the American Association of State Highway and Transportation Officials (1) issued an implementation guide on Asset Management in 2011 (2), defining transportation asset management as a strategic approach to making decisions about transportation facilities over their lifecycles. AASHTO's TAM Strategic Plan (2011-2015) defines Asset Management as a "strategic and systematic process of operating, maintaining, upgrading and expanding physical assets effectively throughout their lifecycle" (3). It recognizes that TAM is not a new concept but rather an evolution in the application of established concepts in an integrated business process cycle within transportation agencies. It further presents TAM as a business process for achieving the agency's mission, noting that agencies may vary widely in their ability to maximize the accomplishment of their missions (3). Fundamentally, TAM implementation is about good management, effective leadership, and achieving the right organizational culture (2).
MAP-21 requires state transportation agencies to develop a risk-based, performance-based, transportation asset management plan. With this requirement, it is expected that transportation agencies will enhance their TAM programs to meet the legislative requirements. However, as clarified by AASHTO's TAM implementation guide (2), there is no "one-size-fits-all' solution to TAM because agencies are at various levels along a continuum of maturing TAM practice. Also, according to AASHTO, TAM decision making should be "based upon quality information" (3). In the context of TAM program development and project-level decision making, "quality information" will include data about an agency's own assets and processes as well as evidence regarding what has worked internally and for peer agencies. In this sense, an evidence-based framework can enhance the assessment of asset management program maturity and help guide agencies to adopt more strategic investments with proven effectiveness. An evidence-based approach to TAM therefore aligns well with the requirement for performance-based approaches, since effective performance management depends on quality information.
Evidence-based approaches have been used in a limited but growing number of applications in transportation decision making. Previous work has indicated that transportation practitioners can learn much from other fields, such as healthcare, where evidence-based approaches are more established (4). The objective of this study is to develop an evidence-based database and planning tool that will allow agencies to incorporate a formalized and more rigorous approach to the development and use of quality evidence in TAM. This tool complements AASHTO's existing framework for TAM program development by providing a formal means to incorporate evidence of successful outcomes. To achieve this objective, this study addresses the issue of evidence and
1

Evidence-Based Decision Making for Transportation Asset Management
performance outcomes in TAM -- specifically what constitutes quality evidence, and how it can be gathered and used to improve TAM practice.
Evidence and Performance-Based Approaches
An evidence-based approach to TAM can be viewed as part of a broader, performance-based approach. Performance-based business practices have become increasingly regarded as important within the transportation field. For example, MAP-21 was the first piece of federal surface transportation legislation to explicitly call for "performance management" (PM) at federal, state, and regional transportation agencies (5). The American Association of State Highway and Transportation Officials (AASHTO), which was instrumental in shaping MAP-21, defines the business practice of PM as "us[ing] performance measures to solve complex management challenges" and "to support investment decision making" (1).
In a transportation context, "the business practice of PM may integrate several of the traditional activities of a transportation agency (or even multiple agencies)" (6), which were previously conceived of as separate or "siloed". For example, PM may link strategic planning, long-range planning, human resource management, and asset management. A performance-based approach is especially important in transportation asset management since deteriorated physical assets can have serious consequences for the public. Indeed, TAM and PM have evolved simultaneously, and often together. For example, measures of bridge health became widely used in the United States after Congress established the National Bridge Inspection Program in response to the deadly collapse of the Silver Bridge in 1967 (7).
Transportation agencies are learning to use PM and TAM to optimize the use of their limited resources, including financial, human, and material resources. State Departments of Transportation (DOTs), in particular, are adapting to develop and use "risk-based, performancebased" asset management. As with PM in other areas, the bedrock of PM in TAM is an effective set of performance metrics. Effective metrics in TAM are founded on sound data, and they support an agency's TAM-related goals and objectives. Beyond metrics, however, PM may be thought of as "a cycle of interlinked processes", a key component of which is to "use performance information in decision making to design new programs or projects, and to periodically update goals, objectives and targets" (2).
According to then AASHTO Director for Engineering and Technical Services, Tony Kane, a successful asset management plan must incorporate "continuous organizational learning" that relies on "evidence-based decision making" (8). This is because transportation agencies need to know what approaches work well in TAM. Also, TAM plans are likely to become more effective as time goes on, as agencies accumulate learning from their own experiences through performance measurement and management, and as they learn from the experiences of their peers. Accumulated experience in TAM, so long as it is well documented, can become an evidence base for more effective asset management plans and decision making in the future.
2

Evidence-Based Decision Making for Transportation Asset Management
CHAPTER 2: APPLICATIONS OF EVIDENCE-BASED THINKING
Evidence, in science, represents the degree of proof that can be gained from practical experience and literature. Evidence-based decision making (EBDM) includes a wide variety of applications, such as evidence-based design, evidence-based planning, evidence-based policy, and evidencebased management. EBDM is not yet widely applied in transportation; however, there have been successful applications in other fields such as healthcare, social policy, education, and organizational management. Much can be learned from these fields, which then can be applied to EBDM in transportation generally, and TAM specifically. An extensive literature review is provided in Appendix A of this report.
Lessons from Healthcare
At the intersection of healthcare and architecture, an evidence-based approach to designing healthcare facilities has led to more effective "healing environments," which increase staff efficiency, decrease stress for patients, reduce the length of hospital stays, and ultimately save money for healthcare providers (9). These sorts of quantifiable outcomes have popularized concepts of evidence-based healthcare practice and evidence-based medicine, which rely on rigorous testing to achieve better outcomes through improved practitioner training, and service delivery. The most common definition of evidence-based practice in healthcare is the "conscientious, explicit, judicious use of current best evidence in making decisions about the care of the individual patient through integrating clinical expertise with the best available external clinical evidence from systematic research" (10).
Significant work has been done in the healthcare context to define and distinguish among different levels of evidence quality (11). One well-cited hierarchy of evidence in healthcare created by Stetler (12) proposes that the highest quality form of evidence is based on a systematic review of multiple experimental studies. A "systematic review" of documented studies may be understood in contrast to a "narrative review". In a narrative, or traditional-style literature review "the criteria for selecting specific sources for review are not always apparent to the reader"; however, a systematic literature review uses "a more rigorous and well-defined approach," that explicitly defines "strict criteria" for inclusion or exclusion of studies in the review, as well as the time frame within which the literature was selected (12).
In Stetler's hierarchy of evidence (12), the highest quality evidence is based on a systematic review of randomized controlled trials (RCTs). An RCT measures the effect of an intervention by randomly assigning individuals to two groups -- one that receives the intervention and another (the "control") that does not -- and then comparing outcomes between these groups. Less rigorous than RCTs, but still acceptable as evidence, are quasi-experimental studies, also called "natural experiments" or "comparison studies." Similar to an RCT, quasi-experimental studies make a comparison between the outcomes for one group that receives an intervention and another group that does not. However, unlike an RCT, the compared groups in a quasi-experimental study are not assigned randomly, and they may be influenced by self-selection bias (13). Nonexperimental studies such as correlational studies, descriptive research, and qualitative research such as case studies, are ranked below quasi-experimental studies on Stetler's hierarchy. Finally, the lowest level of acceptable evidence on Stetler's hierarchy is the consensus opinion of respected authorities, such as guidelines set by a nationally known group.
3

Evidence-Based Decision Making for Transportation Asset Management
Lessons from Social Policy and Programs
Evidence-based approaches to social policy have helped to improve a variety of social programs such as those focused on substance abuse, teen pregnancy and family violence. As with programs implemented in healthcare and medicine, evidence about the effectiveness of social programs is often generated through rigorous experimental studies such as RCTs. However, the literature in this area acknowledges that some social programs cannot (feasibly) or should not (morally) be evaluated using RCTs. In such cases, quasi-experimental studies are often conducted (14).
The social policy literature echoes the need for systematic reviews to identify evidence in support of an intervention. Social scientists generally recognize that the results from a single study can be biased and lead to erroneous conclusions (15). It is the collective body of evidence that is needed to fully understand the relationship between interventions and outcomes (16). The systematic review represents a fundamental practice for locating, appraising, synthesizing, and reporting "best evidence" (16). The methodology used to guide the systematic review should be guided by questions that serve to identify the studies that will be included in the systematic review, the search strategy for identifying relevant studies, and the data to be extracted from the study. Ask a poor question and you will get a poor review (17).
Lessons from Education
The Institute of Education Sciences of the U.S. Department of Education has defined evidencebased decision making as "routinely seeking out best available information on prior research and evaluating findings before adopting programs or practices that demand extensive material or human resources including funding" (18). EBDM has proven useful in education for such applications as improving reading comprehension and decreasing drop-out rates (19). Recommendations for changing educational practices have been categorized according to whether they are supported by strong, moderate, or minimal evidence of effectiveness. Strong evidence of effectiveness demonstrates that the recommended practice causes improvements for a range of students in a range of settings, and can therefore be generalized. In the absence of moderate evidence, minimal evidence to support a change of practice may exist based on strong findings or theories that are indirectly relevant to the specific context or situation being considered (20).
It can be seen from the hierarchy of "strong, moderate, and minimal" evidence in education that evidence may be accumulated over time through the documentation of individual case studies. For example, one team of educators may choose to implement an intervention (a new educational practice) based on minimal, indirectly relevant evidence. If the results of this implementation are well documented, and they clearly demonstrate effectiveness of the practice, then these documented results may become moderate evidence to support the implementation of the same practice by other educators. As more and more educators implement the same intervention, strong evidence may be accumulated to support its broad application. Eventually, quasi-experimental comparison studies or correlation studies may be generated based on a systematic review of the accumulated case studies. Since education may often be among those settings where random controlled trials (RCTs) are not feasible or appropriate, it is important to note that high quality evidence may still be accumulated through rigorous systematic reviews using well-matched comparison studies.
4

Evidence-Based Decision Making for Transportation Asset Management
Lessons from Organizational Management
Evidence-based management (EBMgt) focuses on the effective oversight of staff and resources within organizations. The management literature echoes the need for systematic reviews that rigorously address clearly defined questions and identify evidence from a practice or policy standpoint (21). Furthermore, the management literature outlines eight specific actions that should be taken when implementing an evidence-based framework (21):
1. Use data to identify need, 2. Examine studies and research, 3. Use professional wisdom, 4. Consider contextual constraints, 5. Make the best choice based on information, 6. Monitor and assess implementation, 7. Evaluate outcomes, 8. Revise and improve
According to the EBMgt literature, decisions that are supported by evidence are more likely to result in innovative processes and products, enhanced organizational performance, and fact-based evidence to provide legal justification for actions (21). EBMgt has been defined as "integrating managerial expertise with the deliberate and prudent use of best evidence in making decisions while taking into account the perspective of those who may be affected by them" (22). Depending on the type of organization where EBMgt is being practiced, the stakeholders who may be affected by decisions will vary. It is important to note that these stakeholders may exist both within and outside of a particular organization. This definition of EBMgt essentially treats stakeholder opinions (such as employee opinions, customer opinions, and partner opinions) as complementary to other forms of evidence. For example, employee or partner opinions may provide relevant evidence for evaluating a new organizational process, and customer opinions may provide evidence for evaluating a new product or service.
It is important to note that the organizational management context is not well suited to performing random controlled trials (RCTs). Therefore, as with education, strong evidence of effective organizational practices is likely to be generated over time through the accumulation of case studies, and through comparison studies involving multiple case studies. Furthermore, it is important to note that each organization is different, and also the stakeholders of each organization will have different characteristics. Therefore, researchers in organizational management must be very careful to identify the variables involved in their comparison studies in order to develop strong evidence indicating the effectiveness or ineffectiveness of organizational practices in different organizational contexts.
5

Evidence-Based Decision Making for Transportation Asset Management

Evidence-based Approaches in Transportation
Evidence-based decision making has been used to a lesser extent in transportation. The transportation studies reviewed utilized a variety of methodologies to gather and classify evidence. For example, Classen and Monahan (23) provided an in-depth description of four evidence-based steps in a systematic review (1) review existing literature, (2) classify evidence using established criteria, (3) analyze results and interpret them to identify implementable actions, and (4) recommend interventions (23). In this study, the criteria used to classify evidence were obtained from the medical field -- the American Academy of Neurology. The Classen and Monahan study also categorized interventions from each of their reviewed case studies as Level A (effective) through Level C (ineffective).

Dupont et al. (24) generated evidence by convening a panel of experts who had extensive knowledge in road safety management. The authors used a data matrix to categorize information provided by practitioners according to pre-defined categories such as knowledge, data, methodologies, and tools. This format allowed for more flexibility in the responses given by practitioners when compared to a rigid questionnaire (24). These studies demonstrate that the evidence-based approach can add value to transportation outcomes. As in other fields, transportation professionals assemble practitioner panels to inform their evidence-based work.

A limitation of the work done to date in transportation is the limited availability of studies outside of the traffic and road safety areas. There is therefore an opportunity to extend evidence-based approaches to the policy, planning, and management disciplines of transportation. Table 1 provides a sample of the evidence-based studies found in transportation.

Table 1: Evidence Based Studies in Transportation

Transportation Area

Sample Studies

Occupational Therapy

3

Injury Prevention

9

Road/Traffic Safety

18

Asset Management/

Performance

1

Example References (25), (26), (27)
(28), (29), (30), (31), (24) (32), (33), (34), (35)
(4)

Policy Other*

8

(36), (37), (38), (39), (40), (41)

6

(42), (43)

6

Evidence-Based Decision Making for Transportation Asset Management
CHAPTER 3: TAM EVIDENCE QUALITY AND DOCUMENTAION
Case Study Development
A case study is a descriptive, exploratory, or explanatory analysis of an event (44). An explanatory case study is used to explore causation in order to find underlying principles. Furthermore, case studies may be prospective or retrospective. A prospective case study is one in which criteria are established and cases that fit the established criteria are identified for study and analysis. A retrospective study is one in which criteria are established for selecting cases from a group of established studies for inclusion. Research case studies have four applications: (1) to explain complex causal links in real-life interventions, (2) to describe the real-life context in which the intervention has occurred, (3) to describe the intervention itself, and (4) to explore the situations in which the intervention being evaluated has no clear set of outcomes (45). Social science research recognizes six sources of evidence relevant to most case study research: documentation, archival records, interviews, direct observation, participant observation, and physical artifacts (45). No single source has a complete advantage over the others; rather, they might be complementary and could be used together. Thus a case study should use as many sources as are relevant to the study.
In order for evidence to be useful it should be rigorous but also relevant and well-suited to the context where it is implemented (46). Furthermore, the same hierarchy of evidence quality may not be appropriate for every field (16). A key variable in the development of an evidence-based approach is the method used to gather and generate evidence. In medicine, randomized controlled trials, or true experiments, represent the gold standard for evidence-based approaches. Trueexperimental studies allow investigators to vary or at least control explanatory variables and the intervention (47). However, true-experiments are not always feasible or appropriate in different contexts, such as in organizational management, social services, education, or even transportation. In these contexts, quasi-experimental designs such as time series studies and non-equivalent control group studies are most appropriate.
The results of this work indicate that what is most commonly observed in TAM is what is referred to as pre-experimental study design (47). Examples of pre-experimental studies are the before and after study and the static group comparison. Before and after studies compare performance outcomes for a single context (location, organization, group, etc.) both before and after a particular intervention is applied. Static group comparison studies compare performance outcomes where an intervention is applied and another context where no intervention is applied at one point in time (47). Pre-experimental studies can be used to generate quality evidence, but it is important to be aware that these research mechanisms are more prone to bias than randomized controlled trials. As seen in education and organizational management, however, the accumulated results of many pre-experimental case studies can eventually be analyzed using a systematic review process, leading to decreased bias. TAM professionals would therefore benefit from a standardized framework for documenting important elements of pre-experimental and quasi-experimental case studies, on the basis of which systematic studies can be performed.
7

Evidence-Based Decision Making for Transportation Asset Management
Documenting Evidence in TAM
Transportation agencies have the capability to generate high quality evidence for use in TAM. Better evidence can improve the quality of TAM decision making, leading to more desirable outcomes. As seen in education and organizational management, however, the accumulated results of many pre-experimental case studies can eventually be analyzed using a systematic review process, leading to decreased bias. TAM professionals would therefore benefit from a standardized framework for identifying important elements of the case studies. A common approach to structuring an evidence-based case study or systematic review is to apply PICO (problem, intervention, context, outcome) or CIMO (context, intervention, mechanism, outcome) (16). Building on the work done in other fields, the research team developed the PICMO (problem, intervention, context, mechanism, outcome) framework for application to TAM decision making (48). The PICMO framework is expected to provide structure for systematic reviews and case studies in transportation asset management, providing an opportunity to develop a formalized body of knowledge for evidence-based transportation asset management. The PICMO documentation framework is described in Table 2. The PICMO framework provides a standardized methodology to extract the key details related to program-level interventions, as well as project-level interventions, allowing for ease of information sharing and, with time, accumulation of evidence-based knowledge.

Table 2: PICMO Documentation Framework
Decision Variable Decision Question

P

Problem

I

Intervention

C

Context

M Mechanism

O

Outcome

What problem was being solved? What asset was impacted?
What intervention or investment was used to address the problem?
What was the motivation for investigating this problem?
What other factors affect the possible observed outcomes?
What institutional settings, or wider systems could impact the outcomes?
What is the pathway through which the impacts of the intervention or investment can be observed?
Are the impacts direct or indirect? What rigorous testing method was used to investigate
the relationship between interventions and outcomes? What were the intended/unintended effects of the
intervention or investment? Was the intervention successful, in quantifiable
terms? How successful was the intervention?

8

Evidence-Based Decision Making for Transportation Asset Management
Defining Quality
An evidence-based approach is dependent on high quality evidence that is rigorously tested. The development of evidence-based decision making for TAM therefore requires that steps be taken to do the following: (1) identify components of quality evidence, and (2) define an acceptable standard for rigorous testing. The approach to assessing evidence quality varies from field to field. In most cases a grade from A to E is assigned as a means of communicating evidence quality (49). Grade definitions however are not used consistently. Organizations such as state Departments of Transportation must identify the specific judgments that will influence agency-wide definitions of quality evidence. Having a clearly defined problem statement and using the PICMO framework to link problems, interventions, and outcomes is potentially very useful in these contexts. Value judgments that will have to be made about evidence quality include:
The quality of evidence across studies for each important outcome, The strength of the recommendations for current or future practice, The level of certainty around net benefits, including the balance of benefits and harm, and Overall evidence of the intervention based on industry-established criteria.
The systematic review is a key element of the evidence-based process. Randomized statistical testing represents the gold standard for evidence-based decision making in medicine and other fields where control studies are feasible. However, the systematic review process in transportation asset management needs not take the same form where control studies are not feasible. Instead, an effort can be made to define the systematic review process for TAM in terms of four key components: study design, study quality, consistency and applicability (50):
Study design Was an industry accepted study design used? Study quality How detailed and contextually appropriate were the study methods and
execution? To what extent did the level of detail of the study methods and execution match the context of the problem? Consistency Are similar results observed across studies? Applicability To what extent can the intervention be applied in different contexts?
Study Design Study design for TAM decision making includes case studies, cohort studies, time series analyses, and controlled before and after studies. To develop an evidence-based TAM framework decisions should be taken regarding the relative rigor of one study design as compared to another. The TAM community has not discussed the issue of rigor. However, an attempt has been made to define TAM maturity. As an agency develops an evidence-based TAM program, it should move towards connecting TAM maturity and study design. A program of increasing TAM maturity should use testing methods of increasing rigor to connect investments to outcomes through more robust mechanisms. To the extent possible, there should be a movement away from practitioner observation as the only basis for decision making.
Study Quality Evidence-based literature advocates for the accumulation of documented evidence as a resource for decision making. Therefore it is critical that agencies move towards a systematic methodology for recording decision inputs and outcomes. The PICMO framework (48) is used here as a
9

Evidence-Based Decision Making for Transportation Asset Management
methodology to formally and consistently record decision inputs and outcomes. The PICMO framework can motivate agencies to identify key decision variables (motivation), and identify the strength of attribution (mechanism) through rigorous study methods. As evidence accumulates using the PICMO method, systematic reviews of such evidence can lead to more informed decision making. The quality of a systematic review, and the strength of evidence it provides, will largely depend on the precision and coverage of available data (51). A study providing few or sparse data points could yield lower quality or biased decision results, whereas a more extensive review will decrease bias. In cases where data points are few because directly relevant evidence is not yet accumulated, an agency may still make decisions based on minimal or moderate evidence. However, focus and efforts should be placed on accumulating evidence through the documentation of outcomes, and gathering additional evidence from related contexts as a means of improving the quality of decisions in the future.
Consistency As discussed previously, the randomized control study will not often be an appropriate experimental design for linking TAM investments to outcomes. However, valuable evidence may be gathered through the systematic review of many accumulated case studies. At the highest level of maturity, an agency should use comparative study methods that help identify the level of consistency in case study outcomes. As evidence is accumulated to support decision making, attention should be paid to the potential differences in observed effects, the size of effects, and other unexplained potential inconsistency (51). The presence or absence of such effects can impact the quality of evidence being considered and therefore the potential strength of resulting decisions.
Applicability In TAM decision making, the idea of applicability or whether the decision contexts are similar will remain an important consideration. Factors such as asset age, vehicular traffic composition, funding levels, political environment, organizational environment, and routine maintenance practices may all impact infrastructure condition. In TAM, it is therefore unlikely that any two cases will be the same, or that the results of one intervention will be directly applicable in another case. It is critical that decision makers are aware of the weaknesses or lack of direct applicability in decision points used to inform decisions. The PICMO evidence-based framework used here requires that TAM practitioners document the "context" affecting the decision being taken. By defining the decision context, practitioners can therefore evaluate the direct applicability of other results to the TAM challenges they face providing another lens through which to evaluate evidence quality.
TAM Maturity Hierarchy
AASHTO's "TAM maturity scale" (1) a framework for evaluating TAM programs categorizes transportation agencies into five maturity levels: initial, awakening, structured, proficient, and best practice. The added value of an evidence-based approach is related to the quality of evidence used to make decisions, and the extent to which outcomes are documented. In medicine, evidence quality is assessed based on the mechanism or study design used to generate or rigorously test the data (11). The principles of the evidence-based approach therefore require an increasing
10

Evidence-Based Decision Making for Transportation Asset Management

level of evidence quality and a decreasing level of evidence bias as a program moves along the maturity scale from initial to awakening. In comparison to the prior level, each level of maturity is characterized by an increased understanding of TAM needs. Additionally, the methods used by agencies to generate evidence should increase in rigor, as should the quality of evidence as a program matures. It is expected that as a program matures an agency will use more quasi-experimental methods to generate evidence, such as time series studies and cross-sectional studies, and fewer single case studies or pre-experimental designs. Table 3 illustrates how the concept of evidence quality can be applied to AASHTO's TAM maturity scale to formally incorporate the use of quality evidence in TAM.

Table 3: Role of Evidence in TAM Maturation

Level

TAM

Methods for Generating

Maturity Evidence

1

Initial

- single case study

- practitioner observation, or gut

feeling

2

Awakening - single asset pre-test/post-test

for a snapshot in time

- engineering judgment

supported by limited data

3

Structured - static-group comparison

- pilot studies

- engineering judgment with

extensive data

- in-house research

4

Proficient - time series studies

(observation over extended time)

- cross-sectional studies

- before and after study or

correlation studies

- retrospective studies

5

Best Practice - nonequivalent control group

design of multiple groups and or

variables

- business case analysis (multiple

case studies evaluated for

specified risk and financial

criteria)

Evidence Bias

Decreasing Bias Increasing Quality

Evidence Quality

All mechanisms used to generate evidence in Table 3 can yield "quality evidence," at each given stage of maturity. As TAM practitioners continue to define acceptable standards for study design, study quality, consistency and applicability, the ideas of evidence and a testing mechanism will continue to inform TAM maturity. The most important thing is clear documentation; a strong evidence-based approach is founded on well-documented studies demonstrating failures or successes following the implementation of an intervention. As TAM programs mature, agencies will develop the capacity to generate and use more extensive and more robust evidence.

11

Evidence-Based Decision Making for Transportation Asset Management As more evidence accrues, and is shared by multiple agencies, it is likely to become easier to attribute certain outcomes to certain interventions. For this reason, there is a need for broader contributions to a knowledge base from multiple agencies involved in transportation asset management.
12

Evidence-Based Decision Making for Transportation Asset Management
CHAPTER 4: THE TAM EVIDENCE EXCHANGE
Overview of the TAM Evidence Exchange
A key element of the evidence-based approach is practitioner input towards the formulation and identification of evidence. Likewise, the traditional approach to decision making relies largely on personal experience. Many transportation practitioners use data to make decisions; however decisions are often made based on limited or incomplete information, outdated information, and individual observation (52). Evidence-based approaches provide a method for gathering, accumulating and assessing data and knowledge to improve business outcomes. This ensures higher quality decision making through the use of evidence that has been regulated, controlled, evaluated, and is therefore considered more reliable (46).
The TAM Evidence Exchange, a series of discussions exploring how evidence-based approaches may be applied in TAM, was hosted by the Infrastructure Research Group at Georgia Institute of Technology to allow practitioners to share relevant stories and experiences on their uses of evidence in TAM. Each panel included three or four practitioners from state Departments of Transportation and/or private consultants. Participants included DOTs that had contributed to the Asset Management Implementation Peer Exchange, hosted in April 2012 by the Federal Highway Administration, consultants, and other transportation asset management managers. The TAM Evidence Exchange was hosted as three online videoconferences. Each webinar consisted of a formal PowerPoint presentation delivered by the research team followed by a Q&A session involving the practitioner panel. The formal presentation focused on introducing panelists to the evidence-based approach, using evidence-based design as the example. Practitioners were also introduced to the goals and objectives of the research study, and presented with a draft evidence hierarchy for transportation asset management. Each webinar lasted approximately one hour.
To prepare for the online discussion each practitioner was contacted by phone prior to each webinar. Calls lasted approximately 30 minutes, and consisted of a one-on-one conversation between a member of the research team and the practitioner. The practitioner call was used to understand practitioner thoughts about the project objectives including how practitioners consider evidence in their day-to-day work -- its importance, how it is measured, what value it has added to their work, and other issues that practitioners wanted to highlight related to evidence in TAM decision making. Thoughts expressed by practitioners were also used to shape the Q&A portion of the webinar. Practitioner calls were also used as the initial forum to collect ideas on possible case studies for future research and reporting on TAM decision making.
The Q&A session served as an open forum for all practitioners to discuss their thoughts about the presentation, and share successes and lessons learned through TAM investments in practice. Panelists were asked to help do two things during the Q&A session: first, to identify some TAM investments at the program-level and project-level that, in their experience or observation, had resulted in quantifiable evidence of success (positive outcomes) or failure (negative outcomes). An example of a program-level investment was identified as the development of a performance reporting system, whereas an example of an investment at the project-level could be the selection of a new type of retaining wall that had resulted in quantifiable savings. Second, practitioners were asked to critique the draft PICMO evidence-based framework.
13

Evidence-Based Decision Making for Transportation Asset Management
Webinar Results and Case Studies
Over the course of three webinars, participants made observations about several themes. Related to these themes, participants suggested case studies that demonstrated the use of the evidencebased approach in TAM. The case studies and related themes identified by practitioners are presented below using the PICMO framework. A full webinar report is provided in Appendix B of this report, and additional case studies are presented in Appendix C.
System Integration: General Observations A. An integrated system considers the maintenance tradeoffs that occur when the decision is taken to maintain one set of assets over another. B. Asset managers faced with program development or restructuring should focus their attention on routinely trying new things on a small scale first and then expanding the program as resources become available. The asset management-learning framework is a step-wise one and many failures and successes may be experienced along the way. C. Practitioners must determine whether existing policies and procedures will assist in achieving the desired strategic goals, by asking the right questions, and seeking to make fact-based decisions. There must also be the professional will to alter funding structures or change work plans.

Table 4: Oregon DOT PICMO

Oregon DOT: System Integration Pilot Study

P Problem

Improvement needed with respect to data availability and reliability; ad hoc data collection processes leading to

inconsistencies and the inability to integrate data; limited agency-

wide database availability; only 3 assets being maintained and

I Intervention

tracked through some asset management program: bridges, pavements and Intelligent Transportation Systems (ITS) A pilot study was conducted from the end of 2005 to early 2007, and focused on four highway segments for a total of 75 miles. Under the direction of a Steering Committee and a Technical Committee, the pilot study compiled data about priority assets

and/or roadway features to develop an accurate perspective on

C Context M - Mechanism

how the agency's asset data availability and accuracy affected, asset condition, and data integration capabilities. Agency desire to strategically manage assets given limited funding and lack of comprehensive data Before and after review post pilot study implementation

O - Outcomes

A correlation was found between asset condition and the existence of a centralized data program, and a high level of attention from

maintenance staff; assets with a higher level of asset management

readiness also had generally higher condition levels. Revision of

strategic plan to better reflect data collection needs; Strategic plan redrafted in a simplified format to focus on

more fundamental issues like inventory and data collection

Data information and availability increased

14

Evidence-Based Decision Making for Transportation Asset Management
Recommendation: Preliminary databases should be built to contribute to decisions.

Table 5: Ohio DOT PICMO

Ohio DOT: System Integration and Pilot Study

P Problem

Inability to maintain system preservation and achieve a steady

state condition for pavements and bridges.

I Intervention

Integrated asset management system based on priority levels:

priority (interstate and four-lane divided highways), urban (state

highways within municipalities), and general (primarily two-lane

highways across the state. Biennial review and updates of system

goals to identify short-term actions needed to compliment long-

term goals.

C Context

Observed pavement deficiency in the 1990s where only 80 percent

of the state's freeways had a pavement condition rating over 65 out

of 100

M - Mechanism

Time series review; district to district comparisons.

Biennial review and updates of system goals to identify short-term

actions needed to complement long-term goals.

O - Outcomes

Notable improvements in system condition; documented evidence

supporting outcomes; business case argument

Data, Evidence Quality and Value: General Observations A. Practitioners are less interested in directives that promote additional data. The current challenge is how to make good decisions with limited data. B. Given limited budgets, the focus has shifted to collecting data once and using it multiple times. C. Documentation is key to demonstrating the value of evidence. For example asset managers can make a case for a program structure change based on documented evidence of improved maintenance, or crash rates. D. Evidence was defined as being both quantitative and qualitative. Evidence of success was also defined in terms of behavioral change, or organizational change. There was also emphasis placed on the role of the "gut feeling" or practitioner experience in the decision-making process.
Webinar participants recognized that evidence quality and value are dependent on the mechanisms used to gather information, the context in which it is used, responsibilities to stakeholders, and the timeframe in which the evidence is applied. Program maturity and cost were also believed to dictate the quality of evidence needed for decision making. For example, it may be appropriate for an agency in the initial stage of TAM to use lower quality evidence to make decisions, as their accountability may be lower in this trial and error stage. Intuition, and practitioner experience may therefore play a bigger role in the initial stages of TAM development, but a more mature program may have greater mandates for data-supported decision making. By the same token large
15

Evidence-Based Decision Making for Transportation Asset Management
expenditures may require greater evidence in support of the decision made, as compared to decisions surrounding smaller sums of money.

Table 6: Tillamook County PICMO

Tillamook County: Allocating Funds with Evidence-based Performance Guidelines

P Problem

Lack of understanding of customer expectations, asset condition,

and investment expenditures required to meet performance goals

I Intervention Risk-based approach involving technical analysis on high cost and

high risk assets; risk ratings extreme risk, high risk, medium risk,

low risk; correlation of road investment data with pavement

condition; scenario analysis and optimization software to provide

an indication of future asset condition

C Context

Desire to meet the required level of service in the most cost

effective manner for present and future consumers

M - Mechanism Annual updates and review; and comprehensive reviews every 3

years; county by county reviews

O - Outcomes

A framework connecting desired service outcome with financial

implications.

Evidence Generation and Knowledge Exchange: General Observations A. Practitioners expressed confidence in the case study as a methodology for identifying and applying evidence. Great value is derived from being able to share experiences between TAM professionals. There was also expressed interest in knowing what others are doing particularly within a regional context. A methodological approach that involves case studies was also identified as useful. B. Another mechanism used to provide evidence in support of decision making is research. C. There was expressed interest in further developing the evaluation and feedback stages of the development and implementation process. It is at this point that true connections can be made between investments and decisions, inputs and performance. D. Evidence needs to be framed in a manner that communicates to various stakeholder groups, and addresses the different outcomes of interest to them.
Table 7: North Carolina PICMO
In-House Research at North Carolina DOT North Carolina DOT has found it beneficial to have an engineer on staff with an academic research background. This natural desire to "ask the right questions" and for seeking the evidence that supports decisions in pavement management has allowed the state to better understand what it is, why it is being done, and the value it brings. North Carolina has recognized the value of locally focused research and the evidence it can bring their program. Other useful methodologies for acquiring evidence include the business case, and pilot studies.

16

Evidence-Based Decision Making for Transportation Asset Management

Table 8: AMOTIA Consortium PICMO

AMOTIA: Knowledge Sharing Through Practitioner Consortiums

P Problem

Pre-2007, disjointed approach to asset management; individual

consultants proceeding through and ad-hoc trial and error process

of asset management and operation

I Intervention

In 2007, the formation of partnerships of 20+ multinational

transportation firms - AMOTIA

C Context

Lack of a unified voice representing private sector consultants

involved in the management and operation of transportation assets;

no consensus on best practices; no formalized process for

practitioner consultation or exchange of ideas

M - Mechanism

Knowledge exchange or expert body

O - Outcomes

Expert presentations and panels addressing

FHWA and federal directives

Challenges and successes in public-private partnerships

Industry case studies on innovative practice (e.g. improving

safety, efficiency, and traffic flows around work zones)

Networking opportunities

Opportunities to expand to new partners (e.g. county, city)

Interaction with agencies on performance-based TAM.

Risk Management and Tradeoffs: General Observations A. The decision to expend funds to evaluate maintenance needs often comes at the expense of meeting already established maintenance needs. Tradeoff analysis tools are increasingly being used to evaluate competing needs, and these analyses can produce new evidence to support decision making. However, the quality of evidence generated depends on the quality of the tool, and the quality of the data inputs. B. It is important for practitioners to be able to ask and answer the following questions: What is the business risk across all assets? Can acceptable risk be documented as an agency policy? For example when funds are shifted to the maintenance, repair, and evaluation of one asset class, is it acceptable that this is often at the expense of another asset's condition or performance (pavement preservation and bridge preservation are often competing interests)?

Table 9: Portland Water Bureau PICMO

Portland Water Bureau: Risk Analysis

P Problem

Stakeholders requiring greater justification and accountability for

decisions made

I Intervention Business case analysis and risk based analysis using Consequence

& Likelihood of Failure Evaluation Matrix (CLEM) to rank asset

condition regarding risk on a scale from 1 to 5 based on residual

life

C Context

Do nothing alternative; baseline condition; status quo

17

Evidence-Based Decision Making for Transportation Asset Management

M - Mechanism O - Outcomes

Desire to establish asset criticality rankings; quantitatively diagnose asset action plan; and develop expert knowledge that is supported by quantifiable evidence Risk ratings; detailed business case data - financial data regarding investments; condition and performance data.

18

Evidence-Based Decision Making for Transportation Asset Management
CHAPTER 5: EVIDENCE-BASED DATABASE FOR TRANSPORTATION ASSET MANAGEMENT
Documentation currently available on best practices in Transportation Asset Management speaks mostly to organizational frameworks, agency buy-in, and preparation necessary for asset management integration. Data and literature currently available to asset managers address how to establish an asset management plan and policy. Little focus has however been placed on the formal documentation or reporting of interventions that have provided asset management successes, or identified opportunities for improvement.
The Office of Organizational Performance Management leads GDOT's asset management initiatives. The agency reports performance via a performance dashboard that is available to both internal and external stakeholders. GDOTs interest in building a database of evidence supporting best practices in asset management reflects interest in a formalized approach to using prior investment outcomes to improve future decision making in asset management. The use of evidence from prior investments involves documenting interventions and linking them to specific outcomes. The PICMO framework (48) has been designed to achieve this goal. As a next step the PICMO framework has been applied to build and populate a database that can be used for gathering evidence to support future TAM decision making.
The PICMO database aligns with industry-wide efforts to document and share TAM knowledge. The practice of building knowledge sharing tools has become more prevalent within TAM and other asset management fields such as mega-infrastructure project management and water. The following sections provide brief descriptions of the NCHRP Asset Management Portal, the Netlipse Networking Knowledge, and the Water Infrastructure Knowledge Database (WaterID) as a few such examples. This is followed by a detailed description of the PICMO database, its goals, and overall uses.
Asset Management Evidence Databases in Practice
NCHRP Asset Management Knowledge Portal Recognizing the need for a central location where practitioners can seek out information about asset management, the AASHTO Standing Committee on Planning commissioned NCHRP 0836/Task 125 Transportation Asset Management Knowledge Portal (53). The interactive portal serves to consolidate all existing information about Transportation Asset Management in one location that can be easily searched, and information retrieved in a simple expedited manner. The scope of the Asset Management Knowledge Portal is broad, and serves various asset management needs. The Knowledge Portal is designed to house anything from asset management plans and policies, to best practices in program development and funding, to information about successful interventions and associated outcomes.
The PICMO database can contribute to the Knowledge Portal by providing accumulated evidence of specific interventions. If agencies move towards a systematic process of documentation (PICMO), the Knowledge Portal can provide a platform for information sharing.
19

Evidence-Based Decision Making for Transportation Asset Management
Using the PICMO framework GDOT (and other agencies) will be able to demonstrate successful practices that are supported by documented evidence of related outcomes. Over time, the accumulation of such evidence will inform higher quality decision making, and help to strengthen the asset management knowledge base. In the long term, it is expected that agencies will document and link interventions and outcomes. With the creation of a knowledge portal, the development of a prototype PICMO database creates the opportunity for developing a more comprehensive database that includes experiences from multiple state DOTs.
Netlipse Knowledge Networking Netlipse is a network for the dissemination of knowledge in the management and organization of large infrastructure projects in Europe (54). Netlipse was initiated as a 2-year collaborative research project between European universities, research intuitions, project delivery organizations and private consultants. The initial focus of Netlipse was to support knowledge exchange about the management and organization of large infrastructure projects. The initial outcome of the Netlipse was a book of best practices entitled, "Managing Large Infrastructure Projects: Research on Best Practices and Lessons Learnt in Large Infrastructure Projects in Europe" (54). The findings and contributions of the members of Netlipse are stored in the knowledge center and made available for its members. Netlipse continues its efforts today by offering training and networking opportunities for its members. By constructing a knowledge portal, the transportation asset management community will join in an international trend towards documenting, accumulating and sharing evidence.
Water Infrastructure Knowledge Database - WaterID The WaterID is a National Drinking Water and Wastewater Knowledge Database. WaterID was developed to allow for the accumulation of industry knowledge about water assets including water and wastewater piping systems (55). Water asset maintenance is usually conducted based on a worst-first approach, and true asset management is often not fully understood or implemented. The lack of a central location to store and share industry knowledge created information gaps where practitioners too often struggled to deal with complex asset problems not knowing that their peers or colleagues had experiences to offer. WaterID is a national database developed by Virginia Tech. It serves as a single source of information for water practitioners where they can find information about costs and performance, case studies with real-world application, and technologies that support water asset management. The overall focus of WaterID is to increase the resources available for water asset management.
Evidence-based Database for TAM Development
The purpose of the evidence-based database for transportation asset management (EBD-TAM) is to implement the concepts discussed in this report to achieve the goal of increased documentation of TAM practices in order to accumulate evidence of successful outcomes. The goal, ultimately, is to produce a widely available, online tool that is accessible to multiple parties both to contribute and obtain information. The prototype database was developed in Microsoft Excel, making particular use of the Visual Basic macro coding features. This section describes the components of the database with accompanying illustrative screenshots.
20

Evidence-Based Decision Making for Transportation Asset Management This version (Version 5.0) of the EBD-TAM, is an Excel workbook made up of six sheets with different functions. The opening sheet, the "Welcome and Information" page shown in Figure 1 provides general information about the database in order to get the user acquainted with the tool. From this page, the user can create a new database entry, view the current database or search for specific terms relative to interventions, assets, agencies and related information.
Figure 1: Welcome and Information Page
Choosing to begin a new entry takes the user to the second sheet in the database, the "Contact Information" page. This first form, shown in Figure 2, allows the user to enter their contact information and information about the agency they represent. Currently, the data in this form is available to all users for reference purposes and to allow for follow up. From here, users can navigate back to the Welcome and Information page or continue to provide the actual intervention and outcomes data.
21

Evidence-Based Decision Making for Transportation Asset Management
Figure 2: Contact Information Form
The second part of the two-part data entry process is to provide the specific details of the asset management intervention being documented. The "Record Entry" page allows users to provide this information, following the general PICMO process discussed previously. In addition to the PICMO questions, this form also collects information related to the assets affected and the scope of the intervention (program or project level). This form is shown in Figure 3.
Figure 3: Record Entry Form
22

Evidence-Based Decision Making for Transportation Asset Management After entering the pertinent information related to the intervention, users can enter another intervention record using the same contact information, begin a completely new record with new contact information, proceed to view the database, or return to the Welcome and Information page. The "Done" button on this sheet takes the user to the "Main Database" where all the entries can be viewed. Figure 4 shows the structure of the database a simple Microsoft Excel table.
Figure 4: Main Database Screenshot 1
Figure 5 shows a different view of the main database, showing the evaluative components of Confidence Level, Value Added and Level of Completeness (which is a calculated based on how many cells in each row contain information). While this page is mainly for viewing the entries, records can be filtered out using the dropdown menus in each column heading.
Figure 5: Main Database Screenshot 2
23

Evidence-Based Decision Making for Transportation Asset Management The last feature of this database is the search feature, which allows the user to retrieve records that meet criteria related to the agency type, agency name, intervention scope and the assets included. Figure 6 shows a search being conducted for program-level interventions or interventions conducted by MPOs. The searches pull records that meet at least one of the criteria as shown in Figure 7. Although the last four records are not by MPOs, they are records of program-level interventions and therefore have been included in the search results.
Figure 6: Search Page
Figure 7: Results Page
24

Evidence-Based Decision Making for Transportation Asset Management
Database Completeness, Level of Confidence and Value Added
As shown in Figure 3, two of the questions on the Record Entry form are aimed at characterizing the quality and level of evidence provided by each data entry point. Users are asked to rate both their level of confidence that the intervention influenced the documented outcome and the value that the intervention added to their programming -- on a low-medium-high scale. As previously discussed and based on existing practice, evidence quality can be most appropriately defined with the accumulation of evidence of the same outcome. However, the novelty of this database does not allow for that level of quality definition at the present time. Instead, in the current version of the database, the quality of evidence is defined by the quality of the documentation of each intervention record as captured by the level of completeness of the data record. Thus, evidence quality is associated with the total number of records with the same intervention-outcome, the relative completeness of these records, and similarity of outcomes in these records. The level of evidence is defined by the user-reported variables of confidence level and value added. Confidence level refers to the reporting agency's confidence that the intervention led to the particular outcome, and value added refers to how much value the outcome added to the agency's asset management programming. Generally, a higher confidence level provisionally relates to a higher level of evidence, regardless of the value added.
Over time, an accumulation of records with the same intervention-outcome will indicate a higher level of evidence of that particular outcome for that intervention. In other words, the higher the number of records with the same intervention-outcome, and the more complete the documentation for each record, the higher the level of evidence that the particular intervention results in said outcome for the problem under consideration. Confidence levels can provide an additional layer of information in particular contexts on the level of evidence that a particular intervention does indeed result in a particular outcome. In the initial stages of data gathering for the database, users may rely more on the self-reported confidence levels in the records. However, as the data entries grow, the level of evidence can be determined from the number of records that have the same intervention and outcome. Thus, the self-reported confidence levels may be viewed as a surrogate for the level of evidence until there are multiple records that begin to show accumulation of the same evidence, at which point the database user can mine the data to derive the level of evidence from multiple records.
In the early stages of development of the database, the records that exhibit a high level of confidence and high value added will be potentially of the most value to the database users. At the same time, any record that exhibits a high level of confidence of low added value will indicate it is important to invest additional resources in clarifying the potential added value of such intervention before investing in it, particularly where the investment is significant. If there are multiple records on the same intervention-outcome combination, then the user may obtain a higher level of evidence by pulling and reviewing these multiple records.
25

Evidence-Based Decision Making for Transportation Asset Management
This prototype database has shown a useful way of documenting asset management interventions and their outcomes, providing evidence at various levels. Nonetheless, there are several possible areas of improvement for developing a full database that can be widely accessible to various parties. These include:
i. Deploying the database on a web-based, more user-friendly platform ii. Including the functionality for combination searches (e.g. searching all program-level
interventions by DOTs) iii. Searching keywords in any part of the record iv. Including a contact person repository, relieving users from entering their contact
information each time, and instead allowing them to pull their name from a stored list v. Limiting text response length to maintain brevity
SUMMARY OF FINDINGS
A systematic, rigorously tested approach to evaluating agency decisions is over time expected to improve the quality of TAM decision making. Many agencies are measuring performance; some are reporting their outcomes; but little is being done to directly link investments to their associated outcomes. As funding and other resources become more constrained there is greater need to understand what works. The use of a standardized documentation process offers a potentially substantial step towards building a knowledgebase of evidence in TAM decision making. A PICMO approach to TAM decision making is expected to offer clearer connections between interventions and outcomes.
Moving forward, the widespread use of PICMO to document and attribute case study results in a formal, concise and consistent manner is recommended so that agencies can begin the process of compiling and documenting TAM outcomes to accumulate evidence of successful outcomes. In the long term, the goal should be to develop an evidence-based asset management network of knowledge or a database similar to those available in education, medicine and other science fields. As greater emphasis is placed on the use of quality evidence for improved programming as TAM programs mature, it is expected that a clear hierarchy of evidence will emerge for different asset classes. TAM practitioners will then have an evidence-base of interventions that work for given problems and specific contexts. The accumulation of such intervention-outcomes over time will offer a higher level of evidence of successful outcomes for various interventions. The TAM evidence database contributes to an ongoing effort at the national, state, and local levels to better document decision outcomes. The PICMO framework as presented in the evidence database can be applied to various asset classes and asset programs. This development of a prototype database has demonstrated a useful way of documenting asset management interventions and their outcomes, and providing evidence at various levels as documentation accumulates to higher-level evidence.
26

Evidence-Based Decision Making for Transportation Asset Management
REFERENCES
(1) American Association of State Highway and Transportation Officials (AASHTO). (2007) State Transportation Performance Management Programs: Select Examples.
(2) American Association of State Highway and Transportation Officials (AASHTO). (2011a). AASHTO Transportation Asset Management Guide: A Focus on Implementation, 1st Ed.
(3) American Association of State Highway and Transportation Officials (AASHTO). (2011b). Subcommittee on Asset Management Strategic Plan 2011-2015.
(4) Bones, E., Barrella, E., and A. Amekudzi. (2013) Implementation of Evidence-Based Design Approaches in Transportation Decision Making, Transportation Research Part A, 49, 317-328.
(5) (1) 112th Congress. S.1813: MAP-21. March 2012. <http://www.govtrack.us/congress/bills/112/s1813/text> (Accessed July 15th 2013).
(6) Amekudzi, A., J. Fischer, A. Ingles, and S. Brodie. (2012) "Leading Practices in Organizational and Transportation Systems Performance Management Volume I: State DOT Benchmarking Study" Transportation Asset Management and Risk Review Draft Final Report.
(7) United States Government Accountability Office (USGAO). 2010.Statewide Transportation Planning: Opportunities Exist to Transition to Performance-based Planning and Federal Oversight.
(8) Kane, T. (2010). Performance Measurement/Management and Benchmarking: How to Ensure Data Reliability. Presentation given by the Director of Engineering and Technical Services, American Association of State Highway and Transportation Officials (AAHSTO), to the Association of Government Accountants, Baltimore, MD, October 13, 2010.
(9) Huisman, E, Morales, E., van Hoof, J., and Kort, H. (2012) Healing environment: A review of the impact of physical environmental factors on users. Building and Environment, 5870-5880.
(10) Pati, D. (2011). A Framework for Evaluating Evidence in Evidence-Based Design. Health Environments Research & Design Journal (HERD), 4(3), 50-71.
(11) Cronin P, F. Ryan, and M. Coughlan. (2008) Undertaking a Literature Review: A Step-by Step Approach. British Journal of Nursing, Vol. 17(1), pp. 38-43
(12) Stetler, C. (2002) Evidence-based practice and the use of research: A synopsis of basic concepts & strategies to improve care. Washington, DC: NOVA Foundation.
(13) Whitehouse Office of Management and Budget (2004). What Constitutes Strong Evidence of Program Effectiveness?
27

Evidence-Based Decision Making for Transportation Asset Management
(14) Cooney, S., Huser, M., & Small, S. (2007). Evidence-Based Programs: An Overview. Issue #6. WhatWorks.
(15) Hunter, J. E. & Schmidt, F. L. (2004). Methods of meta- analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
(16) Briner, R. B., Denyer, D., & Rousseau, D. M. (2009). Evidence-Based Management: Concept Cleanup Time? Academy Of Management Perspectives, 23(4), 19-32.
(17) Counsell, C. (1997). Formulating questions and locating primary studies for inclusion in systematic reviews. Annals of Internal Medicine, 127, 380387.
(18) Whitehurst, G. J. (2004). Making education evidence-based: Premises, principles, pragmatics, and politics. Paper presented at the Institute for Policy Research, North- western University, Evanston, IL.
(19) Slocum, T. A., Spencer, T. D., & Detrich, R. (2012). Best Available Evidence: Three Complementary Approaches. Education and Treatment of Children, 35(2), 153-181.
(20) Shanahan, T., Callison, K., Carriere, C., Duke, N. K., Pearson, P., Schatschneider, C., & ... What Works Clearinghouse. (2010). Improving Reading Comprehension in Kindergarten through 3rd Grade: IES Practice Guide. NCEE 2010-4038. What Works Clearinghouse.
(21) Rowley, J. (2012). Evidence-based marketing. International Journal of Market Research, 54(4), 521-541.
(22) Collins, J., Denyer, D . & Turnbull, J .K . (2008) Educating the evidence-based manager: the executive doctorate and its impact on management practice. Paper presented at the Organization Knowledge Learning and Capabilities Conference, Copenhagen.
(23) Classen S, Monahan M. Evidence-Based Review on Interventions and Determinants of Driving Performance in Teens with Attention Deficit Hyperactivity Disorder or Autism Spectrum Disorder. Traffic Injury Prevention, 2013, 14(2):188193.
(24) Dupont E et al. Needs for Evidence-Based Road Safety Decision Making in Europe. Procedia - Social and Behavioral Sciences, 2012, 48:25132522.
(25) Classen S, M. Bewernitz, O. Shechtman. (2011) Driving Simulator Sickness: An EvidenceBased Review of the Literature. American Journal of Occupational Therapy, Vol. 65, Issue 2, pp.179188.
(26) Arbesman, M. and J.M. Pellerito. (2008). Evidence-Based Perspective on the Effect of Automobile-Related Modifications on the Driving Ability, Performance, and Safety of Older Adults. American Journal of Occupational Therapy, Vol, 62, Issue 2, pp. 173186.
28

Evidence-Based Decision Making for Transportation Asset Management
(27) Classen, S., C. Levy, D. McCarthy, W. Mann, D. Lanford, and J. Waid-Ebbs. (2009). Traumatic Brain Injury and Driving Assessment: An Evidence-Based Literature Review. American Journal of Occupational Therapy, Vol. 63, Issue 5, pp. 580591.
(28) Brennan Ramirez, L.K., C.M. Hoehner, R.C. Brownson, R. Cook, C. Orleans, M. Hollander, W. Wilkinson. (2006). Indicators of Activity-Friendly Communities: An Evidence-Based Consensus Process. American Journal of Preventive Medicine, Vol. 31, Issue 6, pp.515524.
(29) Nichols, A., S. Classen, R. McPeek, and J. Briner. Does Personality Predict Driving Performance in Middle and Older Age? An Evidence-Based Literature Review. Traffic Injury Prevention, Vol. 13, Issue 2, pp. 133143.
(30) Hewson, P. (2007). Evidence-Based Practice in Road Casualty Reduction. Injury Prevention, Vol. 13, Issue 5, pp. 291292.
(31) Barffour, M., S. Gupta, G. Gururaj, and A. Hyder. (2012). Evidence-Based Road Safety Practice in India: Assessment of the Adequacy of Publicly Available Data in Meeting Requirements for Comprehensive Road Safety Data Systems. Traffic Injury Prevention, 2012, Vol.13, Suppl 1, pp.1723.
(32) St Bernard, G. (2010). Road User Characteristics, Fatal Road Traffic Crashes and Temporal Patterns in Trinidad and Tobago Evidence Based on Multivariate Statistical Analyses. Injury Prevention, Vol. 16, pp. A176-A177.
(33) Girasek, D.C. (2007). Moving America Towards Evidence-Based Approaches to Traffic Safety. Improving Traffic Safety Culture in the United States - The Journey Forward. AAA Foundation for Traffic Safety, pp. 130-148. www.aaafoundation.org/pdf/Girasek.pdf. (accessed April 14, 2013).
(34) Hauer, E. (2007). A Case for Evidence-Based Road-Safety Delivery. Improving Traffic Safety Improving Traffic Safety Culture in the United States - The Journey Forward. AAA Foundation for Traffic Safety, pp. 329-343. www.aaafoundation.org/pdf/Girasek.pdf. (accessed April 14, 2013).
(35) Munnich, Jr. L. J. (2011). Loveland. Do Americans Oppose Controversial Evidence-Based Road Safety Policies? Transportation Research Record, Vol. 2213, pp. 912.
(36) Meier, A. (2007). The Rise of "Evidence-Based Everywhere": A Brief Guide. Road and Transport Research, Vol. 16, No. 1, pp. 6972.
(37) A Guide for Using Statistics for Evidence-Based Policy. (2010). Australian Bureau of Statistics. http://www.abs.gov.au/ausstats/abs@.nsf/cat/1500.0. (accessed May 1, 2013).
(38) Garcia-Menendez, L., M. (2009). Feo-Valero, European Common Transport Policy and Short-Sea Shipping: Empirical Evidence Based on Modal Choice Models. Transport Reviews, Vol. 29, No. 2, pp. 239259.
29

Evidence-Based Decision Making for Transportation Asset Management
(39) Anable, J., B. Lane, and T. Kelay. (2006). An Evidence-Based Review of Public Attitudes to Climate Change and Transport Behaviour. The Department for Transport, London.
(40) Hutchinson, T.P., and A.J. Meier. (2004). Evidence-Based Road Safety Policy? Evidencebased Transport Policy? A Discussion of Randomized Experimentation and Meta-analysis in The Evaluation of Interventions. Australian Transport Research Forum, 27th, Adelaide, South Australia, Australia.
(41) Goller, B., and J.L. Beck, G.I. Schueller. (2012). Evidence-Based Identification of Weighting Factors in Bayesian Model Updating Using Modal Data. Journal of Engineering Mechanics, Vol. 138. No. 5, pp. 430440.
(42) Badland, H., and G. Schofield. (2005). Transport, Urban Design, and Physical Activity: An Evidence-Based Update. Transportation Research Part D: Transport and Environment, Vol. 10, No. 3, pp. 177196.
(43) Jopson A, May AD, Matthews B. (2004). Facilitating Evidence-Based Decision-Making The Development and Use of an On-Line Knowledgebase on Sustainable Land-Use and Transport. Istanbul, Turkey.
(44) Tellis, W. (1997). Applications of a Case Study Methodology. The Qualitative Report, Volume 3, Number 3. (http://www.nova.edu/ssss/QR/QR3-3/tellis2.html)
(45) Yin, R. (1994). Case study research: Design and methods (2nd ed.). Thousand Oaks, CA: Sage Publishing.
(46) Vishwanath V. Baba, Farimah HakemZadeh, (2012) "Toward a Theory of Evidence Based Decision Making", Management Decision, Vol. 50 Iss: 5, pp.832 867
(47) Campbell, D.T. and J.C. Stanley (1963) Experimental and Quasi-experimental Designs for Research. Houghton Mifin Company, Hopewell, JJ.
(48) Smith-Colin et al. (2014) Evidence-Based Decision Making for Transportation Asset Management: Enhancing the Practice with Quality Evidence and Systematic Documentation. Transportation Research Record (in press).
(49) U.S. Preventative Service Task Fore. (2008) Grade Definitions. http://www.uspreventiveservicestaskforce.org/uspstf/grades.htm (accessed, January 28, 2014)
(50) Boaz, A., and D. Ashby. (2003). Fit for Purpose? Assessing Research Quality for Evidencebased Policy and Practice. ESRC UK Centre for Evidence Based Policy and Practice, Queen Mary, University of London.
(51) Grade Working Group. (2004). Grading Quality of Evidence and Strength of Recommendations. Education and Debate. BMJ, Vol. 328.
30

Evidence-Based Decision Making for Transportation Asset Management (52) Pfeffer, J., & Sutton, R. I. (2006). Evidence-based Management. Harvard Business Review, 84(1), 62-74. (53) Transportation Research Board (2013). Transportation Asset Management Knowledge Portal, NCHRP 08-36/Task 125 [Active] < http://apps.trb.org/cmsfeed/TRBNetProjectDisplay.asp?ProjectID=3684> (accessed, January 6, 2014) (54) Netlipse Networking Knowledge (2004). < http://netlipse.eu/knowledgecenter#.UxCLZv3_yAF> (accessed, January 30, 2014) (55) Development of a Water Infrastructure Knowledge Database. (2014). Journal of Infrastructure Systems, 20(1), 1. doi:10.1061/(ASCE)IS.1943-555X.0000163
31

Evidence-Based Decision Making for Transportation Asset Management
ACKNOWLEDGEMENTS
The authors acknowledge the support of the Federal Highway Administration through the Georgia Tech University Transportation Center -- The National Center for Transportation System Productivity and Management. This effort would not have been possible without the continued dedication to process improvement demonstrated by the Georgia Department of Transportation Office of Organizational Performance Management. The framework for evidence-based transportation asset management decision making was developed, designed, and refined with input from professionals representing the following Departments of Transportation: Arizona, California, Florida, Louisiana, Minnesota, New Mexico, New York, North Carolina, Oregon, Pennsylvania, South Carolina, and Washington. Private sector input was also received from staff at Cambridge Systematics Inc., Volkert, and PBS Consulting Inc.
32

Evidence-Based Decision Making: Developing a Knowledge Base for Successful Program Outcomes in Transportation
Asset Management
Appendix
Prepared for: Georgia Department of Transportation
Point of Contact: Angela T. Alexander
Prepared by: Georgia Institute of Technology
Principal Investigator: Adjo Amekudzi-Kennedy, Ph.D.
Graduate Research Assistants: Janille Smith-Colin Jamie M. Fischer Richard Boadi
Margaret-Avis Akofio-Sowah Stephanie Amoaning-Yankson
Stefanie Brodie Amy Ingles
December 15, 2015

Evidence-Based Decision Making for Transportation Asset Management TABLE OF CONTENTS APPENDIX A: LITERATURE REVIEW REPORT .................................................................. A1 APPENDIX B: WEBINAR REPORT ..........................................................................................B1 APPENDIX C: WEBINAR PRESENTATION............................................................................C1 APPENDIX D: DETAILED WEBINAR NOTES ...................................................................... D1 APPENDIX E: ASSET MANAGEMENT CASE STUDIES ...................................................... E1
Page i

Evidence-Based Decision Making for Transportation Asset Management
APPENDIX A: LITERATURE REVIEW REPORT
A1

Evidence-Based Decision Making for Transportation Asset Management
Introduction
Background and Motivation
The Moving Ahead for Progress in the 21st Century Act (MAP-21), was signed into law by President Obama on July 6, 2012 (1). This law provides funding in the amount of $105 billion for surface transportation programs for fiscal years (FY) 2013 and 2014. MAP-21 is the first long-term highway authorization bill enacted since SAFETEA-LU in 2005. It not only provides needed funding for the national highway network, but transforms the policy and programmatic framework for investments and project delivery. A key element of this programmatic transformation is the transition to a performance and outcome-based program (1). Transportation Asset Management (TAM) is the combination of management, financial, economic, engineering, and other practices applied to physical assets with the objective of providing the required level of service in the most cost effective manner (2). Asset Management uses information from management systems in a performance-based approach for managing the network (3). According to the AASHTO Director for Engineering and Technical Services, Tony Kane, key to a successful asset management plan is an ongoing, systematic approach to improving results through evidence-based decision making, continuous organizational learning and a focus on accountability for performance through performance management. Ultimately, evidence is critical to the successful implementation of a performance-based, outcome-based program.
Formal definitions for evidence have evolved in fields such as law, medicine, and management. TAM practitioners are now faced with the challenge of defining evidence in performance improvement in the context of specific assets such as culverts, retaining walls and sidewalks, and at both the project and program levels. What specific actions have been used to extend the useful life of assets? What specific actions have resulted in improved customer satisfaction? What actions have been proven to lead to successful program outcomes? Beyond best practices, agencies may seek to develop accumulated wisdom or a body of evidence that can be used as a benchmark to assess program performance, and inform decisions to improve existing asset management programs/systems. With clear benchmarks for success, practitioners will have the evidence needed to identify actions needed to achieve high-performance, highmaturity programs.
Objectives & Organization
The objective of this study is to develop an evidence-based framework that can be used to evaluate the outcomes of financial and engineering decisions made in transportation asset management. From this framework an EBD database will be developed as a peer-reviewed and evolving tool that can be used to identify superior actions for achieving high-performance outcomes. An evidence-based approach can be extremely useful given the mandates of MAP-21 for Asset Management Plan development and performance monitoring. The goal is to provide decision makers with a means of formally linking asset performance output with asset management maturity by identifying key evidence parameters that can be used to track asset performance and influence investment in asset management program development. Such
A2

Evidence-Based Decision Making for Transportation Asset Management information will be presented in a planning tool or database that can be used to cultivate and develop an asset management improvement process that is more evidence-based, i.e. evidencebased asset management (EBAM). More explicitly, the objectives of this project are to:
1. Identify a range of practices, including best practices and lessons learned, in asset management program development.
2. Evaluate asset management program levels against key performance indicators. 3. Attempt to characterize the benefits of different actions on program outcomes in order to
isolate actions with a likelihood of higher-performance outcomes. 4. Develop an evidence-based planning tool for asset management. In the following sections of this report, the results of the literature review are presented. An introduction to evidence-based approaches is provided, including a review of evidence; commonly used evidence-based methodologies are offered, and a case study review of evidencebased approaches is presented. Finally, opportunities for the application of EBD to asset management are discussed and a working framework for applying evidence-based principles to asset management is presented.
A3

Evidence-Based Decision Making for Transportation Asset Management
Literature Review
This literature review sheds light on how evidence-based approaches are being applied in other fields and explores how they can be applied to enhance transportation asset management program development. In part, the literature review focuses on the evolution of evidence-based design within the medical field, as well as the application and use of EBD in other fields including transportation, organizational management, occupational safety, and education. An in depth discussion of evidence, evaluating it and assessing its quality is also provided. The effectiveness of EBD is highly dependent on the quality of the evidence used and the methodology employed to obtain that evidence. As such, commonly utilized evidence-based methodologies are reviewed. Finally potential applications of EBD to transportation asset management are discussed. An attempt is also made to identify the knowledge gaps and to outline the data needs for a process that applies an EBD framework to asset management.
What are Evidence-Based Databases?
Evidence-based databases are databases that are systematically developed with formal checks for the quality of evidence, and used to improve the practice in various fields. In other words, they are accepted evaluation and review systems for evidence quality. This section discusses several examples of evidence-based databases.
The Cochrane Collaboration The Cochrane Collaboration is an international network of dedicated people working together to identify the best available evidence for healthcare (4). Cochrane collaborators are from over 100 countries and represent various groups within the healthcare industry including healthcare providers, policy-makers, patients, patient advocates, and caregivers. The Cochrane Collaboration is responsible for preparing, updating and promoting access to the Cochrane Reviews, which are published online in the Cochrane Database of Systematic Reviews a portion of The Cochrane Library (5).
The Cochrane Reviews are recognized as the gold standard for reviewing, analyzing and reporting on evidence related to healthcare practice. The reviews seek to investigate the effects of treatment and rehabilitation interventions. For many, the Cochrane reviews and Library represent the most comprehensive attempt to identify a predefined, rigorous and explicit methodology for evidence review in healthcare. Narrative reviews of healthcare research have existed for many decades, but were often not systematic. Prior to the Cochrane Reviews an attempt had not been successfully made to compile all research and relevant work by topic. The extensive peer review by multiple experts or `author teams' and stakeholders is unique. The Cochrane Reviews are a searchable tool or database. The database content is available in four languages English, French, Spanish, and German. The database content has free Cochrane Summaries, other paid content, and a glossary of terms is provided to assist users with site navigation (6). In addition, the Cochrane Collaboration hosts conferences, training sessions, and educational events to further the education of healthcare stakeholders.
A4

Evidence-Based Decision Making for Transportation Asset Management
GRADE Another commonly accepted evaluation and review system for evidence quality within healthcare has been developed by the GRADE working group, the AGREE Collaboration, and the University of Oxford Center for Evidenced Based Medicine. The GRADE working group has developed a common, sensible and transparent approach to grading the quality of evidence and strength of recommendations (7). GRADE seeks to differentiate between strong and weak recommendations with respect to alternative strategies for outcomes. GRADE has strong representation from a variety of groups already in the business of evaluating evidence, and evaluates recommendations based on a 20-point guideline. GRADE has also developed an electronic tool/software to assist with evaluation.
The methodologies used in the Cochrane Review process have emerged in various fields. This work intends to adapt the process used to systematically identify, detail, and report on evidence in healthcare, and apply this methodology to transportation asset management. The ongoing work of the Cochrane Collaboration, GRADE, and similar organizations will serve as templates for the successful development of an evidence-based database for transportation asset management. This review continues with a discussion of evidence-based approaches.
What is Evidence-Based Design?
Evidence-based design aims to evaluate and apply scientific evidence to a problem in order to arrive at the best possible solution (Rousseau, Denise, 2012). Evidence-based design as a practice has its roots in the healthcare industry. In healthcare evidence-based design, evidencebased practice (EBP), and evidence-based medicine (EBM) have been used interchangeably to describe the process by which the best available evidence is used to maximize resources and ensure optimal patient outcome through methods including improved building design, practitioner training, and service delivery (8).
One of the most commonly used definitions of evidence-based practice was developed by Dr. Sackett in 1996. He defines evidence-based practice as a conscientious, explicit, and judicious use of current best evidence in making decisions about the care of the individual patient through integrating clinical expertise with the best available external clinical evidence from systematic research. The concept of EBD first gained traction in 2003 with a scholarly article highlighting a collaborative approach to facility design based on research and projects evaluations (9). The predominant approach to evidence- based analysis in healthcare is randomized control testing (see section 2.3.3).
One of the appeals of EBD in its application to asset management is that it is a dynamic concept. The knowledge acquired at the time clinicians first receive their training is augmented by their subsequent experience as practitioners and refreshed through knowledge transfer efforts that constantly and reliably convey the latest research evidence to practitioners (10). These characteristics have some similarities to transportation asset management programs where funding, stakeholders, staff, and maintenance goals are often dynamic.
A successful EBD program requires the input of all stakeholders. The steps involved in an evidence-based process are generally as follows:
A5

Evidence-Based Decision Making for Transportation Asset Management

Problem definition and search for wanted sources of information Review of relevant national and international literature (databases are key to this step) Systematic review of existing intervention/strategies Surveys to help identify themes and difficult-to-solve problems encountered in practice Critical evaluation of the information Application of information to the problem Efficacy evaluation of this application to the problem

These steps can be applied to various fields with respect to program implementation, the assessment of intervention outcomes, and strategy evaluation. Evidence-based design is therefore a decision-making support tool. In many cases, the application of EBD to other fields is at its beginning stages. Researchers are still actively reviewing critical literature, gathering tools needed to conduct analysis, and classifying the data needed to support future EBD frameworks.
How are Evidence-Based Approaches Applied?

Though historically rooted in the medical field, evidence-based approaches now have broad application as a resource allocation and maximization strategy within education, information technology, policymaking, organizational management, and even transportation. A variety of evidence-based approaches are currently used in practice with various applications to various fields. Table 1 lists the common variations in evidence-based approaches and their practical applications.
Table 1: Evidence-based Approaches in Practice

Evidence-based Approach

Professional Applications

Evidence-based planning Evidence-based policy; Evidence-based programming Evidence-based medicine; Evidence-based practice
Evidence-based design Evidence-based management; Evidence-based decision making

Social sciences, Education
International Development
Medicine, Healthcare, Nursing Engineering, Information technology,
Medicine, Nursing, Healthcare Organizational management,
Policy and Planning

A6

Evidence-Based Decision Making for Transportation Asset Management
Within each field, there is ongoing discussion around the appropriateness of the evidence-based approach. For example, practitioners at the state level continue to evaluate the efficacy of evidence-based practice versus evidence-based policy (11). Each area of professional practice values an approach to decision making informed by evidence and research, but there are important differences in the way evidence is gathered, interpreted and implemented (9). Based on a review of the literature, the application of EBD methodologies to healthcare, organizational management, and policy-making present great promise for applications to the transportation field and asset management.
Transportation
Evidence-based approaches and specifically evidence-based design have been applied in transportation as a means to predict performance outcomes for traffic safety treatments. In 2003, Retting et al. used EBD to assess engineering modifications to the built environment that reduce the risk of pedestrian injuries. The application of EBD to the safety field is a natural development. Though this article cites an EBD approach, little attempt is made to define evidence within the context of traffic safety treatments, or to clearly define a methodology for assessing the quality of evidence. These elements are critical to an EBD approach.
EBD principles have also been applied to the discipline of urban design/urban form and physical activity. This fact comes as no surprise as there are clear parallels between facility design and the design of the urban environment. In 2005, Badland and Schofield attempted to demonstrate the linkage between traffic calming, neighborhood design and physical activity using descriptive statistics to assess the relationship between urban form and motion. Most recently, the University of Oregon in collaboration with the Oregon Department of Transportation issued a request for proposals to fund a project related to complete street design and EBD.
To date, the application of EBD to transportation, has focused mainly on design be it safety, urban form, or complete streets. The limitations of an evidence-based approach are seen in the transferability of results. Sampling and measurement inconsistencies between studies often make it difficult to perform comparisons and relate results across processes. The application of EBD to asset management therefore constitutes a shift away from a design application to a programmatic/planning application of EBD in transportation.
Organizational Management
From medicine to education, evidence-based approaches aim to evaluate and apply scientific evidence to a problem in order to arrive at the best possible solution. Thus, using scientific knowledge to inform the judgment of managers and the process of decision making in organizations, Evidence-based Management (EBMgt) is the science-informed practice of management. Evidence-based management is practiced in the business world; its focus is on effective oversight of staff and resources (10). The accepted definition of evidence-based management was developed by Collins et al. in 2008: EBMgt is defined as making better decisions by integrating managerial expertise with the deliberate and prudent use of best evidence in making decisions while taking into account the perspective of those who might be affected by them (12). As with other evidence-based approaches the goal for EBMgt is an improved connection between practice and theory, improved managerial practices, and improved
A7

Evidence-Based Decision Making for Transportation Asset Management
decision making. The applications of EBMgt are relevant because of its applications to a knowledge economy in which there is an increasing awareness of the importance of knowledge, skills, and creativity for business performance (12). The benefits of EBMgt include:
i. Innovative process and products
ii. Enhanced performance
iii. Fact-based evidence to provide legal justification for actions
Like many other fledgling schools of thought, EBMgt has its limitations. First, EBMgt is not a rigid process but a family of approaches (13). EBMgt recognizes multiple stakeholders including practitioners, scholars and educators. Lastly, EBMgt requires systematic reviews and an assessment of infrastructure. As with other EB approaches, a major obstacle facing the EBMgt community is a lack of literature; literature on the topic is critical for the development of an evidence base. Moving forward EBMgt researchers need to work with practitioners to establish this evidence-base.
Policy Planning and Development
Evidence-based planning, evidencebased policy, evidence-based decision making, and evidence-based programming are some of the terms used to describe the application of the evidence-based approach to programming and development. Increased pressure for greater efficiency, effectiveness, and accountability in the delivery of government services, constrained budgets, and limited resources have fueled the increased focus of EB approaches. To date much of the focus on these approaches has centered on the delivery of social programs such as those focused on teen pregnancy, substance abuse, and family violence. However, evidence-based planning and policy has also been used in International Development and in other professional applications. Evidence-based policy, planning, and decision making potentially have greater applications for asset management program development than evidence-based design has.
In general, the literature investigating the use of evidence in policy-making is limited. The limitations of literature are even greater within the context of state agencies that develop and implement policies that directly impact the daily lives of citizens (11). As with programs implemented in healthcare and medicine, programs that have been found to be effective in the social sciences have been subjected to rigorous evaluations (14). These social programs have been evaluated rigorously through experimental or quasi-experimental studies. Experimental design typically involves randomized control testing (RCT). During RCT participants are randomly assigned to either the treatment group, which participates in the program, or the control group, which does not (14). In instances where RCT is infeasible, participants are compared to groups that are in many ways similar to the group of participants. Quasi-experimental approaches are not as rigorous because they do not randomly assign participants to groups. However, there are simply instances where random assignment is not feasible or morally appropriate.
A8

Evidence-Based Decision Making for Transportation Asset Management
Education In 2012, Slocum and team investigated three complementary approaches to identify the best available evidence that can be derived from educational research. These include (a) conducting systematic reviews to identify empirically-supported treatments; (b) using methods other than systematic reviews to summarize evidence, and (c) considering research on "treatments" that are not multi-component packages (15). Researchers recognized that none of these approaches on its own could provide an optimal solution, but rather all three approaches working together could increase the breadth of knowledge relevant to educational decision making. Researchers focused on the concept of best available evidence in education attempting to define best available in education in terms of relevance, strength, and methodological rigor. A hierarchy of evidence based on an interventions impact on educational outcomes has been developed, (16). Definitions of strong, moderate and minimal evidence are given below:
Strong evidence - Consistent evidence that demonstrates that the recommended practice causes improvements and that the effects can be generalized to a range of students and settings.
Moderate evidence - Evidence provides either clear demonstration that the practice causes improvement, or strong demonstration of generalization, but not both.
Minimal evidence - The recommendation is based on strong findings or theories in related areas, or other indirect sources of support. However, the panel cannot identify specific research that supports the recommendation and achieves the level of moderate evidence.
The hierarchy of evidence was applied to educational practice guides as a means to evaluate the full range of evidence and make recommendations. Once recommendations were formulated, the strength of the evidence supporting each recommendation was rated as shown in Table 2.
A9

Evidence-Based Decision Making for Transportation Asset Management
Table 2: Using Evidence in Education
Evidence
What is Evidence? Evidence is most widely used in the legal and medical fields to establish a fact base for information gathered. In science, evidence represents the degree of proof that can be gained from practical experience and literature. In science, newly found evidence must be systematically reviewed and evaluated before the evidence can be accepted. Evidence based approaches have adopted a similar methodology. In the case of program development for asset management, evidence may be defined as any outcome that can be measured or tracked at a point or over time, following the implementation of an intervention or action where the performance of the program can be related to the action taken. The methods used for evidence assessment vary widely across fields. In fact, it is safe to say that no commonly accepted method of evidence assessments currently exists. The type of evidence needed to build an evidence-based approach depends heavily on the questions to be answered. Another key issue faced when determining evidence, is the question of who participates in evaluating evidence. The question of what evidence is varies widely based on the perspective from which it is being evaluated be it the practitioner, the researcher or the client. Within a field of practice, it is critical that all stakeholders participate in determining the acceptable characteristics of evidence, and frameworks for assessing evidence.
A10

Evidence-Based Decision Making for Transportation Asset Management
Evaluating Evidence Evidence-based approaches are based on the "best available evidence." However, best available is often time dependent. In medicine and law, centuries of evaluation and assessment have allowed for the development of widely accepted assessment frameworks. In other fields where evidence-based approaches have been adopted, there is some discussion as to whether the best available evidence is simply a function of what evidence is available at a given point in time. The challenge faced by decision makers is deciding what evidence is useful evidence. Several attempts have been made to establish criteria for judging evidence. What constitutes credible evidence? Evidence can be judged based on its strength or amount, and its methodological rigor. Different levels of evidence may be defined as follows:
Strong consistently performing evidence that can be translated to a range of DOTs and settings.
Moderate offers clear demonstration of improvement or strong proof of generalization but not both.
Minimal specific research in support of the evidence and its impacts cannot be identified; based on strong theories and findings from other sources.
Methodologies for evaluating evidence range from the 47 standards for efficacy, effectiveness, and dissemination of programs (17) to the 7 levels of quality evidence (18). In 2005 Shaxson presented five levels or components for robustness: credibility, generalizability, reliability, objectivity and rootedness. The indices of effectiveness and efficiency, effectiveness, appropriateness and feasibility are also often used to define evidence (19). Qualitative attempts to evaluate evidence can be assessed using the following grading questions or criteria. Table 3 presents a commonly used framework for evaluating evidence.
A11

Evidence-Based Decision Making for Transportation Asset Management
Table 3: Evidence Quality Assessment Framework
Attempts have been made to define evidence based on the methodology used to obtain it rather than through qualitative or descriptive terms. It is common practice in evidence-based medicine and evidence-based practice to characterize evidence by a hierarchy of quality. Methodologies for such an approach range from a typical scale, to a graphical depiction. One system of evidence hierarchy developed for EBM and EBP was developed Dr. Stetler (2002), and is based on eight levels of evidence drafted from quantitative and qualitative studies as well as nonscientific pursuits (19) as shown in Table 4. Levels II and I represent the most rigorous form of evidence often based on controlled randomized trial. The "A-D" below each level of evidence represents the four grades that can be assigned to a piece of evidence based on the methodological elements in each tier of scientific strength (19).
A12

Evidence-Based Decision Making for Transportation Asset Management
Table 4: Level of External Healthcare Design Evidence
As depicted above, researchers believe that there is a relationship between evidence grade or quality and the type of systematic review used to collect and analyze the data. This push began with a characterization of this relationship in medicine, where hierarchical grades were introduced as a measure of evidence quality. These grades are affected by various factors and their quality grows as bias diminishes (20). While conducting research on occupational safety and evidence, Annette & Frank defined the relationship between evidence grade and bias in terms of the systematic approach of evaluation as shown below in Table 5.
A13

Evidence-Based Decision Making for Transportation Asset Management
Table 5: Adaptation of the hierarchy of evidence to occupational safety

Source: (20)
What are Systematic Reviews in an Evidence-Based Approach?

A systematic approach is replicable, scientific, and transparent. Systematic reviews rigorously address a clearly defined question usually from a practice or policy standpoint (13). The approach to a systematic review can vary depending on the field of practice. The methodology employed in medicine differs from that which is employed in the social sciences. In interdisciplinary practice, the methodology may again differ. According to Briner, systematic reviews in medicine are often structured according to the PICO approach, while in the social sciences the CIMO approach has been adopted as shown in Table 6.

Table 6: Approaches to Systematic Reviews

Medical Science

P

Group for which evidence is

Patient of required

problem

I Intervention The effects of what event, action,

or activity is being studied?

C

What is the alternative to the

Comparison intervention (e.g.,

placebo/different intervention)?

O

What are the effects of the

Outcomes intervention?

C Context I Intervention
M Mechanisms
O Outcomes

Social Science Which individuals, relationships, institutional settings, or wider systems are being studied? The effects of what event, action, or activity is being studied? What are the mechanisms that explain the relationship between interventions and outcomes? Under what circumstances are these mechanisms activated or not activated? What are the effects of the intervention? How will the outcomes be measured? What are the intended or unintended effects?

A14

Evidence-Based Decision Making for Transportation Asset Management
Evidence-based approaches are intended to eliminate false conclusions based on treatment interventions. A scientific claim constitutes the highest-quality evidence if it is not only based on the opinions of a number of experts, but also verified by the same results from high-quality studies and further by international publications and review (20). These high quality studies are referred to as systematic reviews. Systematic reviews are typically quantitative, but may also be qualitative. In certain instances the systematic review may be a hybrid approach. According to the White House Office of Management and Budget, the most prevalent approach to a quantitative review is achieved through the use of randomized controlled trials (RCTs) (21); qualitative reviews are conducted using non-experimental studies, and quasiexperimental studies are completed to achieve a mixed method or hybrid review of evidence. The approach to these common systematic reviews is described below.
Randomized Controlled Trials (RCT)
An RCT is a study that measures the effect of an intervention by randomly assigning individuals or other dependent variables or units to an intervention group, which receives the intervention, and into a control group, which does not. At some point following the intervention, measurements are taken to establish the difference between the control group and the intervention group, because the control group simulates what would have happened were there no intervention. The difference in outcomes between the groups demonstrates the "overall outcome" or impact one would expect for the intervention more generally.
Quasi Experimental Studies
Like randomized controlled trials, these evaluations assess the differences that result from a specific intervention and the result that would have occurred without the intervention. For example, for a welfare program, the comparison may be between an intervention group that receives the benefits of the program and a comparison group that does not. In this case the control activity (comparison group) is not randomly assigned. Quasi-experimental evaluations are often called "comparison group studies." Under certain circumstances, well-matched comparison group studies can approach the rigor of randomized controlled trials and should be considered if random assignment is not feasible or appropriate. However, use of comparison group studies does increase the risk of misleading results because of the difficulty in eliminating bias in the selection of the control group.
Non-experimental Studies
These evaluations examine only the intervention subject (e.g., group)--the subject (group) receiving the program intervention (e.g., for groups, the intervention may be benefits); there is no comparison subject (group). A common example of this type of evaluation, the "pre-post study," examines only an intervention group (no separate comparison group is selected), with outcomes compared both before and after program benefits are received. "Longitudinal studies" are another example: these studies examine changes over time and relate those changes back to the original condition of the intervention group. Other examples of non-experimental tools and methods include correlation analyses, surveys, questionnaires, participant observation studies, implementation studies, peer reviews, and case studies. These evaluations often lack rigor and may lead to false conclusions if used to measure program effectiveness, and therefore, should be
A15

Evidence-Based Decision Making for Transportation Asset Management
used in limited situations and only when necessary. Such methods may have use for examining how or why a program is effective, or for providing information that is useful for program management.
Pre-post Study
A pre-post study examines whether participants in an intervention improve or become worse off during the course of the intervention, and then attributes any such improvement or deterioration to the intervention. Benefits for example, may be examined before and after an intervention. The problem with this type of study is that, without reference to a randomly-assigned control group, it cannot answer whether the participants' improvement or deterioration would have occurred anyway, even without the intervention. This often leads to erroneous conclusions about the effectiveness of the intervention (21).
Some evaluation literature refer to "pre-post" studies as forms of quasi-experimental evaluation, because a reflexive comparison is made between the group receiving the program intervention and a control group composed of the same group before the intervention. Other sources do not consider such studies to be quasi- experimental.
Longitudinal Studies
Another commonly used experimental design type is the longitudinal study. Similar to the prepost study this design type only examines the subject group receiving the intervention. There is no comparison group against which intervention benefits are compared. Longitudinal studies evaluate changes to the subject group over time and compare these changes back to the "original" condition.
These evaluations often lack rigor and may lead to false conclusions if used to measure program effectiveness, and therefore, should be used in limited situations and only when necessary. Such methods may have use for examining how or why a program is effective, or for providing information that is useful for program management (21).
Applications of EBD: Case Studies
Evidence for Improving Pedestrian Safety
In 2003, a team of engineers applied an evidence-based approach to the complex issue of pedestrian crashes. Researchers provided a critical review of engineering interventions aimed at reducing the risk of pedestrian-motor vehicle crashes. As is required by EBD methodologies, researchers first conducted an extensive review of studies employing engineering countermeasures to improve pedestrian conditions; the Transportation Research Information Services Database was the source for this information. Countermeasures were then classified into 3 categories: separation of vehicles from pedestrians, speed control, and measures that increase the visibility of pedestrians. Study results were then reviewed to determine the
A16

Evidence-Based Decision Making for Transportation Asset Management
countermeasures with the greatest potential for crash prevention, or improved outcomes for pedestrians (22). In this analysis, care was taken to select studies with adequate methodological design. Researchers clearly indicated the intervention, outcome measure, study design, and results obtained from each study. Additionally, study location was noted. The majority of the studies included in this research used before-and-after tests either with or without control to identify evidence, representing Level III evidence per Stetler's hierarchy of evidence. A positive outcome/result was determined by a reduction in crashes. It was determined that modifications to the built environment can significantly reduce pedestrian crashes. The key in moving forward with transportation asset management will be to define the positive outcome/result. Will outcomes be defined in terms of asset life, resource input, reduction in maintenance activity, or the serviceability of the system to users? Will positive outcomes differ based on the asset being considered? Healing Environment: Physical Evidence Healthcare facilities (HCF) are places where patients with health conditions go for treatment, which is provided by specialists and other care professionals. In recent years, there has been a growing interest in the role of technology and the built environment as part of the holistic treatment of patients. Discussions about the importance of the built environment for the patient's health and well-being, and the provision and support of healthcare have become increasingly common. One study presents evidence-based design as the theoretical concept for what are called healing environments. Healing environments can be considered as "smart investments" because they save money, increase staff efficiency, and reduce the hospital stay of the patient by making the stay less stressful (23). In this study, the Cochrane Methodology was used to survey the scientific research on evidence-based healthcare design from the perspective and needs of end-users. The group of end-users is defined as patient, family (PF) and staff in this review. Furthermore, this review distinguishes between empirical data and evidence-based data concerning the patient and staff health outcomes in hospital settings (23). Table 7 lists evidence gathered for the impacts of healing environments.
A17

Evidence-Based Decision Making for Transportation Asset Management

Table 7: Evidence of a Healing Environment

Topics

Subtopics

Patient, Family (PF)

No Errors
Safety and Security

Falls Infection

Enhancing control Privacy
Comfort

Indoor Quality
Comfort Materials Art View

Visual comfort Acoustic
Comfort
Orientation

References and level of evidence
[16]-2, [17]-2, [18]3, [19]-3 [20]-2, [21]-2, [22]2, [23]-2 [24]-2, [25]-2, [26]4, [27]-2, [28]-2, [29]-1, [30]-3, [31]2 [32]-2, [33]-3, [34]3 [17]-2, [36]-1
[37]-3, [38]-2, [39]2 [7]-4 (review), [86]-1 [27]-2, [40]-2 [8]-4 (review), [90]-4 (review) [10]-4, [42]-2, [44]4, [45]-3; [46]-4, [47]-4, [48]-3, [49]2 [50]-3, [51]-3
[55]-1, [56]-1, [57]3, [58]-4 (review), [59]-2, [60]-1, [61]-2, [62]2, [63]-3, [64]-2, [65]-3 [52]-3, [53]-3, [63]3, [67]-4, [68]-1, [74]-2

Total number of references 4 4 8
3 2 3 2 2 2 8
2
11
6

Lowest level of evidence 2 2 1
2 1 2 1 2 4 2
3
1
2

Highest level of evidence 3 2 4
3 2 3 4 2 4 4
3
4
4

Level of evidence [9] 1 = poor (expert opinion); 2 = fair (case series, case reports);

3 = good (cohort studies, case control studies); 4 = excellent (randomized controlled

trials, systematic reviews).

A18

Evidence-Based Decision Making for Transportation Asset Management Evidence Based Management Classifying Evidence Evidence-based management (EBMgt) is the systematic use of best available evidence to improve management practice (10). There have been increased calls for the use of evidencebased management. Much of this push is based on literature stemming from the medical and healthcare fields that promote the use of "best available evidence" in medical decision making. EBMgt encourages managers to make decisions based on scientific evidence as well as local business knowledge. Proponents of EBMgt advocate for the use of evidence in management. Skeptics contend that the diversity amongst organizational structure, business culture, and management styles, does not lend itself to a best management approach that is based on best available evidence. Researchers conducted a review of literature dating back to 2008, to document existing evidence related to EBMgt. The focus of this research was the quantity, quality, and signs of managerial improvements from an evidence-based approach. Researchers classified evidence using Stetler's hierarchy of evidence, translating medical research methodologies into management research methodologies as shown in Table 8. This work provides a template by which asset management practitioners can attempt to define Stetler's hierarchy of evidence in terms of asset management practice.
A19

Evidence-Based Decision Making for Transportation Asset Management
Table 8: Levels of Evidence for Evidence-Based Management
A20

Evidence-Based Decision Making for Transportation Asset Management
Applying an Evidence-Based Approach in TAM Development
Applying an evidence-based approach in transportation asset management will involve the following at the minimum:
1. Defining what constitutes "evidence" of positive outcome when TAM program or system interventions are made (at various levels of decision making, i.e., strategic, network, project, etc.). This will involve some work to determine how to define, isolate and attribute a "positive result" when TAM interventions are made, given that different TAM systems are at different levels of maturity and the outcomes of similar interventions may be different when considered incrementally for different baseline systems or programs. It may also involve considering combined sets of interventions for programs or systems with similar characteristics, i.e., similar maturity levels.
2. Identifying methodologies or approaches for assessing evidence and its quality after TAM interventions have occurred. This may involve the development of a hierarchy of approaches associated with the relative quality of the evidence that one can extract or determine after TAM interventions; higher quality evidence will be more objectively determined, and the results will be more transferable.
3. Developing guidance to support the isolation and organization of evidence (residing as cumulative wisdom within the personnel or systems of various transportation agencies) on TAM interventions in such a manner that it can be used selectively and strategically by transportation agency personnel to influence their TAM program or system development. This effort may involve working with selected agencies that can isolate interventions that are clearly considered to have made a positive and important impact, and working with them to identify, assess and catalogue this evidence.
Implementation Changes
Lack of agreement on what constitutes evidence across policy makers and administrators, and other stakeholders is expected to create difficulties for the implementation of evidence-based approaches in transportation asset management. Historically, programs have been labeled as "best" or "evidence-based" when the quality of information has not been labeled as credible (11). Across the United States and around the world there has been a growing emphasis on evidencebased policy and planning (24). In spite of this increased demand for such programs and the increased availability of data, decision makers continue to make decisions based on opinion, and not facts. Shifting organizational perception relative to the value of evidence therefore presents a barrier to successful integration of an evidence-based approach. This shift in organizational thinking is often rooted in successful knowledge transfer.
Knowledge Transfer
In engineering, as in medicine, a barrier often exists between the transfer of knowledge from researchers to practitioners. In both fields, for EBD to be successful it is critical that researchers translate findings into a format that is meaningful to practitioners. The literature on evidencebased practice in medicine has become increasingly focused on efforts to transfer research
A21

Evidence-Based Decision Making for Transportation Asset Management
findings to the practice setting. This area of research is referred to as knowledge translation (KT) (11). KT was born predominantly out of persistent variation in clinical practice. Variations arise in the process of implementing research evidence and as a consequence create barriers for the practitioners, users, and other stakeholders regarding the applicability of evidence within a new context (11). KT research in health care focuses on understanding how complex interventions, predicated on research evidence and involving multiple providers, recipients, and organizations, can be implemented (Jennings et al, 2009). Within the context of transportation asset management, KT represents opportunities for impacting organizational change, and spurring cultural shifts regarding the efficacy of asset management.
References
(1) Moving Ahead for Progress in the 21st Century Act (MAP-21) A Summary of Highway Provisions (July, 2012). Federal Highway Administration, Office of Policy and Governmental Affairs.
(2) AASHTO transportation asset management guide [electronic resource]: a focus on implementation. (2011). Washington, DC: U.S. Dept. of Transportation, Federal Highway Administration, [2011].
(3) Management systems: driving performance [electronic resource]: a glance at data-driven decision making practices. (2009). Washington, DC: Federal Highway Administration, Office of Asset Management, [2009].
(4) About Us [http://www.cochrane.org/about-us]. Oxford, UK: The Cochrane Collaboration; c2004-2006 [cited 2012.10.11]. Available from: www.cochrane.org.
(5) Cochrane Reviews [http://www.cochrane.org/cochrane-reviews]. Oxford, UK: The Cochrane Collaboration; c2004-2006 [cited 2012.10.11]. Available from: www.cochrane.org.
(6) About The Cochrane Library [http://www.cochrane.org/cochrane-reviews/about-cochranelibrary]. Oxford, UK: The Cochrane Collaboration; c2004-2006 [cited 2012.10.11]. Available from: www.cochrane.org.
(7) Introduction [http://www.gradeworkinggroup.org/intro.htm]. The Grade Working Group; c2005-2010 [cited 2012.10.11]. Available from: www.gradeworkinggroup.org.
(8) Bones, E., Barella, E., and Amekudzi, A. (2012). Evidence-Based Design Approaches in Transportation Decision Making. Transportation Research Part A: Policy and Practice. Forthcoming.
(9) Viets, E. (2009). Lessons from Evidence-Based Medicine: What Healthcare Designers Can Learn From the Medical Field. Health Environments Research & Design Journal (HERD), 2(2), 73.
A22

Evidence-Based Decision Making for Transportation Asset Management
(10) Reay, T., Berta, W., & Kohn, M. (2009). What's the Evidence on Evidence-Based Management? Academy Of Management Perspectives, 23(4), 5-18. doi:10.5465/AMP.2009.4559013.
(11) Jennings, E. T., & Hall, J. L. (2012). Evidence-Based Practice and the Use of Information in State Agency Decision Making. Journal of Public Administration Research & Theory, 22(2), 245-266. (12) Rowley, J. (2012). Evidence-based marketing. International Journal of Market Research, 54(4), 521-541. doi:10.2501/IJMR-54-4-521-541.
(13) Briner, R. B., Denyer, D., & Rousseau, D. M. (2009). Evidence-Based Management: Concept Cleanup Time? Academy Of Management Perspectives, 23(4), 19-32. doi:10.5465/AMP.2009.45590138.
(14) Cooney, S., Huser, M., & Small, S. (2007). Evidence-Based Programs: An Overview. Issue #6. What Works.
(15) Slocum, T. A., Spencer, T. D., & Detrich, R. (2012). Best Available Evidence: Three Complementary Approaches. Education and Treatment of Children, 35(2), 153-181.
(16) Shanahan, T., Callison, K., Carriere, C., Duke, N. K., Pearson, P., Schatschneider, C., & ... What Works Clearinghouse, (. (2010). Improving Reading Comprehension in Kindergarten through 3rd Grade: IES Practice Guide. NCEE 2010-4038. What Works Clearinghouse.
(17) Flay, B. R., Biglan, A., Boruch, R. F., Castro, F., Gottfredson, D., Kellam, S., & ... Ji, P. (2005). Standards of Evidence: Criteria for Efficacy, Effectiveness and Dissemination. Prevention Science, 6(3), 151-175. doi:10.1007/s11121-005-5553-y.
(18) McCall, R. B. (2009). Evidence-Based Programming in the Context of Practice and Policy. Social Policy Report, 23(3), 1.
(19) Pati, D. (2011). A Framework for Evaluating Evidence in Evidence-Based Design. Health Environments Research & Design Journal (HERD), 4(3), 50-71.
(20) Annette, N., & Frank, B. (n.d.). Examples of evidence-based approaches in accident prevention. Safety Science, 48 (Papers selected from the fourth international conference Working on Safety (WOS2008) 'Prevention of Occupational Accidents in a Changing Work Environment', September 30th - October 3rd, 2008, Crete, Greece), 1044-1049. doi:10.1016/j.ssci.2010.02.009
(21) What Constitutes Strong Evidence of Program Effectiveness (2004). Whitehouse Office of Management and Budget.
(22) Retting RA; Ferguson SA; McCartt, A. (2003). A review of evidence-based traffic engineering measures designed to reduce pedestrian-motor vehicle crashes. American Journal of Public Health, 93(9), 1456-1463.
A23

Evidence-Based Decision Making for Transportation Asset Management (23) E.R.C.M., H., E., M., J. van, H., & H.S.M., K. (n.d.). Healing environment: A review of the impact of physical environmental factors on users. Building and Environment, 5870-5880. doi:10.1016/j.buildenv.2012.06.016. (24) Maynard, R. A. (2006). Evidence-Based Decision Making: What Will it Take for the Decision Makers to Care? Journal of Policy Analysis and Management, 25(2), 249-265. doi:10.1002/pam.20169.
A24

Evidence-Based Decision Making for Transportation Asset Management
APPENDIX B: WEBINAR REPORT
B1

Evidence-Based Decision Making for Transportation Asset Management
Executive Summary
The Infrastructure Research Group (IRG) at Georgia Tech conducted a series of panels on the use of evidence in Transportation Asset Management (TAM). These panels are part of a broader research project sponsored by GDOT to develop an evidence-based framework for TAM decision making. Participants were invited to participate based on their expertise. The panels included academics, practitioners and government officials from 10 state DOTs around the country -- known for their TAM effectiveness.
Evidence-based approaches, such as evidence-based design and evidence-based medicine, have been applied for several years in the healthcare industry with notable cost savings (for example, a 35% cost savings over 12 months in lung cancer patients). There have been limited applications in Transportation particularly in the area of safety... The IRG has spent the past few months developing a framework for applying evidence-based approaches in TAM. The advisory panels were convened to help refine the evidence-based framework.
Panelists were asked to help do two things. First to identify some TAM investments at the program-level and project-level that, in their experience or observation, had resulted in quantifiable evidence of success (positive outcomes) or failure (negative outcomes). .(An example of a program-level investment could be the development of a performance reporting system, whereas an example of an investment at the project-level could be the selection of a new type of retaining wall that has resulted in quantifiable savings.) .Secondly, panelists were asked to critique the draft evidence-based framework. The following is an account of the key themes identified by panelists. A transcript of the conversations is provided, and potential case studies for future investigation are presented.
Introduction
Overview
The Infrastructure Research Group at Georgia Tech is working on an evidence-based framework for transportation asset management (TAM) that can be used to formally develop and classify evidence about the impacts of investments in TAM programs. The objectives are to provide examples of project-level and program-level initiatives that demonstrate the value of good quality evidence in TAM, how to obtain evidence, and use it effectively and cost-effectively. This study follows the successful completion of the Asset Management Peer Exchange held in April 2012 and hosted by the Federal Highway Administration (FHWA) and the American Association of State Highway and Transportation Officials (AASHTO). The peer exchange focused on two themes relevant to this study 1) managing asset management performance - being able to monitor if you are meeting your goals is a fundamental part of good asset management, and 2) asset management tools and techniques having good tools and techniques to support your asset management program is a key enabler to success.
The primary goal of an evidence-based approach is to develop a knowledge base for practitioners to use in program and project management. This study elaborates on the findings of the asset
B2

Evidence-Based Decision Making for Transportation Asset Management
management implementation peer exchange by focusing specifically on "evidence" and its use in asset management. The study seeks to address the following key issues:
How are DOTs and other transportation agencies addressing evidence formally in their decision making processes?
What are the costs of increasing the quality of evidence associated with various types of decision support information? What are the benefits?
In what cases is additional evidence or additional quality of evidence justified? What data are used for improving the quality of evidence in decision making? What leadership roles within agencies are necessary for improving the quality of
evidence in decision making? What are obstacles to improved quality of evidence and how can they be surmounted?
The IRG has spent the past 3 months developing a framework for applying evidence-based approaches in TAM. A key element of the evidence-based approach is practitioner input towards the formulation and identification of evidence. Based on their expertise in TAM, a panel of practitioners was invited to participate in the TAM Evidence Exchange. The panel included practitioners, academics and government officials. The advisory panel was convened to help refine the framework.
This report provides the findings of Phase II: TAM Evidence Exchange of the EBDM Framework Project sponsored by the Georgia Department of Transportation. The following provides an overview of the objectives and organization of the practitioner panels, and a detailed summary of the major themes emerging from the discussions held with practitioners. A major contribution of this phase of the study is the case studies identified by practitioners for future investigation. This report serves as a companion guide to Phase I Literature Review.
Objectives & Organization
The TAM Evidence Exchange practitioner panel consisted of three online webinars/ videoconferences hosted between May 2013 and July 2013 (May, 13; May 21; and July 16). IRG team leader Dr. Adjo Amekudzi, and team member Janille Smith-Colin facilitated the webinars. The TAM Peer Exchange Agenda is shown in Figure 1.
B3

Evidence-Based Decision Making for Transportation Asset Management
TAM Evidence Exchange Webinar 3 Tentative Webinar Agenda
1) IRG team presentation (10 - 15 min) a. Project goals and objectives b. Introduction to the evidence-based approach c. Potential applications to asset management
2) Roundtable discussion/ Q&A (20 min) a. What does quality evidence mean in terms of asset management decision making? b. In what cases is having additional evidence, or improved quality of evidence justified? c. What are the costs/ benefits of increasing the quality of evidence? d. Please provide feedback on the proposed evidence-based asset management framework, and evidence hierarchy.
3) Practitioner led evidence exchange (6 min/practitioner) a. Each practitioner will have approximately 6 min to present TAM investments (project level/ program-level investments, or agency-wide initiatives) that have produced evidence of benefit, or opportunities for improvement
Figure 1: TAM Evidence Exchange Agenda
Each webinar consisted of a formal presentation delivered by the IRG research team (see Appendix) followed by a Q&A and discussion session involving the practitioner panel. The formal presentation focused on introducing panelists to the evidence-based approach, using evidence-based design as the example. Practitioners were also introduced to the goals and objectives of the research study, and presented with the draft evidence hierarchy for TAM developed by the IRG team and shown in Table 1. The intent was to have practitioners provide
B4

Evidence-Based Decision Making for Transportation Asset Management

examples of transportation evidence that would assist in the further development of the TAM evidence hierarchy.

Table 9: Draft TAM Evidence Exchange - Evidence Hierarchy

Evidence Grade

TAM Maturity

Type of Study

Description

Transportation Evidence*

A

Best practice

Controlled intervention

Several high

*

studies

quality studies

(high quality)

with consistent

results, one large

high quality multi-

intervention study

B

Proficient

Controlled intervention

One low quality

*

studies

study, few agency

(low quality)

trials

C

Structured

Cross-section studies,

Before and after

*

correlation studies,

study of

retrospective studies

interventions;

study of multiple

variables, study

over time

D

Awakening

Case studies, expert surveys, Expert opinion via

*

best practice

interviews; in

depth study with

limited control

environment

E

Initial

Observational Studies

Careful

*

observation

without

intervention, no

control

environment

*TAM Evidence Exchange used to collect transportation examples for development of evidence hierarchy

The Q&A session served as an open forum for all practitioners to discuss their thoughts about the presentation, and to share successes and lessons learned through TAM investments in practice. Panelists were asked to help do two things during the Q&A and discussion session: first, to identify some TAM investments at the program-level and project-level that, in their experience or observation, had resulted in quantifiable evidence of success (positive outcomes) or failure (negative outcomes)...An example of a program-level investment could be the development of a performance reporting system, whereas an example of an investment at the project-level could be the selection of a new type of retaining wall that has resulted in quantifiable savings. Secondly, they were asked to critique the draft evidence-based framework.

Each panel included three or four practitioners from state Department of Transportation (DOTs) and/or consultants. Practitioners chosen to participate in the webinar included DOTs that had

B5

Evidence-Based Decision Making for Transportation Asset Management
contributed to the Asset Management Implementation Peer Exchange, consultants, and other transportation asset management managers. To facilitate the online discussion each practitioner was contacted by phone prior to each webinar. Calls lasted approximately 30 minutes, and consisted of a one-on-one conversation between a member of the IRG team and the practitioner. The practitioner call was used as a means of understanding practitioner thoughts about the project objectives. Specifically the practitioner calls were used as a means of identifying possible research outcomes that would be useful and applicable to professional practice; how practitioners considered evidence in their day-to-day work its importance, how it's measured, what value it added to their work, and other issues that practitioners wanted to highlight related to evidence in TAM decision making. Thoughts expressed by practitioners were also used to shape the Q&A section of the webinar. Practitioner calls were also used as the initial forum to collect practitioner ideas on possible case studies for future research and reporting of best practices in TAM decision making. Those practitioners who were unable to participate in the webinar were given the opportunity to make their contribution via phone. Practitioners who were unable to participate in the webinar contributed to this work via phone. Some of the issues discussed via phone included:
Evidence as a hard vs. soft parameter Evidence being related to time, context, and maturity TAM risk management and the ISO principles on risk Added value and the cost of improved decision making These and other topics were discussed in depth during the webinars. Table 2 shows the organizations whose officials participated in the webinars.
B6

Evidence-Based Decision Making for Transportation Asset Management

Table 10: TAM Evidence Exchange Participants

Organization

Participation Format

New York DOT

Phone

Pennsylvania DOT

Webinar 3

Minnesota DOT

Phone

North Carolina DOT

Webinar 3

California DOT

Phone

PBS Consulting, Inc.

Phone

South Carolina DOT

Webinar 3

Cambridge Systematics

Webinar 2

Volkert, Inc.

Phone

Washington DOT

Webinar 1

Arizona DOT

Webinar 1

Florida DOT

Webinar 2

Oregon DOT

Webinar 1

Webinar Transcript
Each webinar lasted approximately one hour, and covered themes useful to TAM investment decision making at both the program (high level management of assets at the network or systems level), and the project level (unit level management of infrastructure assets). The formal evidence-based decision making presentation and Q&A session facilitated discussions covering the following topics relevant to TAM investments and decision outcomes:
Webinar 1 System Integration and Data Collection
Webinar 2 Evidence Quality
Webinar 3 Knowledge Exchange, Risk Management and Investment Tradeoffs
The webinars helped to not only identify key issues for future research but case studies at the local, state, and national level for further investigation. The main themes, discussion points, and cases studies to emerge from the practitioner webinars are discussed below. Detailed notes from each webinar can be found in the Appendix.

B7

Evidence-Based Decision Making for Transportation Asset Management
Webinar 1
Data Collection Practitioners expressed that the quantity of data available was sufficient, and that data quality was generally good. However, the limitations of staff time, and resources often prevent agencies from being able to process the large amounts of available data. There is an expressed desire for the development of tools and procedures that use limited data to inform decision making. Good data often provides the building blocks for a sound asset management system. Maintenance, rehabilitation, and preservation decisions rely on the availability of technically sound, and timely data about infrastructure conditions. Under conditions of fiscal constraint data collection has become an expensive prospect for many localities. Practitioners are less interested in directives that promote additional data. The current challenge is how to make good decisions with limited data. There is currently an expressed interest in the use of empirical models to predict infrastructure maintenance needs: models such as deterioration models and risk models. The development of such tools requires that the useful life of data be taken into consideration.
Given limited budgets the focus has shifted to collecting data once and using it multiple times. Less is in fact more and careful consideration should be given to data attribution that can be collected over a reasonable period of time; data that requires 2 years and not 20 years to collect makes more sense. Asset managers faced with program development or restructuring should focus their attention on routinely trying new things on a small scale first and then expanding the program as resources become available. The asset management-learning framework is a step-wise one and many failures and successes may be experienced along the way.
System Integration System integration refers to a holistic approach to decision making that looks at the network as a whole, rather than solely at individual components of the network. An integrated system considers the maintenance tradeoffs that occur when the decision is taken to maintain one set of assets over another. Oregon DOT has adopted an integrated systems approach. The first step taken by the Oregon DOT was to align the asset management plan with the strategic management plan. This approach ensues that the agency vision can be achieved through the management, maintenance, and rehabilitation of assets. An integrated system requires that managers look at the condition of the entire system. This approach may require the revision of existing policies and procedures, and for this reason a champion or program advocate may be needed to ensure success.
Challenge the "Status Quo" Success in asset management may require a challenge to the status quo funding strategies, design requirements, existing policies and regulations. Having a clear strategic vision, and clearly articulated system goals can provide justification for such a challenge. Practitioners must determine whether existing policies and procedures will assist in achieving the desired strategic goals, by asking the right questions, and seeking to make fact-based decisions. There must also be the professional will to alter funding structures or change work plans if it is determined that current practices will not achieve desired outcome. For example, one
B8

Evidence-Based Decision Making for Transportation Asset Management
agency relayed a story where maintenance and safety needs were found to often conflict. It was determined that the schedule for 3R (reconstruction, resurfacing, and rehabilitation) projects was not meeting the maintenance needs of safety attenuators and concrete barriers (often rehabilitated during the course of a 3R project). The maintenance needs for these assets were greater, and the safety implications significant. Asset managers made a case for a program structure change/separation based on evidence that the maintenance backlog for safety attenuators and barriers was increasing, as were the number of crashes and fatalities that could have been prevented. This evidence allowed managers to successfully challenge the status quo.
Webinar 2
Evidence Quality The practitioners involved in this webinar were most interested in the concepts of TAM evidence, and what constituted quality evidence. Evidence was defined as being both quantitative and qualitative. Evidence of success was also defined in terms of behavior change, or organizational change. There was also interest in discussing the role that the "gut feeling" or practitioner experience plays in the decision-making process.
Webinar participants recognized that evidence quality and value are dependent on the methods used to gather the information, the context in which it is used, responsibilities to stakeholders, and the timeframe in which the evidence is applied. Program maturity and cost were also believed to dictate the quality of evidence needed for decision making. For example, it may be appropriate for an agency in its TAM initial stage to use lower quality evidence to make decisions, as their accountability may be lower in this trial and error stage. Intuition, and practitioner experience may therefore play a bigger role in the initial stages of TAM development, but a more mature program may have greater mandates for data-supported decision making. By the same token large expenditures may require greater evidence in support of the decision made, as compared to decisions surrounding smaller sums of money.
Webinar 3
Methods for Achieving an Evidence-based Approach Practitioners expressed confidence in the case study as a methodology for identifying and applying evidence. There is a network of good engineers, doing good work, and great value is derived from being able to share those experiences between TAM professionals. There was also an expressed interest in knowing what others are doing particularly within a regional context, and a methodological approach that involves case studies was found to be useful.
B9

Evidence-Based Decision Making for Transportation Asset Management
Another methodology that has been used to provide evidence in support of decision making is research. In the management of their pavement preservation program North Carolina has found it to be of benefit to have an engineer on staff with an academic research background. This natural desire to "ask the right questions" and for seeking the evidence that supports decisions has allowed the state to better understand what it is, why it is being done, and the value it brings. North Carolina has recognized the value of locally focused research and the evidence it provide can bring their program. Other useful methodologies for acquiring evidence include the business case analysis, and pilot studies.
Communicating to Stakeholders Information is in the eye of the beholder. Transportation asset managers must communicate to various stakeholders each seeking a desired outcome. Benefits do not always have to be framed in terms of revenues, or cost savings. User benefits, customer outcomes, and other non-monetary benefits play a role in defining the success of the project/ program. Evidence needs to be framed in a manner that communicates to various stakeholder groups, and addresses these varying outcomes.
Overall DOTs could improve the process by which they document and communicate the evidence used to support decision making. This documentation process was described as grey matter extraction or information to be extracted from the brain of a single practitioner and shared with all decision makers. An emphasis needs to be placed on developing the evaluation and feedback stages of the program development process. It is at this point that true connections can be made between investments and decisions.
Tradeoff Analysis and Risk Practitioners expressed a need for an increased use of tradeoff analysis tools to evaluate competing needs. For example when funds are shifted to the maintenance, repair, and evaluation of one asset class, this is often at the expense of another asset pavement preservation and bridge preservation are often competing interests. What is the business risk across all assets? Can acceptable risk be documented as agency policy?
In addition, the decision to expend funds evaluating maintenance and evaluation needs often comes at the expense of meeting already established maintenance needs. These situations create tradeoffs. Practitioners expressed excitement about the new tools being used to evaluate such scenarios, and expressed confidence that tradeoff analysis would produce new evidence in support of decisions made. However, there was also concern expressed about the quality of evidence currently being derived from these tradeoff tools. Data confidence is ever changing.
B10

Evidence-Based Decision Making for Transportation Asset Management

Case Studies

As discussed in the webinar, practitioners believe that the case study is a valuable tool in the development of an asset management program. During the webinar several suggestions were made for future case study investigation. Suggestions were made for both project level and program level asset management, and covered a wide range of topics including:
Collaboration and the formation of teams for asset management
Risk analysis and asset management
Trade off analysis as a decision support tool
Business case analysis and other economic tools
A list of the suggested case studies for future study is provided in Table 3 below.

Table 11: Case Studies for Further Investigation

Case Study Business Case Analysis (Portland Water Bureau)
Asset Knowledge Development (Tillamook County Public Works)
System Integration Pilot Study (Oregon DOT) Funding and Performance Strategies (Ohio DOT)

Themes
Business case study and Seattle public utilities; recognized nationally and internationally for successful efforts to embed business case and risk-based decision making as the basis of municipal water infrastructure investment decision making; evaluated specific programs and project candidates as they have been put through the filter of risk management focusing on long term impacts; summary of business case analysis developed by Portland Water Leader in asset management at the local level; developer of transportation asset knowledge and options that inform leadership of the risks and impacts of current and future service levels; alternative funding mechanisms - general obligation county road bond
Integration and coordination of all systems with the goal of achieving strategic management vision through asset management plan Funding allocation based on performance guidelines rather than formula grant; Ohio DOT conducted pilot project using this approach (documented in FHWA/TRB scan on asset management

Trade Off Analysis preventative maintenance vs. reconstruction (Minnesota DOT) AMPER Rating System (Florida DOT)
Liaison Meetings AMOTIA (National/ Florida DOT)
Business Modeling and Business Risk (Paloma County, Ohio) System Health Index (North Carolina DOT) Quality evidence and evidence modeling (New York DOT)

Impacts of increased preventative maintenance; MNDOT pavement analysis tool; tradeoff analysis; expenditure opportunity cost
Asset management contracting and maintenance rating report - allocates a score or grade to contractors (grades them); contractor performance evaluated based on AMPER score; AMPER rating more important to contractors than financial reward/penalties (evidence produced of performance) Partner coordination - identify stuff that works and stuff that does not; statewide maintenance engineer meetings; best practice and practitioner exchange (problem exchange) Business model for decision-making, Business risk; ISO principles; reducing long term costs of your investments Scenario analysis; tradeoff between bridge condition, pavement condition, and maintenance conditions
NYSDOT has been able to model improved pavement quality following a new investment decision as compared to old investment decisions; NYDOT struggling to model the impact of decisions not made.

B11

Evidence-Based Decision Making for Transportation Asset Management
Discussion
The Evidence Exchange attempted to identify some TAM investments at the program level and project level that had resulted in quantifiable evidence of success (positive outcomes) or failure (negative outcomes). This task proved to be more challenging than anticipated. In several instances practitioners were aware of the various steps taken to bolster their asset management program, but could not definitively say what the impacts of these decisions had been, and therefore could not speak to evidence of program outcomes. This result shows that there is an opportunity to create an evidence-based framework for transportation asset management, and to embed this framework into practice. One of the first steps in this process will be to formally define evidence in terms of asset management outcomes, second define acceptable methodologies for identifying and evaluating evidence, third strengthen the feedback and evaluation loop within the asset management planning process; and fourth understand the added value and costs of improved evidence quality in order to assess when it is worth going beyond the status quo of evidence in TAM practice.
As TAM programs evolve to be outcome-based and performance-based as required by MAP-21, greater focus will likely be placed on documentation and the evaluation of program outcomes. Agencies will need to conduct studies to evaluate their programs, taking the time to document outcomes. The need for improved evidence quality will be influenced by the size of the project or program, the political risks involved, the costs of obtaining improved evidence and other factors identified by practitioners above. A literature review/review of existing best practices is the first step in the development of an evidence-based framework and the results of the Evidence Exchange demonstrate that there is room for improvement in this area.
As practitioners begin to document and evaluate outcomes closer attention will have to be placed on evidence quality. In general, participants expressed a level of confidence with the use of the case study as a tool for gathering evidence about best practices and successful investments in TAM. However, there was the recognition that, the term "best practice" is often used loosely. In addition, case studies vary widely by length of time of study, application of context, and methods used for investigation. The general consensus was that there are in fact varying levels or grades of a "case study" and each study type creates different levels of evidence quality. Standardization of the case study process by defining necessary elements for quality by evidence grade, and by providing examples of case studies that correspond to these grades will help practitioners who seek to elevate evidence quality by engaging in more rigorous studies.
Next Steps
Moving forward this work will conduct an in depth investigation of the case studies identified by practitioners as exemplifying a good use of evidence for successful decision making. An attempt will be made to isolate the characteristics of the studies that made them successful. This work will also attempt to grade/categorize the case studies based on their rigor, and the quality of the data generated by the work. Multi-methodological approaches that involve combining case studies with other quantitative or qualitative methodologies will also be explored. Lastly, a
B12

Evidence-Based Decision Making for Transportation Asset Management knowledge base of best practices for asset management will be compiled based on a review of these and other case studies.
B13

Evidence-Based Decision Making for Transportation Asset Management
APPENDIX C: WEBINAR PRESENTATION
C1

12/15/2015
The Transportation Asset Management Evidence Exchange
Dr. Adjo Amekudzi, Janille Smith-Colin, Jamie Fischer, and Margaret Akofio-Sowah
Georgia Institute of Technology School of Civil and Environmental Engineering
Infrastructure Research Group May 20, 2013
Outline
Introductions Presentation
Evidence-based design Evidence quality and bias Evidence-based asset management
Q&A
Evidence-based framework for TAM Evidence quality for TAM decision-making
Roundtable discussion: Evidence exchange Next steps Closing remarks
2
C2

12/15/2015
Transportation Asset Management
Systematic approach to improving results through evidence-based decision making, continuous organizational learning, a focus on accountability and performance management
Evidence is critical to the successful implementation of a performance-based, outcome-based program
(Source: Tony Kane, Former Director of Engineering and Technical Services, AASHTO)
3

Research Objectives
Develop an evidence-based model and framework for transportation asset management
Define evidence quality for TAM decision-making Identify what works and under what conditions
TAM evidence and TAM outcomes

Evidence-based Design (EBD)

Transportation Asset
Management (TAM)

Evidence-based Asset
Management (EBAM)
4

C3

12/15/2015

Evidence-based Design

Emphasizes use of credible data or evidence to influence the design process
Rigorous testing Scientific evidence Practitioner input Improved outcomes
Applications
Healthcare facility design Medicine and nursing Education Policy planning Social services

Systematic reviews
Randomized controlled trials Cohort studies Case control studies
Case reports Editorials, Expert opinion
Rigorous testing hierarchy
(Source: The Center for Health) 5

Evidence Quality and Bias

Grade

Medicine: Rigorous Testing

Bias

Evidence

Quality

Occupational Safety : Rigorous Testing

A Systematic reviews (meta analysis) based on randomized controlled trials (RCT)
B At least one RCT, controlled trial without randomization
C Non-experimental descriptive studies (comparative studies, correlation studies, case control studies
D Opinions and reports from expert groups, consensus conferences, or clinical experiences of recognized persons
E Case studies

Systematic reviews based on high-quality intervention studies
Many intervention studies (high quality) Practical experience (published), best practice portals Prevention/intervention studies (low-quality) Practical experience (not published), best practice
Descriptive studies (cross-sectional studies, correlation studies, case-control studies etc.)
Opinion and reports from expert groups, consensus conferences experiences of recognized persons Emerging topics: expert surveys, case studies

(Source: Annette, N., & Frank, B. (n.d). Examples of evidence-based approaches in accident prevention)

6

C4

12/15/2015

EBD and Healthcare

Natural daylight helps patient heal faster

No cubicle curtain reduces hospital
acquired infections

Patient control of window treatments and
door from bed

Family zone to encourage family members to stay
reduces patient stress

Nature based artwork reduces patient stress

Stain and bleach resistant upholstery reduces
hospital acquired infections
100% Solution died carpet tile helps with acoustics and reduces patient falls

Bedside charting reduces medical errors
Patient toilet door close to side of bed reduces patient falls Easily accessible hand
washing sink near patient room door

(Source: Elting, Douglas, V. " The Impact and Application of Evidence-based Design.")

7

Fable Hospital 2.0 Innovation and Costs Calculations

Innovation
Larger patient bathrooms

Additional Cost
$2,880,000

Large single patient rooms
Larger windows

$13,500,000 $225,000

Enhanced indoor air $374,000 quality
Hand hygiene facilities $235,875

Cost Calculations

An increase of 32 sq. ft. with a 4 ft. doorway for each of the 225 ADA bathrooms: 225 bathrooms @ $12,800/month

Increase all 300 single patient rooms by 100 sq. ft. x 300 beds @ $450/sq. ft.

Increase typical patient room window size from 3 ft. x 5 ft. to 5 ft. x 8 ft. for all single patient rooms: 300 rooms @ $750/room

Improved ventilation: HEPA filtration and increased air change rates for all air handling units serving patient areas: 36 air handling units @ $10,400/unit

Hand-washing sinks in all 300 patient rooms,

automated alcohol-based hand-rub dispenser at each

bedside in all 135 nursing substation: 300 sinks @

$750/sink; 435 alcohol rub dispensers @ $25/hand

dispenser

8

(Source: Sadler, Blair L., Fable Hospital 2.0, Hastings Center Report)

C5

12/15/2015

Fable Hospital 2.0 Innovation and Improved Outcomes

Improved Outcome Patient falls reduced
Adverse drug events reduced Nursing turnover reduced
Length-of-stay reduced
Healthcare infections reduced

Innovation Bundle
Larger patient bathrooms, decentralized nursing substations, patient lifts
Large single patient rooms, medication area lighting
Larger windows, noise reduction measures, staff respite areas
Larger windows, increased natural light, noise reducing measures
Larger single patient rooms, hand hygiene facilities, enhanced indoor air quality

Outcome Reduction Falls reduced by 33%
Drug events reduced by 20% Nursing turnover reduced by 50%
Length-of-stay reduced by 10%
Infections reduced by 20%

Revenue Savings $1,534,166
$617,400 $478,500
$1,092,975
$355,400

(Source: Sadler, Blair L., Fable Hospital 2.0, Hastings Center Report)

9

Evidence-based Asset Management

External evidence

Practitioner experience

DECISION Stakeholders

Context, organizational
culture

Evidence-based Model
A framework that practitioners can use to show program maturity
through evidence of program outcomes based on the types of
interventions implemented

Experience-based Model

10

C6

12/15/2015
What is TAM Evidence?
Any outcome that can be measured or tracked following an intervention or action
Evidence can be tracked at a point or over time
Evidence should be related back to the action or intervention
Evidence does not have to be complex just demonstrate gains (positive or negative)!
11

TAM Evidence Levels

Evidence Level A
B C
D E

TAM Maturity
Best practice
Proficient
Structured
Awakening
Initial

Type of Study

Description

Controlled intervention studies (high quality)

Several high quality studies with consistent results; one large high quality multiintervention study

Controlled intervention studies (low quality)

One low quality study; few agency trials

Cross-section studies, correlation studies, retrospective studies

Before and after study of interventions; study of multiple variables, study over time

Case studies, expert Expert opinion via interviews; surveys, best practices in depth study with limited
control environment

Observational studies

Careful observation without intervention, no control environment

Transportation Outcomes
12

C7

12/15/2015
What We Need from You
3 program level or project level interventions
quantifiable evidence of success (positive outcomes) or losses (negative outcomes)
Discussion of evidence quality in terms of TAM decision making
Input on Evidence-based asset management framework
Thank You
14
C8

12/15/2015
What we need from you
Identify 3 program level and 3 project level interventions that have produced quantifiable evidence of success (positive outcomes) or failures (negative outcomes)

TAM Decision Making Process

Practitioner experience

Stakeholders DECISION

Context, organizational
culture

Experience-based Model
Quantitative and qualitative inputs
Historical data Heavily weighted on
judgment

16

C9

12/15/2015
What We Need from You
Identify 3 program level or project level interventions that have produced quantifiable evidence of success (positive outcomes) or failures (negative outcomes)
Discussion of evidence quality in terms of TAM decision making
Input on EBAM framework
Outcome based? Study based?

Evidence Quality Concepts

Grade

Medicine

Bias

A Systematic reviews (meta analysis) based on randomized controlled trials (RCT)
B At least one RCT, controlled trial without randomization

C Non-experimental descriptive studies (comparative studies, correlation studies, case control studies
D Opinions and reports from expert groups, consensus conferences, or clinical experiences of recognized persons
E Case studies

Evidence Quality

Occupational Safety
Systematic reviews based on high-quality intervention studies

Many intervention studies (high quality) Practical experience (published), best practice portals Prevention/intervention studies (low-quality) Practical experience (not published), best practice Descriptive studies (cross-sectional studies, correlation studies, case-control studies etc.)

Opinion and reports from expert groups, consensus conferences experiences of recognized persons Emerging topics: expert surveys, case studies

(Source: Annette, N., & Frank, B. (n.d). Examples of evidence-based approaches in accident prevention)

18

C10

12/15/2015

TAM Evidence Levels

Evidence Level A
B C
D
E

Evidence Maturity
Best practice
Proficient
Structured
Awakening
Initial

Type of Study
Controlled intervention studies (high quality)
Controlled intervention studies (low quality) Cross-section studies, correlation studies, retrospective studies Case studies, expert surveys, best practices
Observational studies

Description
Several high quality studies with consistent results; one large high quality multiintervention study
One low quality study; few agency trials
Before and after study of interventions; study of multiple variables, study over time
Expert opinion via interviews; in depth study with limited control environment
Careful observation without intervention, no control environment

Transportation Outcome
19

EBD Process

Problem identification

Literature review
(national and international)

Systematic intervention: Rigorous
testing

Efficacy Evaluation/ Outcome summary

Critical evaluation: EVIDENCE

Surveys to identify themes, problems in
practice (practitioner input)
20

C11

12/15/2015
Practitioner Input
Evidence to outcome is not necessarily a one to one relationship
Consider not only interventions but externalities
Bundle externalities (weather, budget, workforce)
Project should involve trade-off analysis Evidence to be gained from other
infrastructure sectors and assets
public works, water management in-house asset management efforts (signs,
non-interstate)
Risk-based analysis framework for decision making (program level)
21
Contributions of the Work to TAM Practice
This outcome-based framework can be used by transportation practitioners to demonstrate program maturity through the types of interventions being made and evidence of program outcomes.
A hierarchy of evidence based generalizable characteristics of evidence level and evidence quality
A database of evidence and strategies for assessing evidence following the implementation of specific interventions
C12

12/15/2015
Next Steps
Compile notes from practitioner calls Schedule follow-up for end of May
Identify format (webinar, teleconference)
Continue work on EBAM framework and evidence hierarchy
Explore potential applications from other infrastructure assets
Tilamook County, Oregon
Identify real-world projects for rigorous tracking and testing
23
C13

Evidence-Based Decision Making for Transportation Asset Management
APPENDIX D: DETAILED WEBINAR NOTES
D1

Evidence-Based Decision Making for Transportation Asset Management
Webinar 1
Practitioners expressed a liking for the evidence-based approach and were interested in tools and methodologies that could assist them in moving towards a more fact-based decision making process a movement away from intuition or gut feelings.
DOTs tend to be risk averse o Liability laws/ tort laws can contribute to this risk aversion
Try things on a small scale first! TAM Evidence Levels comments
o Your approach does not always have to be engineered Maturity may have an impact on your approach to evidence Triaging may be sufficient for those who are starting out Limited resource input with potentially high returns less data intensive
Oregon Original establishment of an asset management strategic plan and an implementation plan based on an individual asset approach (pavement and bridge) Six years ago a pilot effort based on an integrated asset program was undertaken to include more than just pavements and bridges culverts, signs, etc. o less is more Engineers tend to build datasets that are really robust and often infeasible to maintain o Less is more o Data management needs to be looked at collect it once and not again Developed work plan for statewide inventory and maintenance streamlined the process for asset management development Data from Work Plans demonstrated that current practice was not meeting the needs of maintenance safety attenuators Developed a web-based GIS tool to support decision making Reprioritization based on 9 criteria o Working to maintain assets and make info available to all users; commitment identify gap, need, and cost for all assets. Culvert initiative underway to assess gap, needs, and cost Local experience informs choices o Stair-step approach build on the success of pilot and use that knowledge to inform future advances Case study opportunities Pilot Study (Forming groups for decision-making) o Successes and failures of the pilot study o Culverts had 200 attributes Impossible to collect and maintain this data too data intensive o Pilot study led to revision of attributes Basic inventory process Type, size, location, and condition - enough to triage (e.g., catastrophic event at a hospital); assessing in short-order Basic inventory fields became 4 12 fields vs. 200! Info proved to be enough to make basic decisions about asset
D2

Evidence-Based Decision Making for Transportation Asset Management
Follow-up inspections were factored in over time, but brief, and less technically detailed
Minimization of effort does not necessarily mean reduced quality Less can in fact be more! o Data collection can vary by asset DVL, followed by field verification, mobile GPS, mobile-scanner with point cloud data Data integration did not necessarily happen in asset data collection
difficult to maintain attention with data collection of multiple assets 1-mile pass o Internal documentation of approach to Pilot Study and lessons learned
Washington Program structure change separation of safety program from the 3R program o It was discovered that 3R projects were not meeting the safety mandates; number of fatalities and accidents were increasing o Maintenance backlog is decreasing o Safety funds could be used on safety needs, and maintenance funds on maintenance needs o Concrete barriers (non-standard guardrails) in concrete section finally got the attention needed; formally neglected Get down to factual information you will replace a guardrail where it is 1 inch below the standard, but what about the places where there is no guardrail?? o Systems approach o Individual assets do not need to be perfect for the system as a whole to be functional Asking the right questions what approach makes sense??? Use pilots to test preliminary theories about methods for improving o Project level case study possibilities Chip seal overlay example Bridge scouring/ painting Culverts replacement
D3

Evidence-Based Decision Making for Transportation Asset Management
Webinar 2
Evidence as organizational change Evidence that asset management changes the way decisions are made and produces
benefits The gut intuition plays a role hey we think this works
Florida
Asset maintenance contracting program Liaison meetings quarterly
o Met with top executive from each of their asset maintenance contract; all districts DOT maintenance engineers and the contractors
o To talk about what works and what does not Liaison meetings has been combined with the AMOTIA conference, along with the
district maintenance engineers conference o Allows for an exchange of ideas and tools between industry and state officials o A smaller team has been identified to specifically tackle "problems" facing asset managers this group ended up tackling known issues, as well as issues that emerge from the meeting of the minds o Qualitative o Added value achieved from these meetings these meetings cost money, but have resulted in effective practices and knowledge exchange
Maintenance System quantitative evidence of benefits of this program
Framework
Best practice is the ultimate o Is best practice used to loosely? o Can standards be established for what the industry considers best practice
Is there a level of maturity beyond best practice? o Is there a more state of the art way to define a mature program other than best practice? o Does best practice need to be quantifiable? Does it depend on the amount of money being spent? Not everything can be quantified
Risk management is needed - as you move up the scale how can you increase the confidence in the evidence that you get, and reduce the risk of monetary costs o A combination of quantitative data and expert knowledge is needed to achieve the increased confidence
Is there a way to tie the influence factors directly to outcomes o Maybe we don't need to identify the influence factors, we just need to define the evidence gathering procedure
D4

Evidence-Based Decision Making for Transportation Asset Management
Webinar 3
What does quality evidence mean?
Case studies are most valuable to transportation asset management professionals. There is a good network of good engineers, doing good work, and great value is derived from being able to share those experiences between TAM professionals. Case studies are best.
How do we define positive outcomes for transportation asset management? Do outcomes have to be defined in terms of revenue?
Project examples Pavement preservation treatment early in the life of a pavement extends the life of a pavement for years (NC data from Jenifer Brandenburg Dr. Judy Corley-Lay) Bridge preservation program has shown over time a reduction in the rate of the number of bridges becoming deficient every year. o Achieve balance between the bridges that already structurally deficient and the rate at which other bridges will be entering the "structurally deficient pool"
Good evidence or quality evidence/ Is there a standard? Research is used by NC to justify decision making Research allows an organization to show that within a local context national "best practices" can be applied successfully NC has a pavement preservation researcher on staff / manager with a research background this has allowed them to conduct more in depth exploration of existing strategies in pavement preservation and to make informed decisions about how to proceed in NC o More than just a gut decision o State pavement engineer with a doctorate is able to serve as a champion o Researchers' presence really allows them to show/communicate what they are doing/ what they intend to do.
Elements needed for program success: Leadership from the top, buy in from the legislature, by in from the field technicians non managerial champions
Strategies for communicating evidence: Tools are great but a decision has to be made about the confidence level that one has in the data they are getting For example NC has begun to use tradeoff analysis
South Carolina experience Balance has to be achieved between assessing the extent of the problem and repairing known problems
D5

Evidence-Based Decision Making for Transportation Asset Management Review existing standards and regulations; it may be appropriate to lower the LOS for a particular asset o SC now uses a 10 years from date of warranty approach considering a date of installation process Wants to fund a research project on a date of installation and what is a good point of process for replacement
Communicating to stakeholders The information is in the eye of the beholder. Balance between where you make investments and which results you communicate Examples from multiple states how are our neighbors doing it; regional perspective Looking for evidence that we are using evidence. DOTs could improve the process by which they document and communicate the process Grey matter get it out of somebody brain and put it on paper so that everyone knows why are doing what we do Plan evaluation and plan success; the feedback loop needs development
D6

Evidence-Based Decision Making for Transportation Asset Management
APPENDIX E: ASSET MANAGEMENT CASE STUDIES
E1

Evidence-Based Decision Making for Transportation Asset Management
Ohio DOT (ODOT) System Integration and Pilot Study
According to an NCHRP sponsored domestic scan, Best Practices in Transportation Asset Management, ODOT approaches asset management in a comprehensive and sophisticated manner, such that ODOT considers asset management as a core value and function of the organization (Cambridge Systematics Inc., 2007). ODOT efforts to achieve formal asset management started in the late 1990s. Before the inception of this integrated form of asset management only 80 percent of the state's freeways had a pavement condition rating over 65 out of 100. To address this deficiency, ODOT categorized the highway network into three distinct policy systems: priority (interstate and four-lane divided highways), urban (state highways within municipalities), and general (primarily two-lane highways across the State), and focused on the priority road network.
To achieve success with their program, ODOT adapted a series of activities within their business plan that could lead to a successful implementation of the program, as well as improving on the performance of their highway network. Actions were linked to achieving the strategic goals of the organization. Performance measures were then selected to monitor progress and help in making decisions. Transportation infrastructure makes up the majority of ODOT's assets. System Preservation is therefore one of the most emphasized strategic areas in ODOT's asset management system. The goal for the systems preservation strategic area is to achieve a steady state condition for pavement and bridges. To achieve this steady state, ODOT reviews and updates system goals on a biennial basis to identify short-term actions needed to compliment long-term goals. In addition, ODOT undertakes the following procedures to ensure that the system meets its performance targets:
Conducts yearly reviews and holds leaders accountable
Ties yearly appraisal to achieved performance targets
Holds quarterly and mid-year reviews to assess action plan progress
Monitors bridge and road conditions through central office to ensure consistency
Ties budgets are tied to asset condition
Distributes funds to districts via a funds management committee
Gives districts a four-year ultimatum to improve system condition
Shares condition data among district managers to serve as motivation
Major Milestones/Evidence After the inception of the pilot program, ODOT recorded notable improvements in system condition. Documented results from the pilot program also provided ODOT staff with the evidence needed to build a business case for seeking additional funds from decision makers. In fact, due to evidence of system condition, the legislature provided additional dollars for the purpose of increasing capacity (Cambridge Systematics Inc., 2007). Other achievements from
E2

Evidence-Based Decision Making for Transportation Asset Management
ODOT instituting the asset management pilot program are as follows (Cambridge Systematics Inc., 2007):
Pavement degradation in the 90s routinely exceeded 3.3 PCR points per year, compared to a rate of 2.34 in 2010. That is, prior to the pilot program, pavement condition deteriorated at a faster rate when compared to the post-program rate. The steady state achieved through the pilot program reduced maintenance needed to achieve a steady state.
Between 75 and 80 percent of customer respondents rated ODOT performance as "good". This response rate showed that the public was satisfied with the way ODOT was managing the transportation system and infrastructure.
Ohio's priority road network was in very good condition as compared to the conditions in the 80s.
Between 1997 and 2005, statewide system condition deficiencies reduced as follows: Roads 66 percent Bridges--80 percent
References
Cambridge Systematic Inc. and Meyer, M. D. (2007). U.S. Domestic Scan Program: Best Practices in Transportation Asset Management. NCHRP Report 20-68. Washington, D.C. Cambridge Systematics Inc. (2007). U.S. Domestic Scan Program: Best Practices in Transportation Asset Management. Washington, D.C. Federal Highway Administration. (August 2012). Transportation Risk Management: International Practices for Program Development and Project Delivery. Keeney, R. L. (2002). Common Mistakes in Making Tradeoffs. Journal of Operation Research , 50 (6), 935-945. Ohio Department of Transportation. (2012). Capital Programs Dashboard (SFY 2012). U.S. Department of Transportation, Federal Highway Administration, Office of Asset Management. (2001). Data Integration Primer. U.S. Department of Transportation, FHWA-IF01-016.
E3

Evidence-Based Decision Making for Transportation Asset Management
Portland Water Bureau Business Case Analysis
The asset management efforts of the Portland Water Bureau (PWB) were identified by practitioners during the TAM Evidence Exchange as being recognized nationally and internationally for successful efforts to embed business case and risk-based decision making as the basis of municipal water infrastructure investment decision making. The following provides a look at the Buddington Tank replacement as it is put through the filter of risk management in light of long-term impacts.
Portland Water Bureau began its asset management program in 2005 (1). As the asset management program developed so did the pressures for increased accountability, and transparency as stakeholders required justification for decisions made. One strategy used by the Portland Water Bureau to increase the technical legitimacy of decisions made, improve accountability, and strengthen transparency is the business case analysis model for project validation/evaluation. PWB staff described the use of business case analysis as the attempt to support decision making with quantifiable evidence.
Business Case Analysis and Risk Management Business case analysis uses a variety of tools to frame infrastructure investment decisions in terms that decision makers can understand. Business case analysis can be used to support a variety of infrastructure decisions including investment in new technologies, capital improvement, and maintenance and operations decisions (2). Business case analysis uses multiple evidence criteria to evaluate project and program level decision-making - life cycle cost assessment, benefit/cost ratio, risk-cost reduction, net present value, return on investment, cost effectiveness, and fiscal impact analysis. The evidence criteria used by the Portland Water Bureau have been accumulated from both private sector companies and public entities, and applied to achieve asset management goals. The following steps are typically used to develop a business case analysis the purpose of the project or program is defined, the issues to be addressed are identified, and all available alternatives considered (2). To provide a clear picture of the options being considered a do nothing or "status quo" option is often also considered.
Business case analysis is not only used at the project level but also at the program level to inform strategic decision-making efforts. To make this effort successful, a conscientious decision has been made to link business case analysis to, the strategic plan to implement, "a risk-based asset management approach to assist in managing planning and operations for the repair, replacement, and upgrades of our assets through capital projects and maintenance /operational programs." (2) The PWB uses the following working definition for risk, "the threat or probability that an action or event will adversely or beneficially affect an organization's ability to achieve its objectives."(3) PWB uses asset condition ratings to quantify asset risk by evaluating the remaining service life of an asset and the probability or likelihood of failure. As stated earlier, a key focus of the business case analysis process is presenting information in general terms that can be used by non-technical decision-makers. As such, PWB staff developed a Consequence & Likelihood of Failure Evaluation Matrix (CLEM) to rank asset condition regarding risk on a scale from 1 to 5 based on residual life (2) as shown in Table 1. The CLEM likelihood and rate of failure ratings are used by PWB to identify critical assets. Critical assets are then ranked based on risk as follows: extreme risk, high risk, medium risk, low risk, and very low risk as
E4

Evidence-Based Decision Making for Transportation Asset Management
shown in Table 2. Asset managers have developed a recommended course of action based on the asset risk rating as demonstrated in Table 3.

Table 1: CLEM Likelihood and Rate of Failure for Individual Assets (2)

Likelihood Rating

Recurrence Interval for a

Failure Rate of a

Single Asset Failure (yrs)

Population of Assets

5

<=5

0.3

4

5-20

0.1

3

20-50

0.03

2

50-100+

0.013

1

>> 100

<0.1

Table 2: Asset Criticality Risk Ranking (2)

Likelihood

Very Low Low

(1)

(2)

Very Low (1) VL

VL

Low (2) VL

VL

Moderate (3)

L

L

High (4)

L

M

Very High (5)

L

M

Consequence Moderate
(3) L M H H H

High (4) M M H E E

Very High (5) M H E E E

Table 3: Risk Ranking Asset Management Action
Risk Ranking Extreme High
Medium

Asset Management Action Tested, inspected, or repaired immediately Tested, inspected, repaired within 12 months Tested, inspected or repaired within 3 years

Buddington Tank Replacement
In 2011 an inspection of the Buddington tanks identified cracks and exposed rebar on the roof of the tank. The Portland Water Bureau undertook a business case analysis to determine whether to take the investment decision of replacing the tank or to do nothing while risking potentially failure of the tank.
The main goal of conducting the business case analysis was to determine whether to repair and replace the Buddington tank or to completely abandon it (4). Evidence used to support this investment decision included benefit/cost ratio, life cycle cost assessment, and CLEM risk analysis. The results of the CLEM risk analysis are shown in Table 4. Risk analysis results indicated that both alternatives for the Buddington tank would create very low risks to affected
E5

Evidence-Based Decision Making for Transportation Asset Management

parties. The final decision to abandon the Buddington tanks was therefore based on the results of the other investment criteria benefit cost analysis and life cycle cost analysis.

Table 4: Buddington Tank CLEM Risk Analysis (4)

Event/ Failure

Likelihood

Mode

Buddington Tank Buddington Tank Likelihood >> 100

in Service

roof failure renders years L=1

it not usable during

any demand

scenario (regulated

supply not open in

time) with a

standard fire

Consequence

Sufficient supply

w/o tank during 3

hr. Standard fire,

but pressure may

be < 20 psi for a

limited

#

customers

at

higher elevation.

Standard fire flow

met. C=1

Risk Very low

Buddington Tank Offline

Failure of supply lines from Arnold (largest supply to area) during any demand scenario and a standard fire

Likelihood >> 100 years.
L=1

Property and social damages from fire of up to several million dollars and/or possibly dozens of services with pressure < 20 psi. C = 1 to 2

Very low

Application
Portland Water Bureau through their use of business case analysis has demonstrated that multiple evidence criteria can be used to support investment decision-making. PWB has determined that the success of its asset management program requires expert knowledge that is supported by quantifiable evidence. Taking into consideration the TAM EBD evidence hierarchy, PWB has successfully managed to incorporate multiple evidence levels into its decision making process and therefore provides a good example of evidence-based asset management through the use of the business case analysis framework.
In 2010, Portland Water Bureau produced a guide, "Business Case Development Guidebook" to help standardize the business case analysis methodology, and to ensure that the process was repeatable. The efforts of PWB demonstrate how multiple evidence criteria can be applied to assist decision making. A multiple criteria approach is therefore important to ensure technically sound decision making.

E6

Evidence-Based Decision Making for Transportation Asset Management References
(1) Portland Water Bureau Population and Services (2013). The City of Portland Oregon; http://www.portlandoregon.gov/water/29460
(2) Business Case Analysis (3) Luhmann, N. 1996. Modern Society Shocked by Its Risks Social Sciences Research
Centre in association with the Dept. of Sociology, University of Hong Kong (4) Project Concept Report: Buddington Tank, Portland Water Bureau Engineering
Services Group, August 2012
E7

Evidence-Based Decision Making for Transportation Asset Management
Tillamook County Public Works
The Tillamook County Public Works (TCPW) manages the roads and solid waste for the 25,287 residents of Tillamook County. This includes over 386 miles of roads and 100 bridges (1), (2). Tillamook County Public Works is also responsible for managing the county's other transportation services and assets such as structures (including levees and guardrails), drainage (culverts and ditches), maintenance facilities and quarries. Asset Management The department initiated their first asset management policy in 2009. The infrastructure asset management goal of the county is to "meet the required level of service in the most cost effective manner for present and future consumers" (3). To this effect, road assets and service priorities are in line with the strategic priorities outlined in the Tillamook County Comprehensive Plan and the Transportation System Plan. This is a risk-based approach in which technical analysis is performed on high cost and high risk assets. The risks are rated by analyzing threat likelihood and consequence, after which a risk treatment plan is developed to manage extreme and high risks. Identified risks undergo an annual review and the county's Risk Management Plan is updated every 3 years. Risk Treatment Plans also undergo a performance review; 6 monthly performance reviews for council staff performance criteria, and an annual review (1).
TCPW also defines various levels of service for each asset and categorizes them under a) customer outcomes (service delivery standards), b) asset outcomes (technical and operational requirements), and c) asset activities (frequency of repairs and response times). These levels of service provide a link between the community's expected outcomes and the financial requirements. This helps the community understand the condition and needs of road assets. Figure 1 shows an example of the link between pavement outcomes and financial requirements. With this, the community is able to appreciate the financial implications of desired service conditions.
Figure 1 Pavement Outcomes and link to Financial Requirements (1)
TCPW has also correlated the level of road investments with their pavement condition index (PCI). Figure 2 shows a comparison of the 2007-2008 fiscal year investment levels of Tillamook with two other counties, Clatsop and Lincoln. The chart shows the correlation between investment level in dollars per mile and PCI, with Tillamook having the least PCI and investment level.
E8

Evidence-Based Decision Making for Transportation Asset Management
Figure 2: Cost per Mile and Pavement Condition for Northwest Oregon Counties (1)
Following this and best practice, the county uses an optimization software to produce scenarios which give an indication of future pavement conditions with various investment levels. Figure 3 is an example scenario showing current service level and resulting backlog from deferred maintenance.
Figure 3: Current Service Level Road Condition and Backlog (1)
Tillamook County Public Works demonstrates the use of evidence first by using performance reporting of their various strategies to guide decision making. Also, by using the service level system, they give the community a chance to see the tradeoffs between service levels and how much they are willing to invest in the road network. After 3 failed ballots to bond infrastructure needs, the fourth ballot finally passed on May 21, 2013 (4).
E9

Evidence-Based Decision Making for Transportation Asset Management References
(1) PBS Consulting (2008). Asset Management Project: Readiness Assessment. Tillamook County Public Works Department. Tillamook, OR. Retrieved from http://www.co.tillamook.or.us/gov/pw/Documents/AssetManagement/FinalReadinessAss essment.pdf Accessed on August 27, 2013
(2) United States Census Bureau, USCB (2013). "State and County Quick Facts: Tillamook County, Oregon." U.S. Department of Commerce. Retrieved from http://quickfacts.census.gov/qfd/states/41/41057.html Accessed August 27, 2013.
(3) PBS Consulting (2011). Tillamook County Road Performance Report 2011. Tillamook County Public Works, Tillamook, OR.
(4) North Coast Citizen (2013). "Voters give county road measure thumbs up." Retrieved from http://northcoastcitizen.com/ncc_news/article_e083f7f6-c2e7-11e2-bf8e0019bb2963f4.html Accessed on August 27, 2013.
E10

Evidence-Based Decision Making for Transportation Asset Management
Oregon DOT Pilot Study
During the TAM Evidence Exchange, practitioners suggested several examples of what they perceived to be effective evidence-based approaches to TAM at the local, state, and national levels. One example, from Oregon Department of Transportation (ODOT) is summarized here to demonstrate how new evidence can add value to a TAM program, relative to an agency's objectives.
ODOT conducted a pilot study from the end of 2005 to early 2007 in order to investigate the agency's ability to implement an integrated asset management program for data-driven decision making. The pilot study focused on four highway segments for a total of 75 miles. For each of these highway segments, the pilot study compiled data about selected assets in order to determine how data availability and accuracy may affect (a) asset management readiness, (b) asset condition, and (c) data integration capabilities. The study defined asset management readiness as being based on the presence or absence of 19 data/process elements in five categories: basic inventory data (type, size, etc.), additional details (installation date, update cycle, etc.), financial data (original or replacement cost, depreciation rate, etc.), performance and goals (service levels, standards & targets, etc.) and forecast/planning data.
The results of the pilot study indicated some challenges with respect to data availability and reliability. The study found that ad hoc data collection processes lead to inconsistencies and an inability to integrate data for agency-wide use. Furthermore it was determined, based on the presence or absence of 19 data elements, that only 3 assets had some degree of asset management readiness bridges, pavements and Intelligent Transportation System (ITS) sites. Finally, a correlation was found between asset condition and the existence of either (a) a centralized data program or (b) a high level of attention from maintenance staff. Assets with a higher level of asset management readiness also had generally higher condition levels. These results are documented in a pilot project report.
ODOT's pilot study was conducted under the direction of a Steering Committee and a Technical Committee, which convened for this specific purpose. Based on the study results, these committees developed a number of recommendations, referred to as "ideas" for moving the agency forward in asset management maturity:
Establishing priority levels to provide focus and a concentration of efforts;
Increasing data and information availability systematically;
Providing technology solutions to account for maintenance activities and to make
financial data available;
Streamlining resources and process to ensure that regional/divisional data collection
contributes to an overall database; and
Building temporary databases that would contribute to an agency-wide database over
time.
E11

Evidence-Based Decision Making for Transportation Asset Management
ODOT's pilot study is an example of a program-level intervention in TAM. The results of this intervention led to more strategic investments in the agency's TAM implementation process. In the six years since the study's completion, ODOT has seen quantifiable evidence of the success of this intervention. According to the ODOT Asset Management Strategic Plan developed in 2011, the pilot study revealed that the comprehensive TAM Implementation Plan developed in 2006 was too detailed for the current status of data components. As a result, the asset management goals defined in the 2006 plan were refined to focus more on fundamental issues like inventory and data collection. Because of these efforts, ODOT's TAM-ready data availability increased dramatically between 2005 and 2011. According to the 2011 TAM implementation plan, ODOT collected quality inventory data for nine new categories of assets during that time period (including tunnels, signs, sidewalks, and others), and was in progress with five more asset categories. Because of the pilot study as a program-level TAM intervention, ODOT has matured in asset management readiness and TAM programming. There are three types of evidence at work in this example. The first type is asset inventory data, which ODOT can use to make decisions about TAM implementation in the future. The second type is evidence resulting from the pilot study, which indicated that the agency was at a certain level of TAM preparedness, and in need of some changes. ODOT used this self-generated evidence to support its decision to redefine asset management goals and change focus from a broader implementation strategy to basic inventory improvement. This evidence also drove ODOT's decision to refine its TAM implementation process from the 2006 plan. The third type of evidence is provided by ODOT's 2011 TAMimplementation plan, which clearly indicates that conducting the pilot study led to an improvement in ODOT's overall asset inventory within six years. This new evidence can inform other agencies as they make decisions about how to advance their own TAM programs and processes.
E12

Evidence-Based Decision Making for Transportation Asset Management
AMOTIA Consultant Consortium
The Association for the Management and Operations of Transportation Infrastructure Assets (AMOTIA) is a partnership of more than twenty multinational transportation firms. AMOTIA, which was formed in 2007, has two objectives:
Better serve transportation agencies Be the unified voice of the private sector in management and operation of transportation
infrastructure assets.
One of AMOTIA's primary foci is performance-based asset management contracting, which the organization defines as:
"A contractual process in which payments for the management and maintenance of transportation infrastructure assets are directly linked to the contractor successfully meeting or exceeding certain clearly defined minimum performance measures." AMOTIA members enter into performance-based asset management contracts with state DOTs, counties, cities, and toll agencies. These contracts deal with ongoing maintenance activities, large design-build transportation projects, and larger public-private partnerships in transportation management and operations. AMOTIA has its newsletter, posted online, to highlight how members have used innovative methods to achieve high performance. For example, the 2009 newsletter highlights one member for keeping highway roads clear.
E13