Development of the road construction database (ReCORD) system

GEORGIA DOT RESEARCH PROJECT 17-16 FINAL REPORT
DEVELOPMENT OF THE ROAD CONSTRUCTION DATABASE
(ReCORD) SYSTEM
OFFICE OF PERFORMANCE-BASED MANAGEMENT AND RESEARCH
600 WEST PEACHTREE NW ATLANTA, GA 30308

1. Report No.:

2. Government Accession No.:

FHWA-GA-19-1716

4. Title and Subtitle:

Development of the Road Construction Database (ReCORD)

System

3. Recipient's Catalog No.:
5. Report Date: November 2019
6. Performing Organization Code:

7. Author(s): Stephan A. Durham, Ph.D., P.E. (https://orcid.org/0000-0002-

8. Performing Organization Report No.: 17-16

6177-3491); Kyle Johnsen, Ph.D. (https://orcid.org/0000-0003-

1119-0041); Sidney Thompson, Ph.D. (https://orcid.org/0000-

0002-0228-7768); Bryan Knakiewicz, Ph.D.; Justin Cobkit; and

Kevin Goff

9. Performing Organization Name and Address:

10. Work Unit No.:

University of Georgia

College of Engineering Driftmier Engineering Center, Athens, GA. 30602 Phone: (706) 542-9480 Email: sdurham@uga.edu

11. Contract or Grant No.: PI#0015706

12. Sponsoring Agency Name and Address:

13. Type of Report and Period Covered:

Georgia Department of Transportation

Final; August 2017 November 2019

Office of Performance-based Management and Research

14. Sponsoring Agency Code:

600 West Peachtree St. NW Atlanta, GA 30308

15. Supplementary Notes:

Prepared in cooperation with the U.S. Department of Transportation, Federal Highway Administration.

16. Abstract: Documentation of past experiences through the use of a Lessons Learned Program (LLP) is instrumental to an organization's ability to improve by avoiding past mistakes while adapting to new solutions and methods. Feedback from project stakeholders through interviews and meetings, as well as a national survey of state department of transportation professionals was received and used to properly design and construct an LLP database system for the Georgia Department of Transportation. The Road COnstRuction Database (ReCORD) system was designed to include lesson creation, curation, and publication system interfaces with key features such as push notification, mobile compatibility, and GIS capability. The research team trial tested the ReCORD system prototype with feedback received from survey responses and interviews. The evaluation process resulted in system modifications and improvements such as aesthetic changes, interface simplification, and the reorganization of lesson information. The finalized ReCORD system allows users to enter lesson information--such as lesson title, lesson tags, abstract, location, GDOT projects associated with the lesson, lesson summary, and references--and to upload files. The retrieval interface allows users of the system to receive notifications when lessons associated with tags of interest to the user are published. Ultimately, the system will benefit the Georgia Department of Transportation by providing a more efficient method of archiving and dispersing information.

17. Keywords: Database, Construction, Lessons Learned Program

18. Distribution Statement: No Restriction

19. Security Classification 20. Security Classification (of this

(of this report):

page):

Unclassified

Unclassified

Form DOT 1700.7 (8-69)

21. No. of Pages: 22. Price: 166

GDOT Research Project No. 17-16
Final Report
DEVELOPMENT OF THE ROAD CONSTRUCTION DATABASE (ReCORD) SYSTEM
By Stephan A. Durham, Ph.D., P.E. Professor College of Engineering
Kyle Johnsen, Ph.D. Associate Professor College of Engineering
Sidney Thompson, Ph.D. Professor College of Engineering
Bryan Knakiewicz, Ph.D. Assistant Professor Department of Engineering Technology
Justin Cobkit Graduate Research Assistant
Kevin Goff Undergraduate Research Assistant
University of Georgia Savannah State University
Contract with Georgia Department of Transportation
In cooperation with U.S. Department of Transportation Federal Highway Administration
November 2019
The contents of this report reflect the views of the authors, who are responsible for the facts and accuracy of the data presented herein. The contents do not necessarily reflect the official views of the Georgia Department of Transportation or the Federal Highway Administration. This report does not constitute a standard, specification, or regulation.

TABLE OF CONTENTS
CHAPTER 1. INTRODUCTION .................................................................................... 1 OVERVIEW .................................................................................................................. 1 RESEARCH SCOPE .................................................................................................... 3
CHAPTER 2. BACKGROUND....................................................................................... 5 LESSONS LEARNED CASE STUDIES..................................................................... 5 CDOT's US 36 Express Lanes .................................................................................. 5 Boston's Big Dig Tunnel Collapse ............................................................................ 6 STATE DOT SURVEY................................................................................................. 9 Survey Overview...................................................................................................... 10 Survey Findings ....................................................................................................... 10 State DOT Response Summary and Conclusions ................................................. 27 Statistical Analysis ................................................................................................... 30
CHAPTER 3. LITERATURE REVIEW ...................................................................... 35 OVERVIEW OF LLPS............................................................................................... 35 Definition .................................................................................................................. 35 Distinguishing LLPs from Other Knowledge Artifacts ....................................... 36 OPERATIONS OF LLPS ........................................................................................... 38 Psychological Aspect................................................................................................ 38 Benefits ..................................................................................................................... 42 Challenges................................................................................................................. 42 Implementation ........................................................................................................ 45 APPLICATIONS OF LLPS ....................................................................................... 46 LL Systems ............................................................................................................... 46 LESSONS LEARNED FROM LLPS ........................................................................ 50 Colorado Department of Transportation's T-REX Mega-Project ..................... 50 Virginia Department of Transportation's Program ............................................ 52 Pennsylvania Department of Transportation ....................................................... 52 SUMMARY OF LLPS ................................................................................................ 53
CHAPTER 4. PROBLEM STATEMENT.................................................................... 54
iv

RESEARCH SIGNIFICANCE .................................................................................. 54 RESEARCH OBJECTIVES ...................................................................................... 54 CHAPTER 5. EXPERIMENTAL PLAN ..................................................................... 56 PRELIMINARY PLANNING AND PROCESS DEVELOPMENT PHASE........ 56
GDOT Pre-Planning Meeting................................................................................. 56 NASA Interview ....................................................................................................... 57 District 1 GDOT Meeting........................................................................................ 59 Project SR 316 / SR 81 Site Visit ............................................................................ 59 Stakeholders ............................................................................................................. 61 CONCEPTUAL DESIGN........................................................................................... 61 PRODUCT DEVELOPMENT................................................................................... 67 ReCORD Database Features .................................................................................. 67 TESTING AND EVALUATION ............................................................................... 70 Preliminary UGA/SSU Testing............................................................................... 71 GDOT Trial Testing ................................................................................................ 72 CHAPTER 6. EXPERIMENTAL RESULTS .............................................................. 74 CREATION OF THE RECORD SYSTEM.............................................................. 74 System Requirements .............................................................................................. 74 SYSTEM INTERFACES............................................................................................ 74 Create Lesson Interface .......................................................................................... 76 Lesson Management Interface ............................................................................... 81 Help Interface .......................................................................................................... 87 Profile Interface ....................................................................................................... 88 Published Lessons Interface ................................................................................... 89 Lesson Map Interface .............................................................................................. 91 TRIAL TESTING RESULTS .................................................................................... 91 UGA/SSU Trial Testing........................................................................................... 92 GDOT Trial Testing ................................................................................................ 98 CHAPTER 7. DOT IMPLEMENTATION AND USAGE PLAN............................ 104 ORIENTATION TRAINING SESSIONS .............................................................. 105 Training and Materials ......................................................................................... 105
v

ReCORD SERVER AND FUTURE OPERATIONS/MAINTENANCE ............. 107 CHAPTER 8. CONCLUSIONS AND RECOMMENDATIONS ............................. 109
SUMMARY AND CONCLUSIONS........................................................................ 109 RECOMMENDATIONS .......................................................................................... 110 REFERENCES.............................................................................................................. 112 APPENDICES ............................................................................................................... 115 APPENDIX A: STATE DOT SURVEY .................................................................. 116 APPENDIX B: SYSTEM TESTING AND CURATION ....................................... 133 APPENDIX C: LESSON TAGS ............................................................................... 146 APPENDIX D: SOFTWARE SYSTEM SPECIFICATIONS ............................... 147
vi

LIST OF FIGURES
Figure 1. Process. Diagram for a Notification-based Lessons Learned Program....... 3 Figure 2. Photo. Aftermath of Big Dig Collapse (Cornado Common Sense, 2008) ... 8 Figure 3. Illustration. Schematic of Ceiling Tile (Ask MetaFilter, 2006) ................... 9 Figure 4. Map. DOT Respondents Map........................................................................ 11 Figure 5. Graph. Individuals with Access to Input Lessons Learned ........................ 15 Figure 6. Graph. Individuals with Access to Retrieve Lessons Learned ................... 16 Figure 7. Graph. LLPs' Effectiveness in Improving the Construction Process........ 16 Figure 8. Graph. LLPs' Benefit to the DOT and Office/Divisions ............................. 17 Figure 9. Graph. Receptiveness of Contractors to Inputting Lessons into the
Database ................................................................................................................... 18 Figure 10. Graph. Mistakes are Repeated Due to Change in Personnel or Lack of
Documentation......................................................................................................... 19 Figure 11. Graph. Challenges in Adopting an LLP ..................................................... 20 Figure 12. Graph. Documenting Lessons Learned Throughout the Project
Lifecycle.................................................................................................................... 21 Figure 13. Graph. LLP Database Capability for Push Notification for Users ......... 22 Figure 14. Graph. Need for Mobile Application Technology and Cloud
Computing................................................................................................................ 22 Figure 15. Graph. Need for LLP Database to have Ability to Upload and Catalog
Photos/Videos........................................................................................................... 23 Figure 16. Graph. Need for the LLP Database to have GPS Capabilities................. 24 Figure 17. Graph. Usefulness of Common Terminology............................................. 25 Figure 18. Process. KyTC's LLP Process (Goodrum et al., 2003).............................. 48 Figure 19. Process. KyTC's Lessons Learned Process (Goodrum et al., 2003)......... 49 Figure 20. Process. LLP Flowchart ............................................................................... 63 Figure 21. Process. Submission Process ........................................................................ 64 Figure 22. Procss. Curation Process.............................................................................. 65 Figure 23. Process. Notifications and Viewing ............................................................. 66 Figure 24. Image. Login and New User Registration Page ......................................... 75 Figure 25. Image. Submission Form Interface I .......................................................... 76 Figure 26. Image. Submission Form Interface II......................................................... 78 Figure 27. Image. Submission Form Interface III ....................................................... 80 Figure 28. Image. Submission Form Interface IV ....................................................... 81 Figure 29. Image. Lesson Management Interface I ..................................................... 82 Figure 30. Image. Lesson Management Interface II.................................................... 82 Figure 31. Image. Lesson Management Interface III .................................................. 83 Figure 32. Image. Lessons Available for Curating ...................................................... 85 Figure 33. Image. Lessons You are Curating ............................................................... 85
vii

Figure 34. Image. User Management Interface............................................................ 87 Figure 35. Image. Tag Management ............................................................................. 87 Figure 36. Image. Help Interface................................................................................... 88 Figure 37. Image. Profile Interface ............................................................................... 89 Figure 38. Image. Database Search Function............................................................... 90 Figure 39. Image. Lesson Map Interface ...................................................................... 92 Figure 40. Image. UGA/SSU Survey Results................................................................ 94 Figure 41. Image. GDOT Logo and Image ................................................................... 95 Figure 42. Image. Draft Submission.............................................................................. 99 Figure 43. Drawing. Image from Draft Lesson Submission...................................... 101
viii

LIST OF TABLES Table 1. State DOT Lessons Learned Programs.......................................................... 12 Table 2. Hypothesis on One Proportion........................................................................ 33 Table 3. Goodness of Fit Test......................................................................................... 34 Table 4. Types of Knowledge Artifacts (Fisher et. al., 1998) ...................................... 37 Table 5. Types of Knowledge Conversion (Sharif et al. 2005) .................................... 41 Table 6. GDOT Trial Test Survey Response .............................................................. 101
ix

EXECUTIVE SUMMARY
Documentation of past experiences through the use of a Lessons Learned Program (LLP) is instrumental to an organization's ability to improve by avoiding past mistakes while adapting to new solutions and methods. Feedback from project stakeholders through interviews and meetings, as well as a national survey of state department of transportation professionals was received and used to properly design and construct an LLP database system for the Georgia Department of Transportation. The Road COnstRuction Database (ReCORD) system was designed to include lesson creation, curation, and publication system interfaces with key features such as push notification, mobile compatibility, and global information system (GIS) capability. The ReCORD system prototype was trial tested with feedback received from survey responses and interviews. The evaluation process resulted in system modifications and improvements such as aesthetic changes, interface simplification, and the reorganization of lesson information. The finalized ReCORD system allows users to enter lesson information such as lesson title, lesson tags, abstract, location, GDOT projects associated with the lesson, lesson summary, references, and the ability to upload files. The retrieval interface allows system users to receive notifications when lessons associated with desired tags of interest to the user are published. Ultimately, the system designed for the Georgia Department of Transportation will be beneficial by providing a more efficient method of archiving and dispersing information.
x

ACKNOWLEDGMENTS The University of Georgia and Savannah State University acknowledge the financial support for this work provided by the Georgia Department of Transportation. The authors also thank the many GDOT personnel who assisted with this study. A special thanks is offered to John Hancock, P.E., GDOT State Construction Engineer; Beau Quarles, P.E., Assistant State Construction Engineer; Teague Buchanan, Assistant Administrator for IT Applications; and Brennan Roney, GDOT Research Manager. In addition, the authors thank the GDOT project managers who assisted with trail testing the ReCORD system.
xi

CHAPTER 1. INTRODUCTION
OVERVIEW For humanity to continue to advance as a society, past successes and failures must be realized, documented, and reported in a manner to inform others. This not only includes individuals' own successes and failures, but also those of the people around them. Certain procedures and methodology will yield specific patterns of successes and failures. To take advantage of this information, people must be afforded the opportunity to learn this information and integrate it in such a way that aids in the decision-making process. The lessons learned process should include an organized system that reports lessons assembled from a variety of sources given that the content is applicable to individuals trying to learn it. Indeed, learning from the experiences of others forgoes the need to learn by trial and error, which can be costly.
In a professional setting, an organized system of optimized learning is known as a Lessons Learned Information System (LLIS) or Lessons Learned Program (LLP). In the construction industry, there is a rising need for the use of these programs. With substantial growth in the construction sector in recent years, there is an opportunity for valuable information and experiences to be collected from successful and failed projects. LLPs help to properly capture and record this information. Furthermore, LLPs are able to more appropriately organize and disseminate information, known as "lessons," in an efficient manner through the use of an online, searchable database.
The Georgia Department of Transportation (GDOT) requested the development of a construction lessons learned database to store project-related information from its past, current, and future construction projects. GDOT intends to use this construction database
1

to not only document information from its projects, but to allow those working in all aspects of a project (i.e., concept, preliminary design, final design, programing and scheduling, and construction) to access information that may benefit a specific aspect of the project. The Road COnstRuction Database (ReCORD) system that was created as a product of this study will benefit GDOT and those working with the agency by providing information that is not only easily accessible, but pertinent and useful. The lessons learned database will benefit and improve the quality of current and future projects.
This study included a literature review examining the various LLPs and common practices within the construction industry, which the research team used to efficiently design and develop the ReCORD system. Furthermore, a survey was sent to each state department of transportation (DOT) to solicit ideas and feedback from potential users of the system. Additionally, database attributes were explored through this study, such as GIS capability allowing for the lessons learned (LLs) to have a spatial link across the state, photo/video capture and storage, and user communication.
The primary objective was to create a construction database to be used as an LLP and implement it within GDOT's standard practice. This study provides recommendations for implementing the LLP, as well as training for its long-term use and maintenance. Overall, the potential benefit of implementing an LLP to a large-scale organization such as GDOT is very high. When considering the number of construction projects GDOT manages, using an LLP to track important and relevant information may greatly increase both time and financial efficiency for future projects.
The process by which a new lesson is entered into the ReCORD system that resulted from this study includes three major phases (see figure 1): submission of the new
2

information, curation and refinement of the information, and publication of the lesson learned.
Figure 1. Process. Diagram for a Notification-based Lessons Learned Program RESEARCH SCOPE Chapter 2 includes the results and analysis of the responses from a national survey that the researchers conducted with state DOTs. A review of previous research related to lessons learned programs including benefits, challenges, and applications are included in Chapter 3. Chapter 4 documents the need for a lessons learned construction database and states the study objectives and expectations. The design methodology for creating the lessons learned database is documented within Chapter 5, and includes the lessons learned program methodology and database capabilities. Chapter 6 presents the outcomes of database development by documenting the specific details related to database functionality, structure, operation, and maintenance. Chapter 7 provides findings from in-house and
3

GDOT field trial evaluations of the lessons learned database through the examination of example lessons learned. Conclusions and recommendations are included in Chapter 8.
4

CHAPTER 2. BACKGROUND This chapter investigates two case studies of lessons learned reports and construction failures that yielded lessons learned. These studies aim to provide a background review of what potential lessons learned summaries should include when being published in the ReCORD system. Further, this chapter includes findings of a nationwide state DOT survey regarding the current use of lessons learned programs and what information potential users might want in a lessons learned program, should one be available.
LESSONS LEARNED CASE STUDIES
CDOT's US 36 Express Lanes
The Colorado Department of Transportation (CDOT) oversaw the construction of its US 36 Express Lanes project, carried out by AmesGranite Joint Venture. This project was divided into two phases with a budget of $317 million and $180 million, respectively. Phase 1 took three years to complete and yielded valuable lessons learned. It included an 11 mi (18 km) stretch of roadway reconstruction and included the construction of 18 detention ponds and replaced eight irrigation crossings. Additionally, the construction site included three separate Federal Emergency Management Agency (FEMA) classified floodplains.
During Phase 1 construction, temporary drains were installed, but in many cases the location of these drains was not properly documented. At the end of construction, these temporary drains were either forgotten or abandoned once permanent drains were installed. These abandoned drains served no purpose and only cluttered the project site. This lesson learned prompted a change in the outset of future Requests for Proposals (RFPs), which
5

required contractors to document the location of all temporary drains and include the location within the as-built plans. Additionally, it required the contractor to properly dispose of the temporary drains at the project closing.
In this project, not only was the lesson learned with respect to construction practices, but also regarding what information CDOT should include in future RFPs. The RFP for Phase 1 did not include a strict schedule or specific plan for temporary drainage plans; this lack of direction in the matter yielded issues that brought forth new lessons learned. In the future, the temporary drainage plans were to be tied in with the maintenance of traffic (MOT) plans to provide a clear strategy for temporary drainage until the permanent drainage was operational. Additionally, the temporary drain construction was required to be scheduled throughout the project life to avoid last-minute drain construction (CDOT, 2016). The following lessons were learned:
Drainage plans and accommodations must be included in the original RFP. The agency should be specific about its expectations for the drainage plans.
Boston's Big Dig Tunnel Collapse Boston's Big Dig was a megaproject designed to transform the road transportation system in Boston, Massachusetts. The project was spurred by the sheer number of cars that traveled through the Central Artery, a large interstate (I-93) running through the center of downtown Boston, that resulted in significant traffic delays for up to 16 hours each day. The project was completed in 2006 and included an 8 mi (13 km) long road consisting of 810 lanes, of which half of the roadways traveled through underground tunnels (MassDOT, n.d.).
A construction failure did not occur during construction, but rather after the road was operational. The engineers had failed to account for critical properties of polymer
6

adhesives. Epoxy adhesives are by nature viscoelastic and, therefore, are susceptible to creep. In the Big Dig project, the self weight of the tiles was sufficient to cause large longterm sustained loads. Thus, when the ceiling tile load was applied, deformation of the epoxy gradually increased over time. The deformation continued until a large tunnel ceiling panel fell on a car, resulting in the death of an individual. After an in-depth investigation, the cause of the failure was determined to be the result of creep in the epoxy anchor adhesive used to hold the ceiling panel in place. The weight of the ceiling tiles caused the adhesive to deform, resulting in the weakening of the bond strength over time. The anchor bolts were unable to hold the weight of the ceiling panel and the bond failed, causing the ceiling tile to collapse onto the traffic below. The investigation of the incident concluded that numerous tunnel ceiling panels were on the verge of failing, caused by creep in the adhesive. Fortunately, this issue was identified early enough to prevent any further damages or catastrophes. figure 2 and figure 3 show the damage caused by the falling ceiling tile. The following lessons learned were concluded from this case study:
Ultimate load tests should have been performed on the adhesive anchors prior to installation.
Installing adhesive anchors in an overhead application is problematic. This process makes it likely that voids will enter the bonding area, significantly reducing the tensile capacity of the epoxy.
New materials that are introduced to different applications most likely have properties that are not fully understood or addressed correctly. In this case, it added a new failure mode that resulted in tragedy.
7

The Massachusetts Turnpike Authority should have regularly inspected the tunnel ceiling panels for structural integrity.
Figure 2. Photo. Aftermath of Big Dig Collapse (Cornado Common Sense, 2008)
8

Figure 3. Illustration. Schematic of Ceiling Tile (Ask MetaFilter, 2006) STATE DOT SURVEY While GDOT has expressed interest in a lessons learned program in past years, the agency has not implemented a formal program that captures and disseminates lessons learned. In the past, GDOT had used a spreadsheet to track their project successes and failures; however, it was neither widely used nor effective.
To better understand how other agencies use LLPs, a national survey of state DOTs was conducted to determine their opinions about the use of LLPs. A statistical analysis was then performed on responses to determine and validate the importance of LLPs among the transportation construction community.
9

Survey Overview A survey was developed to investigate how state DOTs nationwide view LLPs, as well as information on states' LLP systems, if applicable. The survey intended to discover valuable information pertaining to data management, the transfer of information, and potential areas for increasing efficiency. Ultimately, information received from the survey was used in the design of the ReCORD system. The survey was designed to be simple and concise in order to maximize the number of responses generated. Qualtrics (https://www.qualtrics.com/), a web-based tool, was used to formulate and document the responses. The responses to this survey were expected to provide valuable information for use in this study and in developing an effective LLP system for use by GDOT. The respondents of the survey were state DOT employees who are members of the American Association of State Highway and Transportation Officials (AASHTO) Committee on Construction. These respondents are expected to be the heads of their departments or high-ranking employees within their departments at their state DOTs. A copy of the survey is provided in Appendix A. Survey Findings Survey responses were received from 28 of the 50 states and the District of Columbia, for a 55% return rate (see figure 4). In all, a total of 32 responses were received from the survey, with some state DOTs having multiple respondents. Two responses were received from the states of California, Kansas, Ohio, and Vermont.
10

Figure 4. Map. DOT Respondents Map
The states of South Dakota and Oklahoma did not complete the survey but responded via email that their DOT did not use an LLP. The Kentucky Transportation Cabinet (KyTC) responded via email that they did use an LLP; however, the KyTC database serving their LLP is not accessible by the public.
Question 1 was designed to inquire whether the DOTs responding to the survey utilized an LLP in their construction operations. A total of nine (32%) states replied that their DOT utilized an LLP. As shown in figure 4, Alaska, Colorado, Massachusetts, Delaware, Florida, Kentucky, Michigan, Virginia, and Washington were states that had implemented an LLP system. Additional questions were asked of these states (DOTs utilizing an LLP). Questions included:
How is the LLP managed (e.g., Excel spreadsheets, software, etc.)? Explain. How often and to what degree is the LLP used?
11

Is the LLP system freely available both internally and externally within the DOT?

The techniques in which LLPs are used within the nine states are summarized in table 1.

State Alaska
Colorado Delaware
Florida

Table 1. State DOT Lessons Learned Programs

Software Access Database

Description Top 10 list of LLPs is generated per construction season and then presented to design leaders. Small group meetings are held with the Department of Revenue and the design management team.

Access Partially Internal & External

Input

MS Word

Questionnaire format. Not used regularly; however, there is substantial awareness among DOT employees about the LLP.

Internal & External Access

Spreadsheet Format

Sporadic Use

Internal Use Only

Performance Management Section has complete control over entries and frequency of reporting.

SharePoint

LLP is in the form of a process review. The LLP is separated into different specialty areas such as structures, pavement, materials, and geotechnical.

Internal Use FHWA and its agencies executive team also have access.

Central Office Specialty Engineers & District Construction Engineers to a limited extent.

12

Table 1. State DOT Lessons Learned Programs (Cont.)

State

Software

Description

Access

Input

Kentucky

MS Access MS FrontPage

LLP is located at the Kentucky Transportation Center. LLP system is over 15 years old. System described in Section 3.3.1.4.

Massachusetts SharePoint

District offices are required to select 5 projects per year to include in the LLP. Post construction meetings are conducted on all projects to capture information about success and challenge and provide feedback for future projects.

Internally Available Only

Michigan

ProjectWise

Project meeting minutes with project stakeholders (designer, construction engineers, primary contractors and subcontractors are cataloged. Meetings are required of all projects with plan sets).

Internal Designers can retrieve information

Virginia

SharePoint

Internal Use Only Charge engineers responsible

Washington FileMaker

Use declined over the last few years. Attempting to revitalize the system.

Internal Use Only Previously internal and external use

To further understand the accessibility of users to input and retrieve lessons learned, the survey asked respondents additional questions regarding which individuals of their organization are able to input lessons and retrieve information from the database. As

13

figure 5 illustrates, individuals working as engineers, inspectors, or construction/project managers are most likely to have access to input lessons learned in a DOT's database. Fewer LLP programs allowed responses from maintenance personnel and contractor representatives. The Florida DOT (FDOT) stated that their Central Office specialty engineers and, to an extent, district construction engineers have access to input lessons learned. Further, the Delaware DOT (DelDOT) stated that the agency's Performance Management Section has complete control over the entries and frequency of reporting. Figure 6 provides the responses to the survey question asking DOT respondents which agency employees are capable of receiving and viewing the lessons learned. The responses to this question are similar to those with access to input lessons learned, with the respondents stating that engineers, inspectors, and construction managers have the majority of access. FDOT indicated that FHWA and their agency's executive team have access to retrieve lessons learned. Similarly, DelDOT indicated their Performance Management Section has the ability to receive and view the lessons learned.
14

4

3

# of Respondents

2

1

0

DOT

DOT

DOT

DOT Contractor

Engineers Inspectors ConstructionMaintenance Reps

Managers Personnel

Other

Figure 5. Graph. Individuals with Access to Input Lessons Learned Questions 2 through 5 of the survey asked respondents to rate their level of agreement on statements related to the effectiveness, benefits, and challenges of utilizing LLPs within DOTs. Survey question 2 asked whether "an LLP is effective in facilitation and improving the construction process." The responses, shown in figure 7, were overwhelmingly positive, with 28 of the 32 (88%) respondents agreeing that an LLP is an effective means to improve the construction process. The balance of respondents indicated that they neither agreed nor disagreed.

15

# of Respondents

7

6

5

4

3

2

1

0

DOT

DOT

DOT

DOT

Contractor

Engineers Inspectors Construction Maintenance Reps

Managers Personnel

Other

Figure 6. Graph. Individuals with Access to Retrieve Lessons Learned

# of Respondents

18 16 14 12 10
8 6 4 2 0
Strongly Agree

Agree

Somewhat Neither Somewhat Disagree Strongly

Agree Agree nor Disagree

Disagree

Disagree

Figure 7. Graph. LLPs' Effectiveness in Improving the Construction Process Survey question 3 asked respondents if "an LLP in the form of an archivable
construction database would benefit my organization and the work of my office or division." Similar to question 2, respondents agreed that LLPs would provide great benefit

16

to their agency. Shown in figure 8, 28 of the 32 (88%) respondents agreed with the statement, while only 4 (12%) respondents were indifferent. Respondents were asked in question 4 whether "the general contractors that my DOT typically works with would be receptive to regularly submitting "lessons" into the construction database." Responses to this question were much more varied than others. Illustrated in figure 9, 14 (44%) indicated they agreed with the statement, while 7 (22%) disagreed. Eleven (34%) respondents neither agreed nor disagreed that contractors would be receptive to inputting lessons into the database.

25

20

# of Respondents

15

10

5

0

Strongly Agree Somewhat Neither Somewhat Disagree Strongly

Agree

Agree Agree nor Disagree

Disagree

Disagree

Figure 8. Graph. LLPs' Benefit to the DOT and Office/Divisions

17

# of Respondents

12

10

8

6

4

2

0

Strongly Agree Somewhat Neither Somewhat Disagree Strongly

Agree

Agree Agree nor Disagree

Disagree

Disagree

Figure 9. Graph. Receptiveness of Contractors to Inputting Lessons into the Database
Survey question 5 sought to understand whether mistakes from previous construction projects (materials or procedures), which may occur due to a change in personnel or a lack of documentation, were often repeated. Reinforcing the need for an LLP construction database, figure 10 shows that 24 (75%) of the respondents indicated they believed that their agency made repeated mistakes as a result of a change in personnel or lack of documentation. Smaller percentages of the respondents, 4 (12%) and 4 (12%), indicated that they neither agreed nor disagreed or somewhat disagreed with the statement, respectively.

18

# of Respondents

16

14

12

10

8

6

4

2

0

Strongly Agree Somewhat Neither Somewhat Disagree Strongly

Agree

Agree Agree nor Disagree

Disagree

Disagree

Figure 10. Graph. Mistakes are Repeated Due to Change in Personnel or Lack of Documentation
Understanding an agency's challenges with adopting an LLP is important. Survey question 6 asked respondents, "What challenges would your organization have in implementing an LLP for construction?" Respondents could select multiple challenges as a part of the question. The results are shown in figure 11. Forgetfulness of using the LLP was selected by most respondents (17 of 32, 53%). Other responses, in descending order, included skepticism of using the LLP, lack of project similarities to relate lessons learned, lack of LLP effectiveness, informing DOT personnel of the LLP implementation, and personnel with the inability to operate the mobile app. Additional responses were provided that included:
"Not sure we are discussing apples to apples on this question. Ours is not an app or a queried database."
"They have many responsibilities and adding more may not yield beneficial results."

19

"Turf wars over ownership of the procedures and giving direction based on the outcomes of findings."
Figure 11. Graph. Challenges in Adopting an LLP Survey questions 7 through 11 examined the respondent's level of agreement regarding LLP database capabilities such as notifications, GPS, uploading and storage of photos and videos, and mobile/cloud computing. Question 7 investigated how lessons should be documented and what would be the most effective form of documenting information into the database throughout the project life cycle. Figure 12 shows that 24 out of 30 (80%) responded positively that having information related to lessons documented throughout the project life cycle would be most effective. Only two individuals disagreed with the statement.
20

# of Respondents

16

14

12

10

8

6

4

2

0

Strongly Agree Somewhat Neither Somewhat Disagree Strongly

Agree

Agree Agree nor Disagree

Disagree

Disagree

Figure 12. Graph. Documenting Lessons Learned Throughout the Project Lifecycle Survey question 8 related to the database having the capability for push notification.
This would allow users of the database to be notified when lessons learned specific to the individual's interest are entered into the system. When answering the question "it would be useful for each user of the database to have a profile with specific field interests such that they can be notified when a relevant `Lesson' is entered into the system," 23 of 30, (77%) respondents agreed (see figure 13). In addition, 4 (13%) respondents neither agreed nor disagreed while 3 (10%) respondents somewhat disagreed.
Because of GDOT's e-Construction initiative, field engineers have the ability to complete more site-specific documentation via mobile devices. Question 9 investigated the usefulness of having the LLP accessible through mobile application technology and cloud computing, such that field personnel are able to access it on the construction site. As illustrated in figure 14, 24 of 30 (80%) respondents indicated that mobile application technology and cloud computing would be beneficial to the usage of the LLP database. Only 2 (7%) of the respondents somewhat disagreed.

21

# of Respondents

12

10

8

6

4

2

0

Strongly Agree Somewhat Neither Somewhat Disagree Strongly

Agree

Agree Agree nor Disagree

Disagree

Disagree

Figure 13. Graph. LLP Database Capability for Push Notification for Users

# of Respondents

12

10

8

6

4

2

0

Strongly Agree Somewhat Neither Somewhat Disagree Strongly

Agree

Agree Agree nor Disagree

Disagree

Disagree

Figure 14. Graph. Need for Mobile Application Technology and Cloud Computing For many instances, there is a need for photographic or video evidence to document
the lesson learned. Question 10 of the survey examined the perceived usefulness of the LLP database having the ability of uploading and cataloging photographs of the lessons learned. Most respondents indicated they agreed with the need to have photo/video

22

uploading capabilities, with 29 (97%) positive responses (see figure 15). No respondents disagreed with the need for the database to have such capabilities.
Survey question 11 asked the respondents their agreement regarding the usefulness of the LLP database having global positioning system (GPS) capabilities in order to identify project locations for the lessons learned. Results are shown in figure 16.

# of Respondents

18

16

14

12

10

8

6

4

2

0

Strongly Agree Somewhat Neither Somewhat Disagree Strongly

Agree

Agree Agree nor Disagree

Disagree

Disagree

Figure 15. Graph. Need for LLP Database to have Ability to Upload and Catalog Photos/Videos

23

# of Respondents

12

10

8

6

4

2

0

Strongly Agree Somewhat Neither Somewhat Disagree Strongly

Agree

Agree Agree nor Disagree

Disagree

Disagree

Figure 16. Graph. Need for the LLP Database to have GPS Capabilities

Responses to the question were mixed. Eighteen (60%) of the respondents believed the use of GPS would be beneficial to the LLP database; however, 10 (33%) neither agreed nor disagreed with its use, while 2 (7%) of the respondents disagreed. As a result of the breadth of work that is completed through transportation construction projects, terminology was identified as being a potential challenge. Survey question 12 asked respondents whether the grouping of terms of similar or exact meaning for the input and the retrieval process would be useful. Specifically, the question read:
"Considering LLPs will be used by individuals from various regions, differing terminology may become an issue during the entering/retrieval process. For example, a process or piece of equipment could be referred to by multiple terms. As a user looking to retrieve information from the database, this can cause problems while searching key terms. Would it be useful if the LLP system grouped terms of similar or exact meaning for both input and retrieval processes? (For example, searches for "mesh" would

24

# of Respondents

also show results for "welded wire reinforcement" and "welded wire fabric")."
Results are illustrated in figure 17. Similar to past questions, an overwhelmingly positive response was observed for the use of terminology commonality. In total, 26 (87%) of the respondents felt it would be useful to group terms with common meanings. Only 4 (13%) were indifferent with the question.
14 12 10
8 6 4 2 0
Extremely Moderately Slightly Neither Slightly Moderately Extremely Useful Useful Useful Useful nor Useless Useless Useless Useless
Figure 17. Graph. Usefulness of Common Terminology Question 13 asked respondents for additional input regarding terminology in LLP systems. Suggestions included: Who would maintain this list? (CDOT)
25

The system is currently designed to require all submittals be reviewed and approved by a curator, who coordinates with subject-matter experts (SMEs) and the Attorney General's office, if required, prior to posting. The curator would review and make any final edits; this could help eliminate inconsistencies in terminology (Washington State Department of Transportation [WSDOT])
Maybe allow for descriptors (District of Columbia)
Question 14 asked respondents for additional information that may aid in the development of the ReCORD system for GDOT. Input was received from the Vermont Agency of Transportation (VTrans), District of Columbia Department of Transportation (DCDOT), Rhode Island Department of Transportation (RIDOT), Alaska Department of Transportation and Public Facilities (ADOT&PF), Delaware Department of Transportation, and Virginia Department of Transportation (VDOT). VTrans stated that LLPs would be most beneficial to their in-house design teams and their consultants. DCDOT recommended the use of visual aids, printables, pictures, and anything that would easily add value. RIDOT had attempted to utilize a lessons learned program; however, the "buy in" of the value of lessons learned during planning and design was limited. The ADOT&PF stated that LLPs are only useful if they are integrated into the design/construction process. They found that one-on-one post-construction meetings with the design squad resulted in the process of sharing lessons learned information. DelDOT mentioned that navigating bureaucracy within a DOT is the most significant cause of recurring construction problems; consequently, finding the best use for the results is the most difficult part of getting something like this to work. Additionally, DelDOT believes
26

that the most useful comments are well written paragraphs describing the exact situation. VDOT stated that they transitioned from an LLP to a scenario-based guidance.
State DOT Response Summary and Conclusions
Current LLP Usage The national survey of state DOTs resulted in a 56% (28 states) response rate. In total, 9 of the 28 states reported using an LLP, while others did not actively use an LLP; however, some expressed some interest. The LLPs were primarily managed through Microsoft Office software such as SharePoint, Microsoft Access, and spreadsheets. The Michigan Department of Transportation (MDOT) uses the Bentley System, ProjectWise, while WSDOT uses a FileMaker database. Overall, the level of use varied from state to state, with the majority of respondents experiencing little use of LLPs. Findings from the survey suggest that it would be beneficial to implement the LLP in such a way that would condition users to utilize the program frequently and consistently. Most of the LLPs are only available internally to DOT users. DOT engineers, inspectors, construction managers, and maintenance personnel were listed as the primary individuals allowed to input lessons into the database. The Performance Management Section and the Central Office specialty engineers were able to input lessons into the LLP for FDOT and DelDOT, respectively. DOT engineers, inspectors, construction managers, and maintenance personnel were the most common responses for the people allowed to retrieve lessons from the database. Two other responses indicated access by FHWA and DOT executive teams, as well as the Performance Management Section for one of the DOTs.
27

LLP Challenges Respondents agreed that construction mistakes are normally caused by change in personnel or lack of documentation. This indicates room for improvement and the opportunity to reduce the number of repeated errors in the construction field. An effective LLP is certain to mitigate these errors. DOT respondents indicated that the greatest challenges to the effective use of an LLP are skepticism of the LLP, forgetting to use the LLP, and lack of LLP effectiveness. The majority of these issues are related to the organizational behavior and management. With increased use and adoption within the organization, skepticism will fade and individuals will habitually incorporate the LLP into their normal routine. Another concern is the user's perception that the extra time spent on the LLP is not worth the benefits. Most of the respondents agreed that documenting information throughout the project life cycle was the best method to gather lessons learned information. This process of collecting information throughout the project lifecycle is expected to minimize the amount of important information lost in the confusion of other ongoing tasks.
Most states agreed that an LLP is effective in facilitating and improving the construction process. Only a couple of responses indicated indifference about LLP usage. Similarly, there were positive responses from individuals who believed that an LLP would personally benefit them if applied to their office or division. There was a high variance in the responses regarding whether contractors would be receptive to incorporating the LLP system into their routines and regularly submit lessons into the LLP. These results identify potential issues in implementing the LLP system.
28

LLP Attributes Overall, there was positive support for the use of LLPs. While only 9 out of the 28 state DOTs that responded currently have their own LLPs, many of the states indicated that this would be helpful in improving their current procedures/policies and would reduce the mistakes repeated throughout the design and construction process, which result in loss of time and money. While all LLPs are similar in nature and describe the problem and how that problem might be solved, based on the survey it was suggested that LLPs should also have the following attributes:
a) Notification System A notification system would keep the users actively involved in the LLP, as well as notifying them about topics of interest posted on the LLP.
b) Mobile Access Mobile access would allow information to be submitted to the LLP in the field.
c) Inclusion of Photographs Photographs would help to communicate information and bring a new perspective to the LLP.
d) GIS Capabilities This would allow for location of the lesson learned. e) Terminology grouping Grouping of similar terms would be helpful not only
for input but also for retrieval. It would also create a better experience for the user because of consistent use of terms within lessons.
The research team used the results from this survey to help design a construction LLP for GDOT. Ultimately, the survey was successful in identifying a solid foundation of information from which to begin outlining the basis for the ReCORD system. In addition, use of an LLP would not only be beneficial to GDOT, but many other state DOTs.
29

Statistical Analysis
Data Description The information collected from over 50% of the state DOTs through a national survey was used in a statistical analysis to quantify the need and challenges of a construction LLP database. Calculations performed and figures illustrating the results in this section were generated with the program JMP Pro 13.
Research Questions The fundamental questions examined through the statistical analysis were the following:
Do the majority of state DOT employees agree that an LLP is effective in facilitating and improving the construction process?
Do the majority of state DOT employees agree that an LLP in the form of an archivable construction database would benefit the organization and the work of their office or division?
Do the majority of state DOT employees agree that mistakes from previous construction projects are repeated?
Do the majority of state DOT employees agree that the most effective form of documenting information for input into the database is in increments throughout the project life cycle?
Do the majority of state DOT employees agree that it would be useful for each user of the database to have a profile with specific field interests such that they can be notified when a relevant "Lesson" is entered into the system?
30

Do the majority of state DOT employees agree that it would be useful to have the LLP accessible through mobile application technology and cloud computing such that field personnel are able to access it on the construction site?
Do the majority of state DOT employees agree that it would be useful if the LLP database had the ability to upload and catalog photographs and videos of the lessons learned?
Do the majority of state DOT employees agree that it would be useful if the LLP system grouped terms of similar or exact meaning for both input and retrieval processes?
Are the proportions of state DOT employees the same that either agree, neither agree nor disagree, or disagree that contractors will be willing to cooperate with the use of an LLP the same?
Are the proportions of state DOT employees the same that either agree, neither agree nor disagree, or disagree that GPS capability would be useful?
Data Population Construction representatives from each of the state DOTs nationwide received the survey via email. Thirty-two responses (n = 32) were received from these state DOT employees. Two responses were received from four of the state DOTs. Two states chose not to respond to the last five questions of the survey.
31

Analysis Each of the survey responses was analyzed by one of two tests. The first test is known as the "hypothesis on one proportion" test. The test uses a null hypothesis of H0 = 0.5, an alternative hypothesis of Ha >0.5, and a level of significance of 0.05. The results of these tests are shown in table 2. The second test conducted was the goodness of fit test. It was conducted on survey questions that yielded a high range of answers. This test classified the responses into three different categories: agree, neither agree nor disagree, and disagree. This test determines whether there is a significant difference between each of the response groups. These test results are shown in table 3. Appendix A provides the detailed survey analysis.
Statistical Analysis Conclusion The statistical analysis performed in this section confirms that an LLP construction database would benefit GDOT and other state DOTs. The state DOT employees believe that there is a need to improve the construction process because mistakes are being repeated. Additionally, the findings indicate that DOT employees believe that their office and the entirety of their DOT would benefit from the use of an effective LLP. The only discrepancies lie within how the LLP should be used. The DOT employees had varied responses on the usefulness of GPS capability and the level of contractor cooperation. The inclusion of GPS capability will not harm the effectiveness of the LLP; it can only improve or not affect its working capacity. However, the DOT should expect varying levels of contractor cooperation, which have the potential to complicate the process.
32

Table 2. Hypothesis on One Proportion

Statistical Questions
Do the majority of state DOT employees agree that an LLP is effective in improving the construction process?

Is the proportion significantly different
enough to be considered the
majority?
Yes

Magnitude of agreement
28/32

Do the majority of state DOT employees

agree that an LLP would benefit the DOT and its divisions?

Yes

28/32

Do the majority of state DOT employees

agree that mistakes are repeated due to change in personnel or lack of

Yes

documentation?

Do the majority of state DOT employees

agree that lessons should be documented throughout the project life cycle?

Yes

24/32 24/30

Do the majority of state DOT employees

agree that push notifications would be beneficial for users?

Yes

23/30

Do the majority of state DOT employees

agree that there is a need for mobile application technology?

Yes

24/30

Do the majority of state DOT employees

agree that there is a need for the LLP to have the ability to upload and catalog

Yes

photos and videos?

Do the majority of state DOT employees

agree that it would be useful to have grouping based on common terminology?

Yes

29/30 26/30

33

Table 3. Goodness of Fit Test

Statistical Questions
How receptive will the contractor be to inputting lessons into the database?

Is there a significant difference in proportion of at least two response
groups?

Magnitude (Agree / Neither Agree nor Disagree / Disagree)

No

14 / 11 / 7

Would GPS capability be

useful for the LLP?

Yes

18 / 10 / 2

34

CHAPTER 3. LITERATURE REVIEW
OVERVIEW OF LLPS
Definition A lesson learned is an experience gained by the success or failure of a specific task. When this occurs, knowledge is gained by the individual or group of individuals who worked on the task. Knowledge is defined as "a fluid mix of framed experience, values, contextual information, and expert insight that provides a framework for evaluating and incorporating new experiences and information" (Davenport and Prusak, 1998).
Originally, lessons learned were understood to be guidelines, tips, or a checklist of what went right or wrong (Stewart, 1997). However, in a modern society, guidelines, tips, and a checklist are normally not enough to shift an organization's behavior and reprogram people to have the appropriate working habits and processes. Lessons learned systems have adopted an acceptance criteria for each lesson in order to validate its importance and effectiveness. To be effective, the results must be considered. Some authors make this point in distinguishing a lessons learned. A lesson learned could be defined as the change that results from successfully applying past knowledge. Others argue that stored lessons in a database are simply "identified lessons" (Weber et. al., 2001), as they are not lessons learned because they have yet to be applied. While the lessons have potential value, they are not worth real value until they can be successfully used in a future application. Weber et. al. points to the fact that implementation is the real driver for an effective LLP.
The most complete definition of a lesson learned is: "A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, such as a mishap or
35

failure. In addition, successes are considered sources of lessons learned. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result" (Weber et. al., 2001).
Overall, a successful LLP will promote knowledge reuse, conversion, and sharing in an efficient manner. LLPs have been implemented to serve many purposes, some of which are to avoid wasting resources, protect the safety of employees/clients, and survive and learn. Regardless, LLPs are used to refine an organizations' operations to help achieve the organization's goal. (Weber et. al., 2001)
Distinguishing LLPs from Other Knowledge Artifacts Knowledge artifacts are sources of information that are used solely for individuals or dispersed throughout a large group, organization, or industry. Common knowledge artifacts are lessons learned, incident reports, alerts, corporate memories, and best practices. These knowledge artifacts are compared in table 4 and summarized as follows:
Incident reports normally describe an unsuccessful experience that is used as a manner of documenting safety and accident investigations. They do not typically provide suggestions or recommendations for improvement.
Alerts are similar to incident reports in that they are used to report negative occurrences. However, alerts are used specifically to bring attention to a specific technology or area of technologies that is problematic.
36

Corporate memories do not have a "set in stone" definition. These could be understood to be a repository of different types of knowledge artifacts that all contribute to the understanding and level of knowledge for a specific organization.
Best practices are knowledge artifacts only described as positive experiences with the goal to improve specific activities within an industry. Best practices are the standard to which all processes and tasks should be held. Table 4. Types of Knowledge Artifacts (Fisher et. al., 1998)
Some lessons learned systems combine different knowledge artifacts as input lessons into the database. From the perspective of the user, a mixture of different knowledge artifacts is detrimental, as it results in confusion and makes it more difficult to locate relevant information. Naturally, the solution is to only allow inputs into the LLP in the form of a lessons learned. This formatting consistency will allow users to more quickly
37

familiarize themselves with the thought process of the lessons learned and expedite the learning process.
Additionally, for the ease of verification, each input into the LLP should only have one clear lesson to convey. An input with multiple lessons will not only be complicated to verify and input into the system, but can result in unnecessary confusion. If two lessons are closely tied together, they should be input separately, but have a hyperlink within the file to other closely related lessons (Fisher et. al., 1998).
OPERATIONS OF LLPS
Psychological Aspect According to the book The Power of Habit, 40% of the actions that individuals
make are ascribed to habit (Shedd, 2013). Therefore, many of the decisions made on a dayto-day basis are based on habit and not on deliberate decisions. For an LLP to work effectively, it must be used on a continuous basis and become a habit. An organization that continuously makes using an effective LLP part of their process will realize positive results.
Internalization of Information by Repetition In terms of an LLP, a stimulus is an event that evokes a specific functional reaction. In other words, the stimulus is the incident from which a lesson learned can be synthesized. Repetition of stimulus plays a dual role in the internalization of knowledge: repeated exposure to the same stimulus maintains the information in the primary (short-term) memory, as well as stores the information in the secondary (long-term) memory. Exposure
38

time of information within the primary memory correlates to the internalization of information into the secondary long-term memory (Chabot et al. 1976).
As this relates to the current study, not only can the ReCORD system provide benefits to the users who retrieve information from it, but it can also be helpful to those individuals who are submitting the lessons for review. When a significant success or mistake is made on the jobsite, it is mentally noted and registered to an individual's primary memory. In the proposed process for the ReCORD system, the mistake/experience or success is then to be physically documented and inputted into the database system. This process draws from the wisdom of the old adage "learn from your mistakes," and to do so, it is first necessary to admit that a mistake has occurred. When the the one submitting the lesson documents a mistake, that is the first step of admitting an error. Meanwhile, the process of documenting the mistake additionally provides for a prolonged exposure to the stimulus of information, which facilitates the transfer of information from the primary memory into the secondary memory of the person submitting. Similarly, when a new process or idea is successfully implemented in the field, it is documented within the primary memory of the individual and input into the ReCORD system. The dual nature of memory storage is employed here through the long-term memory of the individual and the storage of information within the database.
Information Types Memory is split into two broad, distinct categories: declarative memory and nondeclarative (or procedural) memory. Declarative memory is the conscious memory that allows individuals to deliberately recall information, whereas nondeclarative memory is much more subtle. Nondeclarative memory is accessed without consciousness, such as the use of
39

motor skills that have been indoctrinated into muscle memory (Curran and Morgan, 2014). By nature, the information that is committed into nondeclarative memory has a higher retention than declarative memory. For example, an individual is more likely to forget a person's name or phone number rather than forget how to tie their shoes or how to ride a bicycle.
Regarding the application of these memory attributes to LLPs, two types of information critical to the use and effectiveness of LLPs are tacit knowledge and explicit knowledge (Dixon, 2000; Inkpen, 1996; Polayni, 2009; Nonaka and Takeuchi, 1995; Von Krogh et al., 2000). Explicit knowledge is straightforward information that is easily transferred or understood, such as a name or phone number. It is easily transferred between people. Tacit knowledge is a more emotional and intuition-based state of awareness. Further, it is linked to personal perspective, experiences, and values. Tacit knowledge is more challenging to articulate or communicate to others.
The requirement to convert knowledge from one form (tacit to explicit or vice versa) to another while distributing information to multiple parties/persons is a demand on any LLP system. Table 5 depicts the methods of knowledge conversion (Sharif et al., 2005).
40

Table 5. Types of Knowledge Conversion (Sharif et al. 2005)
Based upon the theory presented in table 5, there are four types of knowledge transfer: socialization, externalization, internalization, and a combination of methods. Socialization is the method of knowledge transfer that allows for communication of tacit knowledge from person to person. By nature, this is the most difficult of the four methods of knowledge transfer. Tacit knowledge requires time to accumulate and truly understand what is being learned. Thus, it is a challenge to transfer the internalized knowledge of one party and implant it into another. The transfer of knowledge requires the process of socialization; the parties must adequately communicate their tacit knowledge in a specific and detailed manner in order for the information to be properly understood and learned as tacit knowledge.
Externalization is the method of knowledge transfer that converts tacit knowledge into explicit knowledge. This conversion happens within an individual or organization before it can be shared with others. If the knowledge being converted is already tacit knowledge, the conversion process is as simple as vocalizing the knowledge or information. The process must be a deliberate one. Conversion of tacit to explicit knowledge will require introspection of some degree in order to deliberately externalize the implicit knowledge into tangible facts and figures. Once the knowledge is externalized into explicit knowledge, it becomes easier to transfer to other parties.
41

Internalization is the process of converting knowledge from explicit to tacit knowledge. The knowledge must already be possessed as explicit knowledge. Internalization requires a process such as implementation and application of the knowledge. This process is a critical aspect to the type of knowledge conversion that the ReCORD database plans to promote and facilitate.
Combination refers to a multitude of communication avenues to convey a message. Since this method includes the transfer of explicit knowledge from one party to another, it is the simplest process. A prime example of combination is the sharing of a phone number or name with another individual. The sharing of this information occurs in a multitude of manners such as face-to-face meetings, email, text, phone, and other means. The most effective LLP is capable of promoting all types of information conversion and transfer. The input system of the ReCORD system is configured in such a manner to allow for the transfer of explicit knowledge and also promotes two-way knowledge tacit conversion through its documentation process.
Benefits For maximum efficiency and effectiveness, the use of an LLP must be incorporated into an organization's regular and routine system of operations. However, in many cases, LLPs fail and fall out of favor because the LLP is not regularly utilized. However, if an LLP system is used regularly, then it can provide desirable results, such as improved quality, increased working efficiency, and a reduction in costs.
Challenges The Construction Industry Institute's (CII) modeling lessons learned research team conducted a study of 2400 organizations and their use of an LLP. From this study,
42

50 distinct LLP types were identified. CII selected 25 organizations for a more detailed investigation. None of the 2400 lessons learned systems examined in the study used any type of process that proactively "pushed" existing lessons to users who would potentially find them helpful (Fisher et al., 1998). The most reasonable explanations are that either the software used was unable to reinforce the process, the newly found lessons were input into a best practice manual and became untraceable from there, or that the lessons were only presented back to those who made mistakes in the first place. Regardless of the reason, the ability of an LLP to send push notifications to users will make this tool more robust and useful.
In general, LLPs will fall short of their full potential because of two distinct reasons. First, the quality of the lessons within the system are not satisfactory. This usually occurs because the lesson learned does not adequately describe the situations to which it can be applied. Secondly, the performance of an LLP must be strongly linked to the organization's decision-making process. For an LLP to be effective, it must be capable of changing the habitual patterns of the organization as a whole. (Aha, et. al., 1999).
Secutor Solutions is a company that creates lessons learned programs for its clients. According to Bill Brown, the blogger for Secutor Solutions, the three main reasons that LLPs fail are lack of content, poor lesson quality, and inability to efficiently retrieve information. If the LLP does not have a sufficient amount of relevant information, then users will believe the LLP does not contain sufficient information to make repeat usage time worthwhile. Even if the system is able to effectively deliver lessons learned, the quality of the lesson learned must be sufficient to provide useful information for the user. A successful lessons learned program must also have an efficient method whereby
43

individuals can retrieve data. Individuals retrieve data in all different manners (e.g., Boolean logic, browsing topics/categories, timelines). Thus, it is important to have multiple search methods in order for individual users to have a method that is convenient for them (Brown, 2016a and b).
In 2006, a survey was conducted by Ernst and Young for the Project Management Institute about their views on LLPs (Marlin, 2008). Of the respondents, 91% believed that lessons learned are important to the development of the organization; however, only 13% stated that LLPs were used within their own organization. Additionally, only 8% of respondents replied that the point of the LLP was to understand potential benefits and improvements that would help their company grow.
The literature confirms the trend that most people believe LLPs to be useful. However, why are LLPs not more commonly used? A major reason for this trend lies within the implementation of the LLP. There are numerous barriers that contribute to the hindrance of an effective LLP implementation, including:
being unable to find relevant information within the database; the user's belief that using the LLP is a waste of time because it does not
directly contribute to their project; the belief that an LLP is just a "blame game"; the belief that the LLP process is more trouble than it is worth (i.e., too
complicated to use, wastes too much time); and the belief that the LLP is too ideal, something that management only talks
about, but is never able to execute.
44

Implementation The ability of an LLP user to efficiently locate and retrieve information rests with the program administrator. It is the administrator's responsibility to create a simplified system for the users that can be easily searched. The creators of any LLP database should be encouraged to examine programs that future users are currently using and adopt similar features for the LLP, as this will shorten the learning curve. The ReCORD system will have proactive features that are designed to bring newly published lessons to the user. Each user will have their own profile that will include "tags" that identify their specialty/interest area. When lessons are input into the LLP, it will contain tags for relevant fields. Users with the same "tags" as the new lesson will receive a notification to alert them of the lesson, thereby saving the user time by eliminating the need to search for LLs themselves.
Leadership is a critical aspect of instituting an effective LLP. "The lack of leadership involvement in and commitment to the learning process is a critical barrier," asserted Dressler and Palin (2007). Leadership is required to change the attitude of workers and foster the understanding that an LLP is not a "blame game" but rather an acknowledgment that mistakes sometimes do happen and what solutions were used to correct the mistake such that it does not happen again on other projects. This philosophy allows for a more direct method of communication and expedites the problem-solving process. People are often reluctant to admit mistakes because it makes them vulnerable to criticism and puts their pride, reputation, and, quite possibly, their career at risk. However, leadership committed to the LLP process has the potential to reshape the attitude paradigm of an organization.
45

With emphasis placed on creating a simplistic LLP, the study authors recommend that only a single knowledge artifact should be contained within a single lesson. If a single lesson learned contains multiple knowledge artifacts derived from a single project, then the overabundance of information and issues may overwhelm the user. An argument could be made that by keeping each lesson from a single project separate, it will create too many lessons. However, by separating the lessons from one another it ensures that each LL will communicate its core message.
APPLICATIONS OF LLPS
LL Systems The organizations discussed in the following sections have an established LLP. These LLPs range in size and complexity from the federal level down to the state level and are used for different purposes.
NASA The National Aeronautics and Space Administration (NASA) has a lessons learned database known as the Lessons Learned Information System (LLIS). The LLIS is a centralized compilation of reviewed and edited lessons learned that have been submitted by departments across NASA divisions and other organizations, and is used at all 10 NASA centers across the United States (Hoffpauir, 2015). The LLIS was originally established to avoid safety mishaps. It is mostly accessible to the public, with limited access to some files reserved only for NASA employees.
46

U.S. Army The U.S. Army's Center for Army Lessons Learned (CALL) offers a five-day course for Army officers. The course covers information that has been validated through experience and testing. CALL defines lessons learned as validated knowledge and experience derived from observations and the historical study of military training, exercises, and combat operations that has led to a change in behavior at either: (1) the tactical, operational, and strategic levels; or (2) in one or more of the Army's DOTMLPF (Doctrine, Organization, Training, Material, Leadership and Education, Personnel, and Facilities) domains (Global Security, 2015).
U.S. Department of Transportation Currently, the US Department of Transportation (U.S. DOT) is revising their Lessons Learned Program. The most notable aspect about the U.S. DOT's process is that the lessons are written by a third party. The U.S. DOT process begins with the project information being submitted through regular progress reports and end-of-project reviews. Next, a team known as "scourers" searches the end-of-project reviews and compiles lessons for the database after project completion (Greer, E., personal interview, June 18, 2018). Because of this process, there is likely a disconnect between the individuals who experienced the lesson firsthand and those who summarize the lessons for entry into the database. The study authors recommended that GDOT not follow this model, but instead have individuals involved with the project submit lessons directly to the database and curator.
47

Kentucky's LLP The Kentucky Transportation Center, known as KyTC, currently uses a lessons learned database. The KyTC database emphasizes the importance of lesson curation, a centralized system, and integration with the planning process at the outset of construction. In 2003, KyTC conducted a study to formulate and build the Lessons Learned Database, as shown in figure 18). An important lesson that KyTC learned from other LLPs is to keep the lessons learned process as simple as possible. KyTC believes that if a complicated and in-depth lessons learned system is implemented, it has the potential to produce outstanding results, but this process does contain risk in that a complicated system might overwhelm the users and completely turn them away from the system. At that point, all of the work and resources invested into the system would be in vain. Based on KyTC's experience, it is recommended that the process be kept as simple as possible to target high user retention (Goodrum et al., 2003).
Figure 18. Process. KyTC's LLP Process (Goodrum et al., 2003) KyTC had four study objectives: Identify lessons learned systems currently used by other transportation
organizations and other industry organizations Define the desired functional capabilities of a lessons learned system for
KyTC 48

Develop a system design for a lessons learned system Recommend a lessons learned system for integration into KyTC's design and
construction process The process that KyTC used to develop its LLP system was applied to the creation of the ReCORD system. Each of the objectives was met by the ReCORD system project, as well as the creation, testing, and evaluation of the system. A more detailed conceptual design of the KyTC lessons learned process is shown in figure 19.
Figure 19. Process. KyTC's Lessons Learned Process (Goodrum et al., 2003)
49

Database Selection KyTC used MS Access as the database language, based purely on convenience. The software was already installed on the KyTC and the University of Kentucky's computers, since it came pre-installed with other MS Office products. Additionally, KyTC already used MS Access to store and display post-construction reviews conducted on its projects. The team decided to choose MS FrontPage to work as the front end of the database in terms of displaying information. This provided an added bonus of making the database internetbased, reducing the database access requirement to only having internet access.
Curation In the KyTC LLP process, a key part of the cabinet is the gatekeeper. The gatekeeper decides which lessons are worthy of being displayed in the database. Once a lesson is submitted to the database, the gatekeeper is notified. The duty of the gatekeeper is to edit the submitted lesson until it is ready to be put on display in the database. The gatekeeper can make changes to the lesson prior to publication into the system and can solicit help from other cabinet members to aid in the verification of the lesson.
LESSONS LEARNED FROM LLPS
Colorado Department of Transportation's T-REX Mega-Project The Colorado Department of Transportation oversaw a huge transportation expansion project, carried out by Kiewit, known as the Transportation Expansion (T-REX) Project. The project started in 2001 and was completed in 2006; it included 17 mi (27 km) of reconstructed interstate highway, 19 mi (31 km) of double-track rail transit with 13 stations, and construction/reconstruction of 61 roadway bridges (I-25 T-REX Project, n.d.). The
50

massively successful project was attributed to a handful of "keys to success." These keys included:
Remember: "project comes first" Develop a well-defined contract Develop and focus on project goals from the outset Update at the start of major phases
The primary key to the success of the project was the attitude of the team and their positive work environment. This attitude allowed them to prioritize the project over an individual. Their motto was: "One Team, One Voice." It helped the team think in a unified manner that led to mutual respect among themselves. Furthermore, the team co-located all of the T-REX Project team members.
Another key to success was attributed to a well-defined contract. In order to develop a well-defined contract, the team invested time and resources into researching other similar projects and incorporating their lessons learned. This research allowed the team to avoid the mistakes of others while capitalizing on the methods and procedures that worked well. A well-defined contract set proper expectations for the contractors and the client. Realistic expectations from both parties eliminated miscommunication down the line, as well as laid out a standard that could be striven for.
In addition to having a well-defined contract, clear project goals kept the project moving in a positive direction. In the T-REX Project, the goals were updated at the start of all major phases or when situations arose that significantly restricted or complicated the tasks at hand. Updating the goals as the project progressed allowed the entire team to be on the same page (Federal Transit Administration, 2007).
51

Virginia Department of Transportation's Program On April 23, 2009, Thomas Pelnik, P.E., delivered a presentation that took an introspective view on the Virginia Department of Transportation's past projects. The presentation eventually made recommendations for moving forward with VDOT's construction projects:
Begin at the beginning Focus on what you want at the end
"Begin at the beginning" referred to having a clear idea of what needed to be done and planning out tasks. The policy objectives were noted and taken into consideration at all times during both design and construction. Additionally, the planning process was required to consider the cost/benefit of the project, potential risks, and project delivery method.
Focusing on the end goal not only referred to the completion of the project; more importantly, it included the long-term performance, much later than just the project closeout. This included accounting for the project's life cycle, operation, and maintenance. Current construction practices separate construction and maintenance, yet the two are so closely related and dependent upon each other. Maintenance is just as important as the construction itself; otherwise, the construction effort would soon go to waste (Pelnik, 2009).
Pennsylvania Department of Transportation The Pennsylvania Department of Transportation (PennDOT) has an interactive map that marks the location of both ongoing and planned construction projects (PennDOT, 2018). When a marker on the map is clicked, a new page is opened that provides project-specific
52

information such as the contractor name, contact information, type of work being performed, and cost. The interactive map is a simple and effective way to convey project spatial information.
SUMMARY OF LLPS LLPs have the potential to help an organization improve its processes and tasks in order to increase efficiency by saving money, ensuring worker safety, reducing training times, etc. An LLP is the most complete knowledge artifact to learn and gain experience from. LLPs are effective because they emphasize internalization by repetition, as well as promoting multiple types of knowledge conversion and transfer. Based on the findings of the literature review documented in this chapter, the majority of LLP failures are the result of the following issues:
Poor lesson quality / lack of content Inability to recall the lesson properly (during input) Lack of integration of LLP into the decision-making process Lack of cooperation from users or lack of the users' belief in the system Poor administration (curating of lessons) Misuse or lack of overall habitual use Overly complicated input, verification/validation, or retrieval system
A successful LLP requires exceptional leadership and implementation, such that workers feel safe enough to openly admit mistakes. NASA's LLIS is an excellent model for the ReCORD database because it utilizes push notifications and features effective methods of organizing and curating lessons learned.
53

CHAPTER 4. PROBLEM STATEMENT
RESEARCH SIGNIFICANCE LLPs have been implemented by large and successful publicly funded agencies such as the U.S. Army, U.S. DOT, Environmental Protection Agency (EPA), and NASA. The LLPs created and maintained by these organizations typically include database platforms that allow individuals to both store and access information relating to a specific field of interest within their industry. The storage of valuable lessons learned information in an easily accessible database is convenient and advantageous. The process allows for users of the database to access the data in order to learn in advance from the successes and mistakes of past projects, which ultimately results in the saving of time, money and resources. LLP users avoid the same mistakes while implementing successful practices.
The Construction Industry Institute describes LLPs as "essential to the construction industry." As more and more experienced professionals retire from the workforce, documentation of worker expertise is even more valuable. If the information is not recorded or documented, then it will essentially be lost. However, when the information is documented, it can serve as a guideline for the newer, less experienced workers. Additionally, with the growth in publicprivate partnerships, the roles of individuals are also changing within the construction industry.
RESEARCH OBJECTIVES GDOT has never had a formalized LLP. Previously, GDOT utilized a spreadsheet to keep track of project data; however, it was neither widely used nor effective. The development of the ReCORD system will help to unify GDOT's large workforce and footprint throughout the state of Georgia under one system for capturing and reporting of lessons
54

learned. The system will have a systematic and effective method of curating and retrieving lessons learned. As with the adoption of any new method, it is expected that a well thought out and strategic implementation plan should be created to provide support and guidance for the curating and long-term management of the database system. Furthermore, the implementation must illustrate the significance and benefit of the system to its users. The ReCORD system will facilitate the essential task of retaining and dispersing knowledge throughout GDOT, leading to improved design and construction-related decisions being made by employees, thereby, leading to higher working efficiencies throughout the life cycle of projects and organization.
55

CHAPTER 5. EXPERIMENTAL PLAN
The research team established the framework for the ReCORD system through the integration of information collected from the national survey of state DOTs, comprehensive literature review, personal interviews with curators of other industry-based LLPs, and recommendations from GDOT. The design and development plan for the ReCORD system in this study consisted of preliminary planning and process development, conceptual design, product development, and trial testing. A major aspect of this plan is the trial evaluation phase that involves overseeing the use of the ReCORD system with several lessons learned to verify the LLP is operating correctly and producing the results expected by GDOT.
PRELIMINARY PLANNING AND PROCESS DEVELOPMENT PHASE
GDOT Pre-Planning Meeting On March 9, 2018, a meeting was held with GDOT to discuss the ReCORD system framework. This meeting marked the end of the preliminary information acquisition and transitioned into the establishment of the ReCORD system framework. Members from GDOT's Office of Construction and Office of Performance-Based Management and Research were present at the meeting. The group held an open discussion on potential features needed for the ReCORD system. GDOT strongly agreed with the use of the notification system and the development of a mobile version of the ReCORD system. Additionally, the group briefly reviewed NASA's LLIS and decided that more information should be learned about NASA's system. A meeting was scheduled with Michael Bell, the NASA LLIS curator (see the NASA Interview section for details of that interview). GDOT
56

recommended soliciting feedback from potential ReCORD users, such as district project engineers. Thus, a meeting was scheduled with the GDOT District 1 field engineer to discuss and refine operations and features of the ReCORD system. The outcomes of this meeting are summarized in the District 1 GDOT Meeting section of this report.
NASA Interview The LLIS utilizes topic tags to organize its lessons learned. For example, several of the topic tags include: fire protection, ground operations, and pressure vessels. Furthermore, a wide range of topics are categorized within each of the topic tags. This process allows for a more efficient method of organizing the multitude of lessons stored within the database. NASA's LLIS curator, Mr. Michael Bell, stated that the list of topic tags is still fluid and capable of being regularly redefined by new terminology (Bell, M., personal interview, May 4, 2018). Naturally, a user can use these tags to search/retrieve lessons from within the database for viewing.
NASA recommends to its LLIS users to gradually input information as the project progresses rather than at the end of a project. They believe this will minimize the amount of information lost between the realization of the lesson and its formal documentation. In addition, NASA allows for repeat lessons to be input into the system if the situations differ substantially. For example, this could illustrate that a specific problem is not confined to one situation, but could arise within other circumstances as well. While the lessons themselves may not differ much, having both lessons documented and presented to the users will help in the understanding of the issue in a more complex or comprehensive manner.
57

The NASA Lessons Learned Steering Committee (LLSC), and more specifically the system curator, manages the content (Bell, M., personal interview, May 4, 2018). The curator acts as the administrator for the LLP, reviewing, editing, and helping to validate the lessons prior to entry into the system. Before each lesson is validated and input into the system for distribution, the lesson entry requires committee approval. In the NASA LLIS, lessons have different committees for approval because of the diverse nature of the different topics that are covered by the lessons. The time required to verify each lesson varies from lesson to lesson because of the differences in the levels of complexity.
The LLIS features a method of receiving feedback from its users. Users have the opportunity to rate the lesson on its usefulness, applicability, and coherence, among other lesson attributes. As a result, the curator reviews the comments and makes edits to the lesson, if necessary. Additionally, NASA currently monitors the number of times each lesson and topic is viewed. This serves as an indicator of how users view the LLIS. High viewership suggests that the lessons and the system overall are operating in an effective manner that is accommodating to the users' needs.
Formal training for the usage of the LLIS does not yet exist, but users have access to a handbook that discusses the procedure for capturing a lesson learned and submitting it to the LLIS. When Mr. Bell was asked about changes that could have been made to the LLIS in recent years, he responded with three primary suggestions (Bell, M., personal interview, May 4, 2018):
Keep it more simple Think long term Continued promotion of the system
58

NASA's LLIS is a great model for the framework of the ReCORD system. One LLIS feature that will be incorporated into the ReCORD system will be the proactive push notification technology, which notifies users of pre-registered topics tags associated with the entered lessons. The ReCORD system aims to use a similar feature by associating lessons with topic tags. Other features that NASA uses within its LLIS will be adapted and put to practice within the ReCORD system, as well.
District 1 GDOT Meeting A meeting with GDOT's District 1 personnel was held on April 18, 2018. Approximately 30 GDOT employees were in attendance. After a productive discussion regarding the project objectives and the solicitation of system needs from those present, new features were proposed for future incorporation into the ReCORD system framework. These features included anonymous submission, inclusion of links to other suggested lessons while viewing a lesson, and an opportunity for the users to provide feedback on lessons, as well as the whole of the ReCORD system.
Project SR 316 / SR 81 Site Visit To further confirm that the ReCORD system was being designed in a manner consistent with GDOT employees' needs, a site visit was made by a study team member to Project SR 316 / SR 81 in Barrow County, Georgia. The site visit occurred on July 2, 2018. The project is managed by Mr. Luiz Alvarez (Alvarez, L., personal interview, July 2, 2018). The state construction liaison, Mr. Todd Wood, was present on the day of the site visit. Mr. Wood indicated that the project manager should be the primary individual to submit lessons learned into the ReCORD system because the project manager would: a) be located on site, b) have experienced the incident firsthand, and c) have knowledge to understand and
59

document the incident. Further, Mr. Wood stated that while the district offices would be capable of contributing to the submission, the inception of the lesson would need to reside with the project manager (Wood, T., personal interview, July 2, 2018).
Currently, GDOT project managers use ProjectWise, a Bentley Software system, to manage and store documents throughout the life cycle of a project. ProjectWise systematically stores all files for all GDOT projects. Mr. Luiz Alvarez indicated that he uploads files into ProjectWise at project milestones (usually once a month) to keep the system up to date. The files are created outside of ProjectWise, and these files include both digital and hardcopy files. Paper documentation is scanned into the system. Project files typically include pictures and videos; this was previously identified as a key feature of the ReCORD database system. Mr. Alvarez agreed that the use of a ReCORD database on a mobile or tablet app would be helpful as a first entry point for the lessons learned system while on-site. Currently, Mr. Alvarez takes numerous pictures and videos from his phone. A mobile version of the system would facilitate the process of quickly recording the information while on-site and away from a computer. However, this application would be Mr. Alvarez's only use for the LLP app. He stated that a computer would be a more feasible method of finalizing the entry prior to submission. Mr. Alvarez also recommended the development of a tutorial to inform new users how to operate the ReCORD system. The tutorial would be able to assist in familiarizing potential users of the system and making it more likely for the employees to incorporate it into their work day habits (Alvarez, L., personal interview, July 2, 2018).
60

Stakeholders Parties who will primarily interact with the ReCORD system will be the ReCORD system users, subject-matter experts, and the system curator. The ReCORD database users include GDOT employees who are regularly involved in providing planning, design, and construction-related support for projects. The users contribute to the lessons learned system by submitting new lessons, using published lessons, and providing feedback about the system. New lessons are realized by these users through their involvement on active projects. When an incident arises, it triggers the user to submit a form that will capture the lesson.
The curator and subject-matter experts serve as a bridge for the information to be conveyed to users who are looking to retrieve information. They facilitate the knowledge transfer process by refining the raw information that is submitted by the users. The curator will also keep track of the ReCORD database usage, as well as the database's successes and shortcomings. The curator will be the first point of contact for questions or concerns regarding the ReCORD database.
CONCEPTUAL DESIGN A flowchart illustrating the submission, curation, and notification/viewing phases of the proposed LLP is shown in figure 20. The left-hand portion of the flowchart represents the raw data collection and its submission to the system curator. The central portion of the flowchart consists of the refinement of the raw data through the use of the curation system. The right-hand portion of the flowchart depicts the notification and viewing process of a lesson learned once it has been published into the ReCORD system. The raw data collection process occurs on-site where the lesson is first encountered (see figure 21). The lessons
61

learned system process is initiated when the lesson is realized while on a project. The project engineer enters information related to the lesson learned into the ReCORD system via a mobile device or computer.
While mobile access will likely be the initial entry point for most information into the ReCORD system, users will likely perform final edits on the lessons learned submission via computer. The mobile interface will be integrated with capabilities to take and upload photos and videos, or record sound clips. The development team expects that these capabilities will be the features most used on the mobile interface because it is simplest method for inputting this type of information while on the jobsite. The system is designed in such a way that it is possible to start creating a lesson learned submission from the mobile interface, but then save the draft and continue/submit on a computer. During the initiation of a lesson learned submission (draft lesson), the user will be tasked with entering project information, lesson learned, and suggested topic tags. A list of approved topic tags will be available; however, the user will have the option to suggest additional tags not currently included in the system.
62

63

Figure 20. Process. LLP Flowchart

As previously mentioned, the submitter has the opportunity to include photos, videos, or sound clips into the ReCORD system. Additionally, the submitter will be asked to connect the lesson learned to a GDOT project and provide the incident location. Following submission, the lesson learned progresses through the representational state transfer application programming interface (Rest API). The Rest API functions as the "brain" of the database and is capable of receiving input and processing requests. The data within the submission form are considered to be raw data consisting of only the information compiled and submitted by the project engineer.
Figure 21. Process. Submission Process Once the information is submitted, the curator receives a notification of the newly submitted information and the curation process begins. The curator has complete control over how to refine the draft lesson data. The curator is able to organize, edit, and update any information that he or she deems necessary. This process is illustrated in figure 22. If needed, subject-matter experts are identified to assist with the curation task. The SME(s) have an area, or areas, of expertise for which the curator will consult his or her knowledge. Together, the SME(s) and curator will refine the data by making the appropriate changes
64

to the raw data within the LLP editor until it is deemed fit for publication. After the curation is completed, the lesson progresses to the reviewing phase. This phase occurs after the curator and SME(s) finish their curation, but before publication. The original submitter is notified of the completed curation process and is able to view the curated lesson to verify that the original ideas and thoughts have been interpreted correctly by the curator and SME(s). During the review period the original submitter is able to contact the curator to request changes. The lesson remains in review for three business days (excluding holidays) or until a specified date set by the curator. At the end of this review period, the lesson is then automatically published and the lesson is viewable by all users of the ReCORD system.
Figure 22. Procss. Curation Process
65

Once a lesson is published and available for viewing, the notification system will take effect (figure 23). The notification system is designed to proactively disseminate pertinent information to interested viewers when a new lesson learned is available in the system. Each ReCORD system user has a personalized profile that will include subscriptions to an unlimited number of topic tags. Every time a new lesson with that topic tag is published, notifications will be sent through email to users of the ReCORD system with that topic tag in their profile. Included in the notification will be a link that allows users to directly access the newly published lesson learned within the lessons learned viewer.
Management of the database system, whether internal to GDOT or external, is required. This responsibility includes ensuring the ReCORD system is secure, free of bugs, and operating properly. Upon publication, the lesson is accessible by all users of the ReCORD system with the retrieval interface described in Section 6.2.6.
Figure 23. Process. Notifications and Viewing
66

PRODUCT DEVELOPMENT
ReCORD Database Features This section discusses key features of the ReCORD system that ensure its long-term use through effective documentation, storage, and retrieval capabilities. The system's userfriendly design will help shorten the learning curve of the program.
Notification System The major highlight of the ReCORD database is its use of a notification system that alerts system users when a new lesson related to their selected subject matter is entered into the system. Upon joining the ReCORD system, users develop their individual profiles and subscribe to lesson tags that are relevant to their field of work. Each lesson published within the ReCORD system will include a minimum of at least one lesson tag. Upon publication, the system users associated with the lesson tag are automatically notified through email that the lesson can be viewed. Other users are able to view the lesson by searching the newly published lesson in the database, as well.
This attribute distinguishes the ReCORD system from LLPs that have failed in the past due to a lack of consistent use. Notifications keep users connected to the system and establish a habitual standard of practice for the LLP system that assists with the realization and implementation of lessons learned to support GDOT's decision-making process at a project-to-project level.
Curation Process The ReCORD system will be a living and interactive database that encourages its
users to submit new lessons to the program on a regular basis. This enables the database to
67

remain updated with common problems and successful solutions. However, not all lessons submitted to the database will be accepted. As described previously, a curation process will ensure that only lessons of high quality will be accepted into the ReCORD system. Each lesson will be reviewed and verified by the curator. In addition, a subject-matter expert may be used for certain submissions requiring additional examination prior to publishing of the lesson. In the case of more complex lessons, a committee will be formed to aid the curator in the verification of the lesson. If the initial submission for the lesson does not meet the standard required for publication within the ReCORD system, the curator will send the submission back to the submitter asking for additional information or specific changes. This process is repeated until the curator determines that the lesson is acceptable or not acceptable for publication within the system.
GIS Capability The ReCORD system has built-in GIS capabilities that document the lesson location. The system stores the location of the documented lesson. The ReCORD database's retrieval process features the use of an interactive Georgia map. This map includes layers for counties in Georgia as well as GDOT districts. The user has the ability to toggle the layers on and off, based on preference.
The successes or failures of a method or procedure could be entirely dependent upon the location. For example, the soil conditions of Georgia vary significantly throughout the state. North Georgia is a mountainous region with steeper slopes and weak soils, while south Georgia has characteristics of a coastal plain with more stable soil. Therefore, if a similar pavement project was designed and constructed in both north and
68

south Georgia, a pavement performance issue may be encountered in north Georgia, whereas, no issue may exist in south Georgia.
Mobile Compatibility The ReCORD system has mobile compatibility such that field personnel are capable of entering lessons and information through their mobile devices. This system is accessed through the same website as the computer. This process helps to eliminate the loss of information between the occurrence of the event that created the lesson and when the information is recorded. The ReCORD system has the capability to receive pictures, sound files, and video files to aid in the task of effectively communicating the lesson. The mobile interface is utilized as a first response tool to quickly document lessons learned information. The mobile interface has built-in multimedia capabilities such as the capacity to record videos and audio clips, and take pictures. It is expected that the users will finalize their lesson submission via computer, but the mobile version provides a first-response, convenient method of instantly documenting information on site.
Miscellaneous Options A "References" section provides users with a field to include links to relevant codes or specifications from internal (i.e., GDOT) and external (i.e., AASHTO, other) sources. This attribute adds credibility to the lesson and provides further assistance to users in need of additional pertinent information, while providing an additional level of validation and accuracy to the lesson learned. Furthermore, other relevant lessons that provide related information are suggested by the system and provide an option of delving into additional
69

similar information. This process is comparable to Amazon's tactic of displaying items under the heading "Customers who bought this item also bought..."
Suggestions and Feedback The ReCORD system is a dynamic system allowing for the opportunity for future improvement. Each lesson includes a section for users to rate the validity and usefulness of the lesson with the option for the user to record comments and suggestions. The comments and ratings section serves as a feedback-gathering mechanism for the curator. The curator may wish to review and possibly re-edit a lesson that has received poor ratings and a significant amount of constructive criticism. Additionally, there is a suggestion box for the ReCORD system. This suggestion box includes anonymous submission that allows users to submit comments, recommendations, or concerns regarding the operation and functionality of the ReCORD system to the system curator. It is believed that users are more likely to use a product if they have contributed to the improvement of its functionality. The ReCORD system was created with users in mind, and therefore, the priority is to meet their needs.
TESTING AND EVALUATION During its development stage, the ReCORD system was tested and evaluated by two different groups of students at two different sites, along with a limited set of GDOT project engineers. The student groups comprised a small group of graduate students in the College of Engineering at the University of Georgia (UGA) and undergraduate students in a civil engineering course at Savannah State University (SSU). The first phase of evaluation included testing of the ReCORD system by the UGA/SSU students to ensure that the
70

system did not possess any technical bugs and ensure the system satisfied a high enough quality and intuitiveness to move forward in the evaluation process. Based on feedback from the first step of trial testing, a second external evaluation by project managers from each of GDOT's districts and a member of GDOT's Innovative Program Delivery office was performed to confirm usability and operations by the stakeholders.
Preliminary UGA/SSU Testing UGA/SSU preliminary testing included a total of 11 students: 6 students from UGA and 5 students from SSU. Preliminary testing involved submitting the same information into the database. All participants used the same CDOT drainage lesson described in Section 2.1.1 to evaluate the ReCORD system's performance. Along with personal login information, each of the participants receive three files: the "Submission Guide for Draft Lessons," the CDOT drainage lesson learned, and a JPG image. These files are shown in Appendix B. The UGA College of Engineering building served as the project location because the database was not able to accommodate project locations outside the state of Georgia. After completion of the trial test, students were asked to complete a brief survey through Qualtrics. This survey is included in Appendix B. The testing phase was designed to expose system flaws and areas of potential improvement. The students' submissions were verified for accuracy and completion. As the final step of the evaluation process, students were asked to elaborate on their survey answers. The SSU students recorded their explanations in a MS Word document, while the UGA students were each personally interviewed. In the survey, participants were asked about the system's technical functionality, intuitive flow, command accessibility, and ease of retrieval. Technical functionality refers to how well the system works from a coding standpoint (i.e., links are
71

working correctly, the website is responsive, absence of technical errors or bugs, etc.). The intuitive flow references the order in which the fields were listed, while accessibility refers to how conveniently the commands were located within the website and if their use required an excessive amount of navigating. The system's retrieval feature was tested by requiring each student to navigate the system's retrieval interface to locate a specific sample lesson. Once on the lesson, students were asked the coordinates of the lesson location. The lesson coordinates were determined by clicking on the lesson location pin on the map.
For all 11 students the draft lesson entries were examined for accuracy and completion. A high completion and accuracy rating suggested that the system was working well. A lower completion and accuracy rating suggested that modifications were required to optimize the system. The completion and its accuracy were examined in three parts:
Were all the fields in the draft lesson completed correctly? Were images submitted within the draft lesson correctly? Were the sample coordinates correctly located and entered into the
coordinates for the sample lesson learned?
GDOT Trial Testing The GDOT testing phase included both a curator and a project manager from six of the seven GDOT districts, along with the Office of Innovative Program Delivery. Two professional engineers from GDOT's Office of Construction were selected to serve as the curator for the trial test. Each of the project managers were tasked with inputting their own lesson learned into the submission interface, finding and viewing the lesson in the retrieval interface, and testing of the notification system. Each lesson input into the ReCORD system
72

was unique and was from previous real-world experiences associated with each of the project managers' specialty areas. Each of the project managers were given unique login information for the ReCORD system in order to receive notifications to test the retrieval portion of the system. The curator was given access to the ReCORD system after the projects were submitted into the system. The curator was tasked with editing each of the projects and finalizing them for publication.
The aim of the GDOT testing phase was to optimize the simplicity of the ReCORD system for the sake of its users. The goal was to make processes as intuitive and as simple as possible in order to increase user retention. Each project manager used his or her own unique project in this phase. This helped the research team analyze the effectiveness of most of the features in the ReCORD system, while also identifying any missing elements that were not brought to light during the UGA testing phase.
The Phase One survey was again used to properly capture feedback from the project managers and curator. Additionally, a personal interview was held with each of the participants to elaborate on their thoughts, recommendations, and comments.
Similarly, feedback was analyzed based upon accuracy and completion with recommendations for system improvement. High completion and accuracy ratings suggested that the system was working well. Lower completion and accuracy ratings suggested that modifications were required to optimize the system. This completion and accuracy were examined in two parts:
Were all the fields in the draft lesson correctly completed? Were the coordinates for the sample lesson input correctly into the sample
lesson learned?
73

CHAPTER 6. EXPERIMENTAL RESULTS
This section documents the products of the ReCORD system and the trial test results. It includes details regarding the system interfaces and its uses. The trial tests were conducted to verify the performance of the system's features and processes, and complete system improvements prior to turnover to GDOT.
CREATION OF THE RECORD SYSTEM
System Requirements The programming platform of the ReCORD database was created through the use of Java and MySQL. JavaServer Faces (JSF) is responsible for the display of information, while Java Servlet is the "back end" or "brain" of the system and is the connection between JSF and MySQL. MySQL is used to store the data. The user interacts only with JSF and does not have access to either Java Servlet or MySQL. For example, when a user inputs a search query, it will be submitted through JSF. JSF relays the message to Java Servlet, which then retrieves the appropriate information from the MySQL database. Java Servlet then relays the information back to JSF, which displays the information to the user.
SYSTEM INTERFACES All users will begin their interaction with the ReCORD system by either registering or logging into the system. An illustration of the login page is shown in figure 24. This page provides a short description of the GDOT Lessons Learned Database and its purpose. Further, it provides instructions for new users of the system. Individuals have two options: (1) login with their email and password, or (2) register to become a new user of the system.
74

Should users forget their password, they may request that their password be reset by clicking on the "RegisterForgot Password" link at the bottom of the page.
Figure 24. Image. Login and New User Registration Page Six primary interfaces were created for the ReCORD system: Create Lesson, Lesson Management, Help, and Profile, Published Lessons, and Lesson Map. An additional interface only visible to the curator(s) is the User Management interface. The Create Lesson interface was developed for individuals submitting a new lesson learned into the database. The Lesson Management interface allows users to track the progress of a submitted lesson from review to publication. The User Management interface is visible only for the curator. From this interface, the curator is able to manage the list of approved users of the system and their roles (user/curator). The Help interface provides user guides
75

for individuals using the system. The Profile interface allows the user to set personal notification settings and user tags. The Published Lessons interface was developed for users retrieving and viewing the lessons learned. Lastly, the Lesson Map interface allows users of the system to visually see the locations of all published lessons throughout the state and in relation to the county and GDOT district for which they reside. Create Lesson Interface Users of the ReCORD system access the Create Lesson interface when a lesson is realized. ReCORD users access this interface by opening the ReCORD database and navigating to the tab "Create Lesson." The first part of the interface is shown in figure 25. See detailed the submission process in Appendix B.
Figure 25. Image. Submission Form Interface I This form includes lesson title, lesson tags, lesson abstract, lesson location, the project(s) with which the lesson is associated, the lesson learned, references, and file upload. Fields that include an asterisk must be filled out before submission. A navigation bar on the left side of the screen allows the system user to quickly navigate between fields
76

and keep track of progress in inputting the lesson. Once a field is completed, its tab on the navigation bar will turn green and a check mark will appear to confirm its completion.
The "Lesson Tags" section is the key to the organization and distribution of the lessons learned. These tags serve as convenient, searchable labels for the lessons learned. More importantly, the notification system is based upon the topic tags; users subscribe to lesson tags and will get notifications when a lesson with their lesson tag subscription is published. The system curator manages the lesson tags. When a submitter types a tag related to their lesson into the text box, one of two options will occur. If the lesson tag has previously been approved by the curator, it will appear colored blue in the text box. Should the lesson tag be new, it will be highlighted in red with the caption reading "New." During the review process, the curator will have the option of either approving, denying, or modifying the new lesson tag. If the curator approves the new lesson tag, then it will be added to the system and it will become an approved suggested tag for future submitters. The full list of current lesson tags is included in Appendix C.
The "Lesson Abstract" serves as a preview of the full lesson. The abstract must be less than 500 characters, including spaces. The abstract is an overall summary of the lesson learned and helps the database organize the lesson.
Figure 26 shows the "Lesson Location" and "Select Project to Associate" sections. The "Lesson Location" section asks the user to provide the location of the lesson. This is performed by entering an address into the text field or by clicking a location on the map. If users choose to enter an address, they must click "Search" after the address is inputted. A green marker appears on the map, confirming the location. To choose to select the
77

location on the map, the user pinpoints the location on the map and clicks. An address appears in the text field, confirming the location.
Figure 26. Image. Submission Form Interface II The "Select Projects to Associate" section is used to associate the lesson with a GDOT project; each lesson must be associated with one project. Associating a lesson with a project communicates to the ReCORD system which lesson occurred on which project. The system allows for multiple lessons to be associated with one project such that multiple lessons can be learned from a single project. However, it is important that users submit separate entries per project. Thus, only a single lesson is allowed per entry. Doing so
78

simplifies the process by avoiding having an overwhelming number of lessons within an entry.
The ReCORD system is linked to GDOT's GeoPI Dashboard; thus, all GDOT projects with a Project ID number are listed within the system. The list of projects is updated each day such that a current list of projects can be maintained. Projects may be chosen by entering part or all of the GDOT Identification Number (Project ID) or name. Further, users may refine project searches by selecting only certain counties or districts. Lastly, once a project is selected, the user may choose to click on "GeoPI Info" and it will navigate to the GeoPI project page for more project-related information.
Figure 27 shows the "Lesson Body," "File Upload," and "References" text fields. The "Lesson Body" text field includes a description of the incident or incidents that triggered the lesson learned, as well as a description of the lesson itself. The incident description allows readers to understand the circumstances leading up to the lesson learned, which potentially helps them identify similar situations in their own line of work. Additionally, it is recommended that the lesson include a specific approach should the incident arise again. In the case of positive lessons, the user may recommend methods to capitalize on the circumstances. On the other hand, if the lesson is negative, the user should include indicators or symptoms of the potential issue. The information should be written as a useful and relevant resource for future users who may encounter similar circumstances.
The "File Upload" section allows the user to include any media files with the lesson submission. Files such as images, drawings, videos, or audio clips are allowed and serve as supplementary information to explain the lesson learned. Media may be in the form of
79

JPEG, MP3, MP4, etc. The "References" text field allows the users to add links to more official guidelines and requirements, such as a code, reference manual, or specification.
Figure 27. Image. Submission Form Interface III Figure 28 shows the lesson review and submission functions for the Lesson Creation page. The submitter is allowed to "Save as a Draft," "Save and Preview," or "Submit." "Save as a Draft" allows the submitter to save the lesson and come back to complete or edit it at a later time. The user is able to access the lesson through the Lesson
80

Management interface discussed in the next section. "Save and Preview" allows the submitter to save the lesson and preview the lesson prior to submission. The "Submit" option allows the user to submit the lesson to the curator for review and publication.
Figure 28. Image. Submission Form Interface IV After the draft lesson learned has been curated, it will transition into the "Review" stage. This stage lasts for three business days, excluding holidays or an alternative review period as specified by the curator. The submitter will be notified of the lesson's progression into the "Review" stage and have the opportunity to confirm that the curator did not misinterpret the draft lesson's contents. After the time period has elapsed, the lesson will automatically be published into the database. Lesson Management Interface The Lesson Management interface shown in figure 29 allows users to track the progress of their submitted lessons. This interface displays lessons that are saved drafts, submitted to the curator for review, in the review period, and published for a particular individual user. As shown in figure 29, for this user there are 0 drafts, 1 submitted for review, 0 in review, and 0 published.
81

Figure 29. Image. Lesson Management Interface I When the system user selects "Submitted" in this interface, the user is allowed to view the lesson submitted for review or request the lesson back for editing and resubmission (see figure 30). Should the user select the request back option, additional options appear, as shown in figure 31, allowing the user to edit, view, or delete the lesson.
Figure 30. Image. Lesson Management Interface II
82

Figure 31. Image. Lesson Management Interface III
Should the submitter leave the lesson unchanged within the system, the curator will review the submission. The curator serves as the reviewer of the submitted raw data and approver for the published lessons learned to the ReCORD system. The curator is the bridge between the submitter's raw draft lessons and the published lessons for the lesson viewers. Therefore, the curator has complete access to all components and interfaces of the system in order to refine the raw data until it is suitable for publication. A detailed curation guide is provided in Appendix B.
Curation The curator accesses the draft lessons needing review by clicking on the "Submitted" tab of the Lesson Management interface. The interface will display a table titled "Lessons available for curating," as shown in figure 32. The curator will select a draft lesson for curating with the lesson moved to a new table titled "Lessons you are curating." This process enables a lesson being curated to become locked and unable to be edited by the submitter or another curator. Once the lesson has been selected for curating, the curator
83

will click on the edit button shown in figure 33. The submitted draft lesson information will be imported into an editing interface. All fields within the lesson form will be enabled for editing. The curator will carefully review each text field to confirm that the statements are understandable, descriptive, and meet the requirements of a lesson learned.
While the curator will ensure that the text responses meet the definitions of that described in Section 6.2.1, the curator will have the added responsibility of reviewing, approving, or denying lesson tags. Lesson tags are important keywords that identify the lesson topic, and each lesson is required to have at least one lesson tag. The lesson tag is used to both organize the database archive, as well as to disseminate information to the database users. The database archives the lesson based upon its lesson tags. In addition, the tags are the basis of the push notification system. If the lesson tag provided by the submitter already exists in the database and is appropriate for the lesson, no further action is required by the curator. However, if the user has submitted a tag or tags that do not currently exist in the database, then the tag will be highlighted in red and contain a "NEW" symbol. The curator may approve these new tags by leaving them within the lesson submission and, when published, the tag will become automatically approved and included within the database. The list of lesson tags is a dynamic list that the curator can manage at any time through the User Management interface of the ReCORD system. Additional discussion on the tag management is discussed in Section 6.2.3.
84

Figure 32. Image. Lessons Available for Curating
Figure 33. Image. Lessons You are Curating 85

When curators are finished editing a lesson, they have the same options as the user in that they can: Save as Draft, Save and Preview, or Submit. After the curator confirms that the information is finalized, the lesson is transitioned into the final "Review" phase. After "Review," normally three business days, the lesson is published into the database. At this point, the original submitter is notified that the new lesson is available for viewing.
User Management Interface One additional task the curator has in the database is control of user access to the system. An interface tab is located at the top of their page for user management. The curator will control the approval of individuals registering as users of the system. Once a new user registers with the system, the curator approves new accounts with the "User Management" page, shown in figure 34. A GDOT email address is required to register with the ReCORD system. However, the curator has the ability to add users without GDOT email addresses, by directly adding the address into the system at this interface. The curator chooses the role of the user, either as a curator or a regular user. Once the user and role have been approved by the curator, the user will receive an email with a link to set up a password and complete the registration process.
Within the User Management interface, the curator has the ability to manage the users of the database, including activating/deactivating users, as well as changing their role with the system (user/curator). The user list may be sorted alphabetically, or individual users may be searched for using part or all of the email address.
86

Figure 34. Image. User Management Interface In addition, the curator is able to manage the ReCORD system tags within the User Management System. In figure 35, all approved tags are listed, allowing the curator to review current published tags or delete outdated or irrelevant tags. Tags may be sorted alphabetically or found using the search function.
Figure 35. Image. Tag Management Help Interface The Help interface, shown in figure 36, provides users and curators with guides on how to use the ReCORD lesson learned system. ReCORD users and curators have access to three guides: "Submission Guide for Draft Lessons," "Curation Guide to Publish Lessons," and "Usage Guide for ReCORD System." The "Curation Guide to Publish Lessons" is only viewable by those that are curators.
87

Figure 36. Image. Help Interface Profile Interface Users of the ReCORD system will develop a profile. This interface displays the user's Username, User Tags, and Notification Settings (see figure 37). The system user may add or delete only tags that have been approved and published in the database. In addition, the user may select notification sessions, such as receiving messages when a lesson status changes and/or recommended lessons based on newly published lessons related to the user's tags.
88

Figure 37. Image. Profile Interface Published Lessons Interface The Published Lessons interface is for users who wish to view published lessons. ReCORD system users are able to learn of published lessons through four avenues:
Push notification via email (Passive) Text search of the database (Active) Recommended lessons feed / Top lessons (Passive) Visual (geographic) search of the database through the use of the map
(Active)
Push Notifications Each user has the opportunity to subscribe to lessons tags. When users are subscribed to a lesson tag, they receive push notifications when a lesson is published with the associated tag. The user has the option of receiving these push notifications through either email messages or text messages. The push notification received by the user includes a link that requires login information, then gives the user direct access to the newly published lesson. This is an active method for users to be exposed to newly published lessons.
89

Text Search As an alternative method, users are able to access a published lesson by actively performing a keyword search. A user navigates to the Published Lessons tab and performs a keyword search using the text box displayed at the top of the page. The querying process performs a comprehensive review of all text (lesson title, tags, abstract, and lesson body) to produce search results that are most relevant to the user. It does this by recommending lessons based on the frequency and location of the keywords used in the search. Once lessons are displayed, they may be viewed by clicking on the "View Lesson" link. This function is illustrated in figure 38.
Figure 38. Image. Database Search Function Recommended Lessons On the Published Lessons interface page, lessons will be displayed based upon the number of times viewed and rating. Lessons that are viewed more often and have a higher rating will be displayed higher in the order on the initial Published Lessons landing page. The development team chose this manner of organizing the lessons on this interface over others,
90

such as organizing based on order of submission or publication, to provide more visibility to lessons that are viewed more often (and possibly used more often).
Location Search Another form of active searching is the location search feature. A user accesses this search function by navigating the Lesson Map interface. This interface and its ability to aid users in finding lessons is discussed in Section 6.2.7.
Lesson Map Interface The Lesson Map interface allows users to view all published lessons geospatially. A Georgia map, including counties and GDOT districts, is included within the Lesson Map interface (see figure 39). The counties and district layers can be toggled on and off by clicking under "Show Layers" on the left side of the page. A viewing toolbar, located at the top right of the page, provides the user tools for panning and zooming, and general tools for controlling the view of the map. The map includes dropped pins for each published lesson within the ReCORD database. When a pin is clicked, it shows a brief lesson description such that the users can determine whether the lesson is relevant to them. If so, the user may select to view the full lesson by clicking on the "View Full Lesson" link at the bottom of the information box for that location pin. The user has the ability to filter lessons by county or GDOT district. These filters are accessed on the at the top of the page.
TRIAL TESTING RESULTS Throughout the trial tests, the ReCORD system was evaluated by four criteria: appearance, technical functionality, intuitive flow, and accessibility. Each criterion was rated by users on a scale of 1 (poor) to 10 (excellent). The system was modified after each phase of trial
91

testing. By the end of the trial testing, the study team expected each criterion to have an average score of at least 8.
Figure 39. Image. Lesson Map Interface UGA/SSU Trial Testing Submission Accuracy and Completion Upon examination of the UGA/SSU draft lessons submissions, the participants correctly completed all fields, with the exception of image upload and lesson tags. Seven of the users did not submit the image file that was required to be uploaded to the "File Upload" section. This was a result of the upload function process. The users had to click "Choose File," then select the image, and then click "Upload." It was not clear that both steps had to be completed in order to upload a file. As a result, this process has been modified to a single "File Upload" button. When the user selects the image, the system automatically uploads it as part of the draft lesson. Two of the participants did not upload tags into the "Lesson
92

Tags" field. This is the result of the incompatibility between the paste function and this field. However, no changes are required since GDOT users will not be pasting lesson tags into the field.
Survey and Interview Feedback Following the trial testing of the ReCORD system, the participants were asked to rate the system from 1 to 10 in four different areas: appearance, technical functionality, intuitive flow, and accessibility. Based upon the survey results, appearance and technical functionality scored means of 6.8 and 7.5, respectively, which meant modifications needed to be made. The criteria of intuitiveness and accessibility respectively scored 8.5 and 9.0. Therefore, less attention was required to modify the system for those two areas. Figure 40 shows the results of the trial test survey.
Visual Appeal Feedback also was gathered regarding the visual appeal of the database. Comments from the participants are listed in quotations below followed by a short discussion of how the database was modified to accommodate these comments.
"The textboxes are too wide and disproportionate, so it does not look appealing."
"Do not include all of the information on a single page. Each page should only include one field; this will give users a sense of progression and minimize the risk of the user accidentally skipping over a field."
93

"Add a picture or any kind of visual to grab the user's attention," and, "There is too much white space. Add a picture. The interface needs contrast between the background and the textboxes."
Figure 40. Image. UGA/SSU Survey Results The input textboxes for each of the fields was too wide and not large enough. Thus, the text was too spread out and not compact. Textboxes were modified such that they were centered on the page and shortened in width to provide a more proportionate and visual appeal. No change was made regarding the comment to include fields on separate pages. All of the information remained on the same page; this gives the user an idea of how many fields are left, and it does not require the user to go through additional clicks. This comment
94

was reconsidered at a later date and a navigation bar was added to the left-hand side of the page. The navigation bar allows the user to navigate throughout the database easily and keeps track of which sections have been completed.
In order to address the last comment regarding large amounts of white space and the inclusion of graphics, a GDOT logo and image of a book were added to the top of each page (see figure 41). The textboxes now have a blue-gray tint in the background. This provides more contrast for the entry fields, making the submission a more pleasant experience.
Figure 41. Image. GDOT Logo and Image Technical Functionality Feedback was gathered regarding the technical functionality of the database. Each comment represents a challenge that the database faced. These comments are listed in quotations below followed by the database modifications that addressed the challenge.
"The `Lesson Tags' did not allow for copy and pasting." "I was unable to access a saved draft lesson." "There was no confirmation of a successful draft lesson submission."
No changes were made regarding copying and pasting of the lesson tags. In the case of the commenter, multiple tags were pasted into the field at the same time, and the database could not distinguish the end of one tag and the beginning of another. However, it was determined and confirmed that this issue would not arise in practice because users will be
95

typing in one lesson tag at a time. Due to an error in the coding, a system bug did not allow the users to access their saved draft lessons. This system bug has been fixed and is now operational. The database does confirm that a submission is successful; however, the notification only appeared at the top of the screen and was, therefore, not noticeable. A successful submission will now load a new page that clearly confirms the submission.
Intuitive Flow Feedback was gathered regarding the intuitive flow of the database. The only comment given represents a challenge that the database faced. The comment is provided in quotations below, along with the modification.
"The `Incident Summary' should be listed before the `Lesson Abstract'." Upon evaluation, the incident summary field was removed entirely. The development team decided that the field was unnecessary and only served to confuse users. Instead, the user is now asked to include the incident summary within the "Lesson Body" section.
Accessibility Feedback was gathered regarding the accessibility of commands in the database. The comment, listed in quotations, represents a challenge that the database faced. Modifications to the database system are presented following the comment.
"Separate links should be added to help the user navigate the submission interface."
After evaluation, a navigation bar was added to the left-hand side of the screen. The user can use these links to quickly and conveniently navigate the submission interface. The
96

"Create Lesson" navigation bar also allows users to track their status in completing the lesson form by highlighting the form sections that have been completed.
Lesson Location Ten of the eleven users submitted the correct coordinates of the lesson, verifying the use of the retrieval and keyword search interface. The user who incorrectly submitted the coordinates was only incorrect by a single digit, and the research team hypothesized that this was a typographical mistake.
Additional Comments Additional feedback was gathered from the Phase One evaluation participants. Comments (presented in quotations) with the modifications made are presented and discussed below.
"In the `Lesson Location' section, the map should auto zoom to the location once the user inputs the address."
"Under the field `Lesson Abstract' there is a character limit. The character limit includes spaces; this is not consistent with a Word document, so it has the potential to cause problems."
"This system will only work if the users are educated about it. They should be taught how to best absorb and utilize the lessons learned."
Currently, the system does not have this feature; however, it is expected that this feature could be implemented in future updates to the system. The commenter suggesting inconsistency in character limit with a Word document makes an incorrect claim. MS Word has an option to count spaces as characters. Therefore, if users were to copy and paste the abstract from Word, they would still be able to account for the character limit by checking
97

the number of characters with spaces. The last comment regarding system adoption is a good recommendation for implementation and long-term usage. One of the recommendations from this report is that GDOT introduce the system during orientation training sessions and educate users about the best way to utilize the system. GDOT Trial Testing Submission Accuracy and Completion Six of the seven submissions were received from the GDOT participants. Figure 42 shows the most complete submitted draft lesson. This lesson learned regarding the use of concrete material for use in concrete shoulders was submitted from a project engineer from District 4. Appropriately, the lesson included "concrete" and "roadway operations and maintenance" as lesson tags. This lesson was later curated and remains in the ReCORD system.
98

Figure 42. Image. Draft Submission 99

The system initially exhibited an error throughout this test phase that hindered its ability to correctly record the information entered in the "Lesson Body" section for some users. The "Lesson Body" field was only successfully completed for two draft lessons. Several project managers submitted lessons based on projects that did not already exist within the system; thus, the submission guide directed them to add the project information within the "Lesson Body" field, such that the curator could add this appropriately within the lesson. The system successfully received project information from only three of the six participants. Apart from these omissions, the participants were able to successfully complete the remaining fields of the lesson learned submission form. Two of the participants submitted images with their draft lesson submission. One of these images is shown in figure 43. The primary issue encountered during this phase of testing was the submission and saving of lesson content within the "Lesson Body" section of the form. As previously mentioned, a system error prevented multiple users from submitting the lesson body. This problem was rectified.
Survey and Interview Feedback Four of the six GDOT participants completed the feedback survey and interview. The survey results are shown in table 6. Similar to the UGA/SSU trial test, the participants were asked to rate the system from 1 to 10 in four different areas: appearance, technical functionality, intuitive flow, and accessibility. Appearance and technical functionality both received a mean score of 8.75, while both intuitiveness and accessibility received a mean score of 7.75. The scores for appearance and technical functionality significantly increased and are now above the target average score of 8. The scores for intuitiveness and accessibility dropped below the target average score of 8. This was important information
100

because the individuals included during this phase of testing were the future users of the system.

Figure 43. Drawing. Image from Draft Lesson Submission

GDOT 3 GDOT 4 GDOT 6 GDOT 8 Average

Table 6. GDOT Trial Test Survey Response

Appearance

Technical Functionality

Intuitive Flow

9

9

9

8

8

7

8

8

9

8

8

6

8.75

8.75

7.75

Accessibility
9 5 9 8 7.75

Feedback was gathered regarding the overall experience of using the database. Each comment represents a challenge that needed to be addressed. These comments are listed in quotations followed by a brief summary of how the comments was addressed listed alongside.

101

"The search feature for the retrieval of lesson coordinates was not working."
"After completing the `Abstract' field, the recently entered text disappeared."
"There was confusion about what type of lessons should be entered into the system (past, present, etc.)."
"Move the `File Upload' next to the `Lesson Body' section because they are associated."
The search feature was modified to enable an elastic search that allows for a more comprehensive query of the published lesson content. This finalized search process was discussed in detail in the Text Search section of this report. At the time of evaluation, the survey participant was correct that the search feature was not working as the result of a coding error, and this may have led to lower survey scores. The study team confirmed that at the time of GDOT trial testing, there was an error in the system coding causing the abstract to disappear. The coding error was resolved and the abstract no longer disappears. The comment regarding confusion on the type of lesson that should be entered led the study team to realize that training materials and an organized training program should be established for system users. Thus, the team proposes that expectations be clearly explained to system users during GDOT new employee orientations or other departmental training workshops. Additional information on ReCORD system orientation training is provided in Chapter 7. Future orientation sessions will explain to users that the lesson should be recorded at milestones during the duration of the project and at the project closeout. The uploaded media files help to communicate the points made in the "Lesson Body" section,
102

therefore, the "File Upload" section should be directly beneath the "Lesson Body." This positioning allows the media files to better supplement the "Lesson Body" section. As a result, the "File Upload" section was repositioned directly underneath the "Lesson Body" section.
103

CHAPTER 7. DOT IMPLEMENTATION AND USAGE PLAN
The users of the ReCORD system need to understand the expectations of the program. During the preliminary planning and process development phase of this study, GDOT stated that they expect to produce approximately two to three new lessons each month. The system as designed should not be onerous for the submitters or users. Ultimately, the ReCORD system is meant to be a convenient and efficient resource for users. The ReCORD system implementation provides major benefits that will accumulate over time. Visibility of these benefits to system users will only reinforce the LLPs use and increase participation among its users. In part, the system benefits are based on the "pay it forward" mentality, where the work of past user submissions will serve to aid the work of those in the present. Similarly, the work that the users submit now will help those in the future. Potentially, this LLP may help to reduce future training, as the lessons learned are readily available anytime, anywhere, to all departmental personnel. This consistent availability could limit the need to provide specialized workshops on specific failures or successes.
When the users of the ReCORD system realize the benefits, they will be more willing to invest time and effort back into the system. Therefore, the key to a successful implementation is ensuring users are aware of the ReCORD system capabilities and the support it provides its users. Although, the ReCORD system will be widely available to its users, the system must be properly introduced through initial training. This training will provide a thorough understanding of how the database functions, along with the expectation that its users implement the program into their daily routine.
104

ORIENTATION TRAINING SESSIONS Literature review findings and trial testing feedback verified the need to establish a training program for the ReCORD system users. As a result, the study team recommends introducing the ReCORD system to departmental users in two forms. The first would be the introduction of the lessons learned program and web-based system through new employee orientation sessions. This type of training would include the delivery of lecturebased presentation content that can be incorporated with other new employee content. The sessions would serve to familiarize users with the system interface and demonstrate operations. A second form of training may be provided through recorded webinars that discuss the ReCORD lessons learned system, its importance, and its functionality. The webinar can be shared throughout the department and viewed by current employees planning to use the system. The webinar can be an on-demand type of learning.
Continually raising awareness and promoting the system will ensure the system remains visible to users and for its continual use. In most cases, the ReCORD system curator will provide guidance regarding database training sessions and its implementation throughout the department. The ReCORD system works only as well as its use from participants. Thus, a clear and effective implementation will be critical.
Training and Materials User training (new and current employees) should be conducted as if the audience is unaware of LLPs and the ReCORD system. While the information presented to each group of employees will be the same, the delivery type (in-person vs. online webinar) may vary. Regardless of delivery type, the training sessions should begin with a presentation that explains:
105

What an LLP is Contents of an ideal lesson Expectations for employee usage Benefits of the ReCORD system
Following the presentation, audiences should be shown the ReCORD system. To persuade GDOT employees to use the system, they must believe that a lessons learned system is an effective use of their time. Whether in-person or in a webinar, the training session leader should demonstrate how to use the ReCORD system by submitting a sample lesson. The demonstration should begin by illustrating the login page and account registration. It is expected that most training will have only primary users in the audience, so only the Published Lessons, Lesson Map, Create Lesson, Lesson Management, Help, and Profile interfaces will need to be discussed. The training session leader should plan to demonstrate how each interface is used with a sample lesson example. With this process, the training session leader should work through the creation of a sample draft lesson and submit it for curation. Furthermore, the training leader should demonstrate the lesson retrieval process and explain the features within the Help and Profile interfaces. Additionally, the training session leader ought to show how to access the guides within the Help interface of the system. Lastly, the training leader should aim to answer questions regarding database usage and reaffirm the system benefits to the users. After each orientation session, users should understand and be able to utilize the features of the database to their full potential.
106

ReCORD SERVER AND FUTURE OPERATIONS/MAINTENANCE Recently, GDOT's Office of Information Technology (IT) Applications has decided to discontinue the use of systems that require JavaScript. This information was not known prior to the design and development of the ReCORD system. A potential conflict with the ReCORD system is that the majority of the system is programmed in Java. Thus, implementing the system in its current form at GDOT would only be a temporary solution and would ultimately require the system to be rebuilt using a language compatible with GDOT's information technology plan.
As a result, a number of solutions were examined through consultation between the UGA study team, GDOT's Office of IT Applications, and GDOT's Office of Performancebased Management and Research. Option 1 included an external hosting of the ReCORD system by UGA. The advantages of this option included the quickest time to deployment with the system having already been user tested and determined to be functional. However, disadvantages included a lack of integration with preexisting GDOT software and all requests for future maintenance would be routed through UGA. Option 2 examined a redevelopment of the ReCORD system by UGA. The advantage of this option was compatibility of the redeveloped software with the GDOT software ecosystem. Disadvantages of Option 2 were the need for an extension of the contract and a delay of system deployment for general usage. Option 3 involved a redevelopment of the ReCORD system by the GDOT Office of IT Applications. The advantage of this approach is the guaranteed compatibility and integration within GDOT's existing software platform. A disadvantage of this option was an unknown timeline for the redevelopment and subsequent delay for the system's deployment. The last option was a hybrid approach
107

combining Option 1 and 3 whereby UGA would externally host the ReCORD system for a period of time followed by a GDOT redevelopment. The hybrid approach would allow a quick deployment to the users as well as ensure compatibility and integration with the GDOT software ecosystem. A disadvantage would be the need to establish a migration plan to ensure a smooth transition of data between databases.
Through consultation with the GDOT Office of IT Applications, Option 3 was selected with a ReCORD system redevelopment by the GDOT Office of IT Applications using a language compatible with GDOT's information technology plan. Ultimately, the reconfiguration performed by IT Applications will allow for a system that satisfies GDOT's IT plan, while still meeting the required needs of the ReCORD lessons learned program. As a result, the study team developed two documents that were shared with the GDOT Office of IT Applications to aid in the redevelopment of the software. The documents "Software Requirements Specification for Georgia Department of Transportation Lessons Learned Program" and "Supplemental Documentation for Georgia Department of Transportation Lessons Learned Program" were created to inform GDOT of the functional and non-functional requirements of the software and provide the office details specific to the various system interfaces and operations. The system specifications are listed in Appendix D. The supplemental documentation included portions of Chapter 6 of this final report; hence, that part is not included in the appendix due to duplication. In addition, the ReCORD system's source code and supplemental files developed as a part of this study were shared with GDOT.
108

CHAPTER 8. CONCLUSIONS AND RECOMMENDATIONS
SUMMARY AND CONCLUSIONS This study details the development of a lessons learned database, known as the ReCORD system, to be used by GDOT. Other LLPs were studied in order to build the framework of the database, particularly NASA's lessons learned information system. Additionally, a state DOT survey gathered feedback from 28 state DOTs about the usage of LLPs for DOT construction, which found most state DOT employees held an optimistic outlook on the efficacy of LLPs. The database was programmed on Java and is web-based. The six interfaces of the database are: Create Lesson, Lesson Management, Help, Profile Published Lessons, and Lesson Map. In addition, the User Management interface that is utilized to manage the users of the system, as well as lesson tags, is made available to only curators of the system.
During the literature review, it was found that most LLPs failed or lacked effectiveness because of lack of consistent use. In order to address that issue, the ReCORD system was built with an automated push notification system that notifies users of newly published lessons that involve a topic relevant to the user. Other LLPs failed due to poor lesson quality; users do not want to view poorly written lessons or useless lessons. The system was designed to include lesson creation, curation, and publication system interfaces with key features such as push notification, mobile compatibility, and GIS capability. In order to ensure that only high-quality lessons are published, the ReCORD system requires a curator to review and edit lessons prior to publication.
In order to evaluate the effectiveness of the ReCORD system, a two-phase trial evaluation was conducted. The first phase of the trial test was conducted with students from
109

UGA and SSU. These results suggested that the ReCORD system needed to be visually reworked, but received high scores in the criteria of intuitive flow and accessibility. Upon modification of the ReCORD system, a second trial test was conducted that included GDOT project managers from varying GDOT field districts, who submitted draft lessons. All of the UGA/SSU trial test participants and four of the seven GDOT engineers completed a feedback survey and participated in an interview. The feedback was evaluated and used to modify and improve the ReCORD system.
Two documents were created and shared with the GDOT Office of Information Technology. These included "Software Requirements Specification for Georgia Department of Transportation Lessons Learned Program" (Appendix D) and "Supplemental Documentation for Georgia Department of Transportation Lessons Learned Program." These documents inform GDOT of the functional and non-functional requirements of the software and provide the details specific to the various interfaces and operations of the ReCORD system.
RECOMMENDATIONS Based upon the results from the study, the ReCORD system will be most effective if (1) there is continued promotion and education about the system, and (2) if the system remains simple. A measure of success of the database is long-term usage. In order for individuals to use the system, they must first understand how the system works and how it can benefit them. This will be addressed through the use of training for both new and current employees, wherein the users are introduced to the system, shown the system's capabilities, and taught how to use it. Even with participants understanding the expectations and benefits of using a lessons learned system, it is projected that GDOT, as a whole, will only
110

submit several lessons a month. Beyond a basic understanding, the users must recognize that by submitting draft lessons, they are contributing organized information that will greatly benefit the design and construction community within GDOT.
GDOT employees will not use the system--regardless of the benefits--if too much effort is required to use it. Therefore, another recommendation moving forward is to keep the processes simple. Generally, users do not want to learn complex systems that will only be used on occasion. Concerning future changes and updates to the system, the users must be the priority. System capabilities must be designed in a manner to cater to the users; as long as the users understand the benefits of the ReCORD system and are willing to put in the time to properly use it, the ReCORD system will flourish.
111

REFERENCES
Aha, D., Becerra-Fernandez, I., Maurer, F., and Munoz-Avila, H. (1999). Exploring Synergies of Knowledge Management and Case-Based Reasoning. AAAI Technical Report WS-99-10. ISBN 1-57735-094-4.
Brown, B. (2016a, May 03). "How to Make Your Lessons Learned Initiative Succeed." Retrieved January 22, 2018, from http://secutorsolutions.com/blog/Post/302/Howto-Make-Your-Lessons-Learned-Initiative-Succeed.
Brown, B. (2016b, May 03). "What is a Lessons Learned System?" Retrieved January 22, 2018, from http://secutorsolutions.com/blog/Post/301/What-is-a-Lessons-LearnedSystem.
Ask MetaFilter. (2006, July 12). "Construction Equiv. to a Financial Audit?" Retrieved July 31, 2018, from https://ask.metafilter.com/41984/Construction-equiv-to-afinancial-audit.
CDOT. (2016). Lessons Learned US 36 Express Lanes. Research Final Report. Colorado Department of Transportation. Office of Construction, Broomfield, Colorado, pp. 1113.
Chabot, R. J., Miller, T. J., and Juola, J. F. (1976). "The Relationship between Repetition and Depth Processing." Memory and Cognition, pp. 677682. Retrieved December 12, 2017, from https://link.springer.com/content/pdf/10.3758/BF03213234.pdf.
Coronado Common Sense (2008). "$28M Settlement Reached in Big Dig Death Lawsuit." AP News. October 1, 2008. https://coronadocommonsense..html
Curran, H. V., and Morgan, C. A. (2014). "Declarative and Nondeclarative Memory." Encyclopedia of Psychopharmacology, 17. doi:10.1007/978-3-642-27772-6_3392.
Davenport, T.H. and Prusak, L. (1998, January). Working Knowledge: How Organizations Manage What They Know. Book. Harvard Business School Press. doi:10.1145/348772.348775.
Dixon, N. (2000). "Common Knowledge: How Companies Thrive by Sharing What They Know." Article. Harvard Business School Press, Boston.
112

Dressler, D., and Palin, W. (2007, November). "The Challenge of Lessons Learned: Overcoming Barriers to Successful Application." Journal of Petroleum Technology, pp. 4042.

Federal Transit Administration. (2007). Lessons Learned: The Transportation Expansion (T-REX) Mega-project Experience. Project Management Oversight Program, pp. 18.

Fisher, D., Deshpande, S., and Livingston, J. (1998). Modeling the Lessons Learned Process. Research Report 123-11. Albuquerque: The University of New Mexico, Department of Civil Engineering.

Global Security (2015). Center for Army Lessons Learned Services. Handbook No. 1511. Accessed online at https://www.globalsecurity.org/military/library/report/call /call_15-11.pdf

Goodrum, P., Yasin, M., and Hancher, D. (2003). Lessons Learned System for Kentucky Transportation Projects. Report. Kentucky Transportation Center, pp. 168.

Hoffpauir, D. (2015, April 30). "NASA Lessons Learned." Retrieved March 02, 2018, from https://www.nasa.gov/offices/oce/functions/lessons/index.html.

Kiewit. (n.d.). "I-25 T-REX Project." Retrieved July 31, 2018, from https://www.kiewit.com/projects/transportation/roads/i-25-t-rex-project/.

Inkpen, A. (1996). "Creating Knowledge through Collaboration." California Management Review, 39(1), pp. 123140.

Marlin, M. (2008). "Implementing an Effective Lessons Learned Process in a Global

Project Environment." UTD 2nd Annual Project Management Symposium

Proceedings. Accessed online at

http://www.westney.com/wp-

content/uploads/2014/05/Implementing-an-Effective-Lessons-Learned-Process-

In-A-Global-Project-Environment.pdf. Retrieved January 25, 2018.

Nonaka, I., and Takeuchi, H. (1995). The Knowledge Creating Company: How Japanese Companies Create the Dynamics of Innovation. Oxford University Press, New York.

Pelnik, T. (2009). VDOT's 3P Program: Successes and Lessons Learned. Final Report. Virginia DOT, Innovative Project Delivery Program, pp. 1013.

Polanyi, M. (2009). The Tacit Dimension. Book. University of Chicago Press. pg. 109. ISBN 0226672980.

113

Sharif, M.N.A., Zakaria, N.H., Ching, L.S., Fung, and L.S. (2005). "Facilitating Knowledge Sharing Through Lessons Learned System." Journal of Knowledge Management Practice.

Shedd, C. (2013, March 30). "How Habits Can Impact User Behavior." inspireUX. Retrieved March 02, 2018, from http://www.inspireux.com/2013/03/30/howhabits-can-impact-user-behavior/.

MassDOT. (n.d.). "The Big Dig: Project Background." Retrieved April 11, 2018, from https://www.mass.gov/service-details/the-big-dig-project-background.

Stewart, G. (1997). "Supply-Chain Operations Reference Model (SCOR): The First CrossIndustry Framework for Integrated Supply-Chain Management." Logistics Information Management, Vol. 10, pp. 62-67.

Von Krogh, G., Ichijo, K., and Nonaka, I. (2000). Enabling Knowledge Creation: How to Unlock the Mystery of Tacit Knowledge and Release the Power of Innovation. Oxford University Press, New York.

Wald, M. L. (2007, July 11). "Collapse of Big Dig Ceiling in Boston Is Tied to Glue." The

New

York

Times.

Retrieved

June,

2018,

from

https://www.nytimes.com/2007/07/11/us/11bigdig.html.

Wallis, S. (2006, September). "Boston Big Dig Ceiling Collapse." TunnelTalk. Retrieved June, 2018, from https://www.tunneltalk.com/Safety-Sep2006-Ceiling-panelcollapse-in-Boston-Big-Dig-tunnel.php.

Weber, R. Aha D.W., and Becerra-Fernandez, I. (2001, January 15). "Intelligent Lessons Learned Systems." ScienceDirect, 20(1), pp. 1734. Retrieved January 04, 2018, from https://www.sciencedirect.com/science/article/pii/S0957417400000464.

114

APPENDICES
115

1. APPENDIX A: STATE DOT SURVEY STATE DOT SURVEY FORM
116

117

118

119

120

121

122

123

124

125

126

127

STATISTICAL ANALYSIS
Figure A-1. Chart. LLP Effectiveness in Improving Construction Process
Figure A-2. Chart. LLP Benefit to DOT and Division 128

Figure A-3. Chart. Level of Contractor Receptiveness Figure A-4. Chart. Are Mistakes Repeated? 129

Figure A-5. Chart. Documenting throughout Life Cycle
Figure A-6. Chart. Usefulness of Push Notifications 130

Figure A-7. Chart. Usefulness of Mobile App Technology
Figure A-8. Chart. Ability to Upload and Catalog Photos and Videos 131

Figure A-9. Chart. GPS Capability Figure A-10. Chart. Usefulness of Common Terminology
132

2. APPENDIX B: SYSTEM TESTING AND CURATION
TRIAL TESTING
Submission Guide for Draft Lessons
3. Submission Guide for Draft Lessons
Your role as a submitter is to create and submit a draft lesson associated with a current or past project. This guide explains the steps you must complete to create a draft lesson in the GDOT Lessons Learned Database. Once you have created and edited a draft lesson, you will submit it to the system curator for review and approval. Once the curator has approved the lesson, it will be sent back to you for final review before it is published to the database system. Procedure to Submit a Draft Lesson Step #1 Go to https://www.kpgoff.tech/GDOTLLP and log in with your GDOT credentials. Step #2 Use the Creation tab and select Create Draft Lesson (figure B-1).
Figure B-1. Image. Step #2 Create Draft Lesson Step #3 Provide the following lesson information. Please note that all fields marked with an asterisk (*) are required.
a. Lesson Title * Include a concise and descriptive title for the lesson learned. b. Lesson Tags Lesson Tags serve to identify the lessons related fields. Tags are the basis
of the notification system and serve as an easy way to search for lessons. You can select
133

tags from the dropdown menu or begin typing a tag name in the textbox. The system has an autocomplete function that will suggest tags that are already existing in the system. If a tag does not exist, then you have the option of submitting a new tag. The curator will review your tag after submission of the draft lesson (figure B-2).
Figure B-2. Image. Step #3b Lesson Tags c. Lesson Abstract * In 500 characters or less, provide a description of the lesson
learned. d. Lesson Location * The lesson location can be specified by inputting an address into
the text box or by using the cursor to select a location on the map. The features on the right-hand side of the map may be used to navigate (figure B-3).
Figure B-3. Image. Step #3d Lesson Location Entry e. Select Projects to Associate * Select the project for which the lesson is associated.
This is the project that the lesson originated from. Projects can be searched for by entering 134

its GDOT Project Identification number or the project title. Project association is designated by clicking the check box to the left of the project ID. This will link the lesson with project information within the database. If the project does not have a GDOT Project Identification number, include a note within the lesson body for the curator. f. Lesson Body* Describe the event or series of events that triggered the lesson. This will inform users of the circumstances leading up to the lesson. Provide detailed information on the lesson learned from the incident. Be specific! You can also provide photos, videos, or audio clips in the next section. g. References Relevant GDOT specifications or codes that address the issues discussed within this lesson should be entered in this section. Text and website links can be added in this section. h. Upload Reports, pictures, videos, voice recordings, drawings, and other project-related files may be uploaded to support the lesson summary. Files may be uploaded by clicking on File Upload. Then, click Upload to finalize the media upload (figure B-4).
Figure B-4. Image. Step #3h File Upload Step #4 Submit the Draft Lesson. Step #5 After the initial submission, the curator will review and make edits. Once finished, you will be notified and given 2 business days to approve or request changes before the lesson is automatically published. Step #6 Submission Process Complete!
135

Trial Test Sample Information
Disclaimer: The information in this document is for the purpose of testing only. Most of the information is imported from legitimate sources, but some of the information is simplified for the purpose of convenience or system trial testing. Thank you for your willingness to participate in this trial test. The research team appreciates your valuable time and intends to make the most of it. Please follow the instructions in the "Submission Guide for Draft Lessons" document and use the following information to fill out the fields. (You are welcome to copy and paste, if that is convenient for you) Lesson Title: Consider using real time traffic control system to overcome mobility and safety obstacles in a work zone. Lesson Tags: Bridges, Construction, Traffic Incident Management Lesson Abstract: During an extensive bridge and highway reconstruction project on Interstate 55, the Illinois Department of Transportation (IDOT) implemented a Real Time Traffic Control System (RTTCS) to ensure safety and mobility during construction. The system successfully monitored traffic along the busy interstate between Springfield (the state capital) and St. Louis, the location of a busy airport serving southern Illinois and eastern Missouri. Incident Summary: Due to significant traffic backups and higher than average rates of traffic violations during the Interstate 55 bridge reconstruction project, the need for an extra safety measure was required. Lesson Location: 45 Baxter St, Athens, GA 30602 Select Projects to Associate: Interstate 55 Bridge Reconstruction Lesson Body: DOT reported that the system performed well, with little downtime. In addition, IDOT staff stated that they would utilize this type of system again in a similar project. The IDOT shares the following experiences with other implementers that may choose to utilize ITS in a work zone.
136

Involve agencies responsible for 911 and other emergency response operations during system planning and design. This effort can help facilitate a coordinated response to incidents during the roadwork.
Use a proactive approach to building public awareness of the project. Successful techniques include holding press conferences, issuing news releases, and keeping local media up to date.
Assess when it is appropriate to use a work zone ITS application and what type of system best meets the site-specific needs.
Ensure that software/systems engineers and transportation engineers use common terminology during the requirements definition process.
Include the vendor's engineering staff, in addition to vendor marketing staff, in early discussions of vendor capabilities.
Allow significant time for system calibration during initial implementation of queue-length detection systems. The calibration process will likely take longer than the best estimate of the time required. The implementation of this system required system calibration that was complicated by the absence of significant traffic congestion. Consequently, the initial deployment phase lasted longer than anticipated.
Expect the need to recalibrate detection systems during the course of the project. IDOT required that the system be deployed on I-55 and tested two weeks prior to initiation of reconstruction activities. The only difficulty encountered was that there was no significant congestion prior to the start of the reconstruction project, which prevented complete calibration of the traffic detection system. Consequently, recalibration of the system was required after the work zone was in place.
IDOT staff identified several benefit areas for the RTTCS system used in this project. The staff reported benefits in both mobility and safety. Despite the work zone location on a busy interstate, the IDOT staff reported no significant traffic backups while the RTTCS system was in place. For the duration of the construction project, only two crashes occurred which were attributed to other causes than the work zone. In addition, there was a significant downtrend in traffic violations after DMSs started notifying drivers approaching
137

the work zone of the number of citations issued in the work zone. This lesson suggests that using ITS in work zones, such as the RTTCS, can be very successful in ensuring safety and mobility within the work zone. References: https://ops.fhwa.dot.gov/publications/fhwahop17056/fhwahop17056.pdf Notes to Curator: Please write your first and last name and any comments you have. File Upload: (Please upload the JPG file attached in the email.)
138

JPG File 139

Trial Test Survey 140

141

Curation Guide to Publish Lessons
Curation Guide to Publish Lessons
The curator's role is to be the gatekeeper for the database by ensuring the published lessons are relevant, accurate, and of sufficient quality. The curator conducts a thorough review of the draft lesson, requests additional information from the submitter (if needed), seeks expert advice, and ultimately guides the lesson to publication. This guide provides the required steps for the curator to access submitted draft lessons, actively edit and refine the information, and publish the lesson. Procedure to Edit and Publish a Lesson by the Curator Step #1 Go to https://www.ktpgof.techh/DOTLLLk and log in with your GDOT credentials Step #2 Use the navigation bar and select Lesson Management. (figure B-5)
Step #3 After clicking the Lesson Management tab, click on the Submitted tab. The interface will display a table of submitted draft lessons. To lock the lesson for curating, press the Curate (figure B-5) button, and the lesson will be moved to the table of lessons you are curating. Select the lesson that you would like to edit and prepare for publication by selecting the Edit (figure B-6) button. This will automatically import the submitted draft lesson information into the editing interface. Step #4 Editing is now enabled for all fields.
a. Lesson Tags Lesson Tags serve to identify the lessons' related fields. Tags are the basis of the notification system and serve as an easy way to search for lessons. You can select tags from the dropdown menu or begin typing a tag name in the textbox. The system has an autocomplete function that will suggest tags that are already existing in the system.
142

i. Case 1: The user has submitted a tag or tags that already exist within the database system If the submitted lesson tag or tags are appropriate, no further action is required for this field.
ii. Case 2: The user has submitted a tag or tags that do NOT exist within the database system The tag will be highlighted in red and contain the symbol. To approve these tags for general use you are not required to do anything to the tag, just leave it in when you submit the lesson and it will be approved. If you do not wish to allow this tag simply press the x button to remove the tag.
Figure B-5. Image. Step #3 Lessons Available for Curating
Figure B-6. Image. Step #3 Lessons You are Curating 143

Figure B-7. Image. Step #4a Lesson Tag Approval i. For the section that is titled "Select projects to associate", a lesson can be associated to a
project from GDOT. This designates which project that the lesson originated from. A lesson can originate from multiple projects, and a project can also yield multiple lessons. You can search for project by entering the GDOT Project Identification #. You can associate or disassociate the lesson from projects by selecting the checkbox next to each project (figure B-9).
Figure B-8. Image. Step #4b Projects to Associate b. Lesson Location This can be specified by inputting an address or by
selecting the location on the map. You can use the features on the right-hand side of the map to navigate.
144

Figure B-9. Image. Step #4c Lesson Location Step #5 After all the edits have been made, select "Preview and Submit" at the bottom of the page. Step #6 A new page will allow you to review before the submission. Before submitting you have the option to set a date for automatic publication of the lesson. By default, this is set to 3 business days to allow for the submitter to review the lesson before it is published. After confirming the submission, it will be sent back to the original submitter for final verification.
Figure B-10. Image. Step #6 Lesson Location Step #7 You have completed the curation process!
145

4. APPENDIX C: LESSON TAGS
Construction Roads Concrete Bridges Design and Deployment Funding Human Resources Leaderships and Partnerships Legal Issues Management and Operations Policy and Planning Procurement Technical Integration Arterial Management Commercial Vehicle Operations Crash Prevention and Safety Driver Assistance Emergency Management Freeway Management Intermodal Freight Road Weather Management Roadway Operations and Maintenance Traffic Incident Management Transit Management Transportation Management Centers Traveler Information Safety Productivity Energy and Environment Efficiency Customer Satisfaction Drainage Fire Damage
146

5. APPENDIX D: SOFTWARE SYSTEM SPECIFICATIONS
Software Requirements Specification
for
Georgia Department of Transportation Lessons
Learned Program
June 2019
147

Introduction

Purpose

The Georgia Department of Transportation Lessons Learned Program (GDOT LLP) is in its initial beta release by the development team at the University of Georgia. The scope of this software requirements specification is to document the features and implementation of the GDOT LLP system.

Document Conventions

A number of terms will be used in this document and are defined here. Georgia Department of Transportation Lessons Learned Program will be commonly referred to as GDOT LLP or LLP. Additional glossary terms and definitions are listed below.

Term Comments Create Lesson
Curator
Database Draft
Help
Lesson Abstract Lesson Body
Lesson Learned Lesson Location Lesson Management
Lesson Map

Definition A written statement expressing a view regarding the lesson. A system interface that allows users to develop and submit a written lesson learned that includes the following fields: Lesson Title, Lesson Tags, Lesson Abstract, Lesson Location, Projects to be Associated With, Lesson Body, File Upload, and References. This interface allows users to save as a draft, save and preview, or submit. A system administrator that has responsibilities of managing users and their roles, approving lesson tags, and reviewing and approving lessons learned. Collection of all the information monitored by the system. A draft lesson is submitted by a system user through the Create Lesson interface. A system interface that allows users to access documents that provide guidance regarding the use of the LLP system. This interface has three documents: "Submission Guide for Draft Lessons," "Curation Guide to Publish Lessons" and "Usage Guide for the ReCORD System." A brief overview of the lesson limited to 500 characters.
A detailed description of the lesson and the incident it originated from. An experience gained by the success or failure of a specific task.
The location for which the lesson was realized.
A system interface that allows users and curators to access and manage the lesson through the submission, review, and published process. A system interface that allows users to view lessons learned on a map. Users are able to view the lesson in relation to counties and/or GDOT districts.

148

Lesson Tag Lesson Title Profile
Published Published Lessons Rating ReCORD References
Review
Submitted
User User Management

An informative keyword that indicates content within the lesson. A descriptive name for the lesson. A system interface provides user personal information such as the email address associated with the LLP system and allows users to register lesson tags and set notification preferences. A lesson that has been reviewed and approved by the curator. A system interface that provides all published lessons within the LLP system. A score given by users of the system. Road COnstRuction Database A list or link of sources that provide additional or relevant information that supports the lesson. A period for which the lesson has been reviewed and approved by the curator but prior to being published and allows times for the lesson submitter to confirm that any changes by the curator are acceptable. This period is typically 3 business days. A period for which a lesson has been submitted by a user and is either available for curating or under curation. Any individual that submits lessons or reviews published lessons. A system interface accessible by only a curator that allows for the managing of users and lesson tags.

Intended Audience and Reading Suggestions
The intended audience for this specification is project managers and software developers. The goal is to provide an abstract view of the system and more detailed specifics for technical implementation.
Product Scope
The development of the ReCORD system will help to unify GDOT's large workforce and footprint throughout the state of Georgia under one system for capturing and reporting of lessons learned. The system will have a systematic and effective method of curating and retrieving lessons learned. As with the adoption of any new method, it is expected that a well thought out and strategic implementation plan should be created to provide support and guidance for the curating and long-term management of the database system. Furthermore, the implementation must illustrate the significance and benefit of the system to its users. The ReCORD system will facilitate the essential task of retaining and dispersing knowledge throughout GDOT, leading to improved design- and constructionrelated decisions being made by employees, thereby, leading to higher working efficiencies throughout the life cycle of projects and organization.
References
Further info on LLPs and the ReCORD system is found in the "Supplemental Documentation for the Software Requirements Specification" being submitted with this
149

software specification. Additionally, the GDOT RP17-16 Final Report, "DEVELOPMENT OF THE ROAD CONSTRUCTION DATABASE (ReCORD) SYSTEM" provides additional background information and guidance regarding the systems development. Information regarding GDOT's security requirements can be found in the GDOT IT Development Procedures document.
Overall Description
Product Perspective The GDOT LLP is a new system of the GDOT software enterprise, and currently only interfaces with the GeoPI software being used by GDOT to track project information. A flowchart is shown above containing a visualization of the system's interconnections. Product Functions Functionality of the system is separated based on the role of the user interfacing with the system. These functionalities are listed in the following and segregated based on role. All Roles:
User System Login and password reset. Settings for notifications. Ability to associate tags for use in notifications.
Notification System Notify users via email when a change occurs on a lesson they are associated with. If enabled, send notifications for new lessons that a user might be interested in.
150

Lesson Viewing and Retrieval System A list of all published lessons. The ability to search any field of these lessons and filter the results. The ability to view all lessons on a map and access their information.
Lesson Management Shows the user all lessons they are affiliated with and whether the lesson is submitted, in review, or published.
User Role: Lesson Editing Ability to submit lessons containing Lesson Title, Lesson Tags, Lesson Abstract, Lesson Location, Project(s) Lesson is Associated With, Lesson Body, References, and File Upload. Ability to save a lesson as a draft before submitting for curation. Ability to retrieve and edit lessons accessible to the user. Create new tag suggestions for their lesson that are to be approved by the curator. Lesson Management The ability to view submitted lessons and who they are being curated by. The ability to request a lesson back for editing from the curator.
Curator Role: Lesson Management Shows a list of all submitted lessons by users available for curation. The ability to claim a lesson from a user to curate, locking it from editing from the user or by other curators. The ability to send back a lesson to a user for edits. The ability to set an auto-publish date for lessons. Lesson Editing Can edit any lesson they have claimed or have access to. Same editing capabilities as user, but allows the curator to input new tags without requiring approval. User Management Can create and invite new individuals and set their role as either user or curator in the system. Can activate and deactivate users from a control panel.
User Classes and Characteristics
The GDOT LLP has three classes of users: user, curator, and administrator. In the context of the LLP a user is the base level role whose main interactions with the interface will be submitting new lessons and viewing published lessons. A curator is a higher-level role whose work with the LLP will be reviewing and editing submitted lessons and approving for publication. The administrator is a superuser level class, and has the powers of all classes below it as well as full access to all lessons and users.
151

Operating Environment
The current GDOT LLP system runs on Debian 9 Stretch running on a VM Instance on Google Cloud Engine. The current software used to run the server are: Apache 2.4.39, Apache Tomcat 9.0.20, MySQL 5.7.26, Java OpenJDK 1.8, and Manticore Search 3.0.
Design and Implementation Constraints
There are no current implementation constraints.
User Documentation
The LLP includes three documents containing information on the user interface and how users and curators can properly use the system. These documents accessible from the Help interface are titled:
"Submission Guide for Draft Lessons" "Curation Guide to Publish Lessons" "Usage Guide for the ReCORD System"
System Features
User System
Description and Priority This is a high priority system. The LLP must contain a robust user system for linking lessons to users and curators, and in order to notify users of potential new lessons. Stimulus/Response Sequences The user should be able to perform the following actions:
Register using a GDOT email address. Receive an email for registration, access the link, and create a password for logging in.
Log into the system and remain logged in, and have the option to remember the user's device in order to remain logged in. The curator should be able to create users of any role, and has the ability to activate and deactivate any user.
Access a profile containing their personal information and have the ability to change that information.
Have a tagging system on the user profile that will allow the user to receive notifications for lessons related to their tags.
152

Lesson Submission
Description and Priority
This is a high priority system. The LLP must contain a system that allows the base user role to easily submit a lesson from anywhere containing all pertinent information. The lesson must contain a Lesson Title, Lesson Tags, Lesson Abstract, Lesson Location, Project(s) Lesson is Associated With, Lesson Body, References, and File Upload capability to optionally add any number of multimedia files.
Stimulus/Response Sequences
The user should be able to perform the following actions: Quickly access the new lesson creation interface. Create new lessons with Lesson Title, Lesson Tags, Lesson Abstract, Lesson Location, Project(s) Lesson is Associated With, Lesson Body, References, and File Upload capability to optionally add any number of multimedia files. Save the lesson as a draft before submission. Retrieve lessons from their profile for editing.
Lesson Management
Description and Priority
This is a high priority system. The LLP must contain a system that allows the users and curators to view and manage all lessons they are affiliated with. For the users, this allows them to edit and retrieve their own lessons. For curators this allows an interface for viewing all submitted lessons, and an ability to claim and edit them.
Stimulus/Response Sequences
The curator should be able to perform the following actions: View and search all submitted lessons. Claim a lesson for curation, locking it from editing by the submitting user and other curators, unless permission is granted. Send a lesson back to the user for editing. Set an auto-publish date for a lesson after submitting it to the Review stage. Access a lesson they have claimed and edit it with the same interface as a user.
The user should be able to perform the following actions: View a list of all of their lessons in any of the four stages (Draft, Submitted, Review, Published). Edit their lesson only if it is in the draft stage. Request their lesson be brought back to the draft stage by a curator so they can edit the lesson.
153

Lesson Viewing and Retrieval
Description and Priority This is a high priority system. The LLP must contain a system that allows the users and curators to view and retrieve all published lessons in the system. This involves allowing for an extensive search feature. Stimulus/Response Sequences The user should be able to perform the following actions:
View a list of all the published lessons within the system. The viewing should include the Lesson Title, Lesson Tags, Lesson Abstract, and Lesson Location with the ability to view the full lesson via a link.
Search all fields of the lessons (Lesson Title, Lesson Tags, Lesson Abstract, Lesson Location, Lesson Body) for their desired search term.
Search via geolocation and view all published lessons on a map. Refine the search via geolocation by district or county.
View a formatted version of the lesson including all multimedia, and a link to other related lessons.
User Management
Description and Priority This is a medium priority system. The LLP must contain a system that allows new users to register through their GDOT email, or other desired authentication system. It must also allow for individual roles above the user to create, manage, and edit users. Stimulus/Response Sequences System roles above the user should be able to perform the following actions: Register users to any email address, and with any role equal or lower to their
current role. Activate and deactivate users at will.
The user should be able to perform the following actions: Register a new account using an automated process. Change settings on their profile for notifications, tags, and personal
information.
Notification System
Description and Priority This is a low priority system. The LLP must contain a system that will notify users regarding changes to their lessons, profile, or general site changes. In addition, it
154

must have the capability of notifying users when new lessons are published that pertain to their tags. Stimulus/Response Sequences The user should be able to perform the following actions: Change settings on their profile for notifications, tags, and personal
information. Receive notifications when changes occur on an affiliated lesson. Receive notifications when user or site settings change. Receive notifications for relevant lessons related to their tag preferences.
Other Nonfunctional Requirements
Performance Requirements
Given the general nature of the data in the system (textual and small multimedia files) it is unlikely that performance will be an issue in the long-term. However, it is recommended that the server running the search service have at least 2GB of RAM to ensure optimal indexing and search speeds. Safety Requirements All safety concerns can be referred to the GDOT IT Development Procedures document. Security Requirements All security concerns can be referred to the GDOT IT Development Procedures document. Software Quality Attributes
The use of lessons learned databases must be incorporated into a regular and routinely used system of operation for maximum efficiency. In many cases, the lessons learned system is not regularly utilized and falls out of favor. In these instances, it is unlikely that the lessons learned will be of much value and decreases the likelihood of the program being brought back into routine use. However, if the lessons learned system is used regularly, it can provide desirable results such as increased working efficiency and a reduction in resources such as money and manpower. Other performance-based benefits include improved productivity, cost, quality, and overall performance.
155