Curve safety improvements using mobile device and automatic curve sign detection

GEORGIA DOT RESEARCH PROJECT 19-26 FINAL REPORT
CURVE SAFETY IMPROVEMENTS USING MOBILE DEVICE AND AUTOMATIC CURVE SIGN DETECTION PHASE II
OFFICE OF PERFORMANCE-BASED MANAGEMENT AND RESEARCH
600 WEST PEACHTREE STREET NW ATLANTA, GA 30308

TECHNICAL REPORT DOCUMENTATION PAGE

1. Report No.: FHWA-GA-21-1926

2. Government Accession No.: N/A

3. Recipient's Catalog No.: N/A

4. Title and Subtitle: Curve Safety Improvements Using Mobile Devices and Automatic Curve Sign Detection Phase II
7. Author(s): Yichang (James) Tsai, Ph.D., P.E. (https://orcid.org/00000002-6650-2279), Pingzhou Lucas Yu, Tianqi Liu, Ronald Knezevich, Chin Wang. 9. Performing Organization Name and Address: Georgia Institute of Technology 790 Atlantic Drive Atlanta, GA 30332-0355 Email: james.tsai@ce.gatech.edu 12. Sponsoring Agency Name and Address: Georgia Department of Transportation Office of Performance-Based Management and Research 600 West Peachtree St. NW Atlanta, GA 30308

5. Report Date: May 2021 6. Performing Organization Code: N/A
8. Performing Organ. Report No.: 19-26
10. Work Unit No.: N/A 11. Contract or Grant No.: PI# 0016970
13. Type of Report and Period Covered: Final Report; October 2019 May 2021
14. Sponsoring Agency Code: N/A

15. Supplementary Notes:

Prepared in cooperation with the U.S. Department of Transportation,

Federal Highway Administration.

16. Abstract:

A disproportionally high number of fatal crashes (25%) occur on horizontal curves, even though curves

represent only a fraction of the roadway network (5% of highway miles) (FHWA, 2010; FHWA, 2018).

The MUTCD (Manual on the Uniform Traffic Control Devices) (FHWA, 2012) requires various

horizontal alignment warning signs (curve signs) to ensure roadway safety on curves. However, current

transportation agencies' practices for inventorying existing curve signs are typically a labor-intensive,

time-consuming, and manual procedure. This report presents automatic curve sign detection (ACSD)

using low-cost mobile devices such as smart phones, machine learning, and crowdsourcing to develop a

live curve sign inventory. This inventory can be used to analyze MUTCD curve sign compliance for

cost-effective and time sensitive safety improvements. Phase II of this research project focuses on 1)

validating the curve sign computation accuracy, 2) recommending data storage and management

alternatives, and 3) recommending a methodology for MUTCD curve sign compliance analysis

(MCSCA), and 4) presenting a roadmap for implementing the proposed methodology. Based on the

preliminary assessment conducted in Phase I, the proposed ACSD is promising. Based on tests

performed in Phase II, the proposed triangulation method, using the images and GPS data collected

using smartphones, can achieve accuracy within 10 meters for most cases. However, some cases can

have an error greater than 15 meters. Error cases typically occur when there is a significant change in

vertical alignment or a rough roadway surface. Future refinement on the triangulation method that

accounts for the vehicle's pitch angle is recommended. Additionally, data storage and management

alternatives and a methodology for MUTCD curve sign compliance analysis are recommended. Finally, a

roadmap for implementing the proposed ACSD, live curve sign inventory, and MCSCA is presented.

17. Key Words:

18. Distribution Statement:

MUTCD Compliant Curve Sign Detection, Safety, Low-cost

No Restriction

Mobile Device, Machine learning, Crowdsourcing

19. Security Classification (of 20. Security classification (of 21. Number of 22. Price:

this report): Unclassified

this page): Unclassified

Pages: 80

Free

Form DOT 1700.7 (8-72)

Reproduction of completed page authorized.

GDOT Research Project 19-26
Final Report
CURVE SAFETY IMPROVEMENTS USING MOBILE DEVICE AND AUTOMATIC CURVE SIGN DETECTION
PHASE II
By Yichang (James) Tsai, Ph.D., P.E. Professor of Civil and Environmental Engineering
Pingzhou (Lucas) Yu Graduate Research Assistant
Tianqi Liu Graduate Research Assistant
Ronald Knezevich Graduate Research Assistant
Chin Wang Graduate Research Assistant
Georgia Tech Research Corporation
Contract with Georgia Department of Transportation
In cooperation with U.S. Department of Transportation Federal Highway Administration
May 2021
The contents of this report reflect the views of the author(s) who is (are) responsible for the facts and the accuracy of the data presented herein. The contents do not necessarily reflect the official views or policies of the Georgia Department of Transportation or the Federal Highway Administration. This report does not constitute a standard, specification, or regulation.
ii

Symbol
in ft yd mi
in2 ft2 yd2 ac mi2
fl oz gal ft3 yd3
oz lb T
oF
fc fl
lbf lbf/in2

SI* (MODERN METRIC) CONVERSION FACTORS

APPROXIMATE CONVERSIONS TO SI UNITS

When You Know

Multiply By

To Find

LENGTH

inches

25.4

millimeters

feet

0.305

meters

yards

0.914

meters

miles

1.61

kilometers

AREA

square inches

645.2

square millimeters

square feet

0.093

square meters

square yard

0.836

square meters

acres

0.405

hectares

square miles

2.59

square kilometers

VOLUME

fluid ounces

29.57

milliliters

gallons

3.785

liters

cubic feet

0.028

cubic meters

cubic yards

0.765

cubic meters

NOTE: volumes greater than 1000 L shall be shown in m3

MASS

ounces

28.35

grams

pounds

0.454

kilograms

short tons (2000 lb)

0.907

megagrams (or "metric ton")

TEMPERATURE (exact degrees)

Fahrenheit

5 (F-32)/9

Celsius

or (F-32)/1.8

ILLUMINATION

foot-candles foot-Lamberts

10.76 3.426

lux candela/m2

FORCE and PRESSURE or STRESS

poundforce

4.45

newtons

poundforce per square inch

6.89

kilopascals

Symbol
mm m m km
mm2 m2 m2 ha km2
mL L m3 m3
g kg Mg (or "t")
oC
lx cd/m2
N kPa

Symbol
mm m m km
mm2 m2 m2 ha km2
mL L m3 m3
g kg Mg (or "t")
oC
lx cd/m2
N kPa

APPROXIMATE CONVERSIONS FROM SI UNITS

When You Know

Multiply By

To Find

LENGTH

millimeters

0.039

inches

meters

3.28

feet

meters

1.09

yards

kilometers

0.621

miles

AREA

square millimeters

0.0016

square inches

square meters

10.764

square feet

square meters

1.195

square yards

hectares

2.47

acres

square kilometers

0.386

square miles

VOLUME

milliliters

0.034

fluid ounces

liters

0.264

gallons

cubic meters

35.314

cubic feet

cubic meters

1.307

cubic yards

MASS

grams

0.035

ounces

kilograms

2.202

pounds

megagrams (or "metric ton")

1.103

short tons (2000 lb)

TEMPERATURE (exact degrees)

Celsius

1.8C+32

Fahrenheit

ILLUMINATION

lux candela/m2

0.0929 0.2919

foot-candles foot-Lamberts

FORCE and PRESSURE or STRESS

newtons

0.225

poundforce

kilopascals

0.145

poundforce per square inch

Symbol
in ft yd mi
in2 ft2 yd2 ac mi2
fl oz gal ft3 yd3
oz lb T
oF
fc fl
lbf lbf/in2

**SSIIisisththeesysmymboblofolrfothrethIneteIrnntaetrionnaatiloSnyaslteSmysotfeUmniotsf. UAnpiptsro. pAripatperoropurniadtiengrosuhonudlidnbgesmhoaudledtobecommapdlyewtoithcoSmecptiloynw4iothf ASSeTcMtioEn3480o. f ASTM E380. ((RReevviisseedd MMaarrcchh2200030)3)

iii

TABLE OF CONTENTS
EXECUTIVE SUMMARY............................................................................................... 1 CHAPTER 1. INTRODUCTION .................................................................................... 5
RESEARCH OBJECTIVES AND SCOPE ................................................................ 7 REPORT ORGANIZATION ....................................................................................... 8 CHAPTER 2. VALIDATION AND REFINEMENT OF SIGN LOCALIZATION USING TRIANGULATION METHOD ......................................................................... 9 OVERVIEW OF DIFFERENT LOCALIZATION METHODS ............................ 10
Two-Point Triangulation ........................................................................................ 10 Simultaneous Localization and Mapping ..............................................................11 Structure from Motion ............................................................................................11 NAIVE TWO-POINT TRIANGULATION METHOD .......................................... 12 PROPOSED REFINEMENT TO THE TRIANGULATION METHOD .............. 15 VALIDATION OF THE PROPOSED CURVE SIGN LOCALIZATION METHOD .................................................................................................................... 17 Dataset Used for Validation: .................................................................................. 18 Evaluation Test Design: .......................................................................................... 21 Evaluation Results and Interpretation: ................................................................ 21 SUMMARY ................................................................................................................. 27 CHAPTER 3. RECOMMENDATION OF AN ENHANCED PROCEDURE FOR COST-EFFECTIVE MUTCD CURVE SIGN COMPLIANCE ANALYSIS ............ 29 ESTABLISHING THE CURVE SIGN BASELINE USING MUTCDCOMPLIANT CURVE SIGN DESIGN REQUIREMENTS.................................. 31 Curve Geometry...................................................................................................... 34 Curve Type............................................................................................................... 34 Posted Speed Limit ................................................................................................. 34 Curve Advisory Speed Limit.................................................................................. 34 Example Using MUTCD Curve Sign Design for Establishing A Baseline Curve Sign Design .............................................................................................................. 36 DETECT EXISTING CURVE SIGNS USING AUTOMATIC CURVE SIGN DETECTION, CLASSIFICATION, AND LOCALIZATION................................ 38
iv

Detection .................................................................................................................. 38 Classification ........................................................................................................... 39 Localization ............................................................................................................. 39 MUTCD CURVE SIGN COMPLIANCE ANALYSIS (MCSCA).......................... 40 POTENTIAL USE AND BENEFITS OF THE RECOMMENDED COMPLIANCE ANALYSIS ...................................................................................... 41 CHAPTER 4. EXPLORATION AND RECOMMENDATION OF DATA STORAGE AND DATA MANAGEMENT PLAN ........................................................................... 43 ALTERNATIVES OF DATA MANAGEMENT SYSTEM FOR ESTABLISHING AN ACSD-BASED LIVE CURVE SIGN INVENTORY SYSTEM ....................... 44 Alternative 1: Full Field Data Transmission with Centralized Servers for Data Processing. ............................................................................................................... 44 Alternative 2: Optimized Field Data Transmission with On-Device Local Data Processing. ............................................................................................................... 47 EXPLORATION OF THE SMARTPHONE VERSION OF ACSD....................... 50 Network Architecture and Weight Compression ................................................. 51 Evaluation of The Proposed Network ................................................................... 54 SUMMARY ................................................................................................................. 56 CHAPTER 5. ROADMAP FOR MUTCD CURVE SIGN COMPLIANCE ANALYSIS FOR SAFETY IMPROVEMENTS .......................................................... 58 CHAPTER 6. CONCLUSIONS AND RECOMMENDATIONS................................ 66 ACKNOWLEDGMENTS .............................................................................................. 69 REFERENCES................................................................................................................ 70
v

LIST OF FIGURES
Figure 1. Illustration. The geometric relationship between traffic sign and each camera view............................................................................................................................ 14
Figure 2. Illustration. Simplified geometric relationship of the Triangulation method. ... 14 Figure 3. Equation. Equations for computing traffic sign coordinates. ............................ 15 Figure 4. Equation. Equations for computing sign coordinates with deviation angle
change........................................................................................................................ 16 Figure 5. Illustration. The geometric relationship between camera views and the traffic
sign when vehicle trajectory is on a curve................................................................. 17 Figure 6. Photo. Georgia Tech Sensing Vehicle Camera. ................................................. 18 Figure 7. Map. Test data collection sites........................................................................... 19 Figure 8. Photo. Example images of different roadway geometries. ................................ 20 Figure 9. Chart. Impact of roadway geometry on localization accuracy. ......................... 24 Figure 10. Chart. Impact of the data collection system on localization accuracy............. 24 Figure 11. Photo. Error cases on straight roadways, (a) left image case: GTSV-I75-2, (b)
right image case: GTSV-I285. ................................................................................... 25 Figure 12. Photo. Horizon level change in consecutive frames caused by pitch angle
oscillation, the red line indicates the horizon level in the left image, and the orange line indicates the right image horizon. ...................................................................... 26 Figure 13. Chart. Impact of the pitch angle change on localization accuracy. ................. 27 Figure 14. Diagram. Components of the Cost-Effective MUTCD Curve Sign Compliance Analysis Procedure. ................................................................................................... 30 Figure 15. Figure. Horizontal Alignment Warning Signs (MUTCD Figure 2C-1)........... 32 Figure 16. Equation. Equations for computing curve advisory speed. ............................. 36 Figure 17. Illustration. Example of automatically generated baseline curve sign design. 37 Figure 18. Illustration. Example of curve sign compliance analysis using existing curve signs and baseline curve sign design. ........................................................................ 41 Figure 19. Diagram. Key components/modules of the data management system alternatives................................................................................................................. 43 Figure 20. Diagram. Illustration of the data management alternative 1. .......................... 44 Figure 21. Diagram. Illustration of the data management alternative 2. .......................... 47 Figure 22. Photo. Nvidia Jetson device line-up. (Nvidia)................................................. 49 Figure 23. Photo. Example of an assembled stand-alone field data collection unit. ........ 49 Figure 24. Illustration. Graphical representation of different types of computation blocks. ................................................................................................................................... 52 Figure 25. Illustration. Visual representation of the network architecture........................ 54 Figure 26. Chart. Roadmap for MUTCD curve sign compliance analysis for safety improvements. ........................................................................................................... 59
vi

LIST OF TABLES Table 1. Nave Triangulation method (GTSV). ................................................................ 22 Table 2. Nave Triangulation method (Smartphone)......................................................... 22 Table 3. Refined Triangulation method (GTSV). ............................................................. 22 Table 4. Refined Triangulation method (Smartphone). .................................................... 22 Table 5. Horizontal alignment warning sign usage in different roadway conditions ....... 33 Table 6. Comparison of the six methods for establishing curve advisory speeds............. 35 Table 7. Description of the neural network architecture. .................................................. 53 Table 8. Processing speed and accuracy of different architectures................................... 55
vii

LIST OF ABBREVIATIONS

AASHTO ACSD ACSIS AI AID BBI BN CARS CPU DOT FHWA FLOP GPS GPU GTSV IMU IoT LCSI MCSCA MUTCD NCHRP

American Association of State Highway and Transportation Officials Automatic Curve Sign Detection Automatic Curve Sign Inventory System Artificial Intelligence Accelerated Innovation Deployment Ball Back Indicator Batch Normalization Curve Advisory Reporting System Central Processing Unit Department of Transportation Federal Highway Administration Floating Point Operation Global Positioning System Graphics Processing Unit Georgia Tech Sensing Vehicle Inertial Measurement Unit Internet of Things Live Curve Sign Inventory MUTCD Curve Sign Compliance Analysis Manual on Uniform Traffic Control Devices National Cooperative Highway Research Program
viii

PC PT SFM SLAM SR TRAMS

Point of Curve Point of Tangent Structure from Motion Simultaneous Localization and Mapping State Route Texas Roadway Analysis and Measurement Software

ix

EXECUTIVE SUMMARY
In the United States, a disproportionally high number of fatal crashes (25%) occur on horizontal curves, even though curves represent only a fraction of the roadway network (5% of highway miles) (FHWA, 2021). The Manual on the Uniform Traffic Control Devices (MUTCD) (FHWA, 2012) requires various horizontal alignment warning signs (curve signs) to ensure roadway safety on curves. However, many transportation agencies' current practices for inventorying the locations and types of existing curve signs are typically labor-intensive, time-consuming. Moreover, the locations and types of curve signs cannot always be accurately and reliably collected using visual inspection. Therefore, there is an urgent need to develop a method that improves current manual curve sign inventory and supports MUTCD-compliance analysis with frequent curve sign assessment. The objective of this research is to develop and assess a method that uses low-cost mobile devices and machine learning to perform automatic curve sign detection (ACSD) which is the core component for establishing a curve sign inventory base on the live data collected in the field. The live curve sign inventory (LCSI) can be used to effectively identify deficiencies, such as missing signs, in a time-sensitive manner to support a proactive curve sign improvement method for meeting the MUTCD requirements. This report presents an ACSD approach, using low-cost mobile devices (e.g., smartphones), existing agency vehicles, machine learning, and crowdsourcing that can be used to develop a live curve sign inventory and analyze the MUTCD curve sign
1

compliance. This procedure can be used for cost-effective and time-sensitive analysis for proactive safety improvements. Based on the preliminary assessment conducted in Phase I, the proposed ACSD is promising in detecting curve signs collected using images from smartphones, and machine learning. This Phase II research project focuses on validating the curve sign computation accuracy, recommending data storage and management alternatives, recommending a methodology for MUTCD curve sign compliance analysis (MCSCA), and presenting roadmap for implementing the proposed ACSD, live curve sign inventory, and MCSCA. To date, there is no developed method that performs a live curve sign inventory using the combined capabilities of mobile devices and automatic curve sign detection methods with machine learning.
Based on tests performed in Phase II, the proposed triangulation method, using the Global Positioning System (GPS) data collected using smart phones, can achieve the localization accuracy with a median value of 6.18 meters and about 70% of cases can achieve error lower than 10 meters. However, some cases can have an error greater than 15 meters, skewing the average accuracy to 10.86 meters. The error cases typically occur when there is a significant change in the vehicle's pitch angle between the two positions used for triangulation computation. The pitch angle change can be caused by a significant change in vertical alignment of the roadway or from rough roadway surfaces that causes the vehicle to rapidly pitch back and forth. Future refinement that accounts for a vehicle's pitch angle in sign localization computation is recommended.
Two data storage and management alternatives are explored in this study by evaluating their components. However, further study is needed to explore and test different data
2

storage and management alternatives. There are four key components included in the data storage and management alternatives are:
1) Field data collection using low-cost mobile devices. 2) Data transfer and communication using either agency's local wireless network
and/or future 5G infrastructure. 3) Data processing and analysis using ACSD in server and in the field. 4) Data storage and management.
A methodology that utilizes ACSD for MCSCA is recommended to allow for costeffective and timely safety improvements. The recommended methodology includes three components: first, establishing the curve sign baseline using the MUTCD-compliant curve sign design requirement, second, detecting existing curve signs using automatic curve sign detection, and finally, analyzing the MUTCD curve sign compliance by comparing signs that are required or recommended on curves specified by the MUTCD to the existing signs. Finally, and a roadmap for implementing the proposed ACSD, live curve sign inventory, and MCSCA is for curve sign data collection is also presented. ACSD is completed with low-cost mobile devices and machine learning to collect curve sign data such as sign type and location. This data can be used for MUTCD compliance analysis and to effectively delegate appropriate safety improvements. This roadmap is intended to inform GDOT engineers on projects that lie in the future to effectively incorporate MUTCD compliance analysis in standard operating practice with the implementation of ACSD. This project
3

proposed an implementation roadmap for improving roadway safety by applying the innovative ACSD method. The overarching goal of the roadmap is to allow for the cost-effective curve sign data collection and existing sign MUTCD compliance analysis. Three necessary stages are recommended, 1) proof of concept which is completed in Phase I and II, 2) pilot studies from GDOT or the FHWA Accelerated Innovation Deployment (AID) to accelerate the implementation and adoption of innovation, and 3) large-scale deployment of the developed innovation for transportation agencies. Based on the roadmap, it is suggested to:
1) Explore data reduction and management alternatives. 2) Enhance sign location accuracy and speed. 3) Utilize a diverse data set to support a robust ACSD system. 4) Develop a live curve sign inventory (LCSI). 5) Develop standard MUTCD curve sign compliance analysis (MCSCA) practices. 6) Develop a standard QA/QC procedure for the for ACSD and MCSCA. 7) Map the safety improvement needs identified from MCSCA. 8) Conduct pilot studies for small scale implementation. 9) Develop a database to support LCSI and store MCSCA results. 10) Incorporate appropriate policy, training, and standards to implement the
recommended practices.
4

CHAPTER 1. INTRODUCTION
According to FHWA (FHWA, 2021), more than 25% of fatal crashes occur on horizontal curves that constitute less than 5% of highway pavements. The Manual on the Uniform Traffic Control Devices (MUTCD) (FHWA, 2012) requires various horizontal alignment warning signs (curve signs) to ensure roadway safety and reduce the disproportionally high fatal crash rate on horizontal curves. However, many transportation agencies' current practices for inventorying the locations and types of existing curve signs are typically labor-intensive, time-consuming. Moreover, the locations and types of curve signs cannot always be accurately and reliably collected using visual inspection. Therefore, there is an urgent need to develop a method that improves current manual curve sign inventory methods and also supports MUTCD-compliance with frequent curve sign assessment. Due to the safety-sensitive nature of MUTCD curve sign requirements, time sensitive safety assessment and improvement actions are required to identify missing and substandard curve signs. Substandard curve signs can include improper sign types, improper spacing, and damaged signs. It is vital for assessment of road sign condition to be conducted time-efficiently to minimize the potential risk to drivers therefore minimizing the potential liability that poor curve sign design can cause transportation agencies. The Georgia Department of Transportation (GDOT) is actively seeking cost-effective and sustainable approaches to develop a live curve sign inventory and perform MUTCD curve sign compliance analysis in a timely manner. These approaches should identify missing signs and analyze sign types and spacing to ensure they meet MUTCD requirements.
To achieve this long-term endeavor, Georgia Tech has worked closely with GDOT to develop an automatic curve sign detection (ACSD) method that will enable transportation agencies to cost-
5

effectively detect curve signs and classify their MUTCD sign types. The method will also compute curve sign locations using a set of roadway video log images that have been collected by low-cost mobile devices (smart phones) that use machine learning and crowdsourcing. The developed ACSD will enable transportation agencies to develop a live curve sign inventory, and perform cost-effective MUTCD curve sign compliance analysis (MCSCA) for safety improvements in a timely manner; it will identify missing signs and analyze the sign types and spacing to ensure they meet MUTCD requirements. This research project leverages previous research outcomes, including automatic sign detection and curve identification, developed by the principal investigator (PI) in research projects sponsored by the United State Department of Transportation (US DOT) (Tsai and Wang, 2013), the Federal Highway Administration (FHWA), The National Cooperative Highway Research Program (NCHRP) (Tsai and Wang, 2009), and GDOT. This research investigated the use of low-cost smart phones to collect video log images with GPS data to automatically extract curve sign types and MUTCD codes, to compute the locations of signs for a live curve sign inventory, and to analyze for MUTCD compliance. The same low-cost smart phones are used to collect accelerations in x, y, and directions, Gyro data, and GPS data to perform curve safety assessments in a separate research project (sponsored by the NCHRP and co-sponsored by GDOT) to analyze and extract the detailed level of curve sign information (e.g. point of curve, point of tangent, curve radius, deviation angle), compute ball bank indicator (BBI) data, compute super-elevation, and compute advisory speed, etc. for curve safety assessment at network level. BBI is a combined indicator of curve radius, super-elevation, and driving speed that gauges the side friction a vehicle undergoes when navigating a curve.
The original research project was composed of the following sub-tasks:
6

a) adjusting hardware configuration to maximize the automatic curve sign detection; b) collecting sensor data using low-cost mobile devices; c) automatically computing curves, including curve location, radius (R), point of
curve (PC), and point of tangent (PT); d) refining the artificial intelligence (AI) algorithms to better detect curve signs and
recognize their MUTCD sign type; e) computing signs' locations/coordinates using GPS data and camera calibration
parameters; f) generating sign inventory with sign type, location (x-y coordinates), and images; g) recommending a data storage and management plan; h) exploring curve sign applications (e.g., comparing the curve sign inventory with
the MUTCD requirements to identify sign deficiencies and using different time stamps to identify changes).
The research project was divided into two phases with two separate research projects. Phase I includes Sub-tasks a) to d) and focuses on validating and refining the AI algorithms for curve sign detection (Tsai, et. al., 2020). Phase II includes Sub-tasks e) to h) and focuses on developing a method for curve sign location computation using camera calibration and validation of its accuracy. This research project (RP 19-26) is to work on the Phase II.
RESEARCH OBJECTIVES AND SCOPE The objective of the Phase II portion of this research project is to critically validate a curve sign computation method using low-cost smart phones and to recommend data management alternatives for large-scale implementation. The major tasks in Phase II of this research project
7

include the following: 1) Calibrate and validate a sign location computation method. 2) Recommend a methodology for MUTCD compliance analysis for safety improvement. 3) Explore and recommend data storage and data management alternatives. 4) Develop a roadmap for implementing curve sign data collection using low-cost mobile devices. 5) Summarize research findings.
REPORT ORGANIZATION This research project report is organized as follows. Chapter 1 presents the background, the need for, and the objective of developing ACSD, especially focusing on curve sign location computation. Chapter 2 presents a curve sign location computation method (using the triangulation method) and its validation. Chapter 3 recommends an enhanced methodology for cost-effective MUTCD curve sign compliance analysis. Chapter 4 explores and recommends data storage and data management for large-scale implementation. Chapter 5 presents a roadmap for implementing curve sign data collection using low-cost mobile devices. Finally, Chapter 6 presents conclusions and recommendations.
8

CHAPTER 2. VALIDATION AND REFINEMENT OF SIGN LOCALIZATION USING TRIANGULATION METHOD
A frequently updating curve sign inventory is needed to support a live MUTCD curve sign inventory and compliance analysis system that confirms correct sign types and spacings are used and continuously monitors the condition of the traffic signs to identify sign replacement and repair needs. This is especially important for identifying missing signs caused by crashes, natural disasters, and other causes. Using conventional, manual methods to collect and update sign inventory is time-consuming, expensive, and cannot update sign conditions frequently enough to minimize the risks for transportation agencies. Therefore, to solve these problems, an automatic curve sign detection (ACSD) and inventory system is needed. To construct an inventory system like this, the location of the sign is an essential attribute. Nowadays, even though automatic sign detection can be handled by deep learning object detection models using vehicle onboard videos, the location of the sign still needs to be extracted from the videos. Therefore, the techniques for localizing the sign should be developed for completing the inventory system, helping agencies to meet the current MUTCD requirement, ultimately improving the safety of road users therefore, minimizing the risks and liability for transportation agencies.
To ensure a traffic sign localization method can efficiently and accurately localize roadway signs, tests and evaluations should be performed to ensure their accuracy. Therefore, this chapter presents the methodology and evaluation of a proposed traffic sign localization method that can extract traffic sign location from vehicle onboard video to support an automatic sign detection and inventory system. This chapter first present an overview of the available sign localization methods and select one method for further investigation. Then, a refined approach is proposed
9

base on the selected method. Finally, the evaluation design of the proposed method is presented, and the evaluation results are interpreted to identify error-causing factors to formulate conclusion and future recommendations.
OVERVIEW OF DIFFERENT LOCALIZATION METHODS Since identifying the locations of traffic signs is essential for supporting MUTCD curve sign compliance analysis, it is important for an automatic curve sign detection and inventory system to have the capability to perform sign localization and be able to determine the location of detected traffic signs. The objective is to compute the precise location of a sign using video image sequences with their corresponding GPS coordinates. Three sign localization methods are briefly introduced in this study. They are triangulation using camera calibration, simultaneous localization and mapping (SLAM), and structure from motion (SFM).
Two-Point Triangulation The two-point triangulation method requires a traffic sign to be captured in two different camera views. Using camera calibration, the relative direction between the traffic sign and the camera position can be found. When there are two different camera views available, and the GPS coordinates of the camera positions in both views are known, triangulation can be used to pinpoint a traffic sign's coordinates. To be more specific, as the images are extracted from the video, we use the two images that are closest to a sign as two consecutive image locations. When a sign is closer to the camera, the automatic sign detection models have the best chance to accurately detect the position of a sign within each frame. Once we have a sign's position represented as a bounding box, we can use the centroid point of the bounding box as the location ( and ) as shown in Figure 1. As for the distance between two images, because the car cannot
10

drive at a constant speed, the distance can be calculated by inputting the GPS coordinates of two images into the data package in Python. Consequently, the geometric relationship between the locations of two images and the location of a sign can be developed using the position of a sign in each camera view, the focal length of the camera, and the distance between two camera views for the triangulation method.
Simultaneous Localization and Mapping Simultaneous localization and mapping (SLAM) (Durrant-Whyte & Bailey, 2006) uses a sequence of keyframes and tracks obvious key points (or landmarks) in each frame to reconstruct the scene and camera poses across different frames. SLAM involves building a map of the environment while also solving the location of each viewpoint; thus, the GPS coordinate of each viewpoint is not used. To construct the scene, the keyframes, which are extracted in the image sequence, will be used for bringing new information to the localization process. In other words, SLAM addresses the problem of an agent navigating in an unknown environment. The agent seeks to build a spatial map of the environment while simultaneously localizing its position relative to the map. In our case, the agent is the camera used to record the road. However, there are few issues with utilizing SLAM for sign localization. For example, if the camera moves too fast, successive keyframes are too different from each other, causing the algorithm to be unable to reconstruct the scene. In addition, if the camera stops, the location of signs will be falsely computed because the keyframes are identical so that these same points cannot be used in the algorithm to create the 3D map.
Structure from Motion Structure from motion (SFM) (Ullman, 1979) tries to achieve results similar to SLAM; it tries to
11

reconstruct a 3D scene from a sequence of 2D images. Unlike SLAM, the assumption is that the position of each viewpoint is known. SFM refers to the problem of inferring 3D structures from a sequence of images when the camera is in relative motion with respect to the scene. The input of an SFM algorithm is a set of images with known camera positions. The algorithm produces two outputs: the 3D reconstruction of the scene and the sequence of camera poses. However, in preliminary experimentation, SFM had issues when the provided 2D images were too few and sparse, causing only a partial 3D scene to be constructed and leaving part of the scene, which may contain the traffic sign, not to be reconstructed. In addition, SFM is computationally expensive due to the reconstruction process; it tries to map every pixel in the image to the 3D world, leading to long processing times that are unrealistic for batch processing or real-time processing.
Since the localization component should serve as part of the automatic curve sign detection system, the computation overhead should be minimized due to sign detection using object detection, which is already computationally intensive. By comparing the pros and cons of the triangulation method, SLAM, and SFM, the two-point triangulation method is chosen to be explored because it has a faster computation time compared with other two methods and it can be implemented in an easier and more efficient way.
NAIVE TWO-POINT TRIANGULATION METHOD In this section, the nave formulation of the two-point triangulation method is presented. It uses a set of consecutive images with their GPS coordinates taken from a vehicle with an onboard camera. This is illustrated in Figure 1. In Figure 1, two camera views were taken from the timestamp 1 and 2; "d" represents the distance between the two camera views. To implement
12

the two-point triangulation method, the origin of the coordinates is set at the car's position at 1, and the pixel coordinates of signs in the two images will be used to infer the vector direction from the position of each camera view to the traffic sign; this is the method shown in Figure 2. (1, 1) and (2, 2) are the pixel coordinates of signs in the first and second images and are determined by the centroid point of the bounding box, which is created by the detection algorithm. In this coordinate system, the origin of the coordinates is at the optical center of each image. This nave formulation of the two-point triangulation method relies on the following assumptions:
1) GPS locations of each camera view are extremely accurate; 2) the optical axis is parallel to the ground; 3) the optical center and image center are the same; 4) the images considered for determining the position of a particular object have collinear
optical axes; 5) the camera is perfect pinhole camera (no lens distortion).
Based on these assumptions, we can develop the formulas for solving the sign (X, Y) coordinates of the sign shown in Figure 3. In these formulas, "f" stands for the focal length of the camera, "d" represents the distance between the two images, and 1 and 2 stand for the x coordinate of the centroid point of the bounding box, which detects the sign in each image. (X, Y) are the relative distance to the first camera view's location in both the x and y directions.
13

Figure 1. Illustration. The geometric relationship between traffic sign and each camera view.
Figure 2. Illustration. Simplified geometric relationship of the Triangulation method. 14

Figure 3. Equation. Equations for computing traffic sign coordinates.
PROPOSED REFINEMENT TO THE TRIANGULATION METHOD When using the two-point triangulation method in real-world conditions, many assumptions may be invalid. While many assumptions can be enforced through the selection of the camera hardware, some assumptions may be violated due to roadway geometry. For example, when driving on a curved section of roadway or on vertical curves, the optical axes will become noncolinear. When these assumptions are violated, the results may become unstable or inaccurate. Therefore, the nave formulation has the potential to be improved. In this section, other concepts, such as deviation of angle, are introduced into the nave triangulation method to develop a refined version of the two-point triangulation method. To implement the refinement of the triangulation method, a technique called trajectory fitting is used to infer the direction in which the camera was facing in each camera view. Trajectory fitting takes several locations of the camera as input and outputs an approximate trajectory for these points. Once the trajectory is obtained, the forward direction of each camera view is taken as the tangent direction of the two camera viewpoints on the trajectory. The change of direction represents the deviation of angle between two the camera views. Then the deviation of angle can be added to the original formulas of the two-point triangulation method and the refined method can be implemented.
15

After introducing the deviation of angle, the revised x1 and x2 values are shown in Equation 1 and Equation 2 in Figure 4. The original formulas for X, Y coordinates of a sign will be revised as shown in Equation 3 and Equation 4 in Figure 4. To get the deviation of angle, the trajectory fitting technique described in Figure 5 is used. Once we have the trajectory fitted, we can get the tangent lines for each camera. Consequently, these tangent lines can derive the deviation of angle as shown in Figure 5.
Figure 4. Equation. Equations for computing sign coordinates with deviation angle change.
16

Figure 5. Illustration. The geometric relationship between camera views and the traffic sign when vehicle trajectory is on a curve.
VALIDATION OF THE PROPOSED CURVE SIGN LOCALIZATION METHOD This section presents the testing and validation of the proposed curve sign localization method. The purpose of validation is to evaluate the accuracy of the refined two-point triangulation method in various conditions to analyze the performance and error-causing factors. The performance measure of the sign localization method will be assessed quantitatively by calculating the location error between the ground reference GPS location, collected using Lidar, and the location estimated using the triangulation method. Test cases presented in this section include different roadway characteristics, environments, and other factors, like straight roadway sections, curved roadway sections, two-lane local roads or interstate highways, and different video and GPS devices. Based on the outcomes, the feasibility of using the triangulation method to perform curve sign localization is assessed, and recommendations for future improvements are
17

made. Dataset Used for Validation: In order to objectively and comprehensively evaluate the accuracy of the sign localization method, the data with diverse conditions are used for testing. The data used include the following characteristics: Different Types of Data Collection Devices: In this study, the data was collected by recording videos using two types of different 2D cameras: a camera installed on top of the Georgia Tech Sensing Vehicle (GTSV) [Figure 6] and smartphone cameras that were mounted to the dashboard of the data collection vehicle. These two types of cameras were not used at the same time. They are independent and constitute two different monocular systems to collect the data. Each system also collects GPS locations that can associate with the 2D camera images collected.
Figure 6. Photo. Georgia Tech Sensing Vehicle Camera.
Different Roadway Types: Data collection was done in 2017 and 2019 on different types of roads: interstates (I-75 and I285) and non-interstates (West Wesley, Techwood, Northside Drive NW, SR 2) as shown in
18

Figure 7.
e Figure 7. Map. Test data collection sites.
Different Roadway Geometries: The collected data also includes cases of straight and curved road segments. Examples of traffic signs in different roadway geometries are shown in Figure 8. The top left image was recorded on a straight segment of I-75. The top right image was recorded on a curved segment at Exit 254 of I-75 (Moores Mill Road NW). The bottom left image was collected on a straight segment of Northside Drive NW. The bottom right image was collected on a curved segment of West Wesley Road NW.
19

Figure 8. Photo. Example images of different roadway geometries.
For each sign, a set of sequential video images were collected either by the Georgia Tech Sensing Vehicle (GTSV) or by a low-cost smartphone that was mounted on the dashboard of the data collection vehicle. The ground reference location of the traffic sign is provided based on the GPS location extracted from 3D Lidar cloud data collected by the GTSV. By comparing the difference between the ground reference location and the location estimated by the sign localization method, we can calculate the distance offset and use it as the error for evaluating performance. In this dataset, there are a total of 19 traffic signs with GPS locations extracted from 3D Lidar point cloud data, and the signs were collected in different conditions, such as roadway types, roadway geometries, and data collection methods.
20

Evaluation Test Design: In this test, the refined triangulation method will be evaluated using location accuracy. The distance between the estimated sign location and ground reference sign locations is used as the performance measure. In this dataset, diverse conditions are included. Therefore, error-causing factors are identified by analyzing the results and by comparing performance under different conditions. Performing this analysis and evaluation helps us understand the feasibility of the triangulation method and determine the direction of future refinement. Evaluation Results and Interpretation: This section presents the evaluation results of both the nave triangulation method and the refined two-point triangulation method. The result of each test case is shown in Table 1 to Table 4; the tables are separated based on the data collection device (GTSV vs. smartphone), and the description of each case includes route name, route type, sign location relative to a roadway, whether significant vehicle pitch angle change is observed from video, and roadway geometry type.
21

Table 1. Nave Triangulation method (GTSV).

Case Name

Route Type

I75 1 I285 I75 2 I75 3 SR2 1 SR2 2 SR2 3 SR2 4 West Wesley 1 West Wesley 2

interstate interstate interstate interstate non-interstate non-interstate non-interstate non-interstate non-interstate non-interstate

Sign Location
Roadside Roadside Roadside Roadside Roadside Roadside Roadside Roadside Roadside Roadside

Pitch Angle Change
No Yes Yes Yes No No No No No No

Roadway Geometry
straight straight straight ramp straight straight straight curve curve curve

Ground Reference

GPS

Lat.

Lon.

33.8708 -84.4436

33.7569 -84.4945

33.7711 -84.3903

33.8329 -84.4262

34.8820 -83.5751

34.8805 -83.5763

34.8801 -83.5769

34.9124 -83.6191

33.8402 -84.4528

33.8330 -84.4299

Test Results

GPS

Lat.

Lon.

33.8708 -84.4436

33.7568 -84.4946

33.7709 -84.3900

33.8328 -84.4263

34.8820 -83.5751

34.8805 -83.5764

34.8801 -83.5769

34.9122 -83.6191

33.8402 -84.4527

33.8330 -84.4299

Error
Meter 0.36 15.94 35.35 9.76 6.42 5.34 4.96 17.21 3.48 2.44

Table 2. Nave Triangulation method (Smartphone).

Case Name
I75 1 I75 2 I75 3 I75 4 Northside Dr 1 Northside Dr 2 Northside Dr 3 Techwood & 10th West Wesley

Route Type
interstate interstate interstate interstate non-interstate non-interstate non-interstate non-interstate non-interstate

Sign Location
Roadside Overhead Roadside Roadside Roadside Roadside Roadside Overhead Roadside

Pitch Angle Change
No No Yes Yes No No No Yes No

Roadway Geometry
straight straight ramp ramp straight straight straight curve curve

Ground Reference

GPS

Lat.

Lon.

33.8010 -84.4003

33.7966 -84.3951

33.8326 -84.4264

33.8329 -84.4262

33.7881 -84.4075

33.7942 -84.4078

33.7982 -84.4077

33.7816 -84.3923

33.8330 -84.4299

Test Results

GPS

Lat.

Lon.

33.8010 -84.4004

33.7966 -84.3952

33.8325 -84.4265

33.8328 -84.4263

33.7880 -84.4075

33.7941 -84.4078

33.7982 -84.4077

33.7816 -84.3920

33.8329 -84.4300

Error
Meter 4.90 3.41 15.64 9.27 5.61 5.59 6.60 21.93 8.36

Table 3. Refined Triangulation method (GTSV).

Case Name

Route Type

I75 1 I285 I75 2 I75 3 SR2 1 SR2 2 SR2 3 SR2 4 West Wesley 1 West Wesley 2

interstate interstate interstate interstate non-interstate non-interstate non-interstate non-interstate non-interstate non-interstate

Sign Location
Roadside Roadside Roadside Roadside Roadside Roadside Roadside Roadside Roadside Roadside

Pitch Angle Change
No Yes Yes Yes No No No No No No

Roadway Geometry
straight straight straight ramp straight straight straight curve curve curve

Ground Reference

GPS

Lat.

Lon.

33.8708 -84.4436

33.7569 -84.4945

33.7711 -84.3903

33.8329 -84.4262

34.8820 -83.5751

34.8805 -83.5763

34.8801 -83.5769

34.9124 -83.6191

33.8402 -84.4528

33.8330 -84.4299

Test Results

GPS

Lat.

Lon.

33.8708 -84.4436

33.7568 -84.4946

33.7709 -84.3900

33.8330 -84.4261

34.8820 -83.5751

34.8805 -83.5764

34.8801 -83.5769

34.9123 -83.6191

33.8402 -84.4528

33.8330 -84.4299

Error
Meter 0.81 13.61 35.24 23.41 6.18 5.12 5.33 6.72 2.10 4.53

Table 4. Refined Triangulation method (Smartphone).

Case Name
I75 1 I75 2 I75 3 I75 4 Northside Dr 1 Northside Dr 2 Northside Dr 3 Techwood & 10th West Wesley

Route Type
interstate interstate interstate interstate non-interstate non-interstate non-interstate non-interstate non-interstate

Sign Location
Roadside Overhead Roadside Roadside Roadside Roadside Roadside Overhead Roadside

Pitch Angle Change
No No Yes Yes No No No Yes No

Roadway Geometry
straight straight ramp ramp straight straight straight curve curve

Ground Reference

GPS

Lat.

Lon.

33.8010 -84.4003

33.7966 -84.3951

33.8326 -84.4264

33.8329 -84.4262

33.7881 -84.4075

33.7942 -84.4078

33.7982 -84.4077

33.7816 -84.3923

33.8330 -84.4299

Test Results

GPS

Lat.

Lon.

33.8010 -84.4004

33.7966 -84.3952

33.8326 -84.4266

33.8327 -84.4264

33.7880 -84.4075

33.7941 -84.4078

33.7982 -84.4077

33.7816 -84.3920

33.8329 -84.4300

Error
Meter 4.90 3.38 18.86 28.31 5.61 5.36 6.60 21.94 8.37

22

Effects of Roadway Geometry As shown in Figure 9, the evaluation results show the nave triangulation method has an overall accuracy of 9.61 meters, while the refined method, overall, showed a slightly worse accuracy of 10.86 meters. However, the refinement made to the nave method is mainly to address the issue with changes in driving direction between the two camera views, which often occur on curved roadways, suggesting the results should be evaluated based on roadway geometry types. Figure 9 also shows the localization performance based on roadway geometry, which confirms a performance improvement on curved roadways. On straight roadways, the proposed approach showed only marginal improvement, which is expected, as straight roadway sections typically do not violate the assumptions made in the nave model. Interestingly, a significant decrease in accuracy is observed in ramp cases (shown in Figure 8) where the error doubled. This is due to the ramp cases consisting of roadway geometry that includes both vertical alignment change, and horizontal alignment change in the same road segment. This change in vertical and horizontal alignment will cause the optical axes to be non-collinear, which violates one of the model assumptions. While the refined method is designed to handle non-collinear axes that are coplanar in the roadway surface plane (better performance on horizontal curves), it appears to be more sensitive to out-of-plane axes.
23

Figure 9. Chart. Impact of roadway geometry on localization accuracy. Effects of The Data Collection System With ramps identified as one of the error-causing factors, when evaluating the effects of different data collection systems (GTSV vs. low-cost smartphone), ramp cases are removed from both categories (the smartphone dataset contains more ramp cases than the GTSV dataset). This revealed, as shown in Figure 10, that smartphones can achieve similar, if not better, localization accuracy on non-ramp cases than can the GTSV.
Figure 10. Chart. Impact of the data collection system on localization accuracy. 24

Effects of Vehicle Movement and Vibration While ramps' roadway geometry has been identified as one of the error-causing factors, it still does not explain the large errors found in straight roadway cases like the GTSV-I75-2 and GTSV-I285. By closely reviewing the video frames (as seen in the example in Figure 11), it was observed that both cases have roadway conditions that can cause the pitch angle of the vehicle to change rapidly and significantly. In the GTSV-I75-2 case, the traffic sign is located near the end of a straight ramp that contains not only a vertical curve but also a construction joint, which is likely caused the vehicle to bump. Case GTSV-I285 shows a similar situation; the vehicle drove over a slab joint with significant faulting and caused the nose of the vehicle to move up and down. This is evident in Figure 12, where there is a notable change in the horizon level; horizontal lines can be observed even between two consecutive frames, suggesting a rapid change in pitch angle.
Figure 11. Photo. Error cases on straight roadways, (a) left image case: GTSV-I75-2, (b) right image case: GTSV-I285.
25

Figure 12. Photo. Horizon level change in consecutive frames caused by pitch angle oscillation, the red line indicates the horizon level in the left image, and the orange line
indicates the right image horizon.
To quantify how the accuracy impacts of significant pitch angle change, the results are categorized into cases with pitch angle change and cases without. As shown in Figure 13, both the refined method and nave method perform poorly in cases with pitch angle change. In the test, the refined method showed more sensitivities to pitch angle change and provided only a small accuracy improvement in certain cases that are not affected by pitch angle change. Figure 13 also reveals that pitch angle is currently the biggest error-causing factor, as cases with pitch angle change have significant errors, while cases without pitch angle change can yield very accurate results.
26

Figure 13. Chart. Impact of the pitch angle change on localization accuracy.
SUMMARY This chapter presented and evaluated a sign localization method that can extract traffic sign GPS coordinates from sign detection outcomes and onboard video frames with GPS coordinates. The test method can be applied to different camera setups, such as vehicle-mounted cameras and windshield mounted smartphones, with similar performance expected. The evaluation results also suggest the refined two-point triangulation method can achieve an accuracy the localization accuracy with a median value of 6.18 meters and about 70% of cases can achieve error lower than 10 meters. However, the average accuracy (10.86 meters) is skewed by some cases due to the method struggles in situations where roadway geometries or pavement conditions cause the pitch angle of the vehicle to change significantly. This is due to the localization method formulation, which assumes the camera angle is always in the same plane; pitch angle expands the problem domain from "2D" to "3D."
27

This study, although not solving all possible error-causing factors in refinement, did reveal the main error-causing factors of the proposed method. The results showed that, in about 50% of cases, the proposed method can perform accurately with a localization error less than 6 meters. However, the test results also reveal that in some cases, the proposed method may result in errors larger than 15 meters. Therefore, although the validation of the proposed method showed that its capability in most scenarios, the small number of large-error-cases suggest that it is recommended that this localization method be further refined by investigating ways to extract accurate pitch angles from video frames and incorporating the results into the mathematical formulation of the method.
28

CHAPTER 3. RECOMMENDATION OF AN ENHANCED PROCEDURE FOR COST-EFFECTIVE MUTCD CURVE SIGN COMPLIANCE ANALYSIS
This chapter recommends an enhanced procedure for cost-effectively analyzing the Manual on Uniform Traffic Control Devices (MUTCD) curve sign compliance for the required curve sign types and spacings, and identifying the damaged and missing curve signs. The 2009 MUTCD curve sign requirement established the goal of reducing the disproportionately high crashes/fatality rate on roadway curves to save lives and to improve curve safety. However, making the MUTCD-compliant curve sign requirement sustainable (e.g., identifying and replacing the missing and substandard curve signs) is a costly and challenging task. In addition, because MUTCD-compliant curve signs involve safety requirements, there are potential liabilities imposed if transportation agencies cannot identify locations that need safety improvement or provide safety improvements in a timely manner. Consequently, there is a need to perform a real-time or near-real-time assessment to minimize the potential risks to drivers and transportation agencies because of the time-sensitive nature of attending to safety requirements. Therefore, development of a cost-effective, proactive, and time sensitive methodology for identifying and dealing with roadway locations that need safety improvement is vital. GDOT is taking the initiative to actively explore alternatives for improving roadway safety and minimizing potential risks to drivers and transportation agencies. This chapter aims to recommend an enhanced cost-effective methodology and procedure to address these needs.
The methodology recommended in this chapter includes three components to sustainably perform the cost-effective compliance analysis with MUTCD curve sign requirements and check
29

for required sign types, sign spacing and damaged signs. As shown in Figure 14, the three components are:
1) Establishing the curve sign baseline. This component is to establish the curve sign baseline design base on the MUTCD curve sign design requirement.
2) Detecting existing curve signs. This component is to detect and extract existing curve sign data using the developed ACSD.
3) Analyzing the MUTCD curve sign compliance. This component is to analyze the MUTCD curve sign compliance by comparing the baseline curve sign design and existing curve sign extracted using the developed ACSD.
Figure 14. Diagram. Components of the Cost-Effective MUTCD Curve Sign Compliance Analysis Procedure.
30

ESTABLISHING THE CURVE SIGN BASELINE USING MUTCD-COMPLIANT CURVE SIGN DESIGN REQUIREMENTS This section presents the MUTCD curve sign requirements and the design procedure for determining the proper curve sign design in terms of sign type and sign placement that complies with MUTCD safety standards. In an on-going research project (RP 20-22: Enhancing and generating GODT's MUTCD curve sign placement design with CurveFinder and curve sign determination), Georgia Tech research team is processing the raw data collected in the field using RIEKER devices, along with GIS roadway centerline data to extract curve inventory for curve sign design. This extracted MUTCD compliant curve sign design could be used as curve sign baseline design to support MUTCD curve sign compliance analysis. A typical procedure for determining MUTCD compliant curve sign design includes:
1) Collect roadway curve safety assessment data, including GPS trajectory and corresponding driving speed, ball back indicator (BBI), etc. In GDOT's case, the Rieker device is used in field for data collection.
2) Extract curve geometry information, including curve radius (R), point of curve (PC), point of tangent (PT) and curve types, using GPS trajectory or roadway centerline data.
3) Determine appropriate curve advisory speed using the collected curve safety assessment data.
4) Determine the speed differential between posted speed limit and curve advisory speed. 5) Determine the adequate MUTCD-compliant curve sign design base on the curve geometry,
curve type, curve advisory speed, and speed differential between posted speed limit and curve advisory speed.
31

Standards of curve warning signs are covered in the 2009 edition of the MUTCD manual, Chapter 2C: Warning Signs and Object Markers are utilized. The purpose of the curve warning signs is to alert road users of the roadway conditions, especially changes in roadway geometry that may require road users to take various driving actions, such as reducing speed if the curve advisory speed is lower than the posted speed limit (MUTCD, 2C.01). Figure 15 shows the warning signs for informing road users about changes in horizontal alignment.
Figure 15. Figure. Horizontal Alignment Warning Signs (MUTCD Figure 2C-1). The MUTCD manual shows various horizontal alignment warning signs (see Figure 15) based on different roadway conditions summarized in Table 5. For signs such as the advisory speed plaque and horizontal curve warning plaque, which are required to be placed ahead of a curve, MUTCD Table 2C-4 provides guidelines for the advance placement of warning signs based on the amount of deceleration required to get to the advisory speed. Chevron alignment signs must
32

be placed along a curve at pre-determined intervals; the typical spacing on horizontal curves is mandated in MUTCD Table 2C-6 and is based on the curve advisory speed and curve radius. In Table 5, Sdiff refers to speed differential, which is the speed difference between advisory and posted speed limits.
Table 5. Horizontal alignment warning sign usage in different roadway conditions
As shown in Table 5, if a type of warning sign is applicable under certain roadway conditions, its use will be determined based on the speed differential between the posted speed limit and curve advisory speed, as the usage of the sign will either be "Optional," "Recommended," or "Required" based on the MUTCD manual. In summary, the use of particular horizontal alignment warning signs is mainly based on the following factors:
Curve geometry (i.e., curve radius, curve deviation angle) Type of curve (i.e., single curve, reverse curve, and continuous curve) Curve advisory speed limit The speed differential between the posted speed limit and curve advisory speed
33

Therefore, to establish a baseline curve sign design according to the MUTCD standards, available data on the listed factors is required. The following section explains how each required data item is obtained to support establishment of a baseline curve sign design.
Curve Geometry Curve geometry information can be processed from the GPS trajectory data of the data collection device if the information, such as curve radius and curve deviation angle, is not already available within a database. The data processing of the GPS trajectory should extract curve geometry data that includes, curves identified from the GPS trajectory, point of curve (PC), point of tangent (PT), deviation angle (delta), and radius (R) of identified curves.
Curve Type The MUTCD manual requires the use of reverse turn or multiple (winding) turn signs at locations where curves are separated by a tangent distance less than 600 ft. It is also important to group the individually identified curves (from the GPS trajectory) into reverse curves or winding curves to support the determination of the proper MUTCD warning sign.
Posted Speed Limit In addition to curve geometry data, roadway posted speed limit and curve advisory speed limit data are required. The posted speed limit can be extracted from the data collection vehicle's onboard video footage using automatic traffic sign detection and classification.
Curve Advisory Speed Limit Curve advisory speed can be determined through a number of methods. There are six methods available for setting the advisory speed on curves (Milstead et al, 2011), each with advantages
34

and disadvantages that are summarized in

Table 6. The BBI method is currently being used by GDOT for determining advisory speed. This method, similar to the accelerometer method, directly measures the corning side-acceleration, which is proportional to the side-friction demand. The advisory speed is determined by setting a maximum speed that does not make the BBI exceed a maximum threshold (Milstead et al, 2011). This is to limit the friction demand so that the demand does not exceed a safety threshold.

Table 6. Comparison of the six methods for establishing curve advisory speeds.

METHOD Direct Compass GPS
Design BBI Accelerometer

ADVANTAGES Device is commonly available. Suggest advisory speeds
through speed study.
Only need single test run. Devices are easily obtained.
Only need single test run. Devices combine GPS
technology and provide more accuracy in measurements.
No field test running is needed.
Suitable for newly constructed or reconstructed curves.
Device is commonly available. Easy to operate.
Only need one (1) person to run tests.
Device is easy to operate.

DISADVANTAGES
May be subjective to specific traffic flow in the field collection.
Need to determine advisory speed criteria (85th%tile, average, passenger car or truck...).
Need to determine advisory speed criteria (85th%tile, average, passenger car or truck...).
Only to be used on very low-volume roads due to frequent stopping.
Need to determine advisory speed criteria (85th%tile, average, passenger car or truck...).
Method requires TRAMS software, GPS device, a laptop and an electronic ball-bank indicator.
Requires driving the curve at a low speed to achieve high accuracy.
Need to determine advisory speed criteria (85th%tile, average, passenger car or truck...).
Geometry data may not be readily available for most existing curves in service for many years.
Needs two (2) people during test runs. Varied criteria between AASHTO and
MUTCD. Reading ball-bank indicators to determine
the maximum degree of lean can be subjective. Multiple test runs may be needed.

When determining the advisory speed using the BBI method, the advisory computation is based 35

on the maximum allowed side-friction (which corresponds to the maximum allowed BBI), curve radius, and curve superelevation (which corresponds to the measured BBI at field driving speeds). Figure 16 show how the computing advisory speed is computed.

Figure 16. Equation. Equations for computing curve advisory speed.

Where,

: Curve advisory speed (MPH) : Curve radius (ft)

: Maximum allowed side friction

:

Curve superelevation (%)

: Maximum allowed BBI angle for determining advisory speed.

:

Driving speed on curve (MPH)

:

Measured BBI angle that corresponds to the driving speed.

Typically, BBI data is collected with multiple runs on the same curve to compensate for the variations in driving and data collection. The highest advisory speed limit from all runs will be selected to represent the advisory speed for the curve.
Example Using MUTCD Curve Sign Design for Establishing A Baseline Curve Sign Design With all required information obtained, the baseline curve sign design can be automatically generated by following the MUTCD standards. Such a baseline is important for finalizing curve
36

sign design on locations with no existing curve sign; such a baseline can also be used to perform MUTCD curve sign compliance analysis on locations with existing curve signs. The following figure shows an example of an automatically generated baseline curve sign design.
Figure 17. Illustration. Example of automatically generated baseline curve sign design.
The practical implementation used to establish the baseline MUTCD-compliant curve sign design is illustrated in the following example workflow:
1. Collect route data, like that which can be extracted from the GDOT's Rieker CARS portal. This data should include continuous latitude and longitude readings, vehicle speed measurements, BBI readings, vehicle trajectories, advisory speed, and posted speed limit.
2. Identify relevant curves based on route geometry and/or BBI readings. Likely curves can be identified algorithmically, selecting curves with small radii (e.g., <2500 feet) and large deviation angles (e.g., >30 degrees). BBI readings can be used as an alternative or as a supplement to the geometry-based approach, selecting road segments with high BBI readings. 37

3. Group curves that are signed as a single site: compound curves and reverse curves. Reverse curves should be treated as a single site if the tangent section between the curves is less than 600 feet.
4. Refer to MUTCD Chapter 2C.06-2C.10 for guidance on which signs are required or recommended as well as their prescribed placement and spacing. This process can be done algorithmically using a decision tree based on the speed differential and route geometry.
The specific steps of the methodology can be adjusted depending on the data and tools available.
DETECT EXISTING CURVE SIGNS USING AUTOMATIC CURVE SIGN DETECTION, CLASSIFICATION, AND LOCALIZATION The baseline sign design provides "required" and "expected" sign locations and sign types along a curve. To support MUTCD curve sign compliance analysis, the current (existing) sign locations and types that are actually in the field need to be inventoried. A live inventory of the existing curve warning signs requires sign type and sign location to be extracted; this task can be achieved by using an automatic traffic sign detection, classification, and localization system. The following sections briefly describe the procedure for automatic sign detection, classification, and localization.
Detection As explored in Phase I, traffic sign detection from video images is performed using a fine-tuned version of the YOLO v3 deep learning model (Redmon and Farhadi, 2018). This model is chosen because it processes video images faster than many other deep neural network models (it only
38

needs to process the image once). Because of computation speed considerations, the model only detects two superclasses of traffic signs based on the shape of the sign (diamond or rectangular); the exact classification of the MUTCD sign type is handled by another deep learning model in the classification step.
Classification The classification step determines the correct sign type by assigning a MUTCD code to the detected signs. To effectively classify diverse sign types, a deep learning model (MobileNetV2) was deployed to manage this task (Sandler et al., 2018). This network was trained to classify the cropped images within the bounding box of a detected traffic sign, and the cropped image was resized to a 40x40 pixel image size; this allowed clear recognition of the sign class while minimizing the computation overhead.
Localization In a video, a sign is typically displayed in a consecutive set of frames. The objective of sign localization is to determine the GPS coordinates from the data collected in the video and to show the traffic sign detection and classification results. However, before the detection and classification results can be used for localization, clustering must be used to group the images that contain the same physical sign. This allows the localization algorithm to identify the same traffic sign in different video frames and determine the physical location of the traffic sign. As described in Chapter 2, the actual localization of the traffic sign is done by estimating the relative position of the traffic sign to the corresponding GPS positions in each of the consecutive frames in which the traffic sign is detected.
39

MUTCD CURVE SIGN COMPLIANCE ANALYSIS (MCSCA) With the type and location information of the existing curve signs extracted and the baseline MUTCD curve sign design established, the objective MCSCA is achieved by using the baseline design as the "required/expected" curve sign location and comparing it with the existing curve sign location to identify safety improvement needs within the existing curve sign design.
In the MCSCA, the following criteria are evaluated:
Is this the sign placed in the correct location? Is this the placed sign the right sign type? Is the sign placed at the right spacing? Is any sign missing?
Such compliance analysis can be visualized by plotting the expected signs and existing signs on a map, as shown in the example in Figure 18. As shown in this example, after a horizontal curve is identified from the GPS trajectory, the "expected" MUTCD baseline curve sign design is generated, and the expected sign locations are shown as the colored triangles on the map; the expected sign type is shown as the MUTCD code next to the colored triangles. From the automatic traffic sign detection, classification, and localization process, the existing curve signs can be extracted, and the locations are overlayed as blue dots on the map with the sign type shown as the MUTCD code next to the blue dots. Due to variations in data collection and traffic sign installation, even if the curve signs are designed and installed based on MUTCD standards, it is unreasonable to expect existing traffic signs to be found at the exact position according to the baseline curve sign design. Therefore, when performing compliance analysis, a buffer is applied around the expected traffic sign location (shown as the colored boxes on the map); for
40

any sign with the buffer to be considered as correctly placed, the buffer is a 60 by 100 ft area that is designed to allow up to 30-ft deviation along the roadway in both directions and up to 100-ft deviation away from the edge of the roadway in one direction. Finally, besides evaluating the location of traffic signs, the traffic sign type is evaluated by comparing the expected sign type to the actual sign type.
Figure 18. Illustration. Example of curve sign compliance analysis using existing curve signs and baseline curve sign design.
POTENTIAL USE AND BENEFITS OF THE RECOMMENDED COMPLIANCE ANALYSIS It is essential for transportation agencies to meet the MUTCD curve sign design safety requirements and address safety improvement needs in a timely manner, to minimize the potential risks to the road users and potential liabilities to the transportation agencies. Therefore, the recommended "cost-effective MUTCD curve sign compliance analysis procedure" using an automatic curve sign inventory system (ACSIS) could benefit transportation agencies
41

significantly, as it enables transportation agencies to cost-effectively identify roadway sections that need safety improvements. Using the recommended procedure can achieve safety improvements and cost savings. With the ability to perform more frequent curve warning sign inventory and compliance analysis, agencies can save lives by taking proactive safety improvement actions in a timely manner. In the past, this has been very difficult and costly to achieve, and the time and resource required can be infeasible for agencies. To achieve cost-effective MUTCD curve sign compliance analysis, it is recommended that GDOT convert its current curve sign design outcomes into a baseline with sign type and sign location and save the information in an electronic format. It is also recommended that the baseline be combined with an automatic sign inventory system to develop a procedure and tool to implement the proposed compliance analysis approach.
42

CHAPTER 4. EXPLORATION AND RECOMMENDATION OF DATA STORAGE AND DATA MANAGEMENT PLAN
This chapter explores and recommends the data reduction and data management plan that leverages machine learning to support a live curve sign inventory system for MUTCD compliance analysis. Although the proposed automatic curve sign detection (ACSD) method using machine learning has demonstrated promising capability on computers equipped with GPU, the large file size of the collected videos is of concern because could have a major impact on a large-scale implementation. Consequently, data reduction is essential for managing the huge flow of roadway video log data. Figure 19 presents the recommended alternatives for data reduction and management of the proposed ACSD method; it uses machine learning for largescale implementation. Each recommended data management alternative includes four key components/modules, as shown in Figure 19: field data collection module, communication module, ACSD data processing and analysis module, and database module.
Figure 19. Diagram. Key components/modules of the data management system alternatives. 43

In the following section, each data management alternative and its key modules are presented with their corresponding strengths and limitations. ALTERNATIVES OF DATA MANAGEMENT SYSTEM FOR ESTABLISHING AN ACSD-BASED LIVE CURVE SIGN INVENTORY SYSTEM Alternative 1: Full Field Data Transmission with Centralized Servers for Data Processing. One alternative for the data management system is to transmit all raw data collected in the field and rely on off-site, centralized servers for ACSD data processing and MUTCD compliance analysis in a curve sign database. To reduce the data storage needed, after data processing and analysis, only the processing results and a few selected key image frames will be stored in the centralized database; the rest of the field-collected data will be removed permanently to minimize the permanent data storage in the central database. Figure 4-2 below shows the relationship of the modules in this data management alternative.
Figure 20. Diagram. Illustration of the data management alternative 1. 44

Field data collection. In this alternative, all raw data collected in the field by the low-cost mobile devices, including roadway video log data, corresponding GPS coordinates, and other telemetry data, will be transferred to off-site servers for ACSD data processing and MUTCD compliance analysis. This approach offloads the computationally heavy data processing from the data collection devices to off-site servers, where hardware and power consumption are less constrained. Thus, this approach offers the benefits of having lower hardware requirements for the data collection devices and drains less power if the device is battery powered. These benefits come at the cost of having to transfer more data from the data collection devices. Therefore, to minimize the size of the data transfer, this alternative should explore a dynamic video capture technique that dynamically changes the video capture framerate based on the driving speed (lower framerate at low speed) and automatically stop video recording when the vehicle stops (such as at intersections).
Communication. To reduce the efforts needed for the data transfer, wireless data transmission should be considered to automatically or semi-automatically perform the data transfer from the data collection devices to the centralized server. Since the full transfer of the raw data would require significant bandwidth, using the local wireless network should be considered when data collection devices have been returned to the office or the upcoming 5G wireless network could be used for real-time in the field. The benefit of using a local wireless network is it is a fast and existing infrastructure. The upcoming 5G networks promise the benefit of a large bandwidth,
45

which will enable the transfer of a large data set from field. Data processing servers. In this alternative, the ACSD system, including curve sign detection, classification, and localization, will be performed in off-site servers that take in raw data collected in the field. As off-site servers would have fewer constraints on the hardware and power consumption, a centralized place for processing all collected data could be used. However, due to the bulk of the raw data, the size of the collected video data (which, typically, only has a small portion that contains the frames needed for ACSD); permanent storage of the data is not recommended. Therefore, to reduce the data storage needs, it is recommended that a temporary (buffer) data storage be used for incoming raw data that needs to be processed; raw data will need to be removed as it is being processed. The benefit of using buffer storage is to provide a fast I/O environment to minimize potential congestions from simultaneous data transfers and reduce downtime associated with data transfer while allowing the data processing servers to be "well feed" with data in the buffer storage. Centralized live curve sign database. After the information needed for a curve sign inventory is extracted by the processing servers, the data to be permanently stored should include curve sign types, the corresponding locations, timestamp, and a small set of images containing the extracted curve signs. The data stored in the database will be kept and labeled in support of a live MUTCD compliance analysis. The outcomes of MUTCD compliance analysis can be used to generate reports on missing or misplaced curve signs while keeping track of the compliance condition of a curve over time.
46

Alternative 2: Optimized Field Data Transmission with On-Device Local Data Processing. To reduce the bandwidth required for field data transfer, this alternative focuses on optimizing the size of field data for transmission by leveraging local data processing on the data collection device.
Figure 21. Diagram. Illustration of the data management alternative 2. Field data collection. Unlike the other alternative, all raw data collected in the field by the low-cost mobile devices, including roadway video log data, corresponding GPS coordinates, and other telemetry data, will be partially processed to reduce data size before it is transferred to off-site servers for final
47

ACSD data processing and MUTCD compliance analysis.
The partial data processing on the data collection device can be done by performing an initial curve sign detection on mobile devices using a generalized traffic sign detection algorithm. The goal is to find all video frames that are likely to contain curve signs (even if some results are false positives) and only transmit the part of the raw data that is selected by the initial curve sign detection process and sent to the off-site server for final ACSD processing.
This approach should, in theory, drastically reduce the size of the transferred data. Using the SR 2 sign detection results presented in the Phase I report as an example, the videos used in the Phase I report have a total of 65,340 frames that contain 471 actual traffic signs. Assume a generalized traffic sign detection algorithm has a 40% false-positive rate of detecting traffic signs in order to capture all possible frames with actual traffic signs, and for each detected sign, 10 video frames will be transferred; this will result in (471 sign * (1+40% false-positive rate) * 10 frames per sign) 6594 frames that need to be transferred, which is only 10% of raw video data.
This approach optimizes the data needed for transfer to processing servers, which can reduce the bandwidth required but require data collection devices to perform some data processing. There are two possible options to achieve this. One is to explore the use of a smartphone version of traffic sign detection (Six, 2019) so the raw image data can be processed and analyzed in the field. A detailed example of a smartphone version of the sign detection model and machine learning architecture is presented in Section 4.2. The second option is to develop a stand-alone field data collection and processing device from commercially available specialized mobile processing units. For example, Nvidia Jetson devices (Figure 22) with GPU are small computers designed to run neural networks for AI and Internet of Things (IoT) applications (Nvidia); these
48

devices are commercially available and relatively low in cost. Other than a pre-built device, custom hardware is also feasible for a stand-alone device. With the advancement of the Integrated Circuit industry, nowadays, a manufacturer can easily assemble a custom hardware unit with a Central Processing Unit (CPU), Graphics Processing Unit (GPU), GPS, and Inertial Measurement Unit (IMU) at low cost to perform data collection and data processing.
Figure 22. Photo. Nvidia Jetson device line-up. (Nvidia) The following shows an example integration of the standard-alone field data collection unit with ACSD.
Figure 23. Photo. Example of an assembled stand-alone field data collection unit. 49

Communication. Similar to the other alternative, wireless data transmission should be considered to automatically or semi-automatically perform the data transfer from the data collection devices to the centralized server. Since the data transfer size is being optimized in data collection, the bandwidth and transfer time required should be reduced if the communication for transferring is using the local wireless network when data collection devices have been returned to the office, or the data transfer will need to rely on the 5G wireless network for real-time, scheduled data transfer in the field.
Data processing servers. The data processing servers in this alternative will take the data transmitted from the field data collection devices and perform the final data processing using the advanced, desktop version of the ACSD. The functions of the data processing servers are very similar to Alternative 1, but the required buffer storage size should decrease as the incoming data size is reduced.
Centralized live curve sign database. This is the same as Alternative 1. After the information needed for a curve sign inventory is extracted by the processing servers, the data to be permanently stored should include curve sign types, the corresponding locations, and only a small set of images containing the extracted curve signs. The data stored in the database will be kept and labeled in support of live MUTCD compliance analysis.
EXPLORATION OF THE SMARTPHONE VERSION OF ACSD As presented in the Phase I report, using deep learning networks for traffic sign detection shows very promising results. However, this approach is still very heavy in computations and requires a
50

powerful computer with a GPU to process video data effectively. In the field, especially when using vehicles to collect traffic sign data, it is impractical to bring a powerful computer; thus, the collected video data needs to be stored to be later processed on an off-site computer. This approach requires large video files to be stored and transferred, which burdens and hinders the performance of the operation. An alternative to this approach is to scale down the deep learning network to enable local processing on low-power devices, such as smartphones. The following sections present a case study that explores using low-cost mobile devices (Six, 2019) for ACSD while Nicolas Six worked in our research lab in 2019.
Network Architecture and Weight Compression This section presents the network architecture and other techniques to achieve the goal of making the network as lightweight as possible to efficiently run on mobile devices.
The main structure of a deep-learning network is composed of stacked computation blocks. Deciding the type of computation block is an important part of proposing a network architecture. There are different types of computation blocks available. In this case study, their performance is evaluated by comparing their computational speed and accuracy. The types of blocks explored in this study are the following:
Convolution Block The first type of block is the very basic 2D convolution, illustrated in Figure 24 (a). It is simple but has shown great results in the past; even though it has received multiple improvements, it is still a very good baseline. After initial experimentations, this type of block performed much like residual blocks, so the residual block type is used in the final evaluation rather than the convolution block type.
51

Residual Block The second block is a residual block (Szegedy et al., 2017). A residual block allows reduction of much of the gradient vanishing problem, which allows building a deeper network. This is not our goal, as we want to maintain a small size, but this could improve training efficiency at minimal cost. This block is illustrated in Figure 24 (b). Inverted Residual Bottleneck The Inverted residual Bottleneck block was introduced by MobileNetv2 (Sandler et al., 2018) as a structure tailored to run on mobile devices. It is composed of a first expansion layer, a 2D convolution with kernel (1; 1) and is followed by a depth-wise 2D convolution with kernel (3; 3); the final outcome is given by a projection layer, a 2D convolution with (1; 1) kernel. All of that is bundled into a residual block. A graphical representation of this block, from a MobileNet v2 paper (Sandler et al., 2018), is provided in Figure 24 (c).
Figure 24. Illustration. Graphical representation of different types of computation blocks.
The architecture of the network determines how features are extracted from the images. For this study, a lightweight architecture is proposed in order to perform real-time sign detection on lowpower mobile devices. Table 7 shows the feature extraction layers used in this architecture, how
52

the layers are linked, and the information on the use of batch normalization (BN) and residual connections (Res). Figure 25 shows the visual representation of the network architecture. In this architecture, the feature extraction layers are very limited compared to other lightweight deeplearning models; this allows the number of floating-point operations (FLOPs) needed for this architecture to be drastically reduced. The proposed model architecture only requires 39 million FLOPs, while common architecture, such as the tiny Yolo v3 (Redmon and Farhadi, 2018), requires 5.6 billion FLOPs.
Table 7. Description of the neural network architecture.
53

Figure 25. Illustration. Visual representation of the network architecture.
In addition to making the network architecture lightweight, hardware-specific optimization is another technique that can reduce the model in the future. One common approach is weight compression. Weights represent the features a model learned during training; they are what the model uses to determine output from input. Weight compression, at its core, is changing the weight values from the commonly used 32-bit floating-point to 8-bit values.
Evaluation of The Proposed Network In this section, the proposed network with two different types of computation blocks (residual convolution and inverted residual bottleneck) is evaluated based on their processing speed, represented as frames per second (FPS) and detection accuracy, and represented as mean average precision (mAP). Another commonly used lightweight model (Yolov3-tiny) is included in the evaluation to show the performance comparison with the proposed model. Table 8 shows the
54

performance evaluation of the proposed model and Yolov3-tiny. The results show that the Yolov3-tiny model can achieve very high accuracy at the cost of processing speed. Even with a very small input image size, the frame rate is still too slow to be considered for real-time (12.8 FPS) processing; the accuracy of the model is, also, reduced significantly at this speed. The proposed model, when using residual convolution, while unable to match the accuracy of Yolov3, can achieve a much higher processing speed with a smaller performance penalty to accuracy.
Table 8. Processing speed and accuracy of different architectures.
This preliminary study demonstrates that running a lightweight version of the machine learningbased traffic sign detection model on low-cost mobile devices is feasible, although the accuracy will be reduced to 70% (Six, 2019) because of the need for fast computation speed when processing the down sampled low-resolution images. Although a higher detection accuracy can be achieved at the cost of longer processing time, it is worth noting that the study was done in 2019. Because mobile processors are becoming more and more powerful each year and because many system-on-chip (SoC) used by the newer smartphones have begun to use chips designed to
55

run neural networks more efficiently, the processing speed is expected to be even faster with recent smartphone hardware.
SUMMARY ACSD has demonstrated its promising capability to support a live curve inventory and MUTCD compliance analysis system. However, due to the large file size of the collected video files, data storage and management is one of the important factors to consider for large-scale implementation. This chapter has recommended two data management alternatives that leverage machine learning to support a live curve sign inventory system for MUTCD compliance analysis and has detailed their four key modules, including field data collection module, communication module, ACSD data processing and analysis module, and database module. In addition, fast buffer storage is also recommended in the data management design to have fast I/O for data processing and analysis and to minimize the data needed to store permanently.
A preliminary study demonstrated that running a lightweight version of the machine learning based on the traffic sign detection model on low-cost mobile devices is feasible, although the accuracy will be reduced to 70% because of the lowered image resolution for faster processing speed. Although a higher detection accuracy can be achieved at the cost of longer processing time, it is worth noting that the study was done in 2019. Because mobile processors are becoming more and more powerful each year and because many system-on-chip (SoC) devices are being used by newer mobile smartphones to use chips designed to run neural networks more efficiently, the processing speed is expected to be even faster with recent smartphone hardware. Other than smartphone devices, the review on stand-alone data collection models using specialized hardware reveals that the use of low-cost specialized hardware is also recommended
56

for future study. It is recommended that a pilot study be performed to explore different data management alternatives and to deploy the preferable alternatives based on the upcoming technologies. They will be discussed in more detail in the implementation roadmap, which is included in the Chapter 5. In addition, other issues, like privacy, will need to be addressed in the implementation stage.
57

CHAPTER 5. ROADMAP FOR MUTCD CURVE SIGN COMPLIANCE ANALYSIS FOR SAFETY IMPROVEMENTS
This chapter presents a roadmap that uses the proposed automatic curve sign detection (ACSD) with low-cost mobile devices and machine learning to collect curve sign data, including sign types and sign locations. This data can be used for MUTCD compliance analysis and to effectively delegate appropriate safety improvements. In Figure 26, the roadmap is presented and organizes completed and future studies. This roadmap is intended to be informed GDOT engineers on the projects that lie in the future to effectively incorporate MUTCD compliance analysis in standard operating practice for implementing ACSD. Through Phases I and II, the proposed automatic curve sign detection (ACSD) method that leverages machine learning and low-cost smartphones, has demonstrated that it is feasible to accurately detect curve signs using roadway images collected using low-cost smartphones to detect and classify curve signs with their MUTCD codes and to determine curve sign locations. However, steps still need to be taken to improve the machine learning algorithm and the procedure. These steps are outlined in the roadmap. The overarching goal of the roadmap is to allow for the cost-effective collection of curve sign data and optimal analysis of existing sign infrastructure for MUTCD compliance. This analysis can be used to implement safety improvements and decrease roadway curve fatalities and injuries. Items 1-3 have been completed to demonstrate proof of concept in Phase 1 and Phase 2 of this report. It is suggested to initiate the next project to work on items 3-7, 11, and 14 in the near future. All other items can be conducted after.
58

Figure 26. Chart. Roadmap for MUTCD curve sign compliance analysis for safety improvements. 59

Items 1), 2), and 3) have been completed in this study. A review was conducted on ACSD and on curve sign analysis compliance. This led to research on a smartphone-based ACSD system in which signs are detected, classified, and given coordinates. The following section describe the purpose and objectives of the items 4) to 16) listed in Figure 26.
Item 4) In this research report, data reduction management alternatives are listed in Chapter 4. Two methodologies have been proposed to reduce the amount of data storage necessary. Further research is still needed to refine and improve these methodologies. Georgia Tech will research more in-depth on these data reduction methods.
Item 5) The methodology for the localization of signs still needs to be matured. More advanced methods still need to be explored to improve the accuracy of the GPS coordinates of a sign and the post-processing computation speed. The majority of signs were detected accurately within 20 ft (6 meters). Location computation time has shone promising results of one millisecond per image which is near real-time computation speed. However, certain scenarios such as steep vertical grades led to higher levels of inaccuracy. Therefore, there is still room to improve the sign location computation accuracy with more advanced methods. Georgia Tech Researchers are currently working to improve this method.
Item 6) A larger and more diverse dataset is needed to further refine the developed ACSD method. This will allow the algorithm to be more robust and therefore correctly classify signs in edge cases and identify environments in which the algorithm performs poorly to ensure steps are taken to avoid certain scenarios.
60

This diverse dataset should contain images with variations in time of day, weather, image quality, driving speeds, etc. With a more diverse dataset, there can be a better understanding of the gaps in the current ML algorithm, and steps can be taken to train the algorithm appropriately or inform users of the algorithm where inaccuracies could take place, and what those inaccuracies may be. For example, in a dark or shady environment, a sign may be detected correctly, and the GPS coordinates could be correct, but the algorithm may not be able to identify the sign type due to the lack of visibility in the image. Georgia Tech researchers have worked to test and improve the algorithm in diverse environments and will continue to update ML algorithm with a more comprehensive networkwide dataset. Item 7) Develop a procedure to use the ACSD to develop a live curve sign inventory (LCSI) application. Georgia Tech can conduct a networkwide inventory in a selected county or city to illustrate the procedure in which curve sign data can be collected and processed to obtain accurate inventory. Since it is intended to utilize this method while drivers perform other tasks, a procedure to analyze roads that were not immediately collected can be defined. This application will ensure that transportation agencies can obtain adequate count estimates of their signs. Item 8) MUTCD curve sign compliance analysis is discussed in Chapter 3 of this report. This methodology along with the application will enables transportation agency personnel to identify missing signs, assess sign types, and sign spacing. This
61

methodology needs to be developed and refined to ensure analysis is conducted in a cost-effective and time sensitive manner. Item 9) Development of a systematic QA/QC procedure is crucial to verify and curve sign detection, classification, and localization outcomes. It is important to acknowledge that inaccuracies will result from the outcomes of data collection, especially when a diverse dataset has been used. Though inaccuracies may occur, a procedure to modify or verify the outcomes of this data. For example, multiple runs of a roadway can be used to verify a missing sign or identified signs with low confidence levels. Alternatively, video can be used to visually inspect sites. This will allow further confidence in the results of the machine learning algorithm before action is taken to replace or add a sign. Item 10) Once the MUTCD compliance of curve signs is determined, a procedure to map this data should be developed. Explore the potential to integrate the ACSD method with CurveFinder. Data collected via smartphone can be used alongside CurveFinder to develop a systematic cost-effective procedure to conduct systemic safety assessments. These assessments will allow for cost-effective safety improvements to undergo. Additionally, the same low-cost smartphones used to collect sign image and GPS data can be used to collect acceleration and gyro data. The integration of these datasets will enable transportation agencies to conduct cost-effective and time sensitive safety assessments by analyzing MUTCD curve sign compliance, computing advisory speed, and calculating Ball Bank Indicator (BBI). Therefore, not only can MUTD compliance be assessed, but collected data can be used to develop crash prediction models and
62

friction deterioration forecasting models for sites that have undergone friction improvement treatments. This strategy fits into both FHWA's and GDOT's local road safety plans by enabling local agencies to conduct data-driven systemic safety assessments. Item 11) A pilot study will need to be undergone to explore the data reduction management methods described in Chapter 4. The identification of which method is more practical and reliable in real-world scenarios must still be determined. Modifications to the two data reduction methods can be made after the reliability and the practicality are assessed. Georgia Tech researchers are willing to utilize the proposed data reduction methods and identify the most practical option. Item 12) Another pilot study to test the feasibility of both LCSI and MCSCA should be conducted. Various issues may arise when this procedure is implemented at GDOT, and some modifications to obtain necessary data may need to be made. Item 13) A centralized database will also need to be created to support the live curve sign inventory and MUTCD compliance analysis system. This database will include infrastructures such as data processing and analysis servers, a fast temporary data storage server, and permanent data storage servers. The details for such infrastructures should be studied and the implementation can be done by GDOT or outsourced to other companies that can manage the large amount of data collected.
63

Item 14) There is a need to explore change in business operation and policy to establish a more safety-conscious transportation system. Ensuring roadway safety is prioritized in offices such as roadway design, construction, maintenance, etc. will allow for more cost-effective practices in the future and therefore a more sustainable organization. Dr. James Tsai and Dr. Adjo Amekudzi-Kennedy have received a GT CEE seed grant on "Transportation Safety for Smart Cities Using Low-cost Smart Phones, Crowdsourcing, Machine Learning with Real-time Safety Assessment: Toward a More Sustainable Transportation System" to work on this topic. It is expected to work closely with GDOT in this project.
Item 15) Install training programs to guarantee drivers can collect data effectively using the smartphone on their cars and ensure engineers can analyze and interpret the data correctly. Ensuring smartphones are appropriately set up in a vehicle for accurate data collection is vital to obtain useable results. Conveying the current gaps to engineers analyzing the data is also necessary to allow them to make knowledgeable decisions on where repair is needed and where further inspection should be conducted.
Item 16) Identification of what constitutes an acceptable level of MUTCD compliance needs to be determined. The threshold level for a project that should undergo repair or improvements in signage should be determined.
Three stages are necessary for the roadmap to be effective: proof of concept, pilot studies, and large-scale deployment and implementation. Stage 1, the proof of concept, was
64

completed in Phase I and Phase II of this report. Stage 2 are the pilot studies from GDOT or the FHWA Accelerated Innovation Deployment (AID) to accelerate the implementation and adoption of innovation by testing the developed ACSD method in a selected county or city. It is suggested to work with GDOT to prepare the proposal titled "Enhanced MUTCD Curve Sign Compliance Safety Assessment Using Low-cost Smart Phones and Machine Learning" for the FHWA Accelerated Innovation Deployment (AID) program because the developed technology, ACSD using low-cost smart phone and machine, is innovative and will have high impact for improving roadway safety. In addition, it has strong interest from GDOT and other transportation agencies in multiple states. Stage 3 will be comprised of the large-scale deployment and implementation of the developed solution. The key is to make the developed solution scalable for data acquisition, processing, and management.
65

CHAPTER 6. CONCLUSIONS AND RECOMMENDATIONS
To reduce fatal and injury crashes, the Manual on the Uniform Traffic Control Devices (MUTCD) (FHWA, 2012) requires various horizontal alignment warning signs (curve signs) to ensure roadway safety on curves. However, many transportation agencies' current practices for inventorying the locations and types of existing curve signs are typically manual processes that are labor-intensive, time-consuming. A cost-effective MUTCD curve sign compliant analysis methodology that uses intra-agency, low-cost mobile devices (e.g., smartphones), existing vehicles, machine learning, and crowdsourcing is proposed to meet MUTCD requirements Automatic curve sign detection (ACSD) is one of the most critical and core components in the proposed methodology and system. Phase I and Phase II of this research project demonstrated proof-of-concept to evaluate the feasibility of using low-cost smartphones and machine learning to perform curve sign detection. Phase I focused on critically validating curve sign detection accuracy, while Phase II focused on developing the curve sign location computation using a triangulation method and validating its location computation accuracy. This concept study proves it is feasible to use low-cost smartphones and machine learning to cost-effectively collect curve sign data. We further recommend a roadmap to implement ACSD. The roadmap will include a machine learning and smartphone-based data collection module and recommended applications for using ACSD to perform a live curve sign inventory that is cost-effective. Furthermore, a procedure to ensure collected data meets MUTCD requirements is recommended. The roadmap will recommend improvements to the ACSD for large-scale implementation, recommend a data management plan for large-scale implementation, and recommend a
66

new business model and policy. Through the research studies conducted in Phases I and II, the following are the proposed methodology's conclusions and recommendations:
1) Through research studies conducted in Phases I and II, the proposed ACSD has demonstrated that it is feasible to use smartphones and machine learning to detect MUTCD compliant curve signs.
2) The field data collection module using smartphones has been developed. 3) The case study on State Route 2 clearly demonstrates that the proposed automatic
curve sign detection method, which uses deep learning, is promising for implementation. 4) A curve sign location computation method using triangulation has been evaluated and can provide reasonable accuracy, although future improvement is still needed. 5) A MUTCD curve sign compliance analysis methodology using the developed ACSD is recommended to enable transportation agencies to meet MUTCD curve sign compliance requirements and to reduce crashes and fatalities on curved road sections. 6) A roadmap recommended for improving roadway safety by applying the developed ACSD has been presented. 7) Two data management alternatives are recommended; a pilot study is needed to evaluate implementation.
The following are the next steps recommended for developing a live curve sign inventory and MUTCD curve sign compliance analysis applications:
1) Enhance the accuracy of the sign location computation method. 2) Establish a dataset with diverse conditions to support automatic sign detection.
67

3) Develop and evaluate different data reduction and management alternatives. 4) Develop a live curve sign inventory application using the developed ACSD.
68

ACKNOWLEDGMENTS
We would like to thank the Georgia Department of Transportation (GDOT) for its support. The work conducted in this report was sponsored by the GDOT Office of Performance-Based Management and Research (Research Project 19-26). We would like to thank Mr. Andrew Heath and Mr. Sam Harris from the Office of Traffic Operations for their strong support and heavy involvement in the project. We would like to thank Mr. Brennan Roney and Mr. Binh Bui (retired) from the Office of Performance-Based Management and Research. We would also like to thank Mr. David Adams and Mr. Carlos Baker from the Office of Traffic Operations and Mr. Jonathan Peevy, Mr. Shane Giles, Mr. Parker Niebauer from District 1, and other District 1 staff for their support in the field data collection and understanding the curve sign installation and field assessment of missing signs. We would also like to thank the members of the research team at the Georgia Institute of Technology, including Nicolas Six, Mehdi Azabou, Anirban Chatterjee, Murali G Sethuraman, and Don Kushan Saminda Wijeratne for their diligent work on data collection, processing, and analysis in this research project.
69

REFERENCES
AASHTO. A Policy on Geometric Design of Highways and Streets. The American Association of State Highway and Transportation Officials, AASHTO Green Book, Washington DC, 2011.
Ai, Chengbo, and Yi-Chang James Tsai. "Critical assessment of an enhanced traffic sign detection method using mobile LiDAR and INS technologies." Journal of Transportation Engineering 141, no. 5 (2015): 04014096.
Durrant-Whyte, Hugh, and Tim Bailey. "Simultaneous localization and mapping: part I." IEEE robotics & automation magazine 13, no. 2 (2006): 99-110.
FHWA. Manual of Uniform Traffic Control Devices (MUTCD). Federal Highway Administration, U.S. Department of Transportation. 2009 Edition, 2012.
FHWA. Horizontal Curve Safety - Safety: Federal Highway Administration. Safety, 2021. https://safety.fhwa.dot.gov/roadway_dept/countermeasures/horicurves/.
Hu, Z., and Y. Tsai. "Image recognition model for developing a sign inventory." ASCE J. of Comp. in Civil Eng (2010).
Milstead, Robert, X. Qin, Bryan Katz, James A. Bonneson, Michael Pratt, Jeff Miles, and Paul J. Carlson. Procedures for setting advisory speeds on curves. No. FHWA-SA-1122. United States. Federal Highway Administration. Office of Safety, 2011.
Redmon, Joseph, and Ali Farhadi. "Yolov3: An incremental improvement." arXiv preprint arXiv:1804.02767 (2018).
Redmon, Joseph, Santosh Divvala, Ross Girshick, and Ali Farhadi. "You only look once: Unified, real-time object detection." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788. 2016.
Redmon, Joseph, and Ali Farhadi. "YOLO9000: better, faster, stronger." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7263-7271. 2017.
Sandler, Mark, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. "Mobilenetv2: Inverted residuals and linear bottlenecks." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520. 2018.
Six, Nicolas. "Real time detection of traffic signs on mobile device." Master thesis., Georgia Tech, 2019.
Tsai, Yichang James. Using Image Pattern Recognition Algorithms for Processing Video Log Images to Enhance Roadway Infrastructure Data Collection. No. Highway IDEA Project 121. 2009.
70

Tsai, Yichang, Pilho Kim, and Zhaohua Wang. "Generalized traffic sign detection model for developing a sign inventory." Journal of Computing in Civil Engineering 23, no. 5 (2009): 266-276.
Tsai, Yichang, and Yuchun Huang. "A generalized framework for parallelizing traffic sign inventory of video log images using multicore processors." Computer-Aided Civil and Infrastructure Engineering 27, no. 7 (2012): 476-493.
Tsai, Yichang, Six, Nicolas., Heath, Andrew., Bui, Binh. "A Live Curve Sign Inventory for Meeting MUTCD Requirement using Low-Cost Smart Phone and Deep Learning Technologies," The 99th Transportation Research Board Annual Meeting, Washington DC., January 1216, 2020, Paper number: 20-05531.
Tsai, Yichang, and Zhaohua Wang. "A remote sensing and GIS-enabled asset management system (RS-GAMS)." (2013).
Ullman, Shimon. "The interpretation of structure from motion." Proceedings of the Royal Society of London. Series B. Biological Sciences 203, no. 1153 (1979): 405-426.
71