Monitoring and assessing traffic safety at signalized intersections using live video images

GEORGIA DOT RESEARCH PROJECT 14-29 FINAL REPORT
Monitoring and Assessing Traffic Safety at Signalized Intersections Using Live Video Images
OFFICE OF PERFORMANCE-BASED MANAGEMENT AND RESEARCH 15 KENNEDY DRIVE FOREST PARK, GA 30297

1.Report No.:

2. Government Accession No.: 3. Recipient's Catalog No.:

4. Title and Subtitle: Monitoring and Assessing Traffic Safety at Signalized Intersections Using Live Video Images

5. Report Date: October 2018 6. Performing Organization Code:

7. Author(s):

8. Performing Organ. Report No.: 14-29

Jidong J. Yang, Ph.D., P.E.; Ying Wang, Ph.D.;

and Chih-Cheng Hung, Ph.D.

9. Performing Organization Name and

10. Work Unit No.:

Address:

Kennesaw State University

11. Contract or Grant No.: 0013527

Georgia Pavement & Traffic Research Center

1100 South Marietta Parkway

Marietta, GA, 30060

12. Sponsoring Agency Name and Address: 13. Type of Report and Period Covered:

Georgia Department of Transportation

Final Report; January 2016 October 2018

Office of Performance-Based Management

and Research

14. Sponsoring Agency Code:

15 Kennedy Drive, Forest Park, GA 30297-

2534

15. Supplementary Notes: Prepared in cooperation with the U.S. Department of Transportation,

Federal Highway Administration.

16. Abstract: Signalized intersections represent the most hazard spots on a roadway network.

Road users are required to be alert and timely process and respond to a variety of information at

signalized intersections, including traffic signal indications and changes, signage, pavement

marking, road conditions, and a mix of various road users in conflict. Traditional road safety

diagnosis has been conducted in a reactive manner based on crashes that had occurred.

However, to effectively reduce and eventually eliminate crashes, proactive approaches are

needed. Following this direction, traffic conflict events have been collected more frequently and

used as a surrogate safety measure for traffic crashes. The goal of Vision Zero would only be

possible if the inconsequential event data, such as traffic conflicts, can be objectively and

systematically collected and effectively utilized to diagnose and improve road safety such that

consequential crash events can be prevented. In this study, the art of deep learning, multiple

objects detection and tracking were explored and tested in the domain of traffic conflict

monitoring and assessing. As a result, an artificial intelligence (AI) enhanced computational

system was developed to automate the detection and quantification of traffic conflict events as

they occur in real time using traffic monitoring cameras currently installed by transportation

agencies.

17. Key Words: Traffic conflicts, crashes,

18. Distribution Statement:

collisions, detection, tracking, safety, deep

neural networks, images, and analytics.

19. Security

20. Security

21. Number of

22. Price:

Classification

Classification

Pages: 69

(of this report):

(of this page):

Unclassified

Unclassified

Form DOT 1700.7 (8-69)

ii

GDOT Research Project 14-29
Final Report MONITORING AND ASSESSING TRAFFIC SAFETY AT SIGNALIZED
INTERSECTIONS USING LIVE VIDEO IMAGES
Prepared by Jidong J. Yang, Ph.D., P.E., Associate Professor of Civil Engineering
Ying Wang, Ph.D., Associate Professor of Electrical Engineering Chih-Cheng Hung, Ph.D., Professor of Computer Science
Georgia Pavement and Traffic Research Center Kennesaw State University
1100 South Marietta Parkway, Marietta, GA 30060
Contract with Georgia Department of Transportation
In cooperation with U.S. Department of Transportation Federal Highway Administration
October 2018
The contents of this report reflect the views of the authors, who are solely responsible for the facts and accuracy of the data, the opinions, and the conclusions presented herein. The contents do not necessarily reflect the official view or policies of the Georgia Department of Transportation (GDOT), Federal Highway Administration (FHWA), and the Georgia University System. This report does not constitute a standard, specification, or regulation, and its contents are not intended for construction, bidding, or permit purposes. The use of names or specific products or manufacturers listed herein does not imply endorsement of those products or manufacturers.
iii

ACKNOWLEDGMENTS This project was conducted in cooperation with the Georgia Department of Transportation (GDOT) and the Federal Highway Administration. The authors would like to gratefully acknowledge the contributions of many individuals to the successful completion of this research project. This especially includes Mr. David Jared, Ms. Supriya Kamatkar, Ms. Sarah Lamothe, and Ms. Gretel Sims at the GDOT Office of Performance-Based Management and Research for their administrative support and Mr. Alan Davis at the GDOT Office of Traffic Operations for his technical guidance through the course of this project.
Specifically, the authors would like to thank Mr. David Montanye and Mr. Brook Martin at the Cobb County Department of Transportation for their support in accessing traffic cameras and conducting site tests.
iv

TABLE OF CONTENTS

1. INTRODUCTION ....................................................................................................................... 1

1.1 BACKGROUND ............................................................................................................................... 3 1.2 CONFLICT AS A MEASURE OF SAFETY............................................................................................ 5 1.3 SAFETY IMPLICATIONS OF EMERGING TECHNOLOGIES .................................................................. 7 1.4 ORGANIZATION OF THE REPORT .................................................................................................... 8

2. LITERATURE REVIEW ............................................................................................................ 9

2.1 OBJECT DETECTION AND TRACKING.............................................................................................. 9 2.2 TRAFFIC CONFLICT IDENTIFICATION AND QUANTIFICATION ....................................................... 22

3. RESEARCH FRAMEWORK .................................................................................................... 24

4. DETECTION AND TRACKING OF ROAD USERS................................................................ 27

4.1 4.2 4.2.1 4.2.2 4.2.3 4.2.4 4.2.5 4.2.6

APPLICATION DOMAIN CHALLENGES .......................................................................................... 27 DETECTION AND TRACKING......................................................................................................... 30 SSD ARCHITECTURE FOR DETECTION ......................................................................................... 30 SSD TRAINING ............................................................................................................................ 30 SSD TESTING............................................................................................................................... 32 SSD INFERENCE........................................................................................................................... 34 TRACKING BY IOU ....................................................................................................................... 35 RUNTIME COMPUTATIONAL COSTS ............................................................................................. 42

5. CONFLICT DETECTION AND CHARACTERIZATION....................................................... 43

5.1 DETECTION OF CONFLICT EVENTS ............................................................................................... 45 5.2 CHARACTERIZATION AND QUANTIFICATION OF CONFLICT EVENTS............................................. 46 5.2.1 ASSESSMENT OF COLLISION RISK ................................................................................................ 46 5.2.1.1 AFFINITY-TO-COLLISION (ATC) MEASURE AND CONFLICT GRAVITY MODEL (CGM) ........... 48 5.2.1.2 PROCEDURE FOR MODEL CALIBRATION .................................................................................. 53 5.2.2 ASSESSMENT OF CONFLICT SEVERITY ......................................................................................... 53

6. SOFTWARE IMPLEMENTATION.......................................................................................... 55

6.1 6.2 6.3 6.3.1 6.3.2

SYSTEM ARCHITECTURE .............................................................................................................. 55 VIDEO ANALYTICS SYSTEM......................................................................................................... 57 SYSTEM TESTING ......................................................................................................................... 58 SIMULATED ENVIRONMENT ......................................................................................................... 58 TEST ON LIVE CAMERAS.............................................................................................................. 60

7. CONCLUSIONS AND RECOMMENDATIONS ...................................................................... 63

8. REFERENCES .......................................................................................................................... 65

v

LIST OF TABLES
TABLE 2.1 The ConvNet Configurations of VGG-Net (shown in columns)................................ 15 TABLE 2.2 GoogLeNet Incarnation of the Inception Architecture (Szegedy et al., 2015).......... 16 TABLE 4.1 Summary of Runtime Computational Times ............................................................. 42
vi

LIST OF FIGURES
FIGURE 1.1 Steps to Identify Potential Problems (FHWA-HRT-04-092, April 2004).................. 3 FIGURE 1.2 Roadway Safety Management Process (Adapted from the Highway Safety Manual
2010)......................................................................................................................... 4 FIGURE 1.3 Hyden's Safety Pyramid (Hyden, 1987).................................................................... 5 FIGURE 1.4 Illustration of Conflict Points at a Conventional Four-Leg Intersection.................. 6 FIGURE 2.1 An Example of Optical Flow (East-West Connector/South Cobb Drive; Daytime) 10 FIGURE 2.2 An Example of Optical Flow (East-West Connector/South Cobb Drive; Nighttime)
................................................................................................................................ 11 FIGURE 2.3 An Example of ORB Feature Matching .................................................................. 12 FIGURE 2.4 Example of HOG Features...................................................................................... 12 FIGURE 2.5 An Illustration of the Architecture of AlexNet between Two GPUs. One GPU runs
the layer-parts at the top of the figure while the other runs the layer-parts at the bottom. The network has 62.3 million parameters and needs 1.1 billion computation units in a forward pass (Krizhevsky et. al, 2012) .............................. 14 FIGURE 2.6 R-CNN Architecture (Girshick et al., 2014)............................................................ 17 FIGURE 2.7 Fast R-CNN Architecture (Girshick R., 2015) ........................................................ 18 FIGURE 2.8 Faster R-CNN is a Single, Unified Network for Object Detection. (Ren et al., 2016) ................................................................................................................................ 19 FIGURE 2.9 Head Architecture of Mask R-CNN plus Faster R-CNN with FPN (Ren et al., 2016) ................................................................................................................................ 20 FIGURE 2.10 Keypoint Detection Results and Predicted Segmentation Masks (Ren et al., 2016) ................................................................................................................................ 20 FIGURE 2.10 Single Shot MultiBox Detector (Liu et al., 2016) .................................................. 21 FIGURE 4.1 Typical Installation of a CCTV Camera at an Intersection for Traffic Monitoring 28 FIGURE 4.2 An Example of CCTV Camera View (Daytime) ...................................................... 29 FIGURE 4.3 An Example of CCTV Camera View (Nighttime) .................................................... 29 FIGURE 4.4 Illustration of Images Annotation ........................................................................... 31 FIGURE 4.5 Training Progress ................................................................................................... 32 FIGURE 4.6 Procision-Recall Plot (all classes).......................................................................... 33 FIGURE 4.7 Precision-Recall Plot (smaller sizes of vehicles, e.g., passener cars, pickup trucks, minivans, etc.)......................................................................................................... 33 FIGURE 4.8 Precision-Recall Plot (large vehicles, e.g., buses, single-unit trucks, semitrailers, etc.) ......................................................................................................................... 34 FIGURE 4.9 Classification and Detection (Daytime).................................................................. 34 FIGURE 4.10 Classification and Detection (Nighttime).............................................................. 35 FIGURE 4.11 Intersection and Union of Two Bounding Boxes................................................... 36 FIGURE 4.12 Illustration of IoU-based Tracking ....................................................................... 37 FIGURE 4.13 Comparison of Vehicle Tracking Results between the IoU-based Method and the Kalman Filter with Distance Cost Matrix .............................................................. 39 FIGURE 4.14 The Effect of Smoothing (the white lines indicate the smoothed trajectories; while lines in other colors indicate the original trajectories by IoU) .............................. 41
vii

FIGURE 5.1 Conflict Representation in a Rectangular Prism (x, y, t) ................................. 44 FIGURE 5.2 Illustration of Newton's Law of Gravitation. .......................................................... 46 FIGURE 5.3 Extraction of Trajectories, Minimum Separation, and Conflict Points .................. 47 FIGURE 5.4 Proximity to Collision (PTC) Pyramid ................................................................... 50 FIGURE 5.5 Plots of Cumulative and Instant Risk Functions ..................................................... 52 FIGURE 5.6 Estimation of Approaching Velocities based on Conflict Points ............................ 54 FIGURE 6.1 System Setup 1 Distributed System ...................................................................... 56 FIGURE 6.2 System Setup 2 Centralized System ...................................................................... 57 FIGURE 6.3 Illustration of the Proposed Video Analytics System .............................................. 58 FIGURE 6.4 Detection of a Pre-conflict Event ............................................................................ 58 FIGURE 6.5 Detection of a Conflict Event .................................................................................. 59 FIGURE 6.6 Logging of a Simulated Conflict Event ................................................................... 60 FIGURE 6.7 Detection of an Actual Conflict Event (Camera View) ........................................... 61 FIGURE 6.8 Detection of an Actual Conflict Event (Top-down/Plan View) ............................... 62
viii

EXECUTIVE SUMMARY This research study aims to develop an image-based analytical system that is capable of detecting and quantifying traffic conflicts events as they occur in real time. The system, if deployed on a large scale, can help establish an invaluable conflict database, which would be instrumental to the goal of Vision Zero because inconsequential traffic conflict data can be systematically collected and used for road safety diagnosis such that proactive actions can be undertaken to prevent consequential crashes from happening.
The art of deep learning, multi-objects detection and tracking were explored and tested in the domain of traffic conflict monitoring. A smart computational system was developed to extend human perception intelligence to the existing traffic monitoring cameras to detect traffic conflict events as they occur. Road users (e.g., vehicles) are first detected by a convolutional neural network, Single Shot MultiBox Detector (SSD) Inception v2, which was retrained using annotated images from the views of traffic monitoring cameras. The detected road users are then tracked based on an Intersection over Union (IoU) measure, resulting in trajectories of the tracked road users. The trajectories were further processed to remove noises and then analyzed to detect and quantify traffic conflict events. Tests on simulated and actual video images demonstrate the promise of the algorithm for detecting and quantifying traffic conflict events.
ix

1. INTRODUCTION Intersections are planned locations for handling traffic movements in conflict. Different approaches and movements by various road users (e.g., motorists, bicyclists and pedestrians) make the at-grade intersections the most complex and unsafe spots on a roadway network. The information required for road users to process at a signalized intersection is much more complex than on a roadway section. According to the intersection safety issue brief 2 (FHWA-SA-10-005), a disproportionate percentage of fatalities (83 percent) occurred at signalized intersections in urban areas. The safety at signalized intersections has been a top priority for national, state, and local agencies.
Safety evaluation of signalized intersections has traditionally relied on historical crash data (post-event data) and is typically conducted at an aggregate (macroscopic) level due to the lacking of granular time-variant information pertaining to crash occurrence. In practice, evaluating the safety effects of traffic control and geometric features requires aggregation of historical crashes by intersections with similar characteristics. Given crashes as rare events, this traditional approach is somewhat ineffective and cannot be applied in a timely manner, leaving many dangerous intersections undetected until such a time when a significant number of crashes have occurred.
Instead of relying on actual crash data, which will take a long time to accumulate, more recent approaches used surrogate measures for crashes, such as "conflict". Conflict indicates "potential" of crash occurrence and can be observed in the field by way of avoidance behaviors, such as sudden braking or lane change. Nevertheless, the conflict study is subjective, labor-intensive, and often put observers in a dangerous situation. In
1

this respect, Federal Highway Administration (2008) developed a surrogate safety assessment model (SSAM), which combines micro-simulation and automated conflict analysis to analyze the frequency and character of narrowly averted vehicle-to-vehicle collisions. SSAM allows for an expedited safety assessment, but requires considerable efforts in building micro-simulation models for extracting trajectory data and a number of simulation runs for statistical analysis. Instead of observing in the field or modeling through simulations, the microscopic vehicle-to-vehicle "conflict" data could be conveniently extracted from live video images through properly defined surrogate measure(s) based on "real-time" vehicle motion data. This real-time image-based approach provides highway agencies the ability to evaluate and monitor the safety status of intersections in real time such that deficiencies in signal operations (in combination with any specific geometric features) can be timely discerned and corrected before actual crashes occur. Given the proactive nature of this proposed approach, it is expected to reduce the number of crashes that could have occurred otherwise.
For practicality, this research aims to use the existing traffic cameras installed at major intersections, as well as to develop and deploy smart algorithms to scan live images on the fly and extract traffic conflicts in a similar fashion as a trained human observer does. A short-lived history of sequential images (e.g., immediate past 30 successive image frames) is processed to obtain trajectories of the objects (e.g., vehicles) captured in the camera view, based on which conflicts can be detected and quantified. This process continues as each new frame is acquired in real time. Once a conflict event is detected, the conflict characteristic data, which does not contain any personal information, can be retained and logged in a database.
2

1.1 Background Traditional safety analysis at intersections relies on crash or collision data. A typical safety diagnosis process is illustrated in Fig. 1.1.
FIGURE 1.1 Steps to Identify Potential Problems (FHWA-HRT-04-092, April 2004) Highway Safety Manual (HSM 2010) adopted a systematic process (Fig. 1.2) for
safety management, which is driven by established crash databases and involves a live cyclic process for screening and identifying hazard locations, performing diagnosis, selecting appropriate countermeasures, conducting economic evaluations, prioritizing safety projects, and evaluating their effectiveness after implementation.
3

FIGURE 1.2 Roadway Safety Management Process (Adapted from the Highway Safety Manual 2010)
However, the sole reliance on crash data for safety diagnosis and management have two undesirable implications: (1) safety cannot be assessed if there are no crashes occurred, (2) safety diagnosis is conducted in a reactive manner, implying safety issues may not be detected until many crashes have occurred to be considered as overrepresented.
In considering those drawbacks, a proactive approach is necessary to enable effective safety diagnosis in absence of crash data. If this is attainable, proactive safety measures can be undertaken to reduce and eventually prevent crashes. This proactive approach has been supported by Vision Zero, which considers crashes as preventable and aims to eliminate all traffic fatalities and severe injuries, while increasing safe, healthy, equitable mobility for all. First implemented in Sweden in the 1990s, Vision Zero has
4

proved successful across Europe -- and now it is gaining momentum in major American cities. In 2016, Institute of Transportation Engineers (ITE) launched its Vision Zero Task Force to advance the goals of Vision Zero.
1.2 Conflict as a Measure of Safety A surrogate safety measure is needed to permit diagnosis of potential traffic safety issues prior to occurrence of any crash events.
As a surrogate safety measure, traffic conflicts have been collected in the field by observing conflicting behaviors, such as sudden swerving or braking to avoid potential collision. Although more frequent than collisions, conflicts are still considered as rare events. Amundsen and Hyden (1977) defined traffic conflict as an observable situation in which two or more road users approach each other in space and time to such an extent that a collision is imminent if their movements remain unchanged. The safety pyramid (Fig. 1.3) has been used to illustrate the concept and severity levels of conflicts relative to collisions.
FIGURE 1.3 Hyden's Safety Pyramid (Hyden, 1987)
5

Conflicts can be classified by the respective movements in a similar fashion as crashes. For a regular four-leg intersections, there are 32 vehicle-vehicle conflict points as illustrated in Fig. 1.4
FIGURE 1.4 Illustration of Conflict Points at a Conventional Four-Leg Intersection (FHWA-HRT-04-091, August 2004)
Even though a traffic conflict event can be observed in the field or through a video camera by human eyes, it is difficult to quantify because of its inconsequential and evasive nature. Not only will the human assessment be subjective and vary by observers, but result in rather qualitative evaluations that are likely to be inconsistent. As such, a systematic, objective and quantitative approach is required for effectively assessing traffic conflict events.
6

1.3 Safety Implications of Emerging Technologies The transportation sector is currently driving the wave of innovation and has recently been undergoing dramatic transformation by two evolving technologies: (1) ride sharing services (e.g., Uber, Lyft, Didi, etc.), and (2) autonomous vehicle (AV) technology (e.g., Google, Tesla, Baidu, Ford, etc.)
The goal or driving force behind the ride sharing is to maximize the use of the existing capacity of road infrastructure by reducing the number of single-driver vehicles on the road. Not only does it reduce congestion, but create a new job sector for drivers who are willing to participate in the ride-sharing programs. This innovative service has proved to dramatically increase the efficiency of roadway usage by efficiently chaining trips and reducing the number of vehicles on the road. On the other hand, the motivation behind the AV technology lies in increased safety. Knowing that approximately 94% of crashes are attributable to drivers (Singh, 2015), the AV technology is expected to significantly reduce and hopefully eliminate crash occurrence by removing human factors in various driving tasks. Regardless of the progressive effort in the field and foreseeable adoption of AV technology in the near future, many questions remain open. Implementing AV technologies would be a systematic process, which takes time and should be executed with caution. Still many technical and nontechnical barriers exist to overcome. There is likely to be unexpected issues or concerns arising during the implementation process. For example, how autonomous vehicles would and should interact with human drivers and non-motorists (e.g., pedestrians and bicyclists).
The analytics system proposed in this study aims to detect conflict events as they occur in real time and quantify their collision risks correspondingly. Thus, it can serve as
7

an invaluable tool to evaluate various AV technologies as they continue to evolve, especially in their penetration stage where mixed traffic of human-driver vehicles and autonomous vehicles exists on the roads. 1.4 Organization of the Report Following the introduction in Section 1, the relevant research studies and practices are reviewed and discussed in Section 2. The research approach is outlined in Section 3. Section 4 presents the challenges faced in the subject application domain, followed by the proposed methods for road user detection and tracking. Section 5 describes our innovative approach for detecting and quantifying traffic conflicts as they occur in real time. The software implementation is discussed in Section 6. The study is concluded in Section 7 with recommendations and further research.
8

2. LITERATURE REVIEW This section reviews recent research and practices pertaining to the scope of this study by focusing on two aspects: (1) Object Detection and Tracking, and (2) Traffic Conflict Identification and Quantification.
2.1 Object Detection and Tracking Background subtraction is a traditional and well-established approach for object detection. It is simple and computationally convenient, and has been used in many traffic-related applications, such as vehicle detection and counting. Typical assumptions for the background subtraction method include static background, moving foreground objects, a clear contrast between foreground objects and the background, and static cameras. However, these assumptions can be easily violated in field conditions as illumination constantly changes over time, the colors of foreground objects can be similar to those of the background, contrast between foreground objects and the background can degrade significantly at night, various weather conditions could produce a nonstationary background, cameras themselves are not perfectly stationary because of wind or other weather events, foreground objects are not always moving (e.g., a stopped car). Additionally, other factors, such as vehicle headlight at night and shadow cast during a sunny or cloudy day, can blur the boundary between the foreground objects and the background. Because of those various reasons, the background subtraction method does not offer accurate and reliable detection in many real-world situations and thus cannot be used for applications that demand a high level of detection accuracy and reliability.
9

Recognizing some of the drawbacks of the background subtraction method, featurebased tracking methods have been developed and used. One famous tracker is the KanadeLucas-Tomasi (KLT) feature tracker, where some Shi-Tomasi corner points are detected and those points are tracked iteratively using Lucas-Kanade optical flow. The method was based on assumptions that the pixel intensities of an object do not change between consecutive frames and neighboring pixels have similar motion. Large motions are handled by referencing upper levels of the image pyramid. For this study, the KLT method has been applied to detect vehicles in the camera view and the results are shown in Fig. 2.1 and Fig. 2.2, for daytime and nighttime, respectively. The issues are inconsistency of feature points being picked by the algorithm and the changing size of vehicles in the camera perspective view. Additionally, the detection results are unstable and subject to varying lighting conditions as seen in Fig. 2.2.
FIGURE 2.1 An Example of Optical Flow (East-West Connector/South Cobb Drive; Daytime)
10

FIGURE 2.2 An Example of Optical Flow (East-West Connector/South Cobb Drive; Nighttime)
Other features explored in this study include SIFT (Scale-Invariant Feature Transform), HOG (Histogram of Oriented Gradients), and ORB, i.e., Oriented FAST (Features from Accelerated Segment Test) and Rotated BRIEF (Binary Robust Independent Elementary Features). Fig. 2.3 shows the matching of ORB features between two different frames. Similar to the optical flow, the matching based on ORB features are not reliable and subject to varying lighting conditions as well. Sometimes, there are no feature points that can be matched.
11

FIGURE 2.3 An Example of ORB Feature Matching
The HOG describes features based on the distribution of image gradients. The firstorder gradients can capture contour, silhouette and certain texture information. In application, the whole image window is divided into small spatial regions. For each region, a local one-dimensional histogram of gradient over all pixels in the region is generated. A HOG feature image was generated for an image from a traffic camera and is shown in Fig. 2.4.

a) Original Image

b) HOG Feature Image

FIGURE 2.4 Example of HOG Features

As seen in Fig. 2.4, the crosswalks have quite similar HOG features as actual vehicles. This is because the HOG feature descriptor captures only top-layer or "coarse" features of images.
12

Another feature-based detector, Haar Feature-based Cascade Classifiers (Viola and Jones, 2001) has been widely used for object detection, especially face detection. It is a machine learning approach for visual object detection, where a cascade function is trained from plenty of positive and negative images. It is characterized by three key contributions. The first is the introduction of a new image representation called the "Integral Image" which allows the features used by the detector to be computed quickly. The second is a learning algorithm, based on AdaBoost (Adaptive Boosting), which selects a small number of critical visual features from a larger set and yields efficient classifiers. The third contribution is a method for combining increasingly more complex classifiers in a "cascade" which allows background regions of the image to be quickly discarded while spending more computation on promising object-like regions.
The ORB, SIFT or HOG-based methods are computationally fast. However, they rely on hand-turned filter banks and produce only a top layer of features for images. Recently, Convolutional Neural Networks (CNN) has gained momentum in their applications in object detection and classification. Since AlexNet (Alex Krizhevsky et al., 2012) won the 2012 ImageNet Large-Scale Visual Recognition Challenge (ILSVRC), CNN has become the most popular method for image recognition tasks. The ILSVRC 2013 and 2014 winners were also CNN, known as ZFNet and GoogLeNet. The architecture of AlexNet is illustrated in Fig. 2.5.
13

FIGURE 2.5 An Illustration of the Architecture of AlexNet between Two GPUs. One GPU runs the layer-parts at the top of the figure while the other runs the layer-parts
at the bottom. The network has 62.3 million parameters and needs 1.1 billion computation units in a forward pass (Krizhevsky et. al, 2012)
The VGG-Net (Simonyan and Zisserman, 2014; Table 2.1) is the second place of ImageNet ILSVRC-2014. It proved that multiple stacked small kernel size filters in convolutional layers is better than one layer of larger kernel size filter. VGG-Net uses multiple 3x3 kernel size filters instead of the first and second convolutional layers in AlexNet which used 11x11 and 5x5 kernel size filters. This change makes the network feasible to learn more complex features at low cost. The VGG-Net achieved a phenomenal improvement in accuracy in ImageNet ILSVRC competition.
14

TABLE 2.1 The ConvNet Configurations of VGG-Net (shown in columns)
Note: the ReLU activation function is not shown for brevity (Simonyan and Zisserman, 2014). Continued research has been focused on how to make a network deeper with low computational requirements. However, increasing the depth of layers brings two issues: increasing cost of both time and memory and gradient diffusion which can be understood as unavoidable loss during back-propagation of information through layers. The ReLU function is unable to solve the sparsity in earlier layers of a very deep network, meaning the derivative of most nodes are 0. Dealing with those issues, networks with new architectures were proposed. Inception network was designed by Szegedy et al. (2015). So far, Inception has 4 versions. The Inception v1 is also named as GoogLeNet, which won the ImageNet ILSVRC competition in 2014. It introduced a module called Inception,
15

which creatively used 1x1 convolutional kernel for dimensionality reduction. This Inception module offered GoogLeNet a better performance than VGG-Net with less computation time. The GoogLeNet is deeper than VGG 16 but with fewer parameters. The architecture of GoogLeNet is shown in Table 2.2. TABLE 2.2 GoogLeNet Incarnation of the Inception Architecture (Szegedy et al., 2015)
For a direct reference, the recent evolution of CNN for object classification and localization is discussed subsequently. Girshick et al. (2014) demonstrated that rich features generated by a single CNN could be used for both localization and classification tasks. It was found that most features learned in the convolutional layers are general and can be used for feature-based tasks in different domains. Some pre-trained CNNs on the large ImageNet dataset have been successfully tuned for smaller network with minimal
16

modification (Everingham, 2010). The architecture for the Regions with CNN features (RCNN) is shown in Fig. 2.6.
FIGURE 2.6 R-CNN Architecture (Girshick et al., 2014)
Taking an image as input, R-CNN creates region proposals, or bounding boxes, through selective search (step 2 in Fig. 2.6). After the proposals are created, R-CNN warps the region to a standard square size (224 x 224) and passes it through to a CNN, a modified version of AlexNet, to compute features (step 3 in Fig. 2.6). A Support Vector Machine (SVM) is applied on the final layer of the CNN to classify the regions by types of objects (step 4 in Fig. 2.6). Once a region is classified to a certain type of objects, a linear regression is performed on the coordinates of the region to output a tighter bounding box. In R-CNN, three different models are trained separately: (1) the CNN to extract image features, (2) the SVM classifier, and (3) the regression model to tighten the bounding boxes.
Even though the regions proposal by selective search reduces computational time as compared to the sliding window method, it is still quite slow, mainly because every single region proposal requires a forward pass of the CNN. To avoid the repeated passes of the CNN, the Fast R-CNN is proposed and the architecture is shown in Fig. 2.7.
17

FIGURE 2.7 Fast R-CNN Architecture (Girshick R., 2015)
In Fast R-CNN, the Region of Interest Pooling (ROI Pooling) is used, which shares the CNN features from one single forward pass of an image across its sub-regions. Then, the features in each region are pooled. In addition, Fast R-CNN replaced SVM classifier with a softmax layer for the classification task and added a linear regression layer in parallel to the softmax layer to generate bounding box coordinates. As such, a single network can handle both classification and localization. Regardless of all the improvements, there is still one bottleneck in the Fast R-CNN,which is the region proposal by selective search. Instead of executing a separate algorithm for selective search, the Faster R-CNN used the image features extracted from the forward pass of the CNN to generate region proposals. This results in nearly cost-free region proposals (Ren et al., 2016). As seen in Fig. 2.8, a single CNN was used for generation of region proposals and classification.
18

FIGURE 2.8 Faster R-CNN is a Single, Unified Network for Object Detection. (Ren et al., 2016)
Note that the R-CNN, Fast R-CNN, and Faster R-CNN generate bounding boxes, not actual shapes of the objects. Mask R-CNN extends the Faster R-CNN for image segmentation at the pixel level. In Mask R-CNN, a convolutional backbone architecture is used for feature extraction over an image, followed by a network head for the boundingbox recognition (classification and regression). A mask branch is added for mask prediction. The Mask R-CNN with the head architecture of the Feature Pyramid Network (FPN) backbone is shown in Fig. 2.9. Some test results on the COCO (Common Objects in Context) dataset is shown in Fig. 2.10.
19

FIGURE 2.9 Head Architecture of Mask R-CNN plus Faster R-CNN with FPN (Ren et al., 2016)
FIGURE 2.10 Keypoint Detection Results and Predicted Segmentation Masks (Ren et al., 2016)
In balancing accuracy and speed, the recent work in CNN has been focused on onestage object detection, which aims to deliver real-time inference. The two most popular architectures are You Only Look Once (YOLO) and Single Shot MultiBox Detector (SSD). Those models reach detection with one shot or one pass of the network, thus considerably reduce the computational time for inference and are able to deliver real-time classification and detection. The architecture of SSD based on the VGG-16 base network is illustrated in
20

Fig. 2.10. The SSD network combines predictions from multiple feature maps of different resolutions to handle objects of various sizes. It eliminates proposal generation and subsequent pixel or feature resampling stages and encapsulates all computation in a single network, providing a unified framework for both training and inference (Liu et al., 2016). The SSD results in 59 FPS (frames per second) with mAP of 74.3% on VOC2007 test, vs. Faster R-CNN at 7FPS with mAP 73.2% or YOLO at 45 FPS with mAP 63.4%.
FIGURE 2.11 Single Shot MultiBox Detector (Liu et al., 2016)
After experimenting with traditional models and various deep neural network architectures trained with images in the subject domain, the SSD Inception v2 was finally chosen in consideration of both accuracy and speed.
It is worthy of pointing out that a new class of efficient models named MobileNets has been proposed (Howard et al., 2017). MobileNets are often considered as light weight deep neural networks, which aim to reduce computational cost and make CNN feasible on mobile devices by using depth-wise separable convolutions. In this respect, two simple global hyperparameters were introduced to efficiently trade off between latency and
21

accuracy. However, an experiment of MobileNET trained with images in the subject domain showed unsatisfactory detection accuracy.
2.2 Traffic Conflict Identification and Quantification Given the complexity and subtlety of traffic conflict events, a trained human observer is often required to identify a de facto conflict. Recent research studies have been focused on automating conflict identification and quantification by using some safety surrogate measures. Commonly used surrogate measures are Time-to-Collision (TTC), PostEncroachment Time (PET), Potential Time to Collision (PTTC), Difference in Speeds (DeltaS), initial Deceleration Rate of the second vehicle (DR), the Maximum Deceleration of the second vehicle (MaxD), Difference in Velocities (DeltaV), and Safe Stopping Distance. Federal Highway Administration (FHWA, 2008) developed a surrogate safety assessment model, which permits an expedited safety assessment. However, it requires micro-simulation models to extract vehicle trajectories and a significant number of simulation runs for meaningful statistical inferences. Some previous studies were carried out to extract surrogate measures from video images based on spatial or temporal proximity of two or more conflicting road users (Sayed et al., 2013; Oh et al., 2009; Saunier and Sayed, 2006).
In summary, the traditional approaches and methods typically use one or more simplified indicators (e.g., TTC, PET, DR, etc.) to identify a conflict event. However, each of those indictors have its advantages and disadvantages. Given the complexity, variety and subtlety of conflict events, a true conflict may not be identifiable by any of those individual indictors due to the fact that those indicators were mostly based on partial
22

aspects of conflict events. On the other hand, simulation-based conflict analysis approach (e.g., FHWA 2008) relies on predictive modeling of trajectories, and is computationally demanding and not well suited for real-time applications. The accuracy and reliability of micro-simulations might be questionable.
More recently, Laureshyn et al. (2017) proposed an Extended Delta V measure to integrate the proximity to a crash as well as the outcome severity in the event a crash would have taken place, which are both important dimensions in defining the severity of a traffic conflict event. The severity has been estimated based on an Extended Delta V by assuming that road users spent the time available to brake before arriving at the collision point. As such, driver behaviors (e.g., deceleration rates) and collision mechanism (e.g., inelastic collision) are required or have to be assumed for calculating the severity metric.
23

THIS PAGE IS INTENTIONALLY LEFT BLANK.
24

3. RESEARCH FRAMEWORK This section describes our proposed approach for conflict detection and quantification. Considering the practicality, the existing CCTV traffic monitoring cameras currently used by public agencies will be employed as live "sensor" to detect traffic conflict events as they occur in the camera view in real time. Given the subtle and evasive nature of traffic conflict events, accurate and reliable detection and localization of road users in the camera view is critical for detecting a "true" conflict.
Based on the experiments with various detection algorithms described in Section 2, background subtraction and traditional feature-based approaches are not able to deliver desired detection accuracy and reliability for the subject application. As such, we envision utilizing the state-of-the-art deep neural networks for this task. CNN offers significantly improved performance in term of accuracy and reliability in various application settings, however requires considerable computational resources. One or more graphic processing units (GPU) are generally needed at the inference stage for real-time applications. In consideration of accuracy-speed tradeoff, the SSD is finally chosen as the architecture for the detection task. Since our application aims to process live video in real time, the SSDInception v2 network (300x300 input) was adopted and re-trained with the application domain-specific data, i.e., annotated images from traffic monitoring cameras.
Once vehicles in camera view are detected and located by the SSD, they will be tracked based on an Intersection over Union (IoU) measure computed from successive image frames from a live or streaming video source. The transitory trajectories manifested within a "live" or sliding tracking prism, defined by (x, y, t), are extracted and assessed. The relative positions of the trajectories, approaching velocities and
25

acceleration/deceleration are analyzed for detecting conflict events as they occur in real time. For each conflict event, the collision risk and severity are estimated based on an innovative "speed gravity" model, which is discussed in detail in Section 5.
26

4. DETECTION AND TRACKING OF ROAD USERS As described previously, to detect conflict events from a live video source, road users (e.g., vehicles) need to be detected and then tracked in real time. In this section, we first discuss the application domain and challenges, followed by our proposed approach for detecting and tracking of road users. 4.1 Application Domain Challenges The proposed application aims to use the existing CCTV cameras to monitor conflict events as they occur in real time. This requires simultaneously detecting and tracking multiple moving objects (e.g., vehicles) in different directions, which presents the following technical challenges.
(1) For practicality, traffic monitoring cameras are commonly mounted on utility poles in a corner of intersections at certain height, typically in the range of 25-35 feet from the ground. Fig. 4.1 illustrates such an installation. The limited height of the camera results in a 2D perspective view, which introduces potential occlusions, especially when objects are large, such as trucks, and in close vicinity. Additionally, the dimensions of objects in the camera view vary by their locations.
(2) The monitoring locations are subject to various lighting (e.g., daytime, nighttime, street lighting) and weather conditions.
(3) Multiple moving objects in various sizes, colors, shapes are to be detected and tracked.
(4) Potential aerial obstructions (signal heads, mast arms or span wires, utility lines, etc.) are normally present, which inevitably add foreground noises.
27

(5) Lack of uniformity in pavement color and presence of signage and pavement markings add background noises.
FIGURE 4.1 Typical Installation of a CCTV Camera at an Intersection for Traffic Monitoring
An example of camera view in daytime and nighttime are shown in Figs. 4.2 and 4.3, respectively. As seen in Fig. 4.2, aerial signal heads and wires add foreground noises for detection and tracking tasks. The lack of uniformity in pavement color and various pavement markings introduce background noises. Those foreground and background noises could potentially result in missed or false detections depending on the detection methods and/or algorithms. The varying lighting condition throughout a day can significantly affect detection accuracy and reliability as well.
28

FIGURE 4.2 An Example of CCTV Camera View (Daytime)
FIGURE 4.3 An Example of CCTV Camera View (Nighttime)
29

4.2 Detection and Tracking Traditional detection and tracking methods are unable to achieve accuracy and reliability desired for the subject domain of application, characterized by multiple objects, various types and sizes of objects, potential aerial obstruction, complex background scenes, camera perspective view, and various weather and lighting conditions. To tackle those challenges, state-of-the-art objects detection and tracking methods use rich image features extracted by deep neural networks.
4.2.1 SSD Architecture for Detection In selecting appropriate neural network architectures for the detection task, accuracy and speed are two major considerations. Accuracy is needed for reliable conflict detection while speed is required for real-time application. After reviewing and experimenting with various deep neural network architectures, SSD-Inception V2 (300X300 input) was chosen in consideration of both accuracy and speed. Tensorflow Object Detection API (Huang et. al., 2017) was used for implementation.
4.2.2 SSD Training For the purpose of network training and testing, images need to be annotated or labeled. The software (Tzutalin. LabelImg. Git code, 2015) was used for labeling. Rectangular bounding boxes were drawn manually around the objects to be detected and a class (Truck, Bus, etc.) was assigned to the object, as shown in Fig. 4.4.
30

FIGURE 4.4 Illustration of Images Annotation
To achieve desired accuracy, the bounding box was drawn as tight as possible. The resulting data set consists of 2,565 images (2,172 in daytime and 393 in night time) for training and 203 images for testing.
To benefit from the knowledge learned from large datasets, the pre-trained networks were utilized, referred to as transfer learning. In this case, the SSD Inception-v2 model pre-trained with COCO dataset were used and continuously trained with newly labeled images in the subject domain. By experimenting with grouping of different classes, the three-group classification (smaller vehicles, large vehicles, bicyclists/motorcyclists) showed a higher detection accuracy than more refined groupings and as such was adopted.
The training progress is shown in Fig. 4.5, which indicates a stabilized minimum total loss after approximately 3,000 epochs.
31

FIGURE 4.5 Training Progress

4.2.3 SSD Testing

To measure the performance of the trained SSD model, a test dataset was used. The test

dataset contains 203 annotated images, which is independent of the training data set. The

performance measure of mAP (mean average precision) was used for network testing. The

mAP has been widely accepted as a criterion for recognition precision and can be computed

from the recall-precision curve. The recall and precision are defined as:

=

+

=

+

Following the approach proposed in PASCAL VOC 2007, we used a threshold

scale from 0.0 to 1.0 with a step or bin size of 0.1 to generate recall-precision curves. The

true positive in vehicle detection are those predicted bounding boxes which have IoU

greater than 0.5 with one ground truth bounding box in same category. In contrary, the

predicted bounding boxes with all IoUs less than 0.5 are considered as false positive. If a

ground truth bounding box does not have an IoU greater than 0.5 with any predicted

bounding box, it will be considered as false negative. Additionally, the mAP can be

estimated based on the area under the curve. Generally, a step curve is used to describe

mAP so that the area can be easily computed as the sum of rectangular areas of all steps or

32

bins. Based on the test data set, the trained SSD was able to achieve a mAP of 0.82 for all classes, a mAP of 0.97 for smaller sizes of vehicles, and a mAP of 0.79 for larger sizes of vehicles. The corresponding precision-recall curves are shown in Figs. 4.6-4.8.
FIGURE 4.6 Precision-Recall Plot (all classes)
FIGURE 4.7 Precision-Recall Plot (smaller sizes of vehicles, e.g., passener cars, pickup trucks, minivans, etc.)
33

FIGURE 4.8 Precision-Recall Plot (large vehicles, e.g., buses, single-unit trucks, semitrailers, etc.)
4.2.4 SSD Inference The trained SSD was applied for inference task using recorded video clips. The excerpted results are shown in Figs. 4.9 and 4.10, for daytime and nighttime conditions, respectively.
FIGURE 4.9 Classification and Detection (Daytime)
34

FIGURE 4.10 Classification and Detection (Nighttime)
As seen the SSD can successfully detect different classes of vehicles in both daytime and nighttime conditions. Since SSD is based on rich features extracted via different feature layers, the detection is robust to various lighting conditions, shadow, and occlusions.
4.2.5 Tracking by IoU In this section, we first review the state-of-the-art methods for multiple objects tracking, followed by our proposed tracking method. Once objects (e.g., vehicles) are detected, they are tracked frame-by-frame by associating detections between successive frames. This is often referred to as "tracking-by-detection". The performance of a tracking algorithm is largely influenced by the detection accuracy. The inaccurate or unreliable detections by traditional detection algorithms demand additional considerations by tracking algorithms to handle false or missed detections. In this respect, Andriyendo and Schindler (2011) defined a continuous global energy function that included an observation term based on
35

image data; three physically motivated priors for object dynamics, collision avoidance and object persistence; and a regularizer (simplicity prior) for keeping the number of trajectories low. A suitable optimization scheme was constructed o find strong local minima of the proposed energy.
Bae and Yoon (2014) proposed tracklet confidence using the detectability and continuity of a tracklet, and formulated a multi-object tracking problem based on the tracklet confidence, which is solved by associating tracklets in different ways according to their confidence values. An online learning method for discriminating the appearances of objects was used for reliable association between tracklets and detections.
For detecting traffic conflicts, accurate detection and tracking is required. Accuracy in this context implies few or none false or missed detections and tight bounding boxes. More recently, Bochinski et al. (2017) proposed a simpler tracking algorithm based on the intersection over union (IoU) measure, which has a very low computational footprint. The IoU is defined as the ratio of Intersection over Union of two bounding boxes as indicated in Fig. 4.11.
FIGURE 4.11 Intersection and Union of Two Bounding Boxes
36

To minimize the computational cost, we used the IoU measure to associate the detections between successive frames with the highest IoU as proposed by Bochinski et al. (2017). Since the bounding boxes are direct outputs from SSD, IoU can be conveniently computed from those bounding boxes. The assumption is that the larger the IoU is, the more likely the two detection boxes in successive frames are the same vehicle. Note that this assumption is true when vehicles are relative large in the image and/or the frame rate is relatively high. For the purpose of tracking, the max IoU is found for each object and compared against a threshold. If the max IoU is greater than the threshold, same track ID will be assigned to the object detected in previous frames.
Fig. 4.12 demonstrates how the IoU was used for tracking. Note that the numbers in cyan color indicate the IoU values computed from detections in two successive image frames.
FIGURE 4.12 Illustration of IoU-based Tracking
37

The track ID assignment is based on the IoU between the newly detected bounding box and the last bounding box in each existing track. Specially, we used the last bounding box in existing tracks, not necessarily the one in the immediately past frame, to improve the robustness of tracking. This allows some frames to be skipped due to possible lost detections. To assign the new detection to an existing track, we use the maximum IoU with a threshold. The specific steps of IoU-based tracking are described below.
Step. 1. Obtain N detected objects, i.e., bounding boxes from the new frame. Step. 2. Compute IoU between the N newly detected boxes (in Step 1) and the last
M boxes in the M tracks established previously. Step. 3. According to the NM IoU matrix, assign each detected box to the track
based on the max IoU if it is greater than a predetermined threshold. Step. 4. For each existing track, a "skip" counter is used for recording the number
of frames skipped without an assignment. Step. 5. New tracks are created for any unassigned detected boxes. Step. 6. Delete those tracks that are longer than the maximum length threshold for
tracking or have exceeded a limit on the number of skipped frames.
The proposed IoU-based method is compared with the traditional Kalman filter (KF) approach that use a distance cost matrix for assignment. The results are shown in Fig. 4.13. As seen, the IoU-based method produces more stable tracking. Note that the trajectories are not quite smooth due to the imperfect localization of the SSD.
38

(a) IoU turning movement

(b) IoU through movement

(c) KF turning movement

(d) KF through movement

FIGURE 4.13 Comparison of Vehicle Tracking Results between the IoU-based Method and the Kalman Filter with Distance Cost Matrix

To obtain smooth trajectories derived from the IoU-based method, a structured smoothing procedure was applied to remove noises and is described below.

Step. 1. Step. 2.

Screen tracks based on a length threshold. If the number of points in a track is greater than the threshold, execute the following steps. Given the tracks expressed in two dimensions (i.e., an X-Y plane), moving average is performed on "Ys" if (Xmax-Xmin) is greater than (Ymax-Ymin). Otherwise, moving average on "Xs" is performed.

39

Step. 3. Step. 4. Step. 5.
Step. 6.

Fit a quadratic function to the track by X if (Xmax-Xmin) is greater than (Ymax-Ymin). Otherwise, fit the quadratic function by Y. Compute slopes and the mean of the slopes of all points in the fitted curve. Add the absolute differences of the slopes from the mean slope. If the sum is less than a threshold, re-fit the data for this track to a linear function. Compute Euclidean distances between detection points and the corresponding points on the fitted lines. Remove Top-K points (those with largest Euclidean distances) and fit it again.

The effect of implementing the smoothing procedure is shown in Fig. 4.14, where the white lines represent the smoothed trajectories. As seen, minor detection noises have been successfully removed.

40

FIGURE 4.14 The Effect of Smoothing (the white lines indicate the smoothed trajectories; while lines in other colors indicate the original trajectories by IoU)
41

4.2.6 Runtime Computational Costs The runtime experiment was conducted on a hardware platform, consisting of Intel core i7 and GTX 1080 with 8 GB of dedicated GDDR5X VRAM. For video detection and tracking, the total running time includes mainly four parts, as shown in Table 4.1.

TABLE 4.1 Summary of Runtime Computational Times

Computation Task

Time (second)

Percent

1. Image reading and pre-processing

0.0420

44.59%

2. SSD inference

0.0370

39.28%

3. IOU-based tracker update

0.0002

0.21%

4. Processing output and draw image

0.0150

15.92%

Total:

0.0942

100%

Based on the computational time overhead in Table 4.1, image reading and SSD processing consume over 80% of the total computation time. The entire process would run approximately 10.6 FPS. If considering vehicle detection only, it would run approximately 27 FPS which is a real-time performance.

42

5. CONFLICT DETECTION AND CHARACTERIZATION Traditional field conflict study requires trained observers, who are usually positioned at certain vantage points to observe "abnormal" driver behaviors, such as sudden braking or swerving to avoid potential collisions. Similar to a field conflict study, conflict events could be detected by human eyes from CCTV cameras installed by highway agencies for traffic monitoring. However, it would be costly and tedious to have dedicated personnel sitting at a Traffic Management Center (TMC) to monitor potential traffic conflict events. In addition, the monitoring by human eyes is subjective and different persons, even well trained, could perceive and interpret conflict events differently. To overcome those shortcomings, this research study aims to automate the conflict-monitoring process by developing and implementing a smart real-time video analytics algorithm for detecting and quantifying traffic conflict events as they occur in real time.
So, what information would be needed for traffic conflict detection? This can be thought of as what visual cues that human eyes look for in a video for conflict detection. As human eyes scan for potential conflict events, several key pieces of information are constantly searched, which include the closeness of approaching vehicles in conflicting trajectories, the approaching velocities (speed and direction), and the magnitude of deceleration/acceleration if any. Basically, those key pieces of information are well captured by the trajectories of road users in conflict. In other words, a traffic conflict event can be identified by analyzing trajectories manifested in a properly defined spatiotemporal space.
As such, we develop a smart algorithm that extracts trajectories from a live video and processes them to detect and quantify traffic conflict events. To properly capture the
43

trajectories relevant to conflict identification, we define a short and live time window (e.g., t = 1 second or 30 successive image frames from a live or streaming video source) to capture the "live process" of conflict events. It should be noted that since the trajectories reflect the historical positions of road users captured within the short time window (t), conflict detection based on trajectories may be slightly lagging, but less than t.
One potential approach for conflict detection is an end-to-end direct application of deep neural networks (DNN). However, this would require a large amount of conflict events being labeled, which is not practical. Note that even though conflict events are more frequent than crashes, they are still rare events and take time to gather. For practicality, we propose an intuitive approach based on an affinity-to-collision (ATC) measure derived from trajectories captured. The proposed method is described in detail subsequently. After the proposed method has been implemented for a period of time with many conflict events being captured, training a DNN model may become practical if desired.
To scan for a conflict, a "live" rectangular prism, as shown in Fig. 5.1, is defined. The prism slides along the time (t) dimension one frame at a time as a new image frame is acquired in real time. Real-time analytics is performed on the trajectories captured within the prism. An exemplar conflict event captured within the prism is illustrated in Fig. 5.1.
FIGURE 5.1 Conflict Representation in a Rectangular Prism (x, y, t)
44

The sliding rectangular prism in Fig. 5.1 defines a dynamic searching space for conflict events. A conflict event can be quantified or characterized by approaching velocities (speed and direction), and acceleration (deceleration) of the vehicles involved.

5.1 Detection of Conflict Events

The conflict event is detected based on a closeness measure of two or more conflicting

trajectories momentarily manifested in the spatiotemporal searching space. For example,

for two conflicting trajectories i and j, the closeness can be computed as the Euclidean

distance in the spatiotemporal (x, y, t) searching space by Eq. 1.

, = ( - )2 + ( - )2 + ( - )2

(1)

Given the two distinct dimension measures, i.e., space (x, y) and time (t), a

scaling factor may be used, so Eq. (1) can be rewritten as:

, = ( - )2 + ( - )2 + (1 - )( - )2

(2)

where is the scaling factor, 0 1. Practically, is a weighting factor, indicating the importance of spatial closeness versus the importance of temporal closeness. The higher the value, the more important the spatial closeness will be. A lower value give more weight or importance to the temporal closeness. Two boundary conditions are (1) = 0, which indicates only time separation is considered in defining a conflict and (2) = 1, which indicates only spatial separation is considered in defining a conflict.

45

5.2 Characterization and Quantification of Conflict Events Provided the fact that conflicts are unrealized collisions, measuring severity of conflicts cannot be solely based on the potential consequence of conflicts. In other words, collision risk of the conflict should also be considered in deriving conflict severity. If the collision risk can be quantified, the severity of conflict events can be estimated as the product of the potential consequence and the collision risk.
The potential consequence of a conflict has traditionally been measured as velocity difference, delta v. We use a similar concept, but define a "conflict point" on each trajectory that leads to the subject conflict event and estimate the approaching velocity based on the portion of the trajectory prior to the conflict point.
5.2.1 Assessment of Collision Risk To assess the risk of collision for conflict events, we propose a risk model based on the concept of the Newton's law of gravitation, as illustrated in Fig. 5.2. According to the Newton's law of gravitation, two particles with mass m1 and m2 attract each other by a force that is directly proportional to the product of their masses and inversely proportional to the square of the distance between their centers.
FIGURE 5.2 Illustration of Newton's Law of Gravitation.
The Newton's law of gravitation is mathematically expressed in Eq. 3.
46

12

=

21

=



12 2

(3)

where

m1, m2 = masses of two particles

r = distance between the centers of the two particles

G = gravitational constant

By following a similar concept, we define a measure of affinity to collision, C, to

be directly proportional to the product of approaching speed of objects (e.g., vehicles) and

inversely proportional to a power function of a distance or separation measure of the two

conflicting objects. The approaching speeds of two objects and their separation can be

derived from their trajectories captured within a dynamic short-lived time window. Fig.

5.3 illustrates two conflicting trajectories captured from a video source in a spatiotemporal

space defined by a sliding prism (x, y, t), shown in purple. The prism slides along

the temporal (t) dimension one frame at a time as a new frame is acquired in real time.

FIGURE 5.3 Extraction of Trajectories, Minimum Separation, and Conflict Points
47

For example, if a video source (e.g., a camera) is used to obtain trajectories, the subscript i, j index the two conflicting trajectories. The sequence of points on the trajectories are denoted by (, , ) and (, , ). Eq. 4 can be used to obtain the minimum separation di,j between any two conflicting trajectories.

, = m,in,( - )2 + ( - )2 + (1 - )( - )2

(4)

subject to: (, , ) and (, , ) where is a weighting factor between spatial separation and temporal separation.

As shown in Fig. 5.4, the points (circled) on each trajectory, i.e., (, , ) and (, , ), where the minimum separation (,) were measured, are named as "conflict points".

5.2.1.1 Affinity-to-Collision (ATC) Measure and Conflict Gravity Model (CGM)

By resembling the Newton's law of gravitation, Eq. 3, we define an affinity-to-collision

measure, ATC, in Eq. 5.



=



12

(5)

where

V1 = approaching speed of object 1 (e.g., vehicle 1)

V2 = approaching speed of object 2 (e.g., vehicle 2)

d = minimum separation defined in Eq. 2 (,, note the subscripts , were dropped for succinctness).

, = positive parameters, i.e., >0 , >0.

48

Eq. 5 is referred to as Conflict Gravity Model (CGM) in this context. By the

definition in Eq. 5, the higher the speeds for two approaching objects are, the more likely

the conflict would become a collision. On the other hand, the smaller the separation of the

two approaching objects is, the more probable a conflict would turn into a collision.

Note that the affinity-to-collision, ATC, can be computed if the two parameters, ,

, are known or can be estimated. We have proposed a procedure for estimating , , which

will be discussed later. First, we introduce the method for quantifying traffic conflict events.

At this point, let us assume that , are known. So, the affinity-to-collision, ATC, can be

computed.

By inspecting Eq. 5, when V1 = 0 and/or V2 = 0, and d > 0, the affinity-of-collision

measure, ATC = 0, indicating no risk of collision. For ease of interpretation, we define

another measure, proximity to collision, PTC (Eq. 6), which is the inverse of ATC.



=

1

=

12

(6)

By this definition, a collision occurred if PTC = 0 (i.e., physical contact with

velocity greater than 0). The larger the PTC is, the less likely a collision would occur.

Similar to the Hyden's Safety Pyramid (Fig. 1.3), if we sort PTC (obtained from a

sufficiently large observation data set) in an ascending order, the frequency of occurrence

associated with PTC is expected to resemble a similar ranking as indicated by a pyramid

(Fig. 5.4). The normal traffic encounters (associated with larger PTC values) are common

and frequent and are represented by the wider base of the pyramid; traffic conflicts

(associated with smaller PTC values) are rare and less frequent are represented by the upper

tiers of the pyramid with severe conflict events oriented toward the tip of the pyramid.

49

Note that collision can be considered as the extreme case of conflict, where PTC = 0, which is represented by the very tip of the pyramid. It should be pointed out that the word "pyramid" is symbolic and does not necessarily mean that the ordered frequencies by PTC strictly follow the ranking represented by a pyramid shape.
FIGURE 5.4 Proximity to Collision (PTC) Pyramid By sorting traffic encounters in this fashion, PTC serves as an ordered measure, which can be associated with the frequency of traffic encounter events. Note frequency is observable and can be used for calibrating the CGM in Eq. 3. To assess collision risk of any conflict events, we define a risk function (a probability function) to associate collision risk with the PTC measure. There are many function forms that can be used to define the risk function. For simplicity and computational ease, we propose to use a logistic or sigmoid function (Eq. 7), which has been widely used as activation function in artificial neural networks. .
50



=

1 1+ -

(7)

Note that the sigmoid function maps or squeezes x (-, +) to y [0,1]. In

our case, we can not directly substitute PTC for x in Eq. 7 since [0, +).

So,

we

define

x

=

ln( 1 ),


which

maps





[0,

+)

to

y



[0,1].

Substituting

x = ln( 1 ) into Eq. 7 results in:




=

1 1+- ln1

=

1 1+

, [0, +)

(8)

Further, we define the risk of collision, R, as a conditional probability in Eq. 9.

()

=

(

|





)

=

(=0 ) ()

=

(=0) ()

(9)

By substituting PTC = c and y = R(c) in Eq. 6, we obtain Eq. 10.

()

=

1 1+- ln1

=

1 1+

, [0, +)

(10)

As indicated by Eq. 10, the risk decreases as PTC value (c) increases. Consider the

boundary conditions: when c = 0, ( = 0) = 1, indicating a collision occurred; when c

approaches infinity, ( +) 0.

Combining Eqs. 9 and 10 results in Eq. 11.

()

=

(=0) ()

=

1 1+

, [0, +)

(11)

Note that the risk function defined in Eq. 9 is cumulative with respect to PTC. An

instant risk,(), can be obtained by taking the derivative of R(c) with respect to c. As a

result, the instant risk can be written in Eq. 12.

(

=

)

=

()

=

-1 (1+)2

(12)

51

Note that the negative sign in Eq. 12 indicates that risk increases as c decreases. In

addition, r increases to 1 (in magnitude) as c approaches 0. In other words, ( = 0) =

( = 0) = 1 per Eq. 11.

Eq. 12 can also be written in terms of ATC in Eq. 13. Note that negative sign is

removed because of increasing risk function with respect to ATC.





=

1

=

()

=

1 (1+)2

(13)

For illustration purposes, the cumulative and instant risk functions, defined in Eqs.

11 and 12, are plotted in Fig. 5.5.

Collision Risk [0,1]

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1
0 0

R(PTC<=c) r(PTC=c)

5

10

15

20

PTC [0, +inf)

FIGURE 5.5 Plots of Cumulative and Instant Risk Functions

By substituting Eq. 6 into Eq. 11, () becomes a function of d, V1 and V2 (Eq.

14).

()

=

(,

1 ,

2)

=

1 1+12

(14)

Eq. 14 is a relative easy function to use. We will need to estimate parameters and

, which is described in the following section.

52

5.2.1.2 Procedure for Model Calibration

By combining Eq. 11 and Eq. 13, Eq. 15 can be obtained.

()

=

(=0) ()

=

1 1+12

Rearrange Eq. 15 to obtain Eq.16.

(15)

= 1 - 1 = 1-()

12 ()

()

(16)

By taking natural logarithm on both sides of Eq. 16, Eq. 17 is obtained.



ln()

-

ln()

-

ln(1 2)

=



1-()
()

(17)

Rearranging terms in Eq. 17 results in Eq. 18.

ln() - () = 1-(()) 1 2

(18)

Assuming that collision risk order (Fig. 3) would be preserved by the order of

observed conflict event frequencies, () , which is unobservable, can be practically

estimated based on the observed frequencies of conflict events at different PTC levels by

Eq. 19.

()

=

(=0) (<)

=











(19)

5.2.2 Assessment of Conflict Severity Since conflict events are inconsequential, the severity of conflict events cannot be defined in a traditional way. For conflict events, the severity should jointly consider two aspects: (1) the probability (risk) of collision, and (2) the likely consequence if a collision occurred. For the first aspect, we can use the collision risk derived previously. For the second aspect,

53

we propose to use velocity difference. As such, we can compute conflict severity using

Eq. 20.

= ( | = ) = ()

(20)

where () = = (Refer to Eq. 12)

= 2 + 2 - 2 cos() ; =
Velocity and are computed based on the part of trajectories prior to the conflict points as illustrated in Fig. 5.7. Note that conflict points are defined as points on two conflicting trajectories, where the minimum separation (d) between the two trajectories was measured. As shown in Fig 5.6, the short red line indicates the minimum separation (d) between the two conflict trajectories captured in the tracking prism. The two conflict points (one on each trajectory) are the points at which the red line was connected.

FIGURE 5.6 Estimation of Approaching Velocities based on Conflict Points
54

6. SOFTWARE IMPLEMENTATION For software application, Python programming language (Version 3.5.4) was used. OpenCV (Version 3.3.0) was used for image reading and preprocessing. Tensorflow (Version 1.4.0) was used as the backend for implementing the SSD network.
6.1 System Architecture Deployment of the proposed system can be accomplished in one of the two ways: a distributed system or a centralized system. For the distributed system, a processing unit is required to process live video images locally at each site being monitored. The processing unit will perform functionalities, including video processing, detecting, and quantifying conflict events in real time. Once a conflict is detected, the conflict characteristic data will be transmitted via Cloud to a database server, which may be located in a traffic management center. The database server will keep logging all conflicts received from all the sites being monitored and store those conflict data in a database. A schematic of a distributed system is shown in Fig. 6.1. Since the amount of data transmitted over the Cloud is small (i.e., only compiled conflict characteristic data is transmitted), a minimal network bandwidth is required.
55

FIGURE 6.1 System Setup 1 Distributed System For the centralized system, the road processing units are eliminated and the video from field cameras will be streamed via Cloud to the database server directly and all video processing and analytics will be carried out on the server, which may be located in a traffic management center. A schematic of the centralized system is shown in Fig. 6.2. Since all video data will be transmitted (streamed) over the Cloud, the network bandwidth for data transmission needs to be large enough for real-time application.
56

FIGURE 6.2 System Setup 2 Centralized System
6.2 Video Analytics System The video analytics system, which is the core of the application, consists of four major components as illustrated in Fig. 6.3. The system takes a video source as input and processes the sequence of image frames by a DNN module and generates classification and bounding boxes of road users and pass them to the IoU Tracker module. The IoU Tracker module takes the detected boxes in sequential frames and associate them to produce smooth trajectories. The Trajectory Analysis module conduct analysis on the trajectories and generate information pertaining to conflict analysis. The Conflict Detection and Quantification module processes the information from the Trajectory Analysis module to identify conflict events and quantify them accordingly.
57

FIGURE 6.3 Illustration of the Proposed Video Analytics System
6.3 System Testing 6.3.1 Simulated Environment Given the rare nature of conflict events, the program was first tested in a simulated environment, where conflict events can be easily generated. Fig. 6.4 shows an imminent conflict condition. The left window in Fig. 6.4 shows two simulated vehicles (a southbound left-turning vehicle and a northbound through vehicle) that approach each other. The right window in Fig. 6.4 shows the trajectories of the two vehicles. The red lines indicate that a looming conflict as the two vehicles approach each other closely.

a) Vehicle Tracking

b) Trajectories

FIGURE 6.4 Detection of a Pre-conflict Event

58

As the two vehicles continue to approach each other, a de facto conflict event occurs. The event is detected by the algorithm, as indicated by the pink arrows in Fig. 6.5.

a) Vehicle Tracking

b) Trajectories

FIGURE 6.5 Detection of a Conflict Event

Similar to crash records reported by a police, the characteristics of conflict events can be recorded automatically with a reference to the exact location and time as they occur. An exemplar test server was set up to log the conflict event and the web interface is shown in Fig. 6.6. The conflict diagram in Fig. 6.6 indicates that a southbound left turn vehicle failed to yield to an opposing northbound through vehicle, which has right of way. The characteristic information of this conflict event is shown on the left panel of the webpage, including the speeds and angles of approaching vehicles, the velocity difference (delta v), the collision risk and severity.

59

V2 V1
FIGURE 6.6 Logging of a Simulated Conflict Event 6.3.2 Test on Live Cameras The program was further tested on actual images from a live camera at the intersection of Frey Road and Skip Spann Connector, Kennesaw, Georgia. For demonstration, an actual conflict has been successfully captured at this intersection. The actual image when the conflict occurred is shown in Fig. 6.7.
60

FIGURE 6.7 Detection of an Actual Conflict Event (Camera View) The real-time tracking and detection results for the conflict event in Fig. 6.7 are depicted in a top-down view in Fig. 6.8 (a). For this conflict, a northbound (note that the north direction points to the left) right-turning vehicle (the white vehicle) suddenly slowed down, causing the closely following northbound through vehicle (the silver vehicle) brakes considerably. Fig. 6.8 (b) is a callout and shows the schematic of the conflict event in detail. The thicker green arrows indicate the velocities based on the tracked time window (i.e., one second or 30 frames). The thinner red arrows indicate the velocities prior to the conflict points.
61

FIGURE 6.8 Detection of an Actual Conflict Event (Top-down/Plan View)
62

7. CONCLUSIONS AND RECOMMENDATIONS In this study, the art of deep learning, multi-objects detection and tracking have been explored and tested in the domain of traffic conflict monitoring. Specifically, road users (i.e., vehicles) were detected using a state-of-the-art DNN (i.e., SSD) and tracked by an IoU-based method, which requires minimal computation resources. Based on extracted trajectories, a novel method was proposed to detect and quantify conflict events. Although the study has been focused on identifying and tracking vehicles at major signalized intersections, where a CCTV camera is typically installed for traffic monitoring, the proposed method is quite generic and can be employed at any locations with assistance of a high-resolution camera.
Tests on simulated and actual video images show that the proposed algorithm is able to detect conflict events as they occur in real time. The localization noises inherited from the DNN model due to the perspective view of cameras might result in false detection although the extracted trajectories have been smoothed to reduce such a risk. Future research should be directed to further improving detection accuracy and robustness.
Based on the runtime experiment, the computation bottleneck occurs at video image reading and SSD inference. The trajectories analysis for conflict detection and quantification requires minimal computation time. With adequate computing resources being allocated for image reading and processing, the system will be able to process live images and analyze them in real time to detect, quantify and log traffic conflict events.
63

If the algorithm is desired to operate on a central server that simultaneously process multiple streaming video sources, adequate computation resources need to be allocated depending on the number of video sources to be processed in real time.
The top-down camera view is preferred for improving detection and tracking accuracy. For this reason, it would be desirable for the cameras to be installed higher and closer to the center of intersection if practical.
Further research is needed to account for varying grades of intersections to improve accuracy of localization. In this regard, a dual camera setting or a 3D camera may be worthy of further exploration.
A new DNN architecture, RetinaNet, has recently been proposed and shown to improve detection accuracy by focusing on training a sparse set of hard examples (Lin et al, 2018). It is recommended that RetinaNet be tested for road user detection in future research.
64

8. REFERENCES 1. The National Intersection Safety Problem, Intersection Safety Issue Brief 2, FHWA-
SA-10-005, November 2009 2. Signalized Intersections: Informational Guide, FHWA-HRT-04-091, August 2004 3. Tingvall C. and Narelle H. (1999) "Vision Zero - An ethical approach to safety and
mobility". Monash University Accident Research Center, Monash University. https://www.monash.edu/muarc/research/our-publications/papers/visionzero. 4. http://www.ite.org/visionzero/ 5. https://visionzeronetwork.org/ 6. http://essentialelements.naccho.org/archives/8003 7. Highway Safety Manual, 1st Edition, 2010, American Association of State Highway and Transportation Officials (AASHTO). 8. A Policy on Geometric Design of Highways and Streets, 6th Edition, 2011, American Association of State Highway and Transportation Officials (AASHTO). 9. Singh, S. (2015, February). Critical reasons for crashes investigated in the National Motor Vehicle Crash Causation Survey. (Traffic Safety Facts Crash Stats. Report No. DOT HS 812 115). Washington, DC: National Highway Traffic Safety Administration. 10. Surrogate Safety Assessment Model (SSAM), FHWA Publication No.: FHWA-HRT08-049, 2008. 11. Shelby, S.G. Delta-V as a measure of traffic conflict severity. In: 90th TRB Annual Meeting, Washington DC, USA, 2327, January 2011.
65

12. Tarek Sayed, Mohamed H. Zaki, and Jarvis Autey. Automated safety diagnosis of vehiclebicycle interactions using computer vision analysis, Safety Science 59 (2013) 163-172.
13. Jutaek Oh, Joonyoung Min, Myungseob Kim, and Hanseon Cho, Development of an Automatic Traffic Conflict Detection System Based on Image Tracking Technology, Transportation Research Record: Journal of the Transportation Research Board, No. 2129, Transportation Research Board of the National Academies, Washington, D.C., 2009, 4554.
14. N. Saunier and T. Sayed. Clustering Vehicle Trajectories with Hidden Markov Models Application to Automated Traffic Safety Analysis, The 2006 IEEE International Joint Conference on Neural Network Proceedings, July 16-21, 2006.
15. Aliaksei Laureshyn, Tim De Ceunynck, Christoffer Karlsson, Ase Svensson, and Stijn Daniels. In search of the severity dimension of traffic events: Extended Delta-Vas a traffic conflict indicator, Accident Analysis and Prevention, 98, 46-56, 2017.
16. Hoong-Chor Chin and Ser-Tong Quek. Measurement of Traffic Conflicts, Safety Science, Vol. 26. No. 3, 169-185, 1997
17. Paul Viola and Michael J. Jones. "Robust real-time face detection." International journal of computer vision 57.2 (2004): 137-154. DOI: 10.1109/CVPR.2001.990517
18. Aliaksei Laureshyn, Ase Svensson, and Christer Hydn. Evaluation of traffic safety, based on micro-level behavioural data: Theoretical framework and first implementation, Accident Analysis & Prevention, Volume 42, Issue 6, November 2010, 1637-1646.
66

19. Dalal, N. and Triggs, B., "Histograms of Oriented Gradients for Human Detection," IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005, San Diego, CA, USA.
20. David G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, 60, 2 (2004), 91-110.
21. A. Andriyenko and K. Schindler. Multi-target tracking by continuous energy minimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 12651272. IEEE, 2011.
22. S.-H. Bae and K.-J. Yoon. Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 12181225, 2014.
23. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012.
24. Karen Simonyan and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556 (2014).
25. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich "Going deeper with convolutions." CVPR 2015
26. Lin, Min, Qiang Chen, and Shuicheng Yan. "Network in network." arXiv preprint arXiv:1312.4400 (2013).
27. R. B. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," arXiv:1311.2524v5, 2014.
67

28. R. B. Girshick, "Fast R-CNN," arXiv:1504.08083v2, 2015. 29. S. Ren, K. He, R. B. Girshick, and J. Sun, "Faster R-CNN: Towards Real-Time Object
Detection with Region Proposal Networks," arXiv:1506.01497v3, 2016. 30. K. He, G. Gkioxari, P. Dollr, and R. Girshick, "Mask R-CNN", arXiv:1703.06870v3,
2018 31. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, "The
pascal visual object classes (voc) challenge," International Journal of Computer Vision, vol. 88, 303-338, June 2010. 32. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications," arXiv:1704.04861v1, 2017 33. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C-Y Fu, and A. C. Berg, "SSD: Single Shot MultiBox Detector," arXiv:1512.02325v5, 2016 34. J. Redmon and A. Farhadi, "YOLO9000: Better, Faster, Stronger," arXiv:1612.08242v1, 2016. 35. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," arXiv:1506.02640v5, 2016 36. Kuhn, H. W. (2005). "The Hungarian method for the assignment problem". Naval Research Logistics. 52 (1): 721. doi:10.1002/nav.20053 37. A. Mousavian, D. Anguelov, J. Flynn, and J. Kosecka, "3D Bounding Box Estimation Using Deep Learning and Geometry," arXiv:1612.00496v2, 2017 38. A. Sadeghian, A. Alahi, and S. Savarese, "Tracking The Untrackable: Learning to Track Multiple Cues with Long-Term Dependencies," arXiv:1701.01909v2, 2017
68

39. E. Bochinski, V. Eiselein and T. Sikora, "High-Speed Tracking-by-DetectionWithout Using Image Information," IEEE AVSS 2017, August 2017, Lecce, ITALY
40. Huang J., Rathod V., Sun C., Zhu M., Korattikara A., Fathi A., Fischer I., Wojna Z., Song Y., Guadarrama S., Murphy K., "Speed/Accuracy Trade-offs for Modern Convolutional Object Detectors," CVPR 2017.
41. Lin T. Y., Goyal P., Girshick R., He K., Dollar P., "Focal Loss for Dense Object Detection", arXiv:1708.02002v2, 2018.
42. Tzutalin. LabelImg. Git code (2015). https://github.com/tzutalin/labelImg
69