Accident Analysis and Prevention 142 (2020) 105521

Contents lists available at ScienceDirect

Accident Analysis and Prevention

journal homepage: www.elsevier.com/locate/aap

Trajectory-level fog detection based on in-vehicle video camera with T

TensorFlow deep learning utilizing SHRP2 naturalistic driving data
Md Nasim Khan*, Mohamed M. Ahmed
University of Wyoming, Department of Civil & Architectural Engineering, 1000 E University Ave, Dept. 3295, Laramie, WY 82071, United States

A R T I C LE I N FO A B S T R A C T

Keywords: Providing drivers with real-time weather information and driving assistance during adverse weather, including
Foggy weather fog, is crucial for safe driving. The primary focus of this study was to develop an affordable in-vehicle fog
Machine Learning detection method, which will provide accurate trajectory-level weather information in real-time. The study used
Deep Learning the SHRP2 Naturalistic Driving Study (NDS) video data and utilized several promising Deep Learning techniques,
Image Classification
including Deep Neural Network (DNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM),
TensorFlow
Deep Neural Network
and Convolutional Neural Network (CNN). Python programming on the TensorFlow Machine Learning library
Recurrent Neural Network has been used for training the Deep Learning models. The analysis was done on a dataset consisted of three
Long Short-Term Memory weather conditions, including clear, distant fog and near fog. During the training process, two optimizers, in-
Convolutional Neural Network cluding Adam and Gradient Descent, have been used. While the overall prediction accuracy of the DNN, RNN,
Advanced Driver Assistance Systems LSTM, and CNN using the Gradient Descent optimizer were found to be around 85 %, 77 %, 84 %, and 97 %,
Advanced Travel Information Systems respectively; much improved overall prediction accuracy of 88 %, 91 %, 93 %, and 98 % for the DNN, RNN,
Variable Speed Limit LSTM, and CNN, respectively, were observed considering the Adam optimizer. The proposed fog detection
Mobile Weather Sensors
method requires only a single video camera to detect weather conditions, and therefore, can be an inexpensive
option to be fitted in maintenance vehicles to collect trajectory-level weather information in real-time for ex-
panding as well as updating weather-based Variable Speed Limit (VSL) systems and Advanced Traveler
Information Systems (ATIS).

1. Introduction between real-time crash likelihood and visibility using weather station
data. A study by Ahmed et al. (2014) examined the feasibility of using
Foggy weather condition is one of the most frequent causes of traffic weather information collected from nearby airports for the assessment
accidents due to its adverse impact on visibility. Driver performance of road crashes in real-time on highways with frequent fog problem
can be drastically affected by foggy weather conditions because they using Bayesian logistic regression and found that the likelihood of crash
may not be able to accurately assess the visibility distance as well as occurrence might increase with the reduction in visibility. Based on
electing safe speeds for the foggy weather condition (Khan et al., 2018). information from weather stations and real-time traffic data, the study
According to the Federal Highway Administration (FHWA), around in Wu et al. (2018) concluded that fog conditions increase the like-
10,448 injury crashes and 495 fatal crashes occur every year in the U.S. lihood of crashes, especially in congested traffic conditions. Another
due to the presence of fog (FHWA, 2019). Many previous studies have study (Yu et al., 2015) also used weather station data to investigate the
concluded that foggy weather conditions have a significant negative effect of visibility on crash risk for a mountainous freeway and con-
impact on driver behavior and performance as well as one of the major cluded that poor visibility conditions would increase the probability of
causes of weather-related motor vehicle crashes (Abdel-Aty et al., 2011; crashes. Many other studies (Yu et al., 2013; Usman et al., 2010) using
Ahmed et al., 2014; Das et al., 2019; Wu et al., 2018). Therefore, proper weather station data also found similar results.
assessment of road visibility conditions and providing drivers with Fog can cause a sudden reduction in roadway visibility and dra-
appropriate warnings and speed limits are crucial for safe driving, matic changes in driving conditions due to its variable nature as well as
especially in adverse weather, including fog. the small area over which it usually forms. This unpredictability of fog
Considering the negative impact of reduced visibility on traffic formation makes it one of the major causes of motor vehicle crashes.
safety, many studies have thoroughly investigated the relationship Abrupt reduction in visibility due to fog often leads to pileup and more

⁎
Corresponding author.
E-mail addresses: mkhan7@uwyo.edu (M.N. Khan), mahmed@uwyo.edu (M.M. Ahmed).

https://doi.org/10.1016/j.aap.2020.105521
Received 14 November 2019; Received in revised form 7 March 2020; Accepted 22 March 2020
0001-4575/ © 2020 Elsevier Ltd. All rights reserved.
M.N. Khan and M.M. Ahmed Accident Analysis and Prevention 142 (2020) 105521

severe crashes. As an example, a 19-vehicle pileup crashes occurred on Neural Network (DNN), Recurrent Neural Network (RNN), Long Short-
Interstate 75, Florida, due to sudden drop in visibility caused by thick Term Memory (LSTM), and Convolutional Neural Network (CNN).
fog, killing ten and injuring more than twenty people (Hassan and Python programming on the TensorFlow machine learning library has
Abdel-aty, 2013). Despite these adverse effects on roadway safety, the been used for training the deep learning models.
sudden local fog formation sometimes cannot be detected by weather In recent years, various fields of engineering are extensively using
stations due to the fact that weather stations are location-specific and deep learning for classification, detection, and pattern recognition
weather sensors are mostly mounted at higher elevations that may not purposes due to its advantages over traditional statistical and machine
necessarily represent visibility at road surface level (Khan and Ahmed, learning techniques in terms of accuracy, especially when trained with
2019). Therefore, in-vehicle sensors, including video cameras, are more Big Data. In the transportation field, many studies have used deep
promising for better representative real-time visibility determination. learning to detect and predict crashes, conflicts, vehicles, traffic flow,
Different approaches to video camera-based visibility estimation speed, travel time, and driver distraction. A recent study (Huang et al.,
can be found in the literature. For example, Pomerleau, 1997 in- 2020) used several variations of deep learning to detect crash occur-
troduced a system based on the contrast attenuation of the lane rence and predict crash risk using speed, volume, and occupancy data;
markings at a different distance in front of the vehicle. He used Rapidly and compared the results with traditional machine learning techniques.
Adapting Lateral Position Handler (RALPH) method to find and track The study concluded that deep learning models performed better in
roadway features, including lane markings, road/shoulder boundaries, terms of accuracy for crash detection compared to state-of-the-art
tracks left by other vehicles, and pavement discoloration. A study by shallow models. Another study (Bao et al., 2019) proposed a deep
Hautiére et al. (2006a) used a disparity map for visibility determination learning architecture, which they called spatiotemporal convolutional
and tested their system in sunny, foggy, and dense foggy weather long short-term memory network (STCL-Net), to predict crashes. They
conditions. The average visibility distance was found to be approxi- argued that the proposed deep model could leverage both the spatial
mately 250 m, 75 m, and 30 m in sunny weather, foggy weather, and and temporal dependencies in the explanatory variables. The results
dense foggy weather, respectively. Another study (Hautiére et al., indicated that the proposed deep models outperformed the traditional
3
2006b) by the same authors applied Koschmieder’s law (Vmet = k ) to statistical as well as state-of-the-art machine-learning models based on
estimate visibility distance. Here, Vmet is the meteorological visibility several evaluation criteria, including prediction accuracy and false
distance proposed by the International Commission on Illumination alarm rate. A study by Formosa et al. (2020) utilized deep learning to
(CIE), and k is the extinction coefficient of the atmosphere. First, the leverage large imbalanced data for predicting real-time traffic conflicts
Road Surface Luminance Curve (RSLC) was extracted using roadway considering several influencing factors including, speed variance, traffic
images taken from an in-vehicle video camera. Subsequently, a re- density, speed, and weather conditions. The results indicated that the
lationship between the RSLC inflection point and extinction coefficient proposed conflict detection techniques could be effectively integrated
was derived, in which k could be estimated. They showed that the RSLC to improve ADAS, CVs, and AVs. The study by Lv et al. (2015) proposed
model should consider the effects of non-horizontal vision. All RSLC- a novel traffic flow prediction method with Big traffic data using deep
based approaches can work on just a single frame. However, they de- learning. The study claimed that unlike other models, the proposed
pend strongly on a specific road scenario that does not allow for objects model could effectively uncover latent nonlinear spatial and temporal
blocking the view to the horizon. The recent approaches to determine associations from traffic data. They concluded that the deep model is
visibility from video cameras are mainly based on machine vision superior compared to traditional machine learning models in terms of
techniques. Bronte et al. (2009) proposed a real-time fog detection prediction performance. Another study by Napoletano and Schettini
system using an onboard low-cost black and white camera. Their system (2018) proposed a method capable of automatically detect driver dis-
is based on two clues; estimation of the visibility distance, which is traction using deep learning and convolutional neural networks using
calculated from the camera projection equations, and the blurring due video cameras mounted inside vehicles pointing at drivers. Many si-
to the fog. The polarized-based method, which requires two or more milar studies have also used deep learning to detect driver distraction
images of the same scene taken with different degrees of polarization, automatically (Alotaibi and Alotaibi, 2019; Hashemi et al., 2020). Deep
was proposed by Schechner et al. (2001) and Shwartz et al. (2006). The learning has also been used for travel time estimation (Tang et al.,
problem of the polarized based method is that it requires multiple 2019), speed prediction (Li et al., 2019), traffic signal control (Xu et al.,
images of the same scene. To overcome this problem, Tan (2008) de- 2020; Jeon et al., 2018), and vehicle defect detection (Krummenacher
veloped an automated fog detection method that only requires a single et al., 2018). Although many studies have used deep learning to detect
input image. Pavlić et al. (2012) developed an in-vehicle fog detection weather conditions using climate data (Prabhat et al., 2017) or image
system based on global image descriptors and a classification proce- data from stationary cameras (Ibrahim and Haworth, 2019), the use of
dure. Although the granularity of their system could not identify dif- deep learning to detect trajectory-level weather at road surface level
ferent levels of fog, they achieved a classification accuracy of 96 % and using naturalistic data is limited. Considering this research gap in mind,
93 % for clear and foggy conditions, respectively. Zhang et al. (2015) in this study, a novel approach for detecting road surface level fog in
proposed a technique using Support Vector Machine (SVM) and nine real-time based on deep neural networking techniques has been pro-
different features as individual predictors and achieved a maximum posed.
accuracy of 78.5 % (Zhang et al., 2015). It is worth noting that the negative impact of adverse weather on
Most of the current in-vehicle fog detection systems are based on a road surface visibility is more noticeable in foggy weather compared to
consistent object (lane marking, signs, the horizon, etc.) in front of the other weather conditions (Codling, 1971). In addition, fog and snow do
vehicle to operate correctly. These fog detection methods are not de- not usually occur at the same time of the year in some states such as
pendable in everyday situations because the road marking, signs, and Wyoming. Also, snowfall is a very rare environmental condition in the
the horizon can be obstructed by other vehicles, especially in congested southern part of the U.S., including Florida and Texas (Stone, 1936).
traffic conditions. Image-based fog detection technique using deep Considering the reasons mentioned above, this study only focused on
learning may overcome these problems as it only uses the global fea- fog detection in order to eliminate any possible reduction in detection
tures of images. However, the potential of deep learning for real-time accuracy due to the presence of other weather conditions in the model.
road surface visibility identification is not notably researched. This
study proposes some unique techniques to enhance the reliability of 2. Data reduction
real-time detection of road surface visibility from video data collected
from the SHRP2 Naturalistic Driving Study (NDS) dataset. The study is The SHRP2 NDS video data used in this study were acquired from
based on several promising deep learning techniques, such as Deep the Virginia Tech Transportation Institute (VTTI). The SHRP2 is the

2
M.N. Khan and M.M. Ahmed Accident Analysis and Prevention 142 (2020) 105521

Table 1
Criteria for Classifying Weather Conditions during Image Annotation.
Object Visibility

Clear Distant Fog Near Fog

Horizon Visible Undefinable Undefinable

Road markings Visible Visible No/only few road markings are visible
Road signs Clearly readable Readable to some extent Unreadable
Road surroundings (e.g., guardrails, delineators New Jersey barriers, etc.) and ambient Visible Visible Not clearly visible
traffic properly

most extensive study on naturalistic driving behavior until now in the latitude of the trips were imported into the ArcGIS software and sub-
U.S. Many previous studies have used the data from this comprehensive sequently, the non-matching segments were identified and eliminated
study to investigate driver behavior and performance under different from the videos. After that, twelve images were extracted per minute
weather and traffic conditions in an attempt to improve the safety of the from all the videos, which resulted in a total of more than 20,000
roadways. Participants’ vehicles were instrumented with a proprietary images in clear and foggy weather conditions. Note that for detection of
Data Acquisition System (DAS), which was developed by the VTTI so- roadway features, such as signs, signals, and work zones, higher sam-
lely for the SHRP2 program. The DAS includes forward radar; four pling rate is required; however, for weather detection, 12 frames per
video cameras, including one forward-facing color wide-angle camera; minute (e.g., one image in every 5 s) is a reasonable resolution, con-
accelerometers; vehicle network information; Geographic Positioning sidering the fact that weather will most likely be constant within 5 s.
System (GPS); onboard computer vision lane tracking, plus other For training Deep Learning models, correct annotation of images is
computer vision algorithms; and data storage capability (Hankey et al. more important than a higher sampling rate (Khan and Ahmed, 2019).
2016; Smadi et al., 2015). However, for this study, only the data of the Once the extraction of images has been completed, the images were
forward-facing color video camera were used. annotated manually and grouped into three categories: clear, distant
The data reduction started with the acquisition of the video data of fog, and near fog. It is worth noting that the classification of fog is not
interest from the massive SHRP2 NDS dataset. The SHRP2 NDS col- consistent in the literature. The National Oceanic and Atmospheric
lected more than two petabytes of data in different weather conditions Administration (NOAA) classified fog into two categories (Mueller,
(Hankey et al. 2016); therefore, identifying trips of interest was ex- 1959). They classified fog as near if the visibility distance falls below
tremely challenging. To overcome this problem, the research team de- 0.25 miles and light if the visibility distance is between 0.3 miles to 6
veloped two methods to effectively extract video data of trips that oc- miles. In 1992, the South Carolina Department of Transportation
curred in foggy weather (Ahmed et al., 2018). The first method utilized (SCDOT) developed a low visibility warning system, where they defined
the weather station data archived in the National Climate Data Center fog as dense if the visibility falls below 300 feet and light if the visibility
(NCDC). To identify the zones affected by foggy weather, a circular area ranges between 300 feet to 900 feet (Murphy et al., 2012). However, for
of five nautical miles around each station was considered (Ahmed et al., this study, fog was classified into two categories (i.e., distant fog and
2014). Subsequently, NDS trips were requested based on the daily near fog) using qualitative-based measures extracted from the NDS vi-
weather information to identify all trips impacted by foggy weather. deos. The fog was classified based on the visibility of road markings,
The daily weather information was used only to identify the potential readability of road signs, roadside surroundings (delineators, guardrail,
trips that occurred in foggy weather conditions. It is worth mentioning New Jersey barriers, and so forth) and the horizon. It is worth men-
that hourly weather information cannot be used considering the fact tioning that these objects were only used during the manual annotation
that the exact time and date of trips are considered Personally Identi- process because the human cannot perceive individual pixels and
fiable Information (PII) and hence were not provided. PII refers to any change in pixel intensity resulted from fog. However, the proposed
information that has the potential to identify a specific individual. The neural network models use global features of the images and are not
second method used weather-related crash data to identify potential dependent on any arbitrary object and hence, can be applied in ev-
locations of trips that occurred in foggy weather. By using these eryday scenarios, including congested traffic conditions. In clear
methods, video data of trips occurring in foggy and clear weather were weather, it is more likely to have a sudden change in pixel intensity,
acquired from the VTTI data enclave. whereas, in foggy weather, more uniform pixel intensity is expected.
Once the video data of adverse and clear weather were collected, all The deep neural network models can uncover these latent relationships
the videos were observed manually to eliminate NDS trips that did not between change in pixel intensity of the images and weather conditions.
occur in foggy weather. A total of 217 video recordings of naturalistic The criteria for classifying weather conditions during the manual image
trips occurred in foggy weather, and their matched 430 video record- annotation are listed in Table 1, and sample images of weather condi-
ings of trips occurred in clear weather were considered for further tions are shown in Figs. 1–3.
analysis. It is worth mentioning that the initial matching was conducted It is worth mentioning that the initial image annotation might have
by VTTI; however, to ensure more accurate matching, the longitude and some potential error due to subjectivity since all the images were

Fig. 1. Sample images of Clear Weather. a) and b) The horizon, road markings, roadside surroundings are visible; Signs are readable.

3
M.N. Khan and M.M. Ahmed Accident Analysis and Prevention 142 (2020) 105521

Fig. 2. Sample images of Distance Fog a) Road markings are visible, signs are readable, surroundings and traffic can be seen to some extent, the horizon cannot be
seen clearly. b) Road markings are visible, speed limit signs are readable, surroundings and traffic can be seen, the horizon cannot be seen clearly.

annotated by a single observer. Therefore, in order to validate the Table 2

consistency of image annotation, 20 % of the data were randomly se- Summary Statistics of Image Datasets.
lected and provided to three external observers. Subsequently, based on Weather Number of Equivalent video Duration
the weighted results from the observers, the final label of each image Images (min)
was determined and compared with the original label, which revealed
that the accuracy of the initial annotation was 99.7 %, 99 %, and 96 %, Training Dataset Clear 8000 666.67
Distant Fog 6800 566.67
for clear, distant fog, and near fog images, respectively. In order to Near Fog 1200 100
eliminate subjectivity and any potential bias in identifying weather, Total 16,000 1333.33
comprehensive training, and a detailed description of each weather Testing Dataset Clear 2000 166.67
condition with sample images were provided to the external observers. Distant Fog 1700 141.67
Near Fog 300 25.00
However, data from the initial image annotation was used to develop
Total 4000 333.33
the fog detection models since the overall consistency of the initial
annotation is more than 98 %.
Once the manual image annotation was completed, the images were on TensorFlow machine learning library was used.
grouped into two datasets: training dataset and testing dataset. The TensorFlow is a fast, flexible, and scalable open-source Machine
training dataset consisted of 8000 clear weather images, 6800 distant Learning library that can be used to implement a wide variety of ma-
fog images, and 1200 near fog images, which resulted in a total of chine learning algorithms. TensorFlow was developed for conducting
16,000 images and was equivalent to around 1,333 min of video data. machine learning and deep learning research by the researchers
In addition, the testing dataset consisted of 2000 clear weather images, working on Google’s Machine Intelligence research organization
1700 distant fog images, and 300 near fog images, which resulted in a (Google, 2018). The primary building block of any TensorFlow network
total of 4000 images and was equivalent to around 333 min of video is tensor, which is defined as a multi-dimensional array. The nodes
data. As near fog is a relatively rare environment condition, the number inside a TensorFlow network can only receive data in a tensor form.
of near fog images were respectively less compared to other categories. Therefore, all the input data need to be converted into tensor before
Table 2 summarizes the image datasets used in this study. feeding them into the network.
The basic unit of a NN is neuron, also called node, which receives
input from some other neurons, or from an external source, and com-
3. Methodology
putes an output. Each input has an associated weight (W ) , which is
assigned based on its relative importance to the other inputs. The node
In this study, neural network (NN) method was adopted to develop
applies a function (a) , called the activation function, to the weighted
the fog detection method using the annotated image dataset. NN is a
sum of its inputs. The activation function receipts a single number and
machine learning approach which has been extensively used in various
performs a specific mathematical operation on it. The activation func-
field of engineering for image classification, pattern recognition, and
tion either returns one (neuron triggered) or zero (neuron not triggered)
text categorization (Sebastiani, 2002; Abadi et al., 2016) as well as in
depending on the computations performed over the weights and biases.
many traffic safety-related research (Jacobé de Naurois et al., 2019;
The neuron shown in Fig. 4 has two numeric inputs (X 1) and (X 2) with
Wang et al., 2019). Different variations of the traditional NN were
weights (W 1) and (W 2) , respectively. Additionally, there is another
developed to serve specific purposes, e.g., RNN for time-series data
input, known as bias, with weight (B ) associated with it. Inside the
analysis and CNN for image classification. This study explored the po-
neuron, the weighted sum (S ) of all the inputs have been calculated
tential of the traditional NN, as well as the RNN, LSTM, and CNN, for
before passing it through an activation function (a) to generate the
fog detection. The NN models can be developed using various machine
output for the next node.
learning libraries under different programming platforms such as Py-
Another important parameter of a NN is cost, which captures the
thon, Java, and C++. However, for this study, Python programming

Fig. 3. Sample images of Near Fog: a) Only one road marking is visible, signs are unreadable, surroundings, and the horizon cannot be seen properly. b) Few road
markings are visible; surroundings, traffic, and the horizon cannot be seen clearly.

4
M.N. Khan and M.M. Ahmed Accident Analysis and Prevention 142 (2020) 105521

Fig. 4. A Single Neuron.

difference between the predicted and the true class. The cost, also adapts the learning rate of the parameters based on their update fre-
known as loss, is an overall measure of the performance of the trained quency during training and works well with sparse gradients (Duchi,
model and is represented by a single value. The cost is a function of 2011); and Root Mean Square Propagation (RMSProp), which adjusts
weights (W ) , biases (B ) , inputs of the training sample (IT ) and the de- the parameter learning rate based on the magnitude of the recent gra-
sired outputs of the training sample (OT ) (Karn, 2018). The general dients and works well in non-stationary setting (Tieleman and Hinton,
form of a cost function can be described using Eq. (1). 2012).
For optimization, Adam first computes the gradient (1st-order mo-
Cost , C = f (W , B, IT ,OT ) (1)
ment) and element-wise squared gradient (2nd-order moment). These
A cost function must satisfy two properties: first, it must represent moments can be described using the following equations:
the average deviation of the predicted class from the true class; second, mt = β1 mt − 1 + (1 − β1) gt (4)
it should be independent of any activation value of the NN except the
output values. Although, various cost functions including, quadratic, vt = β2 vt − 1 + (1 − β2) gt2 (5)
cross-entropy, exponential, Hellinger distance, Kullback-leibler diver-
gence, and Itakura-satio are currently being used to determine the de- Here mt and vt are the moving averages of the gradient and squared
viation of the predicted class from the true class, the most commonly gradient at time step t , respectively; gt is the gradient of the current
used cost function for a NN model is cross-entropy (Goodfellow et al., mini-batch; and β1 and β2 are the hyperparameters. It is worth men-
2016). The NN models proposed in this study also utilized the cross- tioning that the moments (i.e. mt and vt ) are usually biased towards zero
entropy as a cost function. The cost using cross-entropy can be defined during the initial time steps and can be corrected using the following
by the following equation. equations:
mt
Cost , C = − ∑ [yi ln ai + (1 − yi ) ln (1 − ai )]
i (2) ˆm = 1 − β
t t
1 (6)

Where yi is the predicted probability value for the class i and ai is the vt
v̂t =
true probability for that class (TensorFlow, 2018). 1 − β2t (7)
In a NN model, the cost is minimized at every step using an opti-
mizer, which modifies the weights and biases at each iteration and
feeds the value to the next iteration. The two most commonly used
ˆ
Here mt and v̂t bias-corrected estimators for the first and second mo-
ments, respectively, which are then used to update the weights of the
optimizers in a NN model are Gradient Descent optimizer and Adaptive model using the following equation:
Moment Estimation (Adam) optimizer. The Gradient Descent minimizes
the cost function by changing the model parameters, i.e., weight (W ) wt = wt − 1 − η ˆ
m t

ˆv + ε
t (8)
and bias (B ) , in the opposite direction of the gradient of the cost
function with respect to the parameters. The Gradient Descent can be Here wt and wt − 1 are the weights of the model at timestep t and t − 1, η
described by the following equation. is the step size, and ε is an empirical term introduced to avoid errors
due to divisions by zero (Ruder, 2016).
Pi = Pi − 1 − λ∇f (Pi − 1) (3)
Architecturally, a NN graph consists of three layers: input layer that
Where Pi represents the value of the parameters (e.g., W and B ) for the receives inputs such as images and converts them to a mathematical
next step, Pi − 1 represent the current value of the parameters, λ is a form so that the neuron can interpret; one or multiple hidden layers
weighting factor, and the gradient term ∇f (Pi − 1) is the direction of the that perform necessary numerical computations on the input data from
steepest descent (Donges, 2018). In order for the Gradient Descent to the previous layer; and finally the output layer which is responsible for
reach the optimum values of the parameter, an appropriate learning transferring information from the network to the outside world. A NN is
rate need to be selected. A large learning rate may not provide optimum termed as Deep Neural Network (DNN) if it has more than one hidden
value because it will bounce back and forth between the convex func- layer. In a NN model, the order of computation needs to be determined.
tion of the Gradient Descent. Conversely, a small learning rate will Although the value of the nodes can be calculated separately, the
significantly increase the training time. Therefore, the learning rate common practice of calculating node values is to arrange the nodes into
should be selected in such a way that the parameters achieve optimum layers. This technique is called feedforward. A feedforward network
values within the least possible time. takes the inputs in the lowest layer (input layer). Subsequently, the
While Gradient Descent maintains a constant learning rate for all higher layers (e.g., hidden layers) are calculated until the output is
the parameter updates, the Adam optimizer, which was first proposed generated at the topmost layer. Fig. 5 provides a graphical re-
by Kingma et al. (2014), computes individual adaptive learning rates presentation of a multilayer feedforward DNN consists of three inputs,
for each parameter. Adam is an extension of the stochastic gradient three hidden layers, and one output layer.
descent (SGD) algorithm and was developed mainly for training deep The DNN used in this study consists of one input layer, three hidden
neural networks. Adam optimizer leverages the advantages of two other layers with five hundred nodes in each layer, and one output layer. The
extensions of SGD: Adaptive Gradient Algorithm (AdaGrad), which training of the DNN model started with feeding the training data into

5
M.N. Khan and M.M. Ahmed Accident Analysis and Prevention 142 (2020) 105521

Fig. 5. Fully Connected Deep Neural Network (DNN).

the network. As mentioned earlier, the training dataset consisted of yt = Wyh ht − 1 + by (11)
16,000 annotated images in three different weather conditions: clear,
Here, i , f , o, and c represents the input gate, output gate, forget gate,
distant fog, and near fog. The sigmoid function was used as the acti-
and activation vectors; Wix , Wfx , Wox are the weight matrix of the input,
vation function. For calculating the cost, Cross-entropy has been used,
forget, and output gates; ht and ht − 1 are the hidden vector at time t and
which was calculated using Eq. (2). Two optimizers, including Adam
t − 1; σ is the logistic sigmoid function; and ⨀ represents the ele-
and Gradient Descent, were used to optimize the cost function
mentwise product of the vectors (Li et al., 2020).
One of the shortcomings of the DNN is that it does not consider the
Another improvement of the DNN is the Convolutional Neural
sequence of data, which is essential for time series and video data.
Network (CNN), which is developed primarily for image classification.
Therefore, to leverage the sequential information, a modification of
A CNN consists of input layers, output layers, and many layers in be-
DNN, known as Recurrent Neural Network (RNN), was also used in this
tween. The in-between layers can be categorized into two types of
study. The RNN treated data as a sequence using cyclic connections.
layers: feature detection layers and classification layers. The feature
The RNN stores information from the previous timestamp and uses the
detection layer can perform three types of operation in the data, in-
information as inputs to the network to compute the predictions at the
cluding convolution, Rectified Linear Unit (ReLU), and pooling.
current time step. At time t , if hidden nodes with recurrent connection
Convolution operation can activate certain features from the images by
collect inputs from the current data point x t , and from the hidden node
passing them through a set of convolutional filters. In this study, five
values ht − 1 of the previous step; the output yt at time t can be described
convolutional layers were used during the training process. The first
using the following equations.
convolutional layer took the images as input and applied 32 filters, each
yt = softmax (Wyh ht + by ) (4) with a height and width of 5 pixels. The second and the third con-
volutional layer applied 32 and 64 filters of the same size, respectively.
ht = σ (Whx x t + Whh ht − 1 + bh ) (5)
For the next two convolutional layers, the same number of filters with
Where σ is the activation function; ht is the hidden node value at the the same filter size was applied. After each convolution layer, a ReLU
time t ; Whx is the conventional weight matrix based on the current layer was used to perform a threshold operation on each element of the
input; Whh is the recurrent weight matrix based on the previous hidden input. ReLU layer maps negative values to zero to ensure faster and
states; Wyh is the weight matrix based on hidden state and output; and bh more accurate training. After each ReLU layer, a max-pooling layer was
and by are the bias parameters (Lipton et al., 2015). applied with pooling regions of 3 × 3 pixels. Pooling simplifies the
One of the major limitations of the RNN is that it suffers from the output by performing nonlinear down-sampling to reduce the number
vanishing and exploding gradient problem, especially when optimized of parameters that the network needs to learn.
over numerous time steps. The vanishing gradient problem occurs when After feature extraction, the architecture of a CNN moved to clas-
the gradient of the cost function moves toward zero with the addition of sification. The next layer was a Fully Connected (FC) layer that pro-
more layers to the networks, which makes the network challenging to vided a vector of three dimensions. Finally, the image dataset was
train. On the other hand, the exploding gradient problem occurs when passed into a SoftMax layer, which was the final layer of the CNN. The
the gradients with values higher than one grow exponentially through architecture of CNN used in this study is illustrated in Fig. 7.
the network layers, making learning unstable (Goodfellow et al., 2016).
Long Short-Term Memory (LSTM) can overcome these problems by 4. Results and discussions
replacing the hidden layers with a memory cell that can act as inter-
mediate storage (Lipton et al., 2015). Each memory cell consists of In order to select the best performing model, the hyperparameters of
memory blocks with three multiplicative nodes named input, output, all the models were updated by carefully observing the training pro-
and forget gates, as shown in Fig. 6. gress and the validation results using TensorBoard. TensorBoard is a
The LSTM produces an output probability vector (ht ) from input visualization toolkit inside the TensorFlow library for machine learning
sequence vectors ( x t ) by analyzing the network unit activations using experimentation (Google, 2020). Fig. 7a illustrates the variation in
the following equations. accuracy and cost over the training steps. Note that, in order to better
it = σ (Wix Xt + Wih ht − 1 + Wic ct − 1 + bi ) (6) visualize the training progress, TensorBoard smoothed the curves using
an infinite impulse response (IIR) filter, as can be seen in Fig. 8. While
ft = σ (Wfx Xt + Whf ht − 1 + Wcf ct − 1 + bf ) (7) the accuracy of the DNN model using Adam optimizer gradually in-
creased from 0.52 at the first step to 0.89 at the final step, the accuracy
ot = σ (Wox Xt + Woh ht − 1 +Woc ct + bo) (8)
of the DNN model using Gradient Descent optimizer increased from
ct = ft ⨀ct − 1 + it ⨀tanh (Wcx Xt +Wch ht − 1 + bc ) (9) 0.50 to 0.87. However, the superiority of Adam optimizer is only seen
after about 30 training steps. Considering the cost, the DNN using the
ht = ot ⨀tanh (ot ) (10) Gradient Descent optimizer provided a continuous decrease until it

6
M.N. Khan and M.M. Ahmed Accident Analysis and Prevention 142 (2020) 105521

Fig. 6. Architecture of LSTM Unit (Graves and Abdel-rahman Mohamed, 2013).

reached a minimum value. On the contrary, the cost of the DNN using higher number of training steps in order to reach the maximum accu-
Adam optimizer increased during the initial steps but eventually racy compared to the other models.
reached a minimum value at the final step. During the training process, Once training of the different NN models was completed, the per-
the epoch, also called the training step, was set to 50, considering the formance of the models was evaluated using a test dataset, which
fact that the accuracy of the model became almost constant, and no consisted of 4000 images in different weather conditions. Note that
significant improvement was observed after 50 steps. these images were not used in the training nor validation steps. The
Fig. 8b illustrates the variation of accuracy and cost of the RNN overall detection accuracy of the trained DNN, RNN, LSTM, and CNN
Models in every training step. Although the RNN model using the Adam model was found to be 85.1 %, 77.4 %, 84.2 %, and 97.3 %, respec-
optimizer provided excellent accuracy of 0.91 at the final step of the tively, using Gradient Descent optimizer, as shown in Table 3. The
training, the RNN model using Gradient Descent performed poorly than highest true positive (TP) rate using DNN was found for the clear image
the Adam optimizer with a final accuracy of 0.77. The cost was also group, where 95 % of the images were correctly classified. On the other
found to be much lower at every step of the training process for the hand, the lowest TP rate was found for the distant fog image group
RNN model trained with Adam optimizer compared to the RNN model where 73.7 % of the images were correctly classified. Considering the
trained with Gradient Descent, as can be seen from 5b. Moreover, no RNN model, the highest TP and the lowest false negative (FN) rate were
significant improvement in detection accuracy was found after 500 found for the clear image group, where 9.5 % of the clear images were
training steps. Fig. 8c illustrates the training progress of the LSTM incorrectly classified to other conditions. Similarly, the TP rate of the
model in terms of accuracy and cost. It was observed that after 500 distant fog image group was found to be 73.2 %, meaning 73.2 % of the
training steps, the increase in accuracy and the decrease in cost became distant fog images were correctly classified. As expected, the perfor-
insignificant. Similar to other NN models, the Adam optimizer pro- mance of the LSTM model using Gradient Descent optimizer was sig-
duced better results compared to the Gradient Descent optimizer with nificantly better than the RNN model with an overall detection accu-
an overall validation accuracy of around 93 % at the final steps. racy of 84.2 % compared to 77.4 % using RNN. More specifically, 84.6
Considering the CNN models, the Adam optimizer performed mar- %, 86.7 %, and 67.9 % of the clear, distance fog, and near fog images,
ginally better compared to Gradient Descent optimizer, as can be seen respectively, were correctly classified. The lowest FN rate was found for
from Fig. 8d. The accuracy of the CNN model using Adam optimizer the distance fog image group with only 13.3 % misclassification. The
was about 0.78 at the first step, which increased gradually and reached detection accuracy of the near fog image group for the LSTM model
an accuracy of 0.97 at the last step. The CNN model using the Gradient using Gradient Descent optimizer was lower compared to the CNN
Descent also produced similar results with an accuracy of 0.96 at the model; however, it was significantly higher compared to DNN and RNN
final step. The cost of the CNN model using Adam optimizer decreased model. One possible reason is that the sample size of near fog images
gradually until it reached a minimum value. A similar trend was also was relatively smaller compared to other weather conditions since near
observed for the CNN model using Gradient Descent optimizer, as can fog is a rare environmental condition. More balanced sample size with
be seen from Fig. 8c. Note that the CNN models required a significantly an equal number of images from each category might provide improved

Fig. 7. Representation of a Convolutional Neural Network (CNN).

7
M.N. Khan and M.M. Ahmed Accident Analysis and Prevention 142 (2020) 105521

Fig. 8. Training Progress of The Neural Network Models.

Table 3 near fog detection accuracy. In addition, near fog usually occurs over a
Detection Summary of the Trained Neural Network Models using Gradient small area and often creates a dramatic change in visibility. Since both
Descent Optimizer. LSTM and RNN utilizes the weights from the previous time steps, any
Model Weather TP Rate FN Rate TN Rate FP Rate Overall sudden change in time steps might make these models challenging to
(%) (%) (%) (%) Accuracy train. However, the CNN model was not significantly affected by these
problems and produced a much-improved near fog detection accuracy
DNN Clear 95.0 5.0 90.4 9.6 85.1
of more than 89 %.
Distant Fog 73.7 26.3 94.4 5.6
Near Fog 84.5 15.5 91.1 8.9 Interestingly, the CNN model provided significantly better results
RNN Clear 90.5 9.5 75.0 25.0 77.4 compared to the other models, the overall detection accuracy was found
Distant Fog 73.2 26.8 81.3 18.7 to be 98.8 %, 98 %, and 86.3 % for the clear, distant fog, and near fog
Near Fog 9.2 90.8 98.2 1.8 image groups, respectively. The lowest FN rate was found for the clear
LSTM Clear 84.6 15.4 90.8 9.2 84.2
Distant Fog 86.7 13.3 82.5 17.5
image group, where only 1.2 % of the images were misclassified, as can
Near Fog 67.9 32.1 98.1 1.9 be seen in Table 3.
CNN Clear 98.8 1.2 98.9 1.1 97.5 The performance of the neural network models using Adam opti-
Distant Fog 98.0 2.0 97.1 2.9 mizer is provided in Table 4, which shows that the models using this
Near Fog 86.3 13.8 99.7 0.3
optimizer performed better than the models using Gradient Descent
optimizer. The overall detection accuracy of the DNN, RNN, LSTM, and
CNN model was found to be 88.4 %, 89.3 %, 92.6 %, 98.1 %,

8
M.N. Khan and M.M. Ahmed Accident Analysis and Prevention 142 (2020) 105521

Table 4 dataset was found to be 98.1 %, which indicates that the proposed fog
Detection Summary of the Trained Neural Network Models using Adam detection model can detect weather in every traffic condition. In order
Optimizer. to further investigate the performance of the model under different
Model Weather TP Rate FN Rate TN Rate FP Rate Overall traffic conditions, the weather condition of 150 new images in con-
(%) (%) (%) (%) Accuracy gested traffic (50 in each adverse weather condition) was detected
using the developed CNN model. It was found that out of 150 images,
DNN Clear 94.0 6.0 93.2 6.8 88.4
the model correctly classified 147 images, indicating that the proposed
Distant Fog 86.7 13.3 90.7 9.3
Near Fog 61.1 38.9 96.2 3.8 model can perform equally well in congested traffic.
RNN Clear 95.8 4.2 94.8 5.2 91.4
Distant Fog 91.7 8.3 91.2 8.8 5. Visualization of the proposed model
Near Fog 59.7 4.3 98.6 1.4
LSTM Clear 93.5 6.5 97.9 2.1 92.6
Distant Fog 95.3 4.7 90.5 9.5
As discussed in the previous section, the best performing model for
Near Fog 71.7 28.3 98.9 1.1 detecting fog was found to be CNN with Adam optimizer, which pro-
CNN Clear 99.8 0.2 98.4 1.6 98.1 duced an impressive overall accuracy of 98.1 %. In order to visualize
Distant Fog 97.6 2.4 98.5 1.5 the practical application of the proposed model, a NDS trip pre-
Near Fog 89.1 10.9 99.7 0.3
dominantly traversed on foggy weather on US-22 and I-99 in
Pennsylvania were selected and visualized using the ArcGIS software.
respectively. The highest TP rate and the lowest FN rate of the DNN The length of the trip was around 102 miles, with a total duration of
model were found for the clear image group with only 6% mis- about 1 h and 50 min. The time-series data of the NDS trip was used to
classification. More specifically, 4% and 2% of the clear images were get the location (longitude and latitude) of the vehicle, and the video
misclassified as distant fog and near fog images, respectively. Similarly, data was used to extract the images of the trips. Twenty images were
the TP rate of the distant fog and near fog image group were found to be capture per mile, and using the developed CNN model, the weather was
86.7 % and 61.1 %, respectively. Considering the RNN model, the detected for each image. Subsequently, based on the highest vote, the
highest TP rate was found for the distant fog image group where 95.8 % weather condition was determined for that 1-mile segment. Finally,
of the clear fog images were correctly classified. The detection accuracy using a color-coded map, the weather for the total trip was visualized,
of the LSTM model using Adam optimizer was found to be 92.6 %, as can be seen in Fig. 9a. Since the proposed weather detection method
which is higher than the RNN model. The distance fog image group had was based on only a single video camera, it has a huge potential to be a
the highest overall detection accuracy, where more than 95 % of the cost-effective way of providing comprehensive weather information for
images were correctly classified. Similarly, the detection accuracy of the entire network. In addition, with the evolution of Connected Ve-
the clear and near fog image group was found to be 93.5 % and 71.7 %, hicle technologies, weather data from all the vehicles in a network can
respectively. In addition, the lowest FP rate of 1.1 % was found for the be shared with the Traffic Management Centers (TMCs) to create real-
near fog image group, meaning that 1.1 % of the other images were time weather-based VSL systems. In addition, with the advancement of
wrongly classified as near fog images. As expected, the detection ac- tablet/smartphone cameras and the powerful processors they possess,
curacy of the trained CNN models was found to be much higher com- maintenance vehicles could be easily fitted with tablets/smartphones to
pared to the other neural network models. The trained CNN model collect and process geocoded road weather images that could be easily
provided an outstanding prediction accuracy of 99.8 %, 97.6 %, and classified via an onboard application.
89.1 % for clear, distant fog, and near fog image groups, respectively, as As mentioned earlier, most of the current weather-based regulatory
can be seen in Table 4. The false positive (FP) rate of the clear image VSL system is mainly based on the data from weather stations.
group using the trained CNN model was found to be only 1.6 %, However, the cost of employing weather stations is very high, and
meaning that 1.6 % of the other images were classified as clear images. hence, their widespread implementation is not possible. Therefore, for
It is worth noting that a high FP rate of clear weather is more hazardous non-VSL corridors, the proposed method has the potential to provide
since, in such conditions, drivers will be exposed to adverse weather cost-effective advisory VSL. In addition, the method can also be used to
without any warnings. On the other hand, a high FN rate of clear disseminate cautionary messages, such as “Thick Fog 1-mile Ahead”,
weather will provide frequent adverse weather warnings in clear within the Advanced Traveler Information System (ATIS) over the DMS
roadway conditions, which might affect the compliance rate. The FN to warn the drivers about any potentially hazardous road weather
rate of the trained CNN model for the clear image group was found to conditions where no weather stations are present. Fig. 9b illustrates a
be only 0.2 %, meaning only 0.2 % of the clear images were mis- VSL corridor based on geo-specific trajectory-level weather data. To
classified as other images. maintain the homogeneity of the speed limit, the weather data from all
Overall, the prediction accuracy of the CNN models was found to be the available vehicles within a VSL corridor will be considered, and
significantly higher compared to the accuracy of the other NN models. only one representative weather impact index will be used to calculate
This finding is in line with previous studies. Previous image classifica- the speed limits at the beginning of the corridor. For any particular time
tion studies using CNN also indicated that CNN is capable of providing instance, if the first three vehicles in corridor-1 reported thick fog and
far better results than the other types of NN. For instance, Krizhevsky the last vehicle in corridor-1 reported light fog, the weather for the
et al. used a subset of the ImageNet database to classify 1000 different whole corridor-1 will be considered as thick fog and will be used to
classes. ImageNet is a massive image dataset developed for object re- calculate the speed limit at DMS-1. Since the weighted average from all
cognition research and contains over 15 million labeled images be- the available vehicle data from corridor-1 will be used, the effect of any
longing to around 22,000 classes. The results of their study showed that false detection will not be that significant. The National Center for
CNN is capable of reaching record-breaking accuracy in a highly de- Atmospheric Research (NCAR) is also using similar methodologies to
manding dataset (Krizhevsky et al., 2012). Several other studies also reduce random errors during weather forecasting (Delle Monache et al.,
adopted the CNN structure developed by Krizhevsky et al. with a slight 2011).
modification to classify the ImageNet dataset and found satisfactory The proposed fog detection method can be implemented using the
results (Simonyan and Zisserman, 2015; Sermanet et al., 2014). existing infrastructures and facilities. Currently, many apps, such as
It is worth mentioning that the test image dataset consisted of Waze, support manual reporting of weather; however, manual re-
images in all traffic conditions, including congested flow. As mentioned porting could be dangerous because drivers should pay greater atten-
earlier, the overall detection accuracy of the CNN model using this test tion to the roadway instead of interacting with a mobile app. But the
proposed model can automatically detect weather conditions from

9
M.N. Khan and M.M. Ahmed Accident Analysis and Prevention 142 (2020) 105521

Fig. 9. Visualization of the Detection Model using Geo-Specific Trajectory Level Weather Data.

video data without any human involvement. The system can be im- 6. Conclusions
plemented by utilizing the road users' smartphones for automatic de-
tection and reporting of weather conditions. However, regular vehicles The primary objective of this study was to develop a method that
might have to cancel their trips under extreme adverse weather con- can detect trajectory-level foggy weather conditions in real-time. The
ditions, which makes using regular vehicles for weather data collection study utilized the massive SHRP2 NDS video data and used a promising
not always applicable. In such extreme weather conditions, the pro- machine learning technique known as Neural Network (NN). Python
posed weather detection model could be a keystone of developing im- programming on the TensorFlow machine learning library, which is
plantable systems for Traffic Management Centers (TMCs) using developed by Google, was used for training different models, including
snowplows and maintenance vehicles as well as integration within the Deep Neural Network (DNN), Recurrent Neural Network (RNN), Long
511 apps for automated weather identification. The maintenance crews Short-Term Memory (LSTM), and Convolutional Neural Network
from most the Departments of Transportation (DOTs) usually provide (CNN). First, a database of images was created from the acquired NDS
service on interstates, high volume highways, principal arterial, and video data. Subsequently, the images were annotated manually and
urban routes up to 24 h a day with a goal of maintaining clear roadways grouped into three categories; clear, distant fog, and near fog; 80 % of
for driving safely at reasonable speeds (WYDOT, 2019). In current which were then used to train the NN models, and the remaining 20 %
practice, reporting of real-time road surface weather information is were used to test the detection accuracy on the models. Two optimizers,
mostly based on snowplow drivers. Snowplow truck drivers manually including Adam and Gradient Descent, were used during the training
select a code to describe the prevailing surface weather condition of a process. While the overall prediction accuracy of the DNN, RNN, LSTM,
road segment based on his/her experience and report the code to the and CNN using the Adam optimizer was found to be 85 %, 77 %, 84 %,
TMCs. Manual reporting, especially in adverse weather, is extremely and 97 %, respectively, the overall prediction accuracy of the DNN,
dangerous because it will create distractions. Therefore, to reduce the RNN, LSTM, and CNN considering the Gradient Descent optimizer was
risk of manually reporting weather conditions, the maintenance ve- found to be around 88 %, 91 %, 93 %, and 98 %, respectively.
hicles could be equipped with an automated weather detection system The findings from this study can be applied in Connected Variable
via smartphones/tablets using deep learning and computer vision. The Speed Limit (VSL) systems, as illustrated previously. The proposed
smartphones/tablets will automatically collect and process geocoded method has the potential to improve roadway safety, especially in ad-
road weather images that could be easily classified via an onboard verse weather conditions, including fog. The majority of the current
application to get accurate weather conditions, especially in extremely VSL systems are based on data collected from weather stations, which
harsh weather when enough regular on-road drivers are not present. may not provide real-time continuous visibility conditions. In addition,
Furthermore, the existing roadside webcams or surveillance cameras the weather stations are expensive and hence set a limit to their ex-
can also be utilized to implement the weather detection model. Al- tensive deployment. As the fog detection method developed in this
though the proposed detection models are based on the videos captured study is capable of providing trajectory-level weather information in
from moving vehicles, the model can easily be updated using images real-time with excellent prediction accuracy, in a Connected Vehicle
from existing roadside cameras. environment, it would improve the VSL systems significantly, especially
on roadways with no weather stations.
In addition, with the rapid advancement in connectivity, processing
power, and camera quality of the smartphones, the maintenance vehicle

10
M.N. Khan and M.M. Ahmed Accident Analysis and Prevention 142 (2020) 105521

can be equipped with these mobile devices capable of collecting and Acknowledgments
processing geocoded road weather images that could be easily classified
via an onboard application. The road weather images correspond to the This work was conducted under the second Strategic Highway
images that are taken from inside moving vehicles representing tra- Research Program (SHRP2), which is administrated by the
jectory-level weather conditions at road surface level from driver per- Transportation Research Board of the National Academies, and it was
spective. From the roadway safety point of view, weather conditions at sponsored by the Federal Highway Administration in cooperation with
the road surface levels are more important than the atmospheric the American Association of State Highway and Transportation Officials
weather. It is worth mentioning that the proposed fog detection system (AASHTO), and the Wyoming Department of Transportation (WYDOT).
closely resembles mobile mapping technology. Mobile mapping is the
method of compiling geospatial data from a mobile vehicle, typically References
equipped with an array of different sensors such as camera, radar, laser,
and LiDAR (Escalera and Radeva, 2004). However, in this study, we are Jacobé de Naurois, C., Bourdin, C., Stratulat, A., Diaz, E., Vercher, J.L., 2019. Detection
proposing the use of smartphone cameras from inside a vehicle as a and prediction of driver drowsiness using artificial neural network models. Accid.
Anal. Prev. 126 (November), 95–104. https://doi.org/10.1016/j.aap.2017.11.038.
source of moving weather station. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S.,
Results from this study can also be used to develop an affordable Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D.G.,
Advanced Driving Assistance System (ADAS). A simple mobile phone Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., Zheng, X., 2016.
TensorFlow: a system for large-scale machine learning. OSDI 16, 265–283.
application with proper data connectivity can effectively be fitted in Abdel-Aty, M., Ekram, A.A., Huang, H., Choi, K., 2011. A study on crashes related to
maintenance vehicles and used to detect roadway fog and visibility visibility obstruction due to fog and smoke. Accid. Anal. Prev. 43 (5), 1730–1737.
conditions in real-time, and real-time data could be disseminated to https://doi.org/10.1016/j.aap.2011.04.003.
Ahmed, M.M., Abdel-Aty, M., Lee, J., Yu, R., 2014. Real-time assessment of fog-related
Traffic Management Centers (TMCs). The Machine Vision methodology
crashes using airport weather data: a feasibility analysis. Accid. Anal. Prev. 72,
provided in this research could be extended to identify work zones, 309–317. https://doi.org/10.1016/j.aap.2014.07.004.
pedestrians, crashes, road closures, and so forth. With the evolution of Ahmed, M.M., Ghasemzadeh, A., Hammit, B., Khan, M.N., Das, A., Ali, E., Young, R.K.,
Eldeeb, H., 2018. Driver Performance and Behavior in Adverse Weather Conditions:
the Connected Vehicle technologies and 5G communications, these data
An Investigation Using the SHRP2 Naturalistic Driving Study Data-Phase 2. Final
can easily be shared with other vehicles in a Vehicle-to-Vehicle (V2V) Report WY-18/05F.
environment for the provision of appropriate warnings and related in- Alotaibi, M., Alotaibi, B., 2019. Distracted driver classification using deep learning. Signal
formation based on real-time conditions of the roadway. Therefore, the Image Video Process. https://doi.org/10.1007/s11760-019-01589-z.
Bao, J., Liu, P., Ukkusuri, S.V., 2019. A spatiotemporal deep learning approach for city-
system proposed in this study has the potential to provide affordable wide short-term crash risk prediction with multi-source data. Accid. Anal. Prev. 122
ADAS with multiple functions on a single hardware platform without (October), 239–254. https://doi.org/10.1016/j.aap.2018.10.015.
using costly sensors. Bronte, S., Bergasa, L.M., Alcantarilla, P.F., 2009. Fog detection system based on com-
puter vision techniques. In: 2009 12th International IEEE Conference on Intelligent
It is worth mensioning that, despite having groundbreaking Transportation Systems. November. https://doi.org/10.1109/ITSC.2009.5309842.
achievements in many fields, including artificial intelligence, deep Codling, P.J., 1971. Thick Fog and Its Effect on Traffic Flow and Accident. Transport and
learning has several limitations. Deep learning models may uncover the Road Research Laboratory TRRL Report LR 397.
Das, A., Ahmed, M.M., Ghasemzadeh, A., 2019. Using trajectory-level SHRP2 naturalistic
latent patterns in the image pixels, but they cannot understand what the driving data for investigating driver lane-keeping ability in fog: an association rules
patterns actually mean. Therefore, deep learning can provide excellent mining approach. Accid. Anal. Prev. 129, 250–262. https://doi.org/10.1016/j.aap.
performance for properly annotated benchmark datasets but sometimes 2019.05.024.
Delle Monache, L., Nipen, T., Liu, Y., Roux, G., Stull, R., 2011. Kalman Filter and Analog
can perform poorly for slightly different real-world data. To overcome Schemes to Postprocess Numerical Weather Predictions. pp. 3554–3570.
this problem, deep learning, especially supervised learning, requires a Donges, Niklas. Gradient Descent in a Nutshell – Towards Data Science. https://
large amount of annotated data in every possible scenario, which is towardsdatascience.com/gradient-descent-in-a-nutshell-eaf8c18212f0. (Accessed 25
July 2018).
extremely hard to get considering the fact that annotated images are
Duchi, J., 2011. Adaptive subgradient methods for online learning and stochastic opti-
not readily available and human-curation and/ or human annotation of mization. J. Mach. Learn. Res. 12, 2121–2159.
a training data set is a challenging and time-consuming task. Therefore, Escalera, S., Radeva, P., 2004. Fast greyscale road sign model matching and recognition.
in order to reduce the need for supervision, future studies will explore Recent Adv. Artif. Intell. Res. Dev., vol. 2 (1), 69–76.
FHWA. How Do Weather Events Impact Roads? - FHWA Road Weather Management.
the potential of using other variations of deep learning methods, such as https://ops.fhwa.dot.gov/weather/q1_roadimpact.htm. (Accessed 20 September
unsupervised learning, weakly supervised learning, semi-supervised 2019).
learning, and transfer learning (Goodfellow et al., 2016). Another Formosa, N., Quddus, M., Ison, S., Abdel-aty, M., Yuan, J., 2020. Predicting real-time
traffic conflicts using deep learning. Accid. Anal. Prev. 136 (July). https://doi.org/
limitation of deep neural networks is weight adjustment during 10.1016/j.aap.2019.105429.
training. The weights are adjusted in each iteration using optimizers; Goodfellow, I., Bengio, Y., Courville, A., 2016. Deep Learning. MIT Press.
however, there is no rule of thumb what optimizer will provide the best Google. TensorFlow. https://opensource.google.com/projects/tensorflow. (Accessed 26
July 2018).
possible set of weights for a particular dataset. Although this study Google. TensorBoard: TensorFlow’s Visualization Toolkit.
utilized two optimizers (i.e., Gradient descent and Adam), future re- Graves, Alex, Abdel-rahman Mohamed, G.H., 2013. Speech Recognition with Deep
search will also include other optimizers to improve the performances Recurrent Neural Networks. arXiv:1303.5778. .
Hankey, J.M., Perez, M.A., McClafferty, J.A., 2016. Description of the SHRP 2 naturalistic
of the deep neural networks by finding the best set of weights. database and the crash, near-crash, and baseline data sets. Virginia Tech
Transportation Institute.
Declaration of Competing Interest Hashemi, M., Mirrashid, A., Shirazi, A.B., 2020. Deep Learning-Based Driver Distraction
and Drowsiness Detection. arXiv preprint arXiv:2001.05137. .
Hassan, H.M., Abdel-aty, M.A., 2013. Predicting reduced visibility related crashes on
The authors declare that they have no known competing financial freeways using real-time traffic flow data. J. Safety Res. 45, 29–36. https://doi.org/
interests or personal relationships that could have appeared to influ- 10.1016/j.jsr.2012.12.004.
ence the work reported in this paper. Hautiére, N., Labayrade, R., Aubert, D., 2006a. Real-time disparity contrast combination
for onboard estimation of the visibility distance. IEEE Trans. Intell. Transp. Syst. 7
(2), 201–211. https://doi.org/10.1109/TITS.2006.874682.
CRediT authorship contribution statement Hautiére, N., Tarel, J.P., Lavenant, J., Aubert, D., 2006b. Automatic fog detection and
estimation of visibility distance through use of an onboard camera. Mach. Vis. Appl.
17 (1), 8–20. https://doi.org/10.1007/s00138-005-0011-1.
Md Nasim Khan: Conceptualization, Methodology, Software, Huang, T., Wang, S., Sharma, A., 2020. Highway crash detection and risk estimation using
Formal analysis, Validation, Writing - original draft, Writing - review & deep learning. Accid. Anal. Prev. 135 (April), 105392. https://doi.org/10.1016/j.
editing. Mohamed M. Ahmed: Conceptualization, Methodology, aap.2019.105392.
Ibrahim, M.R., Haworth, J., 2019. WeatherNet: recognising weather and visual conditions
Software, Validation, Writing - review & editing, Supervision, Project from street-level images using deep residual learning. Int. J. Appl. Earth Obs. Geoinf.
administration, Funding acquisition.

11
M.N. Khan and M.M. Ahmed Accident Analysis and Prevention 142 (2020) 105521

8 (12), 549. 1109/CVPR.2001.990493.

Jeon, H., Lee, J., Sohn, K., 2018. Artificial intelligence for traffic signal control based Sebastiani, F., 2002. Machine learning in automated text categorization. ACM Comput.
solely on video images. J. Intell. Transp. Syst. 0 (0), 1–13. https://doi.org/10.1080/ Surv. (CSUR) 34 (1), 1–47.
15472450.2017.1394192. Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., Lecun, Y., 2014. OverFeat:
Karn, U. Introduction to Neural Networks. https://ujjwalkarn.me/2016/08/09/quick- integrated recognition, localization and detection using convolutional networks.
intro-neural-networks/. (Accessed 24 July 2018). ICLR.
Khan, M.N., Ahmed, M.M., 2019. Snow detection using in-vehicle video camera with Shwartz, S., Namer, E., Schechner, Y.Y., 2006. Blind haze separation. 2006 IEEE
texture-based image features utilizing K-Nearest neighbor, support vector machine, Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2
and random forest. Transp. Res. Rec. 2673 (8), 221–232. https://doi.org/10.1177/ 1984–1991. https://doi.org/10.1109/CVPR.2006.71.
0361198119842105. Simonyan, K., Zisserman, A., 2015. Very Deep Convolutional Networks for Large-Scale
Khan, M.N., Ghasemzadeh, A., Ahmed, M.M., 2018. Investigating the impact of fog on Image Recognition. arXiv preprint arXiv:1409.1556,. .
freeway speed selection using the SHRP2 naturalistic driving study data. Transp. Res. Smadi, O., Hawkins, N., Hans, Z., Bektas, B.A., Knickerbocker, S., Nlenanya, I.,
Rec. 2672 (16), 93–104. https://doi.org/10.1177/0361198118774748. Souleyrette, R., Hallmark, S., 2015. Naturalistic Driving Study: Development of the
Kingma, D.P., Lei, J., Adam, Ba, 2014. A Method for Stochastic Optimization. Roadway Information Database. SHRP 2 Report: S2-S04A-RW-1. p. 108.
arXiv:1412.6980,. . Stone, R.G., 1936. Fog in the United States and adjacent regions. Geogr. Rev. 26 (1),
Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012. ImageNet classification with deep 111–134.
convolutional neural networks. Adv. Neural Inf. Process. Syst. 1097–1105. Tan, R.T., 2008. Visibility in bad weather. Computer Vision and Pattern Recognition
Krummenacher, G., Ong, C.S., Koller, S., Kobayashi, S., Buhmann, J.M., Member, S., CVPR 2008. pp. 1–8.
2018. Wheel defect detection with machine learning. IEEE Trans. Intell. Transp. Syst. Tang, K., Chen, S., Khattak, A.J., Pan, Y., 2019. Deep architecture for citywide travel time
19 (4), 1176–1187. https://doi.org/10.1109/TITS.2017.2720721. estimation incorporating contextual information deep architecture for citywide travel
Li, L., Qu, X., Zhang, J., Wang, Y., Ran, B., 2019. Traffic speed prediction for intelligent time estimation incorporating. J. Intell. Transp. Syst. 0 (0), 1–17. https://doi.org/10.
transportation system based on a deep feature fusion model. J. Intell. Transp. Syst. 1080/15472450.2019.1617141.
2450. https://doi.org/10.1080/15472450.2019.1583965. TensorFlow. Cross-Entropy with Logit. https://www.tensorflow.org/api_docs/python/tf/
Li, P., Abdel-Aty, M., Yuan, J., 2020. Real-time crash risk prediction on arterials based on nn/sigmoid_cross_entropy_with_logits. (Accessed 25 July 2018).
LSTM-CNN. Accid. Anal. Prev. 135. Tieleman, T., Hinton, G., 2012. Lecture 6.5-Rmsprop, Coursera: Neural Networks for
Lipton, Z.C., Berkowitz, J., Elkan, C., 2015. A Critical Review of Recurrent Neural Machine Learning. Technical Report.
Networks for Sequence Learning. pp. 1–38. https://doi.org/10.1145/2647868. Usman, T., Fu, L., Miranda-moreno, L.F., 2010. Quantifying safety benefit of winter road
2654889. maintenance: accident frequency modeling. Accid. Anal. Prev. 42 (6), 1878–1887.
Lv, Y., Duan, Y., Kang, W., Li, Z., Wang, F., 2015. Traffic flow prediction with big data: a https://doi.org/10.1016/j.aap.2010.05.008.
deep learning approach. IEEE Trans. Intell. Transp. Syst. 16 (2), 865–873. https:// Wang, J., Kong, Y., Fu, T., 2019. Expressway crash risk prediction using back propagation
doi.org/10.1109/TITS.2014.2345663. neural network: a brief investigation on safety resilience. Accid. Anal. Prev. 124
Mueller, F.H., 1959. History of Observational Instructions on Fog. Key Meteorological (December), 180–192. https://doi.org/10.1016/j.aap.2019.01.007.
Records Documentation No. 3.031. pp. 1–8. Wu, Y., Abdel-Aty, M., Lee, J., 2018. Crash risk analysis during fog conditions using real-
Murphy, R., Swick, R., Hamilton, B.A., Guevara, G., 2012. Best Practices for Road time traffic data. Accid. Anal. Prev. 114 (May), 4–11. https://doi.org/10.1016/j.aap.
Weather Management. Version 3.0. 2017.05.004.
Napoletano, P., Schettini, R., 2018. Recognition of driver distractions using deep learning. WYDOT, 2019. Wyoming Report on Traffic Crashes – 2018. Highway Safety Program,
2018 IEEE 8th International Conference on Consumer Electronics - Berlin (ICCE- Wyoming Department of Transportation, Cheyenne, WY.
Berlin) 1–6. Xu, M., Wu, J., Huang, L., Zhou, R., Wang, T., Hu, D., 2020. Network-wide traffic signal
Pavlić, M., Belzner, H., Rigoll, G., Ilić, S., 2012. Image based fog detection in vehicles. control based on the discovery of critical nodes and deep reinforcement learning. J.
IEEE Intell. Veh. Symp. 1132–1137. https://doi.org/10.1109/IVS.2012.6232256. Intell. Transp. Syst. Technol. Plan. Oper. 24 (1), 1–10. https://doi.org/10.1080/
Pomerleau, D., 1997. Visibility estimation from a moving vehicle using the RALPH vision 15472450.2018.1527694.
system. Proc. IEEE Conf. Intell. Transp. Syst. doi:10.1109/ITSC.1997.660594. Yu, R., Abdel-aty, M., Ahmed, M., 2013. Bayesian random effect models incorporating
Prabhat, M., Racah, E., Biard, J., Liu, Y., Mudigonda, M., Kashinath, K., Beckham, C., real-time weather and traffic data to investigate mountainous freeway hazardous
Maharaj, T., Kahou, S., Pal, C., O’Brien, T.A., Wehner, M.F., Kunkel, K., Collins, W.D., factors. Accid. Anal. Prev. 50, 371–376. https://doi.org/10.1016/j.aap.2012.05.011.
2017. Deep Learning for Extreme Weather Detection. Am. Geophys. Union Fall Meet. Yu, R., Xiong, Y., Abdel-aty, M., 2015. A correlated random parameter approach to in-
Ruder, S., 2016. An Overview of Gradient Descent Optimization. arXiv:1609.04747,. pp. vestigate the effects of weather conditions on crash risk for a Mountainous Freeway.
1–14. Transp. Res. C 50, 68–77. https://doi.org/10.1016/j.trc.2014.09.016.
Schechner, Y.Y., Narasimhan, S.G., Nayar, S.K., 2001. Instant dehazing of images using Zhang, D., Sullivan, T., O’Connor, N.E., Gillespie, R., Regan, F., 2015. Coastal fog de-
polarization. In: Proceedings of the IEEE Computer Society Conference on Computer tection using visual sensing. OCEANS 2015-Genova, IEEE, Vol. 1–5.
Vision and Pattern Recognition. IEEE Comput. Soc. pp. 325–332. https://doi.org/10.

12