Innovative Infrastructure Solutions (2023) 8:274

https://doi.org/10.1007/s41062-023-01232-4

TECHNICAL PAPER

A LiDAR‑camera fusion approach for automated detection

and assessment of potholes using an autonomous vehicle platform
Sk Abu Talha1 · Mohammad A. Karasneh1 · Dmitry Manasreh1 · Alfarooq Al Oide1 · Munir D. Nazzal1

Received: 24 March 2023 / Accepted: 29 August 2023 / Published online: 27 September 2023
© Springer Nature Switzerland AG 2023

Abstract
Pothole detection and repair are critically important responsibilities for transportation agencies since potholes pose a sig-
nificant safety concern to the traveling public. Substantial time and effort are needed continuously to keep an updated
inventory of the potholes. Connected and autonomous vehicles (CAVs) have the potential to be used for pothole detection
because CAVs are integrated with sensors that are able to collect data from their surroundings. The aim of this study was
to investigate the use of sensors integrated into the CAV platform to detect the location and the severity of potholes. To this
end, a comprehensive framework was proposed which included real-time pothole detection and quantification using CAV.
A research-grade Autonomous Vehicle (AV) platform integrated with a camera and light detection and ranging (LiDAR)
was used for pothole detection and quantification. As a part of the proposed framework, the you only look once (YOLO)
v7 object detection model was used for pothole detection. In addition, the data collected using a camera and LiDAR were
fused by camera-LiDAR calibration to estimate pothole size and depth. The accuracy of the camera-LiDAR calibration was
validated by measuring the dimension with the fused data of known objects. The validation results showed that the accuracy
of the fused data was within 0.8 cm (8%) compared to the true measurement. Furthermore, the ability of CAV to detect and
determine pothole dimensions and severity using the development framework was validated in a field section. The results
of this validation indicated that the CAV was able to evaluate the severity of the pothole with 100% accuracy and determine
the pothole dimension and location in real time and at highway speed. Thus, the developed framework can potentially be
integrated as part of CAV to automate the pothole detection task which will allow the transportation agencies timely repair
of potholes in the future and will improve the safety of the traveling public.

Keywords Pothole inspection · Sensor fusion · Computer vision · LiDAR · Connected and autonomous vehicles (CAVs)

Introduction Besides, if the potholes are not repaired on a timely basis,

the adjacent pavement area of the potholes also starts dete-
Potholes are one of the most ubiquitous and troublesome riorating. Recognizing the severity of the issue, the Federal
forms of pavement deterioration. Potholes are typically ini- Highway Administration (FHWA) considers pothole repair
tiated by severe interconnected alligator cracks existing on to be a critical component of roadway maintenance programs
the pavement. The majority of the potholes are formed dur- [2]. Therefore, tackling potholes is crucial, and it requires
ing the winter season. Due to severe freeze and thaw cycles a combination of repair and preventative measures such as
in the winter, the interconnected alligator cracks are eas- regular pavement maintenance and pothole patching.
ily worn off due to continuous traffic flow, resulting in pot- Each year, the state and local departments of transporta-
holes [1]. These potholes pose a major safety concern to the tion invest millions of dollars in pothole repair programs [1].
traveling public with a high potential for damaging vehicles. As such, a significant portion of money and time is invested
in detecting potholes. The purpose of pothole detection is
twofold: (1) detecting the location of the pothole and (2)
* Munir D. Nazzal
nazzalmd@ucmail.uc.edu assessing the severity of the pothole from its dimensions.
Transportation agencies also require accurate pothole
1
Department of Civil and Architectural Engineering dimensions in order to estimate repair materials accurately.
and Construction Management, University of Cincinnati, Currently, most transportation agencies rely on manual
Cincinnati, OH 45221, USA

13
Vol.:(0123456789)
274 Page 2 of 14 Innovative Infrastructure Solutions (2023) 8:274

data collection methods through visual inspection. In some framework that possesses significant potential to fulfill all
cases, the severity and the geolocation of the potholes are the requirements is the utilization of connected and autono-
assessed from the digital image collected by a digital inspec- mous vehicles (CAVs). While full automation still remains
tion vehicle [3, 4] Although these visual inspection meth- to be a distant goal, more and more production-level con-
ods are useful, they require significant time and effort from sumer vehicles are being equipped with loads of various
transportation agency workers and can be subjective and sensors and computational power to facilitate Advanced
inaccurate. Due to the subjective nature of the work, many Driver Assistance Systems (ADAS) in their vehicles. It is no
potholes may remain undetected which might be unsafe for secret that the amount of sensory and computational power
the traveling public and also cause accelerated deterioration used to support the driver is only going to increase before
of the adjacent area of the pavement. In addition to public finally reaching the full potential of CAV technology. Typi-
safety, the agencies also end up over or underestimating the cally, these vehicles are equipped with a global positioning
required patching materials due to inaccurate volume estima- system (GPS), an inertial measurement unit (IMU), up to
tion. Occasionally, transportation agencies use commercial several cameras, and recently, light detection and ranging
digital inspection and data collection systems to identify and (LiDAR) sensors. Unlike the geodetic grade sensors, the sen-
detect potholes. Although these systems can provide highly sors installed on the CAVs are relatively less expensive [5].
accurate data, they are not regularly used for pothole inspec- Developing a pothole detection and evaluation frame-
tion for two reasons. Firstly, although the raw data obtained work that leverages the sensory and computational resources
from these systems is accurate, it typically requires process- available in CAVs has several advantages. At a minimum,
ing and doesn't provide a report of the pothole information, the developed approach can be implemented on transporta-
which means that experienced personnel must thoroughly tion agency vehicles to enable automated inspection using
investigate the huge amounts of raw data. Secondly, the available resources. Alternatively, in a more optimistic
cost of operating these commercial data collection systems scenario, the developed approach can be implemented on
is usually prohibitively expensive to execute on a regular data obtained from road user vehicles which would allow
basis [5–7]. Therefore, the use of such data collection sys- for a much higher rate of data collection and update of
tems is limited to the holistic pavement evaluation programs pothole distresses. Of course, utilizing road user data has
that are typically employed annually or bi-annually [5, 6]. other challenges such as logistical challenges and data reli-
Nonetheless, the formation of potholes, especially in winter ability. Nonetheless, these challenges are outside the scope
seasons, is quite frequent and therefore requires a more fre- of this study. This study focuses on utilizing the sensors
quent inspection from agencies in order to generate accurate (GPS, Camera, and LiDAR) widely used on CAV platforms
assessments. to perform pothole detection and evaluation. The developed
An optimal approach for regular pothole inspection would framework presents a Camera-LiDAR fusion approach. The
possess three key properties [8, 9]. Firstly, the system must camera data is mainly used to detect the pothole using the
be accurate. The accuracy of the system is defined by (1) state-of-the-art object detection algorithm YOLO v7 [10],
its ability to detect potholes and not raise false alarms, (2) while the LiDAR data is used to obtain accurate measure-
the accuracy of the reported pothole dimensions which are ments of the potholes including the dimensions, area, and
important to determine its severity, maintenance priority, depth, followed by a clustering algorithm to accurately
and the expected amount of effort and materials needed to determine the volume. Finally, the GPS system is utilized for
mitigate it, and (3) the precision of the reported location accurate georeferencing. The approach is implemented in a
of the pothole (accurate georeferencing). Secondly, the sys- robot operating system (ROS) environment to be compatible
tem must be relatively inexpensive to purchase and operate. with widely recognized CAV development tools.
This can be achieved by exploiting inexpensive sensors and
reducing the required operation and data processing time to
be performed by experienced personnel. Finally, an optimal Background
approach would be able to utilize road users’ vehicle data to
obtain frequent observations. While the last property could Many studies have evaluated the use of sensors used in
be considered optional, proper application of this property CAVs such as LiDAR and cameras for pavement condition
would not only allow for regular and frequent evaluation but assessment [4, 11, 12]. These studies can be classified into
would also increase the reliability of the reported results. two main categories: camera-based computer vision and
To achieve all three components of the optimal approach, LiDAR point cloud based. In the camera-based approach,
it is imperative to select a framework that can satisfy the Koch and Brilakis [3] introduced a method to detect pot-
second and third requirements. Subsequently, the frame- holes from the images captured by a low-cost high-speed
work needs to be complemented with a sound algorithm camera. The proposed algorithm includes segmenting the
in order to achieve efficiency and accuracy. A promising images into defect and non-defect regions. Then the shape

13
Innovative Infrastructure Solutions (2023) 8:274 Page 3 of 14 274

of the pothole was approximated from the geometric char- the overall cost of the system would be significantly high
acteristic of the pothole. Finally, the pothole was identified since the system used two very high-grade LiDARs (Riegl
based on the comparison between the texture of the pothole VUX-1HA and ZþF Profiler 9012).
and the surrounding area. Baek and Chung [13] used the You Both camera-based and LiDAR point cloud-based stud-
Only Look Once (YOLO) object detection algorithm where ies have their own advantages and limitations. Vision-based
the potholes are classified based on the extracted feature approaches are mostly used for detection purposes. But it
characteristics of the edge of the pothole. The average accu- fails to provide necessary information such as the depth of
racy of the detection method was found to be 83%. Zhang the pothole. Point cloud-based approaches can overcome
et al. [14] proposed a stereo vision-based pothole detection such limitations. However, post-processing requires a huge
system from the disparity map generated from the dispar- amount of storage since this approach needs to deal with
ity calculation algorithm. The authors reported the sever- millions of point clouds representing the spatial coordi-
ity of the potholes from the size, volume, and position of nates [20]. One of the most efficient ways of eliminating
the pothole. However, this algorithm was not flawless since the limitations of both approaches is sensor fusion. Using
some false pothole detection was observed due to dispar- the sensor fusion technique, the object can be detected by
ity calculation errors. Besides, the 3D imaging technique a camera and the geometry of the detected object can be
includes a high computational cost. Ouma and Hahn [15] obtained from the spatial coordinates of the point cloud.
introduced an unsupervised clustering technique to detect This technique is widely adopted in the automotive industry
a pothole from the color 2D images. The authors reported for object detection purposes, such as pedestrians, vehicles,
very high accuracy in detecting potholes. However, like any lane marking, signs, etc. However, the data collected by the
other 2D vision-based approach, this method cannot provide sensors installed in CAVs are mostly used for the navigation
any information on the depth of the potholes. of the vehicle [21]. Using a proper sensor fusion technique,
The application of LiDAR in pavement evaluation is the extracted data from the sensors of a CAV can be utilized
increasing rapidly. Previously, LiDAR has been widely used for pavement pothole detection with the corresponding depth
in various applications, including rock fall hazard detection, and area. Successful implementation of sensor fusion will
pavement marking, lane detection, crack detection, and road- enable a real-time pothole detection system which will help
side safety detection [15–18]. Many studies have success- the transportation agency to maintain an updated road condi-
fully used LiDAR in detecting potholes as well. Kang and tion database at a very low cost.
Choi [19] utilized two 2D LiDARs to obtain the pavement
surface information. Their proposed pothole detection algo-
rithm includes four steps: filtering, clustering, line extrac-
tion, and gradient determination of the extracted line. The Objective and scope
accuracy of the proposed algorithm was determined by com-
paring the dimensions obtained from the algorithm with the This paper evaluates the application of sensors installed on
manually measured dimensions of a known object. The esti- a CAV platform to detect the location, size, depth, and vol-
mated depth and the width of the object obtained from the ume of potholes. To this end, a framework was developed
point cloud based on their proposed algorithm were within that included real-time pothole detection and quantification
5 percent of the actual depth and width of the pothole. How- using the data collected by camera and LiDAR at highway
ever, for more accurate and faster detection of potholes, they speed. The pothole detection was done using YOLO v7.
suggested using an image-based detection technique in com- Both extrinsic and intrinsic calibration of the camera and
bination with point cloud. Besides, the algorithm requires LiDAR were performed in order to fuse the data collected
the pothole detection system to move at a maximum speed from LiDAR and camera. The pixel coordinates provided
of 2.7 km/h which may be impractical to implement in the by the YOLO bounding box were utilized to filter out the
actual field condition. Ravi et al. [4] proposed a methodol- point cloud outside the bounding box. K-means cluster-
ogy to classify the 3D point cloud of the pavement surface ing was performed on the filtered point cloud in order to
obtained by a mobile mapping system that includes two separate the point cloud inside the pothole from the point
high-grade LiDARs, two cameras, an IMU, and a GNSS/ cloud that belongs to the pavement surface. Also, the depth
INS navigation system. In this study, the k-nearest neighbor was obtained by subtracting the elevation of the point cloud
clustering algorithm was applied to the point cloud to define inside the pothole from the average elevation of the point
the roadway surface. A plane was fitted to the point cloud cloud on the pavement surface. Finally, the volume was cal-
to estimate the parameter of the roadway surface. Finally, culated from the clustered point cloud using the minimum
a height threshold was applied to distinguish between the convex hull algorithm. Everything including the data acqui-
roadway surface and the potholes. The study reported very sition and processing was implemented in ROS, which are
high accuracy in detecting the pothole dimension. However, integrated with the CAV platform.

13
274 Page 4 of 14 Innovative Infrastructure Solutions (2023) 8:274

Description of the AV system this study was a Velodyne VLP32 and Allied vision Alvium
1800 U-158 camera was used to collect the image data. The
The autonomous vehicle research development program data acquisition, storage, fusion, and post-processing were
used in this study is a platform built on a Lexus RX450h conducted using ROS noetic, a framework typically used in
by AutonmousStuff company. The platform is designed to a CAV platform.
facilitate research for autonomous and connected driving.
Figure 1 shows the schematic diagram of the components
used in the platform. Figure 2 shows the sensors and the
setup that was used to collect the data. The LiDAR used in

Fig. 1  Schematic diagram of the system integrated with the autonomous vehicle

13
Innovative Infrastructure Solutions (2023) 8:274 Page 5 of 14 274

Fig. 2  Sensor setup and a snapshot of the autonomous system

Proposed framework The schematic diagram of the real-time implementation

of pothole detection and analyses is shown in Fig. 4. As
In general, the proposed framework to obtain the dimen- shown in Fig. 4, the coordinate of the pothole location was
sion based on the sensor fusion can be divided into four obtained from the timestamp synchronization between the
main stages, which include: (1) obtaining the extrinsic and camera and the GPS sensor installed on the AV platform. At
intrinsic matrices through the camera and LiDAR calibra- the end of the process, a small file was generated containing
tion; (2) detecting pothole using YOLO object detection information related to georeferenced potholes which can be
model, (3) use the information from the bounding box uploaded to a cloud database and maintained by the agen-
obtained from YOLO to filter out the point cloud, and (4) cies via the onsite internet or saved on the computer. The
analyze the point cloud to obtain the geometric informa- pothole information would help the agencies to fix the pot-
tion of the pothole. Figure 3 shows an overview of the holes in a timely manner. In addition to providing location
approaches that were implemented in this study to obtain data, pothole volume is also calculated in order to provide
the dimensions of the pothole. As shown in Fig. 3, the first severity information and required repair material quantities.
step was to calibrate the two sensors, including LiDAR and Details about framework steps are described in the following
camera to obtain the extrinsic and the intrinsic matrices. subsections.
After that, the extrinsic and intrinsic matrices were used
to fuse the LiDAR point cloud with the image pixels. The
accuracy of the fused point cloud data was validated by Sensor calibration and fusion
comparing the size information of known objects extracted
from the point cloud. In the second stage of the frame- Sensor calibration is a fundamental task required to fuse
work, YOLO v7 object detection model was implemented the data collected from multiple sensors. The calibration
to detect potholes in real time. The pixel coordinates pro- is required to provide the system with information about
vided by the YOLO bounding box were utilized to filter the relative positions and orientations of the sensors in
out the point cloud outside the bounding box. Finally, a real-world coordinate system by comparing the rela-
the filtered point cloud was further clustered using the tive position of a known feature [22]. This calibration is
K-means clustering algorithm in order to estimate the called extrinsic calibration. A successful extrinsic calibra-
depth and the volume of the pothole. It is important to note tion provides a rotation and transition matrix that converts
that both LiDAR and the camera can produce hundreds of the LiDAR frame into the camera frame (Fig. 5a). In this
gigabytes of data from a single evaluation trip given the study, the extrinsic calibration between the LiDAR and the
vast road network system that the agencies may need to camera was performed using the method provided by Belt-
cover. The agencies will require huge storage to maintain rán et al. [23]. This method is available as a ROS package.
and store the collected data from these sensors. Therefore, The package was successfully utilized to obtain the rota-
the analysis of this huge amount of data should be done in tional and translational matrices shown in Eq. (1). Another
real time to eliminate the need for storage. The proposed important parameter required for the fusion is the projec-
approach can be implemented to process the sensors data tion matrix of the camera which can be obtained from the
to provide pothole information in real time without need- intrinsic calibration of the camera. The projection matrix
ing to store the sensor data.

13
274 Page 6 of 14 Innovative Infrastructure Solutions (2023) 8:274

Fig. 3  Flow chart of the proposed framework

Fig. 4  Real-time implementation of pothole detection and analysis

was obtained by intrinsic calibration of the camera using

[ ]
R t
a software package integrated with ROS. Figure 5b shows T=
0 1 (1)
the fused data obtained from the extrinsic calibration of
the LiDAR and the camera. The calibration packages also where R = 3 × 3 rotational matrix and.
provided the rotational and translational matrices that were t = 3 × 2 translational matrix.
used to merge the point cloud with the image pixels.

13
Innovative Infrastructure Solutions (2023) 8:274 Page 7 of 14 274

Fig. 5  Camera-LiDAR calibra-

tion, a LiDAR and camera
coordinate, b projected LiDAR
point cloud on image after
calibration

Successful projection of the point clouds on the image is This can be achieved by dividing Xi and Yi with the corre-
one of the most important steps of the proposed framework. sponding Zi value. Once the point cloud is projected on the
This step was implemented using Eq. (2) [24], where the image, the borderline pixel coordinates information of the
point cloud was multiplied with the extrinsic and intrinsic image is utilized to filter out the point cloud out the FOV of
calibration matrix obtained from the calibration step men- the image. Figure 6 shows an example of the projected point
tioned earlier. cloud on the image pixels.
Y2D = PTX3D (2)
where Y = the pixel coordinates in 2D (n × 2 matrix).
P = camera projection matrix.
Validation of the fusion technique
T = Transformation matrix.
The successful fusion of point cloud data and the image
X = 3D point cloud.
pixels is the key component of the overall accuracy of the
As described in Eq. (2), the 2D pixel information of the
framework. The accuracy of the fused data collected from
image is directly related to the 3D LiDAR point cloud the
the LiDAR and camera was evaluated by validating the
projections matrix of the camera (P), and the transformation
dimensions of known objects obtained from the point cloud.
matrix (T) of the camera coordinate with respect to LiDAR
To this end, three objects with known dimensions were
coordinate. Using the transformation matrix obtained from
placed next to each other within the field of view (FOV)
the extrinsic calibration the LiDAR coordinate was trans-
of the camera. Both camera images and LiDAR scans were
formed into the camera coordinate. Subsequently, the 3D
taken with the calibrated setup. Figure 7a shows the fused
point cloud was projected on the image by using the projec-
image with the known objects. As indicated by Fig. 7a, point
tion matrix obtained from the intrinsic calibration of the
clouds were matched with the corresponding pixels of the
camera. The 3D to 2D conversion was obtained by normaliz-
image, which gives the visual validation of the fusion. The
ing the X and Y coordinate with respect to the Z coordinate.
LiDAR point clouds of the known objects were extracted
using the coordinates of the pixels surrounding the known
objects (Fig. 7b). Finally, the dimensions obtained from the
point cloud were compared with the manual measurement.
Table 1 shows the comparison between the manual measure-
ment and the measurement obtained from the point cloud. It
is noted that the deviation of the dimensions obtained from
the point cloud is almost negligible. In fact, for all cases,
the deviations were well within 1 cm. Therefore, the results
suggest that the fusion of the image and LiDAR point cloud
data can be used to detect and measure the pothole dimen-
sion accurately as long as the pothole is within the FOV of
the image. Figure 7c and d shows the scatter plot of the point
Fig. 6  Projected point cloud on the image
cloud of Object 1 extracted from the fused data.

13
274 Page 8 of 14 Innovative Infrastructure Solutions (2023) 8:274

Fig. 7  Validation of sensor

fusion, a a snapshot of the fused
data with the validation objects,
b filtered point cloud using a
bounding box, c extracted point
cloud of object 1, and d scatter-
plot of Object 1 on xz-plane

Pothole detection algorithm on an NVIDIA GTX 3070ti GPU workstation with a total
of 1000 epochs.
Over the past decade, numerous deep learning-based object The performance of the models was measured by
detection models have been introduced. Among them, mAP@0.5, recall, and precision, where mAP@0.5 repre-
YOLO has been widely popular due to its outstanding per- sents the average mean precision when positive detection
formance over other models in terms of speed. Since this occurs if the intersection over union (IoU) is larger or equals
study was focused on real-time pothole detection, YOLO 0.5. In other words, the accuracy of the proposed model
was chosen for pothole detection. YOLO is an object detec- was determined by measuring the percentage of overlap
tion model which utilizes an end-to-end convolutional neural between the predicted bounding box and the ground truth.
network to detect the bounding box along with the class of Precision is defined by the of the ratio number of potholes
the object. Several studies have already used different ver- correctly detected to the total number of detected potholes.
sions of YOLO for pothole detection [17, 25]. For this study, Finally, recall is the ratio of the number of potholes correctly
YOLO v7, the most recent version of YOLO (released 8th detected to the actual total number of targeted potholes. In
of July 2022), was implemented for pothole detection. It is the following steps, the detected bounding box from the
known to be one of the fastest object detection models of YOLO object detection was used to eliminate the point cloud
its kind [23]. that was not inside the bounding box.
The YOLO v7 model was trained, validated, and tested
using the same dataset. A total of 521 pothole images were
collected, annotated, and analyzed in this study. Pothole Estimation of pothole depth and volume
images were collected from all over Ohio at a highway
speed using the experimental setup shown in Fig. 2. The As mentioned previously, the detected bounding box from
data were split into 75 percent training, 15 percent valida- the YOLO object detection was used to filter out the point
tion, and 10 percent testing. It is worth noting that the data cloud outside the bounding box. Finally, the filtered point
set was manually annotated using an online annotation tool clouds were clustered based on the z-direction (elevation)
named makesense.ai. The training of the models was done using the K-means clustering algorithm to separate the point
clouds inside the pothole from the point cloud which belongs

Table 1  Validation results

Objects Measured length Error (cm) %Error Measured width (cm) Error (cm) %Error Measured height Error (cm) %Error
(cm) (cm)
Manual Point cloud Manual Point cloud Manual Point cloud

Object 1 30.00 30.18 0.18 0.60 24.80 25.60 0.80 3.23 10.00 10.80 0.80 8.00
Object 2 25.20 24.59 0.61 2.42 20.00 19.75 0.25 1.25 5.88 6.12 0.24 4.08
Object 3 65.50 65.37 0.13 0.20 37.50 37.30 0.20 0.53 14.85 15.02 0.17 1.14

13
Innovative Infrastructure Solutions (2023) 8:274 Page 9 of 14 274

Fig. 8  Estimation of depth and volume from point cloud

to the pavement surface. Subsequent clustering was also per- measured manually for comparing the manual measure-
formed along X and Y directions to filter out the point clouds ment with the measurement obtained from the point cloud.
that do not belong to the vicinity of the pothole area. The The pothole dimensions were measured using a level and
depth of the pothole was obtained by subtracting the eleva- the depth was measured using a caliper. The depth of the
tion ( Zi ) of the point cloud inside the pothole from the aver- pothole was measured based on the guideline provided
age elevation ( Zsurf ) of the point clouds that belong to the by the federal highway administration (FHWA) [27]. The
pavement surface. Finally, the maximum difference between maximum depth of the pothole was compared with the
Zsurf and Zi is considered to be the depth of the pothole. Fur- maximum depth obtained from the proposed framework.
thermore, the area and the volume of the pothole were esti- In this case, the manual measurement was assumed to be
mated using the minimum convex hull approach proposed the ground truth. Figure 9b shows the snapshots taken dur-
by Sampath and Shan [26]. All the analysis to implement the ing the data collection process in the experiment section.
proposed framework to obtain the geometry of the pothole
was performed in Python. Figure 8 illustrates the stepwise
depth and volume calculation from the filtered point cloud Results and discussion
obtained using the YOLO bounding box.
Accuracy pothole detection algorithm

Implementation of the proposed framework The performance of the model during the training after each
epoch is shown in Fig. 10. As shown in Fig. 10, the model
To verify the performance of the proposed framework to performed exceptionally well as all the performance indi-
detect and estimate the pothole dimension using the sys- ces improved with each epoch. The model obtained scores
tem, a 1-mile-long test section was selected in Cincin- above 0.9 for recall, precision, and mAP@0.5. Figure 10
nati, Ohio (Fig. 9a). The section included 16 potholes of also displays the average mAP at various IoU thresholds
different shapes and depths. It is noted that such a sec- ranging from 0.5 to 0.95 in 0.05 increments. The growing
tion was selected in order to make sure that the proposed mAP@0.5:0.95 also suggests that the model is getting better
methodology is able to detect and estimate the geometric after each epoch in detecting and classifying the pothole.
information of potholes of various shapes and depths. The All these results demonstrate that YOLO v7 can be used to
data were collected in the experimental section using three detect potholes effectively and accurately.
different speeds, including 30, 40, and 50 mph to check the YOLO v7 was tested with images captured at different
efficiency of the proposed framework at different speeds. speeds (30, 40, and 50 mph) that had not been used in the
Furthermore, the dimensions of the potholes were also training or validation process of the model. Figure 11 shows

13
274 Page 10 of 14 Innovative Infrastructure Solutions (2023) 8:274

Fig. 9  Data collection, a Location of the experimental study, b manual pothole data collection

a few examples of images where the pothole was detected

successfully by YOLO v7. It is important to note that YOLO
v7 provided a detection speed of 3.8 ms on a batch size of
32 images, which is the fastest known detection speed ever
when it comes to pothole detection. Besides, the speed of
the vehicle did not affect the accuracy of the detection of
the model. Such detection speed and accuracy was a key
element of this study as this study was focused on real-time
pothole detection. Therefore, the proposed framework could
potentially be used for pothole detection even when a car is
moving at a highway speed.

Accuracy of pothole depth predictions

The FHWA Pavement Distress Identification Manual

[27] classifies the severity of the pothole based on depth.
Therefore, estimation of depth is very important for high-
way agencies to prioritize the pothole repair program. Fig- Fig. 10  Performance metrics progress against the training epoch
ure 12 shows the percentage of error obtained from the of the YOLOv7 road lane marking detection model. a Precision, b
manual depth measurement and the measured depth from Recall, c mAP@0.5 and d mAP @0.5:0.95
the point cloud. It is important to note that the maximum
depth obtained from the proposed framework was compared # 6 (approximately 29 percent), whereas the minimum error
to the maximum depth obtained manually. As indicated by was obtained for pothole #5 (approximately 1.5 percent). It
Fig. 12, the errors obtained for the depth estimated from is worth mentioning that although the area obtained from
the point cloud compared to the manual depth measurement the point cloud was compared against the manual measure-
were well below 20 percent, where the maximum error was ments, the manual measurements by no means represent the
observed for pothole #15 (approximately 17 percent) and true dimension of the potholes. The manual area estimation
the minimum error was observed for pothole #3 and #10 was simply obtained by multiplying the maximum width
(approximately 1 percent). Figure 13 shows the error rate and the length of the pothole assuming that the pothole is a
obtained from the point cloud compared to the manual area perfect rectangle.
measurement. The maximum error was observed for pothole

13
Innovative Infrastructure Solutions (2023) 8:274 Page 11 of 14 274

Fig. 11  Testing results showing successful pothole detection by YOLO v7

Accuracy of pothole severity predictions

Table 2 shows the severity classification of the pothole esti-

mated from both manual calculation and automated calcu-
lation. It is important to note that the severity classifica-
tion was compared based on the guideline provided by the
FHWA [27]. According to FHWA, the severity of a pothole
is classified as low if the depth is below 25 mm, medium if
the depth is between 25 and 50 mm, and high if the depth is
above 50 mm. As shown in Table 2, there is a 100% match
in severity classification based on the manual measurement
and the automated measurement from the point cloud. The
results suggest that the framework can be effectively used
Fig. 12  Error percentage in depth for each pothole by transportation agencies to estimate the severity of the
pothole. This can help the agencies in the decision-making
process by prioritizing the activity for the pothole repair-
ment program.

Pothole volume estimation

The geometric information, including the depth, area, vol-

ume, and location of the 16 potholes based on the manual
and automated calculations are summarized in Table 3. In
addition, the geolocation is also shown in Fig. 14 for visual
observation. As mentioned earlier, the area and the volume
of the pothole were calculated based on the minimum con-
Fig. 13  Error percentage in area for each pothole vex hull algorithm. The main objective of volume estimation

13
274 Page 12 of 14 Innovative Infrastructure Solutions (2023) 8:274

Table 2  Severity classification

Pothole # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Severity based on
manual Medium High Low Medium Medium High Medium Medium Medium Medium Medium High Medium High Medium High
measurement
Severity based on
automated depth Medium High Low Medium Medium High Medium Medium Medium Medium Medium High Medium High Medium High
calculation

Table 3  Depth and volume information of the potholes

Pothole# Depth (manual Depth (Developed Area (manual Area (Developed Volume (Developed Geolocation
measurement) AV Framework) measurement) AV Framework) AV Framework)
(cm) (cm) ­(cm2) ­(cm2) ­(cm3)

S1 3.2 3.603 4529 3333 12,010 39.135941, − 84.518617

S2 9.1 10.061 3958 2948 29,660 39.136127, − 84.518503
S3 2.47 2.479 266 335 830 39.136262, − 84.518416
S4 3.7 3.605 859 662 1990 39.136352, − 84.518388
S5 4.1 4.565 1394 1371 6260 39.137160, − 84.517601
S6 6.4 7.04 3110 2206 12,730 39.137248, − 84.517457
S7 3.8 4.381 1290 1072 3820 39.137390, − 84.517257
S8 3.9 4.411 1548 1120 4060 39.138152, − 84.516120
S9 4.3 4.855 1452 1308 5090 39.138386, − 84.515824
S10 4.7 4.731 3200 2720 12,870 39.138467, − 84.515733
S11 3.8 3.735 1206 900 3360 39.138474, − 84.515708
S12 7 8.259 4087 2994 16,470 39.137115, − 84.514949
S13 3.2 3.479 2032 1535 5340 39.137061, − 84.514930
S14 6.4 7.115 2477 1819 5830 39.136982, − 84.514945
S15 3.6 4.339 1149 942 3220 39.136933, − 84.514942
S16 6.6 7.67 19,742 16,979 84,130 39.136873, − 84.514959

was to provide the transportation agencies with a tool to Finally, the results obtained from the point cloud analysis
estimate the quantity of the patching materials required to fill were validated against the data collected manually from
the pothole. This may save a significant amount of money by the experimental section.
preventing the transportation agencies from over or underes- The YOLO v7 object detection model successfully
timating the amount of required pothole repairing materials. detected potholes with very high accuracy. The overall
detection speed was observed to be 3.8 ms for a batch of
32 images which is one of the fastest compared to any
Summary and conclusion detection speed reported in the literature. The accuracy
of the fused data from the validation was observed to be
This paper proposed an automated framework using a within ± 0.8 cm (8%), which clearly indicates that sensor
CAV platform for real-time detection and quantification of fusion can be used to accurately quantify the geometry of
pothole geometry at varying speeds. Two sensors, includ- a pothole. The error calculated by comparing the maxi-
ing a camera and LiDAR, were utilized for detection and mum depth obtained from the point cloud with the maxi-
quantification purposes. Potholes were detected using the mum depth from manual measurement was well below 20
most recent version of YOLO object detection model. To percent. In addition, the severity of the pothole damage
obtain geometrical information about the pothole the data obtained from the manual calculation and point cloud was
from two sensors were fused using extrinsic and intrinsic compared. The results indicated that the proposed methods
calibration. The bounding box obtained from the YOLO can be successfully used to obtain the severity of the pot-
model was used to filter out the point cloud that did not hole damage. Besides, the framework was able to provide
belong to the pothole area. K-means cluster analysis was a volumetric estimation of the pothole that can be used to
conducted on the filtered point cloud. The clustered point estimate the patching materials for the pothole repairment
cloud was used to obtain the depth and volume of potholes. program.

13
Innovative Infrastructure Solutions (2023) 8:274 Page 13 of 14 274

Fig. 14  The pothole locations obtained from the time synchronization from the proposed framework

The outcomes of this study suggest that sensors integrated 2. Federal Highway Administration. LTPP – Beyond FY 2009: What
into AV platforms can be applied to accurately detect and needs to be done? Technical Report, Publication No. FHWA-
HRT-09-052, website: http://​www.​fhwa.​dot.​gov/​pavem​ent/​ltpp/​
assess potholes at varying speeds using the proposed frame- pubs/​09052/​09052.​pdf. (Nov. 10, 2011).
work. Therefore, a CAV platform integrated with the pro- 3. Koch C, Brilakis I (2011) Pothole detection in asphalt pavement
posed framework can potentially be used by transportation images. Adv Eng Inform 25(3):507–515
agencies for both pothole detection and severity quantifica- 4. Ravi R, Habib A, Bullock D (2020) Pothole mapping and patching
quantity estimates using LiDAR-based mobile mapping systems.
tion in real time which may save a significant amount of time Transp Res Rec 2674(9):124–134
and effort. Furthermore, the framework that was developed 5. Manasreh D, Nazzal MD, Talha SA, Khanapuri E, Sharma R,
can be integrated with the existing systems in automobiles Kim D (2022) Application of autonomous vehicles for automated
equipped with ADAS technology featuring sensors utilized roadside safety assessment. Transportation Research Record,
03611981221095090.
for this research study. 6. Coenen TB, Golroo A (2017) A review on automated pavement
distress detection methods. Cogent Eng 4(1):1374822.
Declaration 7. Bansal K, Mittal K, Ahuja G, Singh A, Gill SS (2020) DeepBus:
Machine learning based real time pothole detection system for
Conflict of interest On behalf of all authors, the corresponding author smart transportation using IoT. Internet Technol Lett 3(3):e156
states that there is no conflict of interest. 8. Sathya R, Saleena B (2022) A framework for designing unsu-
pervised pothole detection by integrating feature extraction
Ethical approval This article does not contain any studies involving using deep recurrent neural networkk. Wireless Pers Commun
human participants or animals performed by any of other authors. 126(2):1241–1271
9. Pandey AK, Iqbal R, Maniak T, Karyotis C, Akuma S, Palade
Informed consent For this study formal consent is not required. V (2022) Convolution neural networks for pothole detection of
critical road infrastructure. Comput Electr Eng 99:107725
10. Wang CY, Bochkovskiy A, Liao HYM (2023) YOLOv7: Train-
able bag-of-freebies sets new state-of-the-art for real-time
object detectors. In: Proceedings of the IEEE/CVF Conference
References on Computer Vision and Pattern Recognition. pp 7464–7475.
11. Litman T (2020) Autonomous vehicle implementation predic-
1. Dong Q, Huang B, Zhao S (2014) Field and laboratory evaluation tions: implications for transport planning. Victoria Transport
of winter season pavement pothole patching materials. Int J Pave- Policy Institute, Victoria, Canada
ment Eng 15(4):279–289

13
274 Page 14 of 14 Innovative Infrastructure Solutions (2023) 8:274

12. Rojas-Rueda D, Nieuwenhuijsen MJ, Khreis H, Frumkin H 21. Cao W, Liu Q, He Z (2020) Review of pavement defect detection
(2020) Autonomous vehicles and public health. Annu Rev Pub- methods. IEEE Access 8:14531–14544
lic Health 41:329–345 22. Yeong DJ, Velasco-Hernandez G, Barry J, Walsh J (2021) Sensor
13. Baek JW, Chung K (2020) Pothole classification model using and sensor fusion technology in autonomous vehicles: a review.
edge detection in road image. Appl Sci 10(19):6662 Sensors 21(6):2140
14. Zhang Z, Ai X, Chan CK, Dahnoun N (2020) An Efficient Algo- 23. Beltrán J, Guindel C, de la Escalera A, García F (2022) Automatic
rithm for Pothole Detection using Stereo Vision. In: 2014 IEEE extrinsic calibration method for lidar and camera sensor setups.
International Conference on Acoustics, Speech and Signal Pro- IEEE Trans Intell Transp Syst 23(10):17677–17689
cessing (ICASSP). IEEE, New York, pp. 564–568. 24. Barea R, Pérez C, Bergasa LM, López-Guillén E, Romera E,
15. Ouma YO, Hahn M (2017) Pothole detection on asphalt pave- Molinos E, López J (2018) Vehicle detection and localization
ments from 2D-colour pothole images using fuzzy c-means using 3d lidar point cloud and image semantic segmentation. In:
clustering and morphological reconstruction. Autom Constr 2018 21st International Conference on Intelligent Transportation
83:196–211 Systems (ITSC), pp. 3481–3486). IEEE, New York.
16. Inzerillo L, Di Mino G, Roberts R (2018) Image-based 3D 25. Park SS, Tran VT, Lee DE (2021) Application of various yolo
reconstruction using traditional and UAV datasets for analysis models for computer vision-based real-time pothole detection.
of road pavement distress. Autom Constr 96:457–469 Appl Sci 11(23):11229
17. Majidifard H, Jin P, Adu-Gyamfi Y, Buttlar WG (2020) Pave- 26. Sampath A, Shan J (2007) Building boundary tracing and regu-
ment image datasets: a new benchmark dataset to classify and larization from aAirborne LiDAR point clouds. Photogramm Eng
densify pavement distresses. Transp Res Rec 2674(2):328–339 Remote Sens 73(7):805–812
18. De Blasiis MR, Di Benedetto A, Fiani M, Garozzo M (2020) 27. Miller JS, Bellinger WY (2003) Distress identification manual for
Assessing of the road pavement roughness by means of LiDAR the long-term pavement performance program. Report no. FHWA-
technology. Coatings 11(1):17 RD-03-03. United States. Federal Highway Administration. Office
19. Kang BH, Choi SI (2017) Pothole detection system using 2D of Infrastructure Research and Development.
LiDAR and camera (2017). In: 2017 Ninth international confer-
ence on ubiquitous and future networks (ICUFN), pp 744–746, Springer Nature or its licensor (e.g. a society or other partner) holds
IEEE, New York. exclusive rights to this article under a publishing agreement with the
20. Nguyen DT, Kaup A (2022) Learning-Based Lossless Point Cloud author(s) or other rightsholder(s); author self-archiving of the accepted
Geometry Coding Using Sparse Tensors. In: 2022 IEEE interna- manuscript version of this article is solely governed by the terms of
tional conference on image processing (ICIP). IEEE, New York, such publishing agreement and applicable law.
pp 2341–2345

13