International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.

6, November 2023

SMART CROSSWALK: MACHINE LEARNING AND

IMAGE PROCESSING BASED PEDESTRIAN AND
VEHICLE MONITORING SYSTEM
Hiruni J.M.D.K, Weerakoon L.M.R, Weerasinghe T.R, Jayasinghe
S.J.A.S.M.S, Jenny Krishara, Sanjeevi Chandrasiri

Department of Computer Science and Software Engineering, Sri Lanka Institute of

Information Technology, Malabe, Sri Lanka

ABSTRACT
The conventional pedestrian crossing system's shortcomings require urgent reform to enhance the safety of
pedestrians and improve urban mobility. Issues such as insufficient time for pedestrians to cross, prolong
waiting times, neglection of emergency vehicles, and the absence of effective 24/7 response mechanisms at
traditional crosswalks present significant safety concerns in urban areas. Our primary intention is to
develop a cutting-edge pedestrian crossing system that relies on deep learning and image processing
technologies as its foundation. This research addresses to innovate an advanced smart crosswalk
consisting of four essential components: a real-time Pedestrian Detection and Priority System customized
for individuals with special needs, a responsive system for detecting road conditions, vehicle availability
and speed near crosswalks, a real-time Emergency Vehicle Detection and Priority System strengthened by
rigorous verification procedures, and a robust framework for identifying pedestrian accidents and
violations of crosswalk rules. The entire system has been meticulously designed not only to enhance
pedestrian safety by identifying potential dangers but also to optimize traffic flow. In essence, it aims to
provide an improved pedestrian crossing experience characterized by increased safety and efficiency.

KEYWORDS
Pedestrian Safety, Image Processing, Machine Learning, Deep Learning, YOLO

1. INTRODUCTION
Pedestrian safety presents a pressing challenge in today's urban transportation systems, with
crosswalk accidents and fatalities posing significant problems for cities worldwide. A primary
contributor to these issues is the lack of coordination between pedestrians and vehicles,
emphasizing the need for advanced safety solutions. Data from the "Pedestrian safety, A road
safety manual for decision-makers and practitioners" [1] reveals that pedestrians account for
more than 20% of the annual 1.24 million traffic-related fatalities, highlighting the necessity for
targeted interventions.

The urban landscape surrounding traditional crosswalks is fraught with immediate concerns.
Pedestrians often grapple with insufficient time to safely crossroads, a predicament amplified for
individuals with special needs. The lack of attention to these vulnerable populations underscores
the demand for impartial solutions. Furthermore, the common occurrence of pedestrians waiting
near crosswalks in the absence of vehicular traffic leads to chronic time wastage, hindering urban
productivity and causing frustration among citizens. Neglecting to prioritize emergency vehicles
like ambulances at these points poses a substantial risk, potentially resulting in avoidable
accidents. The aftermath of vehicular-pedestrian accidents and widespread rule violations at
DOI:10.5121/ijaia.2023.14603 21
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023
traditional crosswalks underscores a concerning absence of structured education and continuous
monitoring protocols. Addressing these multifaceted challenges at traditional crosswalks is
crucial for the safety, efficiency, and inclusivity of urban spaces.

The focal point of this research revolves around the innovative concept of a "Smart Crosswalk,"
which integrates Machine Learning, Deep Learning, and Image Processing techniques. This
system comprises four distinct components, each significantly enhancing crosswalk safety and
efficiency.
The primary emphasis is on optimizing pedestrian crossing times through real-time Pedestrian
Detection and Priority Systems. Customized to accommodate individuals with specific needs, this
approach utilizes algorithms to anticipate crossing times based on the number of individuals
present. The dynamic adjustment of pedestrian traffic light signals that results facilitates secure
and efficient pedestrian movement, promoting inclusivity and equitable access.

The research also delves into reducing pedestrian idling times and determining the optimal times
for pedestrians to cross the road. This is achieved through a responsive system that considers
real-time vehicle availability and uncontrollable speeds, departing from conventional static
systems to adjust signal timings according to dynamic traffic patterns.

In emergency scenarios, the research introduces a Real-time Emergency Vehicle Detection and
Priority System, reinforced by verification mechanisms. This technology swiftly identifies and
prioritizes emergency vehicles, minimizing disruptions to both pedestrian and vehicular traffic.
Additionally, the research addresses informing authorities about accidents and enforcing
pedestrian and vehicle rule violations. Employing image processing and machine learning
methodologies, the system identifies pedestrian accidents and rule violations, promoting safer
urban environments and reinforcing adherence to traffic regulations.

Each facet of the "Smart Crosswalk" system operates in synergy, embodying a comprehensive
approach to crosswalk safety and efficiency. Through the integration of computer vision, deep
learning, and advanced image processing techniques, this research seeks to bridge the divide
between technological innovation and urban well-being.

2. LITERATURE REVIEW
In urban settings, optimizing pedestrian safety and crosswalk efficiency is paramount. This
involves addressing challenges like inadequate crossing time, extended pedestrian waiting
periods, prioritizing emergency vehicles, and ensuring swift post-accident responses. This
literature review thoroughly examines these challenges, pinpoints areas where further research is
needed, and lays the groundwork for potential remedies. Tackling these issues is vital for
enhancing pedestrian safety, urban transportation, and the efficacy of relevant policies.

The study authored by Yuejin Wang et al. [2] introduces an automated image processing-based
system designed for pedestrian detection and enumeration. This system integrates various image
processing algorithms including background subtraction, Gaussian mixture model (GMM), and
blob analysis.

The research “Real Time Traffic Density Count Using Image Processing [3]” proposes an
algorithm to intelligently control traffic signals by determining the volume of the traffic on both
sides of the road. A density counting algorithm is used to compare real-time video frames to a
reference image and identify vehicles solely within a specific area of focus especially the road
area. The volume of the vehicles in the road is then used to control the traffic signal in a smart

22
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023
manner by comparing it with the traffic density on other directions of the road. Here they have
used the vehicle count to examine the density of the roads.

The research “A Self-Adaptive Traffic Light Control System Based on Speed of Vehicles [4]”
present a system that utilizes V2I communication, whereby vehicles transmit their information
regarding the speed to the traffic lights and them controlling the traffic lights considering
situations. Using this data, the signal timings are dynamically adjusted in real-time with the aim
of optimizing vehicle flow across the intersection and lowering traffic congestion on main
roadways. Furthermore, the method used in this study relies on the presumption that the driver
can regulate the speed of the car. The technology has the potential to greatly decrease traffic and
enhance the security and dependability of transportation networks by utilizing real-time data and
non-orthogonal signals.

The traffic light control system designed by Bhoomika G M has shown the potential for using
image processing and neural networks to improve traffic management. The proposed ambulance
detection system builds on this work to identify and prioritize ambulances in traffic. By using
CNN and YOLOv5, the system can accurately detect ambulances even in crowded traffic
conditions. The system's ability to switch traffic signals green for 30 seconds to allow
ambulances to pass through intersections can significantly reduce the time it takes for ambulances
to reach their destinations, ultimately saving lives [5].

The research paper "Sound Sensors to Control Traffic System for Emergency Vehicles”
addresses urban ambulance congestion. Employing two wireless sound sensors with Xbee
protocol and Arduino, it detects ambulances at 100 meters, turning the relevant lane green for 2
minutes. This allows safe passage to the next sensor at the signal. After passage, the green signal
extends by 2 seconds for added safety. Cost-effective and adaptable for high-priority vehicles, the
system aids emergency responders, benefiting urban traffic flow [6].

In a study conducted by Hadi Ghahremannezhad et al. [7] a novel and efficient framework for
detecting accidents at junctions in traffic monitoring applications is introduced. This proposed
architecture comprises three hierarchical stages: precise and swift object identification through
the utilization of the YOLO_v4 technique, object tracking employing a Kalman filter in
conjunction with the Hungarian algorithm for association, and accident detection through
trajectory conflict analysis.

Marjan Simončič [8] conducted a separate study focusing on a collection of traffic incidents
involving various combinations of motor vehicles, pedestrians, bicycles, and motorcyclists in
Slovenia. Subsequently, the logistic regression technique was employed to scrutinize this specific
group of incidents.

A study by Jianqing-Wu et al. [9] focuses on potential collisions between pedestrians and moving
vehicles, a critical concern for pedestrian safety. Techniques such as object tracking, grouping,
classification, background filtering, and lane identification are employed. Three key indicators,
post encroachment time (PET), percentage of stopping distance (PSD), and crash possibility
index (CPI), are used to assess conflict risk. Case study results affirm the effectiveness of this
approach in identifying near-crash situations between pedestrians and vehicles.

The paper by J. Z. Zhang et al. [10] offers a comprehensive summary of current research
concerning pedestrian crossing detection and behavior analysis. It underscores a range of
methods and strategies employed in pedestrian identification, monitoring, and behavior
assessment.

23
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023
A traffic light control system was proposed by Divij N, Divya K, and Anuradha Badage, which
aims to detect the siren sound of approaching emergency vehicles and prioritize their passage
through intersections. The proposed system integrates a sound detection sensor, camera, and
microcontroller into a smart object, which processes the data. To facilitate communication
between the smart objects and a centralized Decision Support System installed at the signal
junction, LoRa technology is used. By utilizing the Decision Support System, the system is able
to make informed decisions about clearing traffic in the lane where the emergency vehicle is
passing.

In the first phase of the proposed system, the smart object detects the emergency vehicle on the
road through sound detection sensor and camera. The smart object compares the moving object
on the road with the stored dataset to determine whether it is an emergency vehicle. The smart
object sends a message to the Decision Support System at the signal junction if both conditions
are met. In the system's second phase, the Decision Support System will determine the
appropriate course of action to ensure the emergency vehicle can safely navigate through the
traffic lane. The system is also equipped with acoustic sensors near the intersection, which work
on Receding Doppler Effect, to confirm the departure of the emergency vehicle [11].

Research by a group of Indian researchers, which is “Modelling pedestrian road crossing

behavior under mixed traffic conditions [12]” is a study that investigates road crossing patterns of
the pedestrians at midblock locations which are uncontrolled in India under contrasting traffic
conditions. In the research, linear regression techniques are applied to develop a model that
predicts the size of vehicular gaps on the road near pedestrian crossings. Additionally, a choice
model in statistics is used to analyze the decision-making process of pedestrians when it comes to
accepting or rejecting gaps between vehicles. The findings indicate that there are various aspects
that have a considerable impact on pedestrian safety at uncontrolled crossings, such as the
driver's willingness to yield, the size of the rolling gap, and the frequency of crossing attempts.
The findings of this study can be valuable in improving pedestrian safety at such crossings. This
research paper uses statistical models to determine the vehicle gaps on the road.

The research paper named “Automatic Traffic Using Image Processing [13]” proposes an
adaptive traffic light system that uses image processing and traffic density calculation as the
parameters. The system aims to address the issues of heavy traffic jams which occurred by
conventional traffic lights that work based on a timer. The system uses a server to collect data
and control traffic light mechanisms at a crossroads. Here the algorithms which are used for
vehicle density calculation and to adjust timing of traffic lights are validated and tested through
conditions on an actual road, and the results illustrates good accuracy in detecting traffic density
and successfully calculating the timing of traffic lights.

The research paper “Traffic Signal Violation Detection using Artificial Intelligence and Deep
Learning [14]”, which concentrates on the detection of traffic violations, proposed research
extends its focus to encompass both accident prevention and the enforcement of traffic rules.
While their work aims to detect various traffic infractions, such as signal jumping, speeding
vehicles, and vehicle counting, proposed research combines YOLO-based object detection with
custom YOLO models for accident detection and vehicle rule violation detection during red lights
at crosswalks. By addressing both accident prevention and rule enforcement simultaneously,
proposed research offers a more comprehensive solution to enhance pedestrian safety and
promote adherence to traffic regulations, ultimately contributing to a safer road environment for
both pedestrians and drivers.

Giovanni Pau et al.'s study, titled “Smart Pedestrian Crossing Management at Traffic Light
Junctions through a Fuzzy-Based Approach [15]”, delves into the imperative task of enhancing

24
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023
pedestrian safety at signalized crossings. With urbanization and rapid population growth
presenting formidable challenges, the need for intelligent solutions in urban planning has never
been more apparent. Pau and his co-authors address this concern by leveraging Information and
Communications Technologies (ICT) to implement a fuzzy logic-based system. This system
dynamically adjusts traffic light phases, accounting for variables such as time of day and
pedestrian volume. Through rigorous analysis and simulation using Vissim, their work
significantly advances discussions surrounding urban mobility and pedestrian safety.

“A dynamic traffic light management system based on wireless sensor networks for the reduction
of the red-light running phenomenon [16]” research describes the problem of accidents at traffic
light junctions and the potential of Intelligent Transport Systems to improve safety of the roads in
these areas. In addition, the article states that conventional traffic light control systems may not
always be successful and can result in accidents, especially the risk of Red-Light Running. This
situation arises when drivers must decide whether to stop or keep driving through an intersection
as the state of the traffic light changes from green to yellow. If the vehicle driver does not stop at
a red-light signal, they are breaking the law and putting themselves and others in danger. Red
light running is a frequent reason for collisions at junctions with traffic lights. The authors
propose that real-time data from WSNs may be utilized to dynamically adjust traffic signal cycles
and monitor traffic volumes. The study makes the argument that incidences of red-light running
can be reduced by reducing the amount of time people have to wait at traffic signals. It describes
the intended structure in detail and assesses its efficiency.

The research paper “Density Based Traffic Control System Using Image Processing [17]”
presents a real-time dynamic traffic regulation system that utilizing methods of image processing
to tally the quantity of automobiles present in each stage of traffic light and allocate timings
accordingly. Here as the density, they have considered the number of vehicles present. A camera
has been installed to record footage of the highway. Each frame of the video is compared to the
original image that was captured as it is constantly recorded in successive frames. Image
processing algorithms are used to count vehicles.

Another study by Alisa-Makhmutova et al [18] “Intelligent Detection of Object’s Anomalies for

Road Surveilance Cameras”. This study focuses on computer vision and machine learning
approaches that enable applications to accomplish various tasks in real time without a human,
such as object recognition, anomaly detection, and incident detection. In this article, they
examined how our AI based object recognition and tracking model is affected by picture
preprocessing techniques like grayscaling. Additionally, they used machine learning algorithms
to find recurring object paths and anomalies in real-time footage from traffic surveillance
cameras. Based on this, they proposed a method to identify illegal trajectories being followed by
cars or people.

3. METHODOLOGY

Figure 1. High level overall system diagram

25
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023
3.1. Real-Time Pedestrian Detection and Priority System for People with Special
Needs on Crosswalk

The research embarks with the collection of a diverse range of datasets, encompassing video
feeds of pedestrians in proximity to crosswalks. These datasets encapsulate a spectrum of
scenarios, including varying weather conditions, demographics, and urban settings. Preceding the
actual training process, extensive data preprocessing is undertaken. This includes video frame
extraction, resizing, and augmentation to ensure the quality and diversity of the dataset.
Importantly, the data is meticulously segregated for training and testing purposes.

The heart of the research lies in the selection and deployment of sophisticated models. The
YOLOv8 object detection framework is chosen as the backbone for precise pedestrian detection
due to its well-regarded real-time object recognition capabilities. Additionally, a distinct
YOLOv8s model is implemented for the specialized task of disabled pedestrian detection. The
selection of these models hinges on their proven performance under real-world conditions and
their versatility in addressing various pedestrian scenarios.

Crucial to the success of the system is the fine-tuning of these selected models using the collected
datasets. This training process involves the optimization of model parameters to enhance both
accuracy and overall performance. A notable aspect of this fine-tuning process is the special
emphasis placed on training the models to accurately identify mobility aids used by disabled
pedestrians, which include crutches, push wheelchairs, walking frames, and wheelchairs.

Upon completion of the model fine-tuning phase, the trained models are seamlessly integrated
into the pedestrian detection and time estimation system. This system is adept at processing video
feeds captured by high-angle CCTV cameras, enabling the real-time identification of pedestrians
and their associated mobility aids. It is of utmost importance that the system excels at detecting
and accurately categorizing disabled pedestrians, thereby ensuring their prioritization in the road-
crossing process.

In tandem with the detection system, the research encompasses the development of a precise
pedestrian counting mechanism. Employing advanced image processing techniques and mask
creation, the system can accurately quantify the number of pedestrians who are prepared to cross
the road. This data subsequently forms the basis for time estimation, a core component of the
research.

The time estimation aspect involves the creation of a machine learning-based algorithm capable
of estimating the time duration required for safe pedestrian crossing. This algorithm exhibits
remarkable adaptability, catering to diverse pedestrian demographics that encompass children,
adults, and particularly disabled individuals. It's noteworthy that the model takes into careful
consideration the specific mobility challenges faced by disabled pedestrians, thus prioritizing
their safety and convenience throughout the estimation process.

3.2. Real-Time Responsive System on Road Condition, Vehicle Availability, and

Uncontrollable Speeds Near Crosswalks

This research component encompasses the development of a dynamic traffic light control system
which is responsive to real-world data and traffic conditions. The objective is to minimize idle
time for pedestrians near pedestrian crossings through the integration of machine learning
methodologies, image processing, and sophisticated traffic control logics.

26
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023
The foundation of this research is built upon the collection and annotation of image data.
Highangle CCTV camera footage is used to capture real-world traffic scenarios. The data
includes instances of various road conditions, such as "Bare Roads" and "Filled Roads," along
with the presence or absence of vehicles on the road. These 1000 data points are meticulously
annotated and pre-processed using augmentation methods and resized to train the machine
learning model effectively.

The core of the research methodology revolves around the utilization of the YOLOv8 machine
learning model. YOLOv8 serves as the primary tool for detecting and classifying road conditions
and the presence of vehicles in the CCTV footage. The model is trained to achieve high accuracy
in real-time object detection and classification, forming the basis for the subsequent traffic
control logics.

To distinguish between "Bare Roads" and "Filled Roads," a specialized methodology is

developed using YOLOv8. This classification is crucial in adapting traffic control measures to
real-time road conditions. The methodology is designed to work seamlessly with high-angle
CCTV camera footage, ensuring accurate road classification.

The "Vehicle Availability Confirmation Checker" is a critical component of the system where a
pretrained YOLOv8 model is used to detect vehicles in real-time and confirm their presence in
front of a predetermined limit line. This involves training the model on a dataset of frames with
labeled vehicle presence, enabling it to deliver "Available" or "Unavailable" results. The checker
plays a pivotal role in ensuring that pedestrian safety is a top priority in traffic control decisions.
Another vital aspect of the methodology is the "Uncontrollable Speed Detection" system. It is
designed to identify and measure vehicle speeds accurately. Vehicle speeds are measured using
two benchmark lines and using the equation,

Speed (m/s) = Distance(m) / Time(s)

This real-time speed detection system leverages the capabilities of the YOLOv8 model to identify
high-speed incidents, contributing to pedestrian safety and efficient traffic management.

The core of the traffic light control system consists of four intelligent traffic control logics. These
logics are executed within specific timeframes, responding to the results of the custom machine
learning model, the availability confirmation checker and the uncontrollable speeds.

• Traffic Control Logic 01: This logic ensures a predictable traffic signal pattern by
transitioning to red when both sides of the road are "Available" and "bare."
• Traffic Control Logic 02: Adaptability is the hallmark of this logic, which responds to the
presence or absence of vehicles in real-time, reducing idling time and optimizing traffic
flow.
• Traffic Control Logic 03: Addressing high-speed incidents, this logic remains green when
uncontrollable vehicle speed is detected, promoting pedestrian safety and reducing
accidents.
• Traffic Control Logic 04: Balancing efficiency and safety, this logic prioritizes pedestrians
right of way, while ensuring green traffic lights are maintained till the vehicle crosses the
crosswalk.

27
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023

Figure 2. High level traffic light setup on road conditions

3.3. Real-Time Emergency Vehicle Detection and Priority System with Verification
for Crosswalk

This research component encompasses the development of a state-of-the-art smart crosswalk

system, with a primary objective of promptly identifying emergency vehicles, particularly
ambulances, and prioritizing their secure passage through congested crosswalks. The core
methodology revolves around the utilization of the YOLOv5 machine learning model, recognized
for its proficiency in real-time object detection tasks.

To achieve this objective, a comprehensive dataset of real-world urban traffic scenarios was
gathered. This dataset encompasses a diverse range of environmental conditions, traffic densities,
and emergency vehicle configurations. High-resolution images and corresponding metadata were
collected using strategically positioned cameras near intersections. The images were meticulously
annotated to highlight the presence and location of emergency vehicles, thus creating a labeled
dataset for model training. In total, 2500 data points were collected, ensuring a robust foundation
for model development and evaluation.

To initiate the process, two sound sensors are strategically positioned near the crosswalk to detect
the distinct siren sound emitted by approaching emergency vehicles. By employing advanced
triangulation techniques, the system accurately determines the direction of the sound source. This
auditory data is relayed to the system for further processing. The INMP441 Omnidirectional
Microphone Module, integrated via a 12S connection to the ESP32 Development Board, forms
the sensory backbone. The ESP32, acting as the central processing unit, leverages its WiFi and
Bluetooth capabilities for seamless networking. The Arduino IDE serves as the development
environment for programming and integrating these components. The ESP32 captures and
processes the audio data, first identifying the siren sound amidst ambient noise. Leveraging the
inherent capabilities of the INMP441 module and the processing power of the ESP32, the system
effectively hones in on emergency vehicle sirens.

28
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023
Subsequently, the system employs machine learning algorithms to analyze real-time footage from
strategically positioned CCTV cameras at the crosswalk. This analysis serves a dual purpose:
firstly, to verify if the identified vehicle is indeed an ambulance, and secondly, to track its path
across the crosswalk. For ambulance verification, a fine-tuned YOLOv5 model is utilized,
ensuring accurate identification even in complex traffic scenarios. Once the ambulance presence
is confirmed, the system proceeds to the next step.

The system then leverages IoT components to ascertain the approaching direction of the
ambulance based on the detected siren sound. Through this integration, the system gains the
ability to precisely identify the angle from which the ambulance is approaching the crosswalk.

Upon confirmation of the ambulance's presence and its approaching direction, the system
communicates with the traffic control infrastructure, requesting priority passage. This
coordination synchronizes traffic lights and associated devices, ensuring a safe and unhindered
path for the ambulance. Simultaneously, the system maintains vigilant surveillance over the
ambulance's movement, ensuring it crosses the intersection smoothly. After a successful transit,
the system promptly switches the traffic lights to green, allowing pedestrians to cross without
undue delay, while still prioritizing their safety following the ambulance's passage.

3.4. Pedestrian Accident and Rule Violation Detection on Crosswalk

To create an effective accident detection model, a diverse dataset was assembled. It combined
real-world accident images, when accessible, and synthetic images generated through graphical
tools. Synthetic images were carefully designed to mimic real accident scenarios with varied
conditions. Each image was meticulously annotated with precise bounding boxes to label
pedestrians and vehicles involved in accidents. The dataset was thoughtfully divided into training
(70%), testing (20%), and validation (10%) sets to facilitate robust model development and
evaluation, preventing overfitting. This comprehensive dataset is essential for training and testing
the accident detection model.

The You Only Look Once (YOLO) architecture, specifically YOLOv5, was selected as the
foundation for accident detection. YOLOv5 is esteemed for its real-time object detection
capabilities and efficient use of resources. It adopts a single-stage approach to object detection,
making it suitable for applications requiring speed and accuracy. The architecture consists of a
backbone network, neck, and head, collectively responsible for efficient feature extraction, object
detection, and bounding box regression. The model employs anchor boxes to predict object
locations and confidence scores. The choice of YOLOv5 over other architectures is rooted in its
aptness for real-time accident detection within crosswalks, where time is of the essence.

To equip the YOLOv5 architecture for the task of pedestrian-vehicle accident detection within
crosswalks, a comprehensive training process was undertaken. The hyperparameters were
methodically fine-tuned to ensure optimal model convergence and accuracy. The learning rate
(α), batch size, anchor box dimensions, and confidence threshold were all meticulously adjusted.
Learning rate (α) was a critical component that significantly influenced the model's optimization
process. Through rigorous experimentation, a learning rate of α = 0.001 was empirically selected.
This value was integral in controlling the step size of each gradient descent iteration.

In conjunction with hyperparameter tuning, the training process involved the execution of
multiple epochs. With the synthetic dataset containing around 800 images, a crucial consideration
was preventing overfitting. Hence, the training strategy encompassed a total of 20 epochs,
thoughtfully chosen to strike a balance between convergence and generalization. This approach

29
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023
prevented the model from becoming excessively tailored to the training data while allowing it to
effectively learn the underlying patterns necessary for accurate accident detection.

The custom loss function, incorporating object detection (L_obj), no-object detection (L_noobj),
classification (L_cls), and bounding box regression (L_reg), was meticulously defined to guide
the training process. The loss function was calculated as follows:

L(Θ) = (1 - β) * L_obj + β * L_noobj + L_cls + λ * L_reg

The parameter β, set to 0.5, ensured a balanced approach to object and no-object detection. This
balance was critical in maintaining the model's ability to distinguish objects and background
effectively. The coefficient λ, representing the influence of regression loss, was assigned a value
of 1.0, indicating the equal significance of precise bounding box regression in the context of
accident detection.

4. RESULTS AND DISCUSSIONS

The implementation of the Pedestrian Detection and Time Estimation System has generated
highly promising outcomes across its diverse components. Anchored by the YOLOv8 object
detection framework, the system achieves a notable accuracy rate of 93.87% in pedestrian
detection, marking its potential to significantly enhance urban traffic management through
responsive intersection control and heightened road safety standards.

Figure 3. Pedestrian detection

Figure 4. Confusion matrix of disabled pedestrians

30
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023

Figure 4. Precision, recall, and overall accuracy

In the domain of disabled pedestrian identification, the system also excels with an accuracy rate
of 82.34%, effectively categorizing individuals based on their mobility aids to prioritize safety
and convenience during road crossing. Moreover, the pedestrian counting mechanism,
underpinned by advanced image processing, delivers precise counts and offers valuable insights
into pedestrian behavior patterns. The real-time time estimation algorithm, meticulously tailored
for different demographics.

The research outcomes underscore the efficacy of four distinct traffic control logics in optimizing
road conditions and ensuring pedestrian safety. In the initial phase between 0 to 50 seconds,
(Fig.6.) fosters predictability and safety by rapidly transitioning to red when both road conditions
are "bare" and the availability confirmation checker signals "Available." This logic aligns traffic
light behavior with real-world data, reducing unnecessary idling time and enhancing pedestrian
safety. In the first 50 seconds to 100 seconds (Fig. 7.), the system promptly turns traffic lights red
when both custom models concur on either "bare-roads" or “Filled Roads” and the availability
confirmation checker indicates "Available," when no vehicles are in sight. This ensures a
pedestrian-friendly environment and reduces idling time for pedestrians, aligning traffic lights
with road conditions and improving overall efficiency.

Between 100 to 120 seconds, (Fig.8.) introduces a dynamic response to uncontrollable vehicle
speed. When high-speed incidents are detected, the traffic lights remain green, optimizing traffic
flow efficiency while addressing the seriousness of high-speed scenarios. In the subsequent phase
after 120 seconds, (Fig.9.) strikes a balance between pedestrian crossings and safety. It permits
pedestrian crossings when road conditions are favourable and high-speed incidents are absent,
prioritizing both safety and mobility.

Figure 5. Traffic control logic 1

31
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023

Figure 6. Traffic control logic 2

Figure 7. Traffic control logic 3

Figure 8. Traffic control logic 4

The ambulance detection component demonstrates its effectiveness in enhancing intersection

safety. The fine-tuned YOLOv5 model boasts an impressive 88% accuracy and a low 47% loss
rate. Leveraging confusion matrix (Fig.11.) and detection images (Fig.10.) provides valuable
performance insights. The integration of auditory cues via strategic sound sensors, supported by
advanced triangulation, ensures precise ambulance approach detection, strengthening the system's
efficacy. The system swiftly responds to an approaching ambulance, coordinating with IoT
components for a seamless, unobstructed passage. It monitors the ambulance's movement,
ensuring a smooth intersection crossing. Post-successful transit, traffic lights promptly shift to
green, enabling uninterrupted pedestrian crossing with continued priority for safety. This
approach not only expedites emergency response but also optimizes intersection operations,
greatly enhancing urban traffic safety.

32
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023

Figure 9. Ambulance detection

Figure 10. Ambulance confusion matrix

In the evaluation of our accident detection model (Fig.12.), we achieved compelling results.
Notably, the model exhibited an impressive confidence level of 0.80 in detecting real-world
pedestrian accidents, reflecting its robustness. Furthermore, during testing, it demonstrated an
accuracy rate of about 86%, reinforcing its competence in accident detection. Key metrics, such
as precision (0.87), recall (0.85), and F1-score (0.86), emphasize the model's effectiveness in
minimizing both false positives and false negatives, quantifying its accident identification
capabilities. Importantly, with increasing training epochs, the model's performance improved,
peaking at 88% accuracy after 20 epochs, indicating its adaptability and potential for further
refinement. Additionally, with an average inference time of about 40 milliseconds per image, the
model is suitable for real-time traffic surveillance, contributing significantly to road safety and
accident prevention. The vehicle rule violation detection leverages a pre-trained YOLO model. It
involves masking the crosswalk area and continuously monitoring traffic light status, flagging
red-light violations, enhancing road safety (Fig.13.).

Figure 11. Accident detection

33
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023

Figure 12. Rule violation when red

5. CONCLUSION AND FUTURE WORKS

In the realm of urban transportation, the emergence of the "Smart Crosswalk" system represents a
paradigm shift in addressing pedestrian safety concerns through innovative technological
solutions. The research outlined a comprehensive framework integrating advanced technologies
like deep learning and machine learning to revolutionize pedestrian safety, traffic management,
and urban mobility. The "Smart Crosswalk" system has demonstrated its efficacy in real-time
Pedestrian Detection, Priority Systems, adaptive vehicle detection, and Emergency Vehicle
Detection, showcasing its potential to enhance safety and efficiency in urban environments.

As the study looks to the future, several avenues for further exploration and improvement present
themselves. One crucial aspect is the augmentation of the dataset for pedestrian accidents, aiming
for diversity in scenarios, lighting conditions, and environmental factors. This expansion will
fortify the accident detection system's robustness, ensuring its effectiveness across a broader
range of real-world situations. Accurate time estimation for pedestrian crossings could be refined
through extended data collection, encompassing various pedestrian actions and environmental
conditions to enhance the model's accuracy.

Furthermore, the research suggests automating the identification of crosswalk areas and road
lanes through computer vision techniques. This would eliminate the need for manual
intervention, increasing the system's adaptability to different crosswalk configurations and
streamlining its implementation. User experience and accessibility considerations remain
paramount, with future research focusing on user studies and feedback collection, particularly
from pedestrians with special needs. Iterative improvements based on this feedback will enhance
the system's inclusivity and overall functionality.

In conclusion, the "Smart Crosswalk" system not only stands as a testament to the transformative
power of innovation in urban transportation but also serves as a foundation for ongoing research
and development. The suggested future work addresses key areas such as dataset augmentation,
time estimation accuracy, dynamic identification of crosswalk areas, and user experience
considerations. This collective effort aims to create a safer, more efficient, and inclusive urban
future, where technology seamlessly integrates with the well-being of all inhabitants. As cities
evolve, the "Smart Crosswalk" system offers a beacon of progress, inviting collaboration and
exploration for the continued advancement of urban safety and mobility.

REFERENCES

[1] W. H. Organization, “World Health Organization,” [Online].

Available: https://www.who.int/publications/i/item/pedestrian-safety-a-road-safety-manual-for-
decisionmakers-and-practitioners.
[2] N. Abbas, N. Abbas and T. M. Qadri, “Real Time Traffic Density Count using Image Processing”.

34
 International Journal of Artificial Intelligence and Applications (IJAIA), Vol.14, No.6, November 2023
[3] J. Li, Y. Zhang and Y. Chen, “A Self-Adaptive Traffic Light Control System Based on Speed of
Vehicles”.
[4] B. G. M, “Ambulance Detection using Image Processing,” International Journal of Advanced
Research in Science, Communication and Technology, vol. 6, no. 1, 2022.
[5] G. Iswarya, B. H. P and V. V. Reddy, “Sound Sensors to Control Traffic System for Emergency
Vehicles,” International Journal of Applied Engineering Research, vol. 13.
[6] H. Ghahremannezhad, H. Shi and C. Liu, “Real-Time Accident Detection in Traffic Surveillance
Using Deep Learning,” IEEE International Conference on Imaging Systems and Techniques (IST),
2022.
[7] M. Simoncic, “Road accidents in Slovenia involving a pedestrian, cyclist or motorcyclist and a car”.
[8] J. Wu, H. Xu, Y. Zhang and R. Sun, “An improved vehicle-pedestrian near-crash identification
method with a roadside LiDAR sensor,” Journal of Safety Research, vol. 73, 2020.
[9] R. Zhang, F. Li, J. Zhou and F. You, “A Review on Pedestrian Crossing Detection and Behavior
Analysis,” 2015.
[10] Y. Wang, S. Guo and H. Huang, “The Pedestrian Detecting and Counting System Based on
Automatic Method of CCD,” 9th International Conference on Advanced Infocomm Technology,
2017.
[11] D. N, D. K and A. Badage, “IoT based Automated Traffic Light Control System for Emergency
Vehicles using LoRa,” International Journal of Science Technology & Engineering, vol. 6, no. 1,
2019.
[12] P. V. B Raghuram Kadali 1 , “Modelling pedestrian road crossing behaviour under mixed traffic
condition,” 2013.
[13] A. H. Akoum, “Automatic Traffic Using Image Processing,” Journal of Software Engineering and
Applications, 2017.
[14] R. J. Franklin and Mohana, “Traffic Signal Violation Detection using Artificial Intelligence and
Deep Learning,” 2020.
[15] G. Pau, T. Campisi, A. Canale, T. Campisi, A. Severino, M. Collotta and G. Tesoriere, “Smart
Pedestrian Crossing Management at Traffic Light Junctions through a Fuzzy-Based Approach,”
2018.
[16] A. Makhmutova, R. Minnikhanov, M. Dagaeva, I. Anikin, T. Bolshakov and I. Khuziakhmetov,
“Intelligent Detection of Object's Anomalies for Road Surveilance Cameras,” 2019.
[17] U. E. Prakash, A. Thankappan, V. K. T. and A. A. Balakrishnan, Density Based Traffic Control
System Using Image Processing, 2018.
[18] M. Collotta, G. Pau, G. Scatà and T. Campisi, “A dynamic traffic light management system based
on wireless sensor networks for the reduction of the red-light running phenomenon.,” 2014.

35