Capstone Design (SOC4150–004)
Final Assignment project report
Devs_Team
Team mebers:
1. Mirzakhmedov Shokhrukhmirzo – u1610230(Team leader)
2. Abbosjon Kudratov - u1610001
3. Solikh Mirzaev – u1610234
4. Boburjon Borataliyev – u1610053
Lane Detection for Autonomous vehicle
Problem statement
To detect the lane lines in a road sample video and show the detected lines in the frame.
Make prediction according to detected lines about turns car is about make.
Algorithm
The process of lane lines detection can be divided into the following steps:
1) Image denoising
2) Edge detection from binary image
3) Mask the image
4) Lines detection using Hough transform technique
5) Left and right lines separation
6) Drawing the complete line
7) Predict the turn
Algorithm implementation steps (code explanation)
Pre-implementation steps:
We used in this project OpenCV library to do main tasks along with the help of other libraries
like numpy, imutils and matplotlib
import cv2
import numpy as np
import imutils
import matplotlib.pyplot as plt
We defined the function to process the frames of video. As video is a sequence of image
frames, first we need to load the video and we can process each frame as individual image.
cv2.VideoCapture() function returns 2 variables – first of them ret is a state boolean
variable(true/false). We will open the video while ret is true and process frames by sending
them to function process_image()
# open the video file
videofile = cv2.VideoCapture('./road.mp4')
while True:
# ret = frame capture result
# frame = captured frame
ret, frame = videofile.read()
frame = process_image(frame)
try:
# cv2.imshow("video", frame)
cv2.imshow('with lines', frame)
except:
pass
If the variable ret becomes false it means we reached the end of the video. To quit from the
loop we define waitKey() function with trigger some key pressed to break:
if ret is False:
print('end of video!!: break!')
break
q = cv2.waitKey(1) & 0xff
if q == ord('q'):
print('q is pressed: quit')
break
1) Image denoising
All code about processing the frame image we defined inside the function process_image()
Passed as argument image has is shape property. As it is colored image frame it has inside
shape property height, width and channel count. With help of resize() function of cv2 we
resize and store the image and load to another variable:
def process_image(img):
# resize the frame image:
if img is not None:
h, w, d = img.shape
frame = cv2.resize(img, (int(w / 1.4), int(h / 1.4))) # store the resized image in another variable
To do denoising of image – one of the best ways to make it using Gaussian filter or
Gaussian Blur with kernel matrix 5x5:
lane_img = np.copy(frame)
# apply Gaussian filter to smooth the image:
blurred = cv2.GaussianBlur(lane_img, (5, 5), 1)
Note: We applied the GaussianBlur to the copy of the image
Original image frame and Blurred
2) Edge detection from binary image
Before detecting edges we need to convert our colored 3-channel blurred image to 1-
channel grayscale image using cvtColor() function of cv2
# convert to grascale
gray = cv2.cvtColor(blurred, cv2.COLOR_BGR2GRAY)
Resulting grayscale image
Now to binarize the image we apply thresholding technique. This code determines the
threshold automatically from the image using Otsu's method and returns tuple of binarized
image with its threshold value:
(thresh, im_bw) = cv2.threshold(gray, 128, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)
Binarization takes all values till 128 – converts into black and other values greater than 128 till 255 ->
into white
Binarized image
To detect edges we use Canny edge technique. We need 1-channel image to detect edges.
# apply canny edge detection with binarized image
canny = auto_canny(im_bw)
We defined auto_canny() function to make the edge detection better that gives us additional
parameters (sigma and median values), but main thing here is Canny() of cv2:
def auto_canny(image, sigma=0.33):
# compute the median of the single channel pixel intensities
v = np.median(image)
# apply automatic Canny edge detection using the computed median
lower = int(max(0, (1.0 - sigma) * v))
upper = int(min(255, (1.0 + sigma) * v))
edged = cv2.Canny(image, lower, upper)
# return the edged image
return edged
Display the found edges:
3) Mask the image
We are interested in only some defined region of the image. To find our ROI (region of
interest) we need points to crop accordingly.
We display the image using matplotlib to find some points on it to define ROI. Find the
approximate locations of these red dots to crop road ROI:
ROI_frame = [
(200,500),
(800,500),
(450,350) ]
To mask the image means leave out only the ROI and cover other useless parts. To do so
we have defined the function ROI_area_masking()
Call this function and store the returned image:
cropped_masked_img = ROI_area_masking(canny, np.array([ROI_frame], np.int32))
This function is defined very simple. We need image and vertices(points) as arguments.
Channel count is not defined for our canny edge detected image so we comment it out. With
cv2.fillPoly() we fill the polynomial defined by edges and then make ‘bitwise AND’ to apply
as a mask:
def ROI_area_masking(img, vertices):
mask = np.zeros_like(img)
# channel_count = img.shape[2]
match_mask_color = (255,) # * channel_count
cv2.fillPoly(mask, vertices, match_mask_color)
# we can use cv2.bitwise_and() to do masking:
masked_img = cv2.bitwise_and(img, mask)
return masked_img
So we receive the masked cropped image like this with only detected edges (because we
passed to it our canny edge detected image) in our ROI:
4) Lines detection using Hough transform technique
Expressing lines in the Polar system by using Hough transform.
We should convert Lines from x,y space to polar mc space. But first we use HoughLinesP()
probabilistic function to determine points for us and store in lines variable:
lines = cv2.HoughLinesP(cropped_masked_img, rho=4, theta=np.pi / 180,
threshold=3, minLineLength=130, maxLineGap=40)
With function avg_slope_intercept() we collect 4 points: x1,x2,y1,y2. For input argument we
pass lines from HoughLinesP and our lane image:
averaged_lines = avg_slope_intercept(lane_img, lines)
So the function avg_slope_intercept() is defined like this: it takes 4 points from passed line
and fits into polyfit from where it gets the slope and intercept as parameters array:
def avg_slope_intercept(img, lines):
left_points = []
right_points = []
if lines is not None:
for line in lines:
x1, y1, x2, y2 = line.reshape(4)
parameters = np.polyfit((x1, x2), (y1, y2), 1)
slope = parameters[0]
intercept = parameters[1]
# we can differentiate lines whether on right or left by their slope values:
5) Left and right lines separation.
The slope helps us to identify which points are on the left or right:
if slope < 0: # points on the left have negative slope
left_points.append((slope, intercept)) # so we add these points to our left_points array
else:
right_points.append((slope, intercept)) # else these points are on the right side
To be able to draw the continuous line we have to find averages of these left or right points
arrays:
left_points_avg = np.average(left_points,
axis=0) # axis should be 0 because we want averages of columns in array
right_points_avg = np.average(right_points, axis=0)
And to draw them we will also need the coordinates and then we can return the arrays of
lines left and right. For making coordinates we will have to define another function
make_coordinates()
# we need the coordinates to draw the line:
right_line = make_coordinates(img, right_points_avg)
left_line = make_coordinates(img, left_points_avg)
return np.array([left_line, right_line])
make_coordinates() function helps us to make coordinates for the line with given slope and
intercept:
def make_coordinates(img, line_parameters):
try:
slope, intercept = line_parameters
except TypeError:
slope, intercept = 0.001, 0
# slope, intercept = line_parameters
y1 = img.shape[0]
y2 = int(y1 * (3 / 4.3))
x1 = int((y1 - intercept) / slope)
x2 = int((y2 - intercept) / slope)
return np.array([x1, y1, x2, y2])
6) Drawing the complete line
To draw the lines use the function draw_the_lines() :
image_with_lines = draw_the_lines(lane_img, averaged_lines)
Defined like this - it takes the argument image and averaged lines with defined points and
draws using cv2.line() function:
def draw_the_lines(img, lines):
img = np.copy(img)
blank_image = np.zeros((img.shape[0], img.shape[1], 3), dtype=np.uint8)
if lines is not None:
for line in lines:
x1, y1, x2, y2 = line.reshape(4)
# for x1, y1, x2, y2 in line:
try:
cv2.line(blank_image, (int(x1), int(y1)), (int(x2), int(y2)), (255, 255, 0), thickness=5)
except OverflowError:
pass
img = cv2.addWeighted(img, 0.8, blank_image, 1, 0.0)
return img
We enclosed cv2.line() function inside try statement to avoid OverflowError (as we
encountered this during runtime)
For better effect of line detection:
We defined function morph_closing() to refine lines – make them thicker and connect by
using the effect of morphological closing:
def morph_closing(image):
# P.S. erosion and dilation => ONLY APPLIED TO GRAY IMAGES!
# create 5x5 kernel with 1s
kernel = np.ones((5, 5), np.uint8)
kernel2 = np.ones((7, 7), np.uint8)
# apply dilation
#morphological closing = dilation , then erosion
frame_dilation = cv2.dilate(image, kernel2, iterations=2)
frame_erosion = cv2.erode(frame_dilation, kernel2, iterations=2)
#or alternatively
closing = cv2.morphologyEx(image,cv2.MORPH_DILATE,kernel2,iterations=2)
cv2.imshow('morpholofical closing effect',closing)
return closing
The canny edge
detected lines and these lines after morphological closing(refining) effect
The RESULT
The resulting image with drawn detected lines: we have a problem identifying points to draw
lines simultaneously for right and left sides with function cv2.HoughLinesP() and its
parameters minLineLength and maxLineGap – we could not find the optimal values.
lines = cv2.HoughLinesP(cropped_masked_img, rho=4, theta=np.pi / 180,
threshold=3, minLineLength=130, maxLineGap=40)
The written source code can be found inside the archive and it is named:
“capstone-team53.py”
The video demonstration has been uploaded to Youtube and the link is the following:
https://www.youtube.com/watch?v=d497UY59Uok
The source code also has been uploaded to Github for pulic use. Link to the repository:
https://github.com/abbosjon-kudratov/line_detection_opencv
