See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/326034733

Mind-Wave Controlled Robot: An Arduino Robot Simulating the Wheelchair for

Paralyzed Patients

Article · June 2018

DOI: 10.5430/ijrc.v1n1p6

CITATIONS READS
0 644

3 authors, including:

Seifedine Kadry
Beirut Arab University
244 PUBLICATIONS   637 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Call For Chapter "Finite Element Book" by Intechopen View project

FAIMA Journal View project

All content following this page was uploaded by Seifedine Kadry on 29 June 2018.

The user has requested enhancement of the downloaded file.

http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

ORIGINAL ARTICLES

Mind-Wave Controlled Robot: An Arduino Robot

Simulating the Wheelchair for Paralyzed Patients
Chi Hang Cheng1 , Shuai Li∗1 , Seifedine Kadry2
1
Department of Computing, Hong Kong Polytechnic University, Hong Kong, China
2
Beirut Arab University, Beirut, Lebanon

Received: May 7, 2018 Accepted: June 6, 2018 Online Published: June 27, 2018
DOI: 10.5430/ijrc.v1n1p6 URL: https://doi.org/10.5430/ijrc.v1n1p6

A BSTRACT
This project attempts to implement an Arduino robot to simulate a brainwave-controlled wheelchair for paralyzed patients with an
improved controlling method. The robot should be able to move freely in anywhere under the control of the user and it is not
required to predefine any map or path. An accurate and natural controlling method is provided, and the user can stop the robot any
time immediately to avoid risks or danger. This project is using a low-cost brainwave-reading headset, which has only a single
lead electrode (Neurosky mind wave headset) to collect the EEG signal. BCI will be developed by sending the EEG signal to the
Arduino Mega and control the movement of the robot. This project used the eye blinking as the robot controlling method as the
eye blinking will cause a significant pulse in the EEG signal. By using the neural network to classify the blinking signal and the
noise, the user can send the command to control the robot by blinking twice in a short period of time. The robot will be evaluated
by driving in different places to test whether it can follow the expected path, avoid the obstacles, and stop on a specific position.

Key Words: Rain-computer interface, Electroencephalogram, Neural network, Neurosky sensor, Wheelchair

1. I NTRODUCTION Advances in the neural network and human computer inter-

action technologies have caused concern to brain computer
Paralyzed patients face many difficulties in their daily life.
interface (BCI).[2] By employing BCI technology, human
It is hard for them to make use of motor neurons to control
can use brain wave to interact with physical devices easily.
muscle. People suffer from motor disabilities may some-
In this project, authors will make an Arduino robot car that
times be very stiff and even cannot speak as they want. They
controlled by human brain wave using the BCI technique. Ar-
need the help from others to perform daily activities. For
duino Mega is chosen because it is a low-cost microcontroller
example, fully paralyzed patients may need someone’s help
and it is more powerful than an Arduino UNO.[3] The human
to control the wheelchair. In the past, many technologies
brain wave will be captured using a low-cost Neurosky mind
have grown and become mature for disabled people to in-
wave headset.[4] The techniques used in this project can be
teract with physical devices, such as the electromyogram
further extended to a wheelchair for paralyzed people.
(EMG) arm, finger gesture recognition application and voice-
controlled wheelchair.[1] However, most of them are relying The objective of this work is to implement an Arduino robot
on muscles, body movements or speech commands. Obvi- to simulate a brainwave-controlled wheelchair for paralyzed
ously, they are not convenient for paralyzed people perform patients with an improved controlling method. The outcomes
these actions. of this project should fulfill all the following requirements. In
∗ Correspondence: Shuai Li; Email: shuaili@polyu.edu.hk; Address: Department of Computing, Hong Kong Polytechnic University, Hong Kong,

China.

6 ISSN 2577-7742 E-ISSN 2577-7769

http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

terms of mobility, no any map or path need to be predefined ment of attention and meditation level are based on human
and the robot should be able to move freely in anywhere un- mental activity. Tightening or relaxing the muscles may not
der the control of the user. In terms of accuracy, a controlling have an immediate change in the strength of attention and
method with at least 85% of accuracy should be adopted. meditation.[11]
In terms of safety, an immediate command should be pro-
vided to stop the robot immediately to avoid risks or danger.
In terms of cost-effective, the time cost of each controlling
command selection should be less than 1 second. In terms
of simplicity, the robot should provide a natural controlling
method that does not require left blinking or right blinking.
Autonomous obstacle avoidance should be included to avoid
the jerky movement.

2. N EUROSKY MIND WAVE HEADSET

Brain computer interface provides a communicating method
between physical devices and human brain.[5] This technol-
ogy used the neural activity of the human brain as a signal.
The objective of BCI is to create an output channel for the
brain to benefit the people suffered from motor disabilities, so
that they can control the physical devices with their mind.[6]
However, BCI is not a very stable interface and most of the
time it needs some sort of reliable control.
Electroencephalography (EEG) is the most studied non-
invasive interface of BCI. It is an instrument to record the
electrical activity of the human brain. The EEG signal is
represented in waveforms and they have a random and small
amplitude. Generally, waveforms can be classified into sev-
eral power spectrums such as alpha (closed eyes and re-
laxed), beta (thinking), theta (sleepy or disappointed), and
delta (sleeping).[7] Although this non-invasive brain wave
measurement is not harmful to the body, the measurement is
inaccurate because the noise of brain wave keeps producing
miscellaneous signals. NeuroSky mind wave headset (see
Figure 1) is a low-cost module for reading mental activities Figure 1. A NeuroSky mind wave headset
of the human using a single lead electrode. According to a
BCI hardware study, the neurosky mind wave headset is the 3. R EQUIREMENTS OF PARALYZED PA -
lowest price modular EEG device with the highest usability
TIENTS
score.[8] By recording the EEG signal, this headset is able
Paralysis refers to one or more muscle groups completely
to identify EEG power spectrum, and output the RAW sig-
loss the muscle function. The patient may lose the feeling
nal, EEG power spectrum, attention strength and meditation
in the affected area which causes the mobility problem. For
strength.[9] The raw brainwaves and the eSense data ob-
some of the patients, like the amyotrophic lateral sclerosis
tained from the NeuroSky mind wave headset are calculated
(ALS), the eye muscles are the only part that they can easily
to determine the strength of the eye blinks. It is possible to
control even in the final stages of the disease.[12] The rea-
find other existing techniques for blink detection, such as
sons for paralysis are generally damaged the nervous system,
EMG activity detection and image processing. However, the
stroke, trauma, poliomyelitis or ALS.[13] A survey by the
NeuroSky mind wave headset is more efficient than other
Christopher and Dana Reeve foundation indicated that in
eye blinks detection techniques and it requires fewer acces-
every 50 people, about 1 of them are paralyzed.[13]
sories with a faster processing time.[10] The eSense value
also indicating the strength of user’s attention and meditation Paralyzed patients need help from others in the daily life. Es-
level. The range of the level is from 0 to 100. The measure- pecially the fully paralyzed patients, they require the support
Published by Sciedu Press 7
http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

at all the time. However, it is impossible to find someone that and the biggest challenge of EMG system is the electrical
can provide the support in every moment. In the worst situa- noise.[10] Therefore, the idea of EMG system is possible, but
tion, they can only lie in bed when they are alone in a room. it is not an effective way to detect the eye blinks.
Although wheelchair is a good tool when walking is difficult
As mentioned the most commonly used techniques for blink
or impossible, paralyzed patients cannot control them easily.
detection before, it is obvious that the existing blink detection
For fully paralyzed patients, they can go to somewhere only
approach required lots of data and algorithm to handle the
under the help of someone. There is no doubt that a mind
identification of eye blinks. A new way to detect eye blinks is
wave-controlled wheelchair can help in their daily life.
using the EEG signals. By employing this method, only one
brain wave reading device is required for the whole system
4. C OMPARISON OF WHEELCHAIR CON - to extract the EEG signals. Compared to use the built-in
TROLLING METHODS webcam with various lights, employing EEG approach has a
higher mobility. This new technique can also overcome the
Almost all the existing wheelchair controlling methods are deficiencies of the electromyography and image processing
relying on the physical motion of the human. Even the elec- techniques for detecting blinks. For example, according to
tric wheelchair, it required the control of the joystick. The the website of Neurosky, the accuracy of the brainwave data
muscles of the mouth are used for the speech-controlled obtained is about 96%.[16] And this method involved fewer
wheelchair. These wheelchairs controlling methods are not algorithms and computation, which means it has a less time
suitable for the paralyzed patients in the final stage of the cost and able to command the wheelchair rapidly.[10] Also, it
disease. is a lightweight device which required fewer accessories.[10]
As mentioned in the problem statements, some of the paral- Hence, compared to the existing blink detection approach,
ysis like the ALS, their eye muscles are functional even in analyzing the EEG signals is more efficient and effective.
the final stages of the disease.[12] Therefore, the preferable It should be the best solution to employ for controlling the
wheelchair controlling methods for them should be eye blinks wheelchair.
and non-physical motion, like brain motion. A lot of tech-
niques can be used for the eye blink detection while most
5. C OMPARISON OF EEG WHEELCHAIR SYS -
of them have some limitations. One of a new approach is to
use a build-in webcam for the face and eye detection base on TEMS
the image processing, but the first tests showed some people Brice Rebsamen and his team introduced an indoor
cannot successfully control and communication through this wheelchair controlling method using though.[17] This system
solution.[14] was introduced in 2007. The idea of this system is to build
a mind-controlled wheelchair that able to navigate inside
Using image processing techniques to detect the eye blinks
the hospital or a typical house autonomously. The control
also has some limitations. First, the face should always be
mechanism of this system is based on the P300 EEG BCI
steady in a specified position. Otherwise, the camera cannot
which allows the patients to choose a destination listed on
take a clear frame for computation. The second limitation
the menu. The destination listed on the menu will flash one
is adequate light should be prepared for the detection. Fur-
by one. By identifying the positive potential peak of the
thermore, it is hard to implement to a real-time system as the
EEG signal, the system can know the patient is focusing on
time cost is varied. Based on these limitations, this design
which destination. Although the system provides a simple
is not an effective way to obtain eye blinks for controlling
controlling method, the usage of the wheelchair is only lim-
the wheelchair. Another approach that commonly adopted
ited to a specific environment with predefined paths. If the
at present for the eye blink detection is using three small
environment changed, a new map is required to load into
electrodes.[15] These electrodes are stuck to the human skin
the system. Moreover, the wheelchair can only arrive at a
around the orbicularis oculi muscle to get the electromyog-
specific point, but the patient cannot adjust the position. It
raphy (EMG) data. Obviously, an EMG system can detect
seems that the usage environment is the biggest limitation of
the eye blinks efficiently, but this approach is not very ac-
this system.
curate.[10] The accuracy of EMG system is easily affected
because of variability skin conductance and sight position- A new wheelchair controlling method through thought has
ing changes. The raw EMG data of the muscles are hard been introduced by Vaibhav and his team.[18] A monitor mod-
to obtain. It is because the baseline will easily drift when ule is used in this system for patients to select the movement
patient changing the sight positioning. Moreover, the skin option. The selection pointer moves from column to column
conductance is not a constant as it can change dynamically, with a defined time interval. Patients can perform left-hand
8 ISSN 2577-7742 E-ISSN 2577-7769
http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

motor imagery (MI) or right-hand motor imagery to select 6. S YSTEM ARCHITECTURE

the row of options. If the patient does not take any MI in the Figure 2 showed the system architecture of this project. The
time interval, the pointer will move to the next column. This system is formed by a Neurosky headset (EEG reader), a
system allows patients to control the wheelchair freely by computer included MATLAB, and an Arduino robot car. The
selecting the command, but the time cost of each command EEG reader contains a TGAM1 chip that can capture the
selection may take up to 7 seconds. Also, the time cost will human brain signal. TGAM1 also provided the signal filter
be double if the patient forgets to take an MI or the MI cannot and signal amplification. After that, the EEG signal will
be identified by the system. Although wheelchair is slow be digitized and sent to the Bluetooth transmission module
for safety purpose, the time spends of command selection (HC-06). Finally, the EEG signal will be transmitted to the
should be minimized to enhance the cost-effective. computer for further analysis.
Arzak and his team introduced a method using single lead
EEG module provided by Neurosky to collect the attention
level of the patient.[9] The attention level is used as a com-
mand of controlling the wheelchair to move forward, left or
right according to the range of attention level. Although this
system provides a real-time controlling method which is cost-
effective, it is tired for patients to keep in a high attention
level. Moreover, the attention level may easily be affected
by emotions.
Ramya Stephygraph and his team purposed an eye blinks
commanding method to control the wheelchair effectively.[19]
They used the NeuroSky mind wave sensor to obtain the eye
blinks information by recording the EEG signal. Various
eye blink characteristics can be used for different commands.
Such as the frequency, strength and time duration. Although
this controlling method seems cost-effective, it is not very
convenient for the patients to perform only left blink and right
blink. Moreover, the accuracy of the system is still a question.
The success rate of the command has not been mentioned. It
is also possible that the success rate will drop because of the
inconvenience left blink and right blink. Besides, the stress
blink is also hard to perform for some patients as this action
may require tightening the muscles.
Base of the analysis of the related works. We can find that
these systems have a lot of limitations. The most serious Figure 2. System architecture
problem is first three systems do not provide an immediate
command to stop the robot. This limitation may cause the MATAB in the computer will be used for noise filtering. It
safety issues. And these three systems required to wait for will check whether the user is sending a robot controlling
a long time to do the command selection, the time cost of command, or the blinking are just natural blinking of human.
each controlling command is too long so users cannot control If the blinking is belonging to a robot controlling command,
the robot in real-time. And the last two systems have not MATLAB will also determine which command does the
included autonomous obstacle avoidance, so the user may blinking representing.
need to send lots of commands in order to avoid the obstacles.
Finally, users may have some possible difficulties when con- The Arduino robot car is consisting of 7 components. They
trolling the robots because the controlling method requires are Arduino Mega, one Bluetooth HC-05 module,[20] two DC
the user focus, imagine, keep in a high attention, and perform motors, one L298N DC motor driver,[21] five ultrasonic range
unnatural blinking like left blinking, right blinking or stress detectors (SR-04), three infrared range sensors, and one
blinking. LEDs board. When the Bluetooth HC-05 module received
the controlling command from the computer, the Arduino
board will consider the data received from the ultrasonic
Published by Sciedu Press 9
http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

sensor and infrared sensor to make the final decision of the Otherwise, maintain the motion in the last time step and run
car movement. The ultrasonic sensors will detect the obsta- obstacle avoidance. And the LEDs will display the status of
cles and avoid it automatically. The infrared sensors will the controlling command (e.g. the front LED will on when
detect the distance between the robot body and the ground the user sends a “forward” movement command).
to prevent falling from the stair. If the command from eye
Below is the flow chart showing how the system works (see
blinking is received, the robot will follow this command.
Figure 3).

Figure 3. A descriptive flowchart

7. T HE FIRST PROTOTYPE considered as controlling command only if the two blinking

were occurred within a short period and strength of that two
For the first prototype, the RAW EEG signal can be collected blinking are higher than the natural blinking.
and send to the Arduino through Bluetooth. After collecting
the EEG signal, the Arduino will do the work of eye blinking By default, the robot will stop until the user sends an eye
detection by identifying the wave of eye blinking in the RAW blinking command to it. At the same time, the 4 LED in-
EEG signal. Generally, when the eye blinking occurs, a pulse stalled on the robot will blink one by one with a short delay.
will be sent to the RAW EEG data, and this prototype is able 4 LEDs are working on the robot to represent the direction of
to identify the pulse. After detecting the eye blinking, the forward, backward, left, and right. Users can blink twice in
robot will follow the eye blinking command to move in dif- a short period to start the robot. The movement of the robot
ferent directions. However, eye blinking is a natural activity is depending on which LED is lighted up at that moment. At
of human. In order to identify whether the eye blinking is any time, the user can blink twice in a short period to change
a controlling command or a natural human activity, some the direction of the robot. Once the user blinked twice, the
eye blinking identification strategy need to be used in the movement of the robot will be changed immediately base
prototype.[22] At the current stage, the eye blinking will be on which LED is lighted up at that moment. As mentioned
10 ISSN 2577-7742 E-ISSN 2577-7769
http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

before, a safety mechanism is the most important part of 8.2 Prototype using 2 LEDs
designing the robot. To avoid the danger caused by care- By default, the robot will move forward until the user sends
less or incorrect control, the system required to provide an an eye blinking command to it. At the same time, the LED
immediate command for the user to stop the robot. In this installed on the robot will blinks one by one with a short
prototype, the user can blink three times or more in a short delay. 2 LEDs are working on the robot to represent the
period to stop the robot. The robot will stop immediately, direction of left and right. User can blink twice in a short
while the LED will keep lighting up one by one. User can period to change the direction of the robot. The movement
blink three times or more in a short period to restart the robot.of the robot is depending on which LED is lighted up at that
The movement of the robot is depending on which LED is moment. If the left LED lighted up, the robot rotates left.
lighted up at that moment. If right LED lighted up, the robot rotates right. During the
This first prototype can work fine base on the eye blinking rotation of the robot, the user can blink twice at any time to
command. Although the accuracy is quite low, it can move end the rotation session. After that, the robot will go forward
in different directions and stop at any time. Also, it is a bit as a default. Again, during the testing, although the accuracy
difficult for the robot to follow a specific path. It is because is not very high, this prototype can work fine base on the eye
the 4 LEDs light up one by one with a short delay. Although blinking command. Compared to the first prototype, it has
the delay is short, the 4 LEDs light up one by one cost a huge fewer LEDs, so it can turn in just about one second, which
of time. The user needs to wait until a proper LED light up is acceptable. Moreover, it can follow a specific path like a
and send the eye blinking command. rectangle, and even some irregular path accurately. Overall,
it is a good controlling method.

8. F OUR PROPOSED CONTROLLING METH - 8.3 Prototype using stress blinking

ODS By default, the robot will move forward until the user sends
For the following prototypes, the stop/start function and eye an eye blinking command to it. User can blink twice in
blinking signal filtering functions are same as the first pro- a short period to change the direction of the robot. If the
totype mentioned above. For the brief testing part of the strength of first blinking is larger than the second one, the
following prototype, the following common testing method robot rotates left. If the strength of first blinking is less than
will be used: level 1 (The robot should follow the rectangle the second one, the robot rotates right. During the rotation
drawn on the floor) and level 2 (The robot should follow the of the robot, the user can blink twice at any time to end the
irregular path draw on the floor) are used. rotation session. After that, the robot will go forward as
a default. Again, this prototype can work fine base on the
8.1 Prototype using 7 LEDs eye blinking command. Compared to the other prototype, it
does not rely on the LED. The user can send any command
By default, the robot will move forward until the user sends
immediately and no need to wait for a proper LED light up.
an eye blinking command to it. At the same time, the LED in-
However, during the testing, the accuracy of the eye blinking
stalled on the robot will blinks one by one with a short delay.
command is a bit low because of the difficulty of performing
7 LEDs are working on the robot to represent the direction of
the stress blinking.
forward, various degrees of left, and various degrees of right.
User can blink twice in a short period to change the direction 8.4 Prototype using speed of blinking
of the robot. The movement of the robot is depending on
By default, the robot will move forward until the user sends
which LED is lighted up at that moment. If the front LED
an eye blinking command to it. User can blink twice in a
lighted up, the robot moves forward. If other LED lighted
short period to change the direction of the robot. If the time
up, the robot will turn in that direction in different turning
of two blinking is less than 350 ms, the robot rotates left.
degrees.
Otherwise, if the time of two blinking is between 350-700 ms,
During the testing, although the accuracy is not very high, the robot rotates right. During the rotation of the robot, the
this prototype works fine base on the eye blinking command. user can blink twice at any time to end the rotation session.
Compared to the first prototype, it has more LEDs, so it can After that, the robot will go forward as a default. During the
turn in a more accurate direction and do not require to stop testing, this prototype can work fine base on the eye blinking
and rotate. However, it is a bit difficult for the robot to follow command. Compared to the other prototype, it does not rely
a specific path. It is because the 7 LEDs lighted up one by on the LED. The user can send any command immediately
one with a short delay. The user needs to wait until a proper and no need to wait for a proper LED light up. Moreover,
LED lighted up and send the eye blinking command. it can follow a specific path like a rectangle, and even some
Published by Sciedu Press 11
http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

irregular path. Overall, it is also a good controlling method. wave pattern will be stable.

8.5 Comparison and summary of the prototypes

During the brief testing, method B and method D have the
best performance.
For all the controlling methods proposed, an immediate com-
mand has been provided. The user can stop the robot at any
time by blinking three times in a short period. Moreover, all
the controlling methods proposed can move in different direc-
tions base on the command sent by users. It is not required
to predefine any map or path.
Method B is relying on the LED and the user may need to
take up to 1 second for the direction selection. It may see
as a reasonable time cost even in a real situating when con-
trolling the wheelchair,[23] and the user can indicate a clear Figure 4. Example of EEG noise
and accurate direction selection by the help of LED. Method
D is not relying on the LED but relying on the speed of the In the graph below, waves labeled in green color are human
eye blinking. The user does not need to wait for the LED blinking while red color is some significant noise that have a
light up to perform the direction selection. User can select similar wave pattern as human blinking (see Figure 5).
the direction immediately by the rapid blinking. The speed
of blinking determining the moving direction of the robot.
However, for all the prototypes above, the classification of
the blinking and noise can be improved to enhance the accu-
racy. As in the prototypes, eye blinking will be considered
as controlling command only if the two blinking is within a
short period and strength is above the natural blinking, but
some noise having the similar pattern of the blinking wave
may be considered as human blinking which causes to the
incorrect command.
To conclude, after implementing the controlling methods
Figure 5. Example of blinking in EEG
proposed, we can find that method B and method D have the
best performance. However, in terms of immediate control,
In the prototype, eye blinking is considered as controlling
method D is better than method B. User may need to wait
command if the peak and trough of the wave are higher and
up to 1 second for the direction selection in method B, while
lower than a specific value as the example (see Figure 6).
method D does not. Considering some complicated path
and the efficiency, method D is more safety and less time-
consuming. Therefore, method D is chosen as the controlling
method to be adopted in this project. However, in this stage,
the accuracy of the prototype is still a big concern and it
should be further optimized to have a better classification of
the blinking and noise to enhance the accuracy.

9. N OISE FILTERING USING NEURAL NET-

WORK
The neural network has been adopted in this project to clas-
sify the blinking and the noise in the raw EEG signal. Below
are some examples showing the wave pattern of human blink-
ing and noises. A lot of noise will appear for several minutes
after the user wears the headset (see Figure 4). Then, the Figure 6. Noise filtering adopted in the prototype
12 ISSN 2577-7742 E-ISSN 2577-7769
http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

However, the wave of noise may also have a peak higher than If the peak is higher than the specific value X, it will come to
the specific value, so it is not a good classification method the the next stage. At this stage, several points are used to record
identify whether the wave is human blinking or not. In order the wave patterns as the example (see Figure 8).
to have a higher accuracy in the robot controlling process, an
When the strength reaches value X, the timer will start to
artificial neural network was adopted.
record the time until the peak is reached. The time spent in
In this project, MATLAB is used to build the neural network this part is represented by T1.
and send the controlling command after classifying the wave
The peak of a wave is represented by Strength1. When the
of noise and the human blinking. The wave pattern of human
peak is reached, the timer will start again until the strength
blinking will be further identified to different controlling
of wave drops back to value X. The time spent in this part is
command, such as blinking twice rapidly belong to the turn
represented by T2.
left command, blinking twice slowly belong to the turn right
command. When the strength drops below value X, the timer starts until
the strength of the wave drops below value Y. The time spent
10. C LASSIFY THE BLINKING AND NOISE in this part is represented by T3.
Wave in the EEG signal is captured for two stages of filtering. When the strength is below value Y, the timer start until the
Firstly, if the peak is lower than a specific value X as labeled trough is reached. The time spent in this part is represented
in red color below, it will not count as a blinking (see Figure by T4.
7).
The trough of a wave is represented by Strength2. When the
trough is reached, the timer starts until the strength of wave
reach value Y. The time spent in this part is represented by
T5.
Below are the pseudocode showing the wave pattern captur-
ing process.
If(raw>X)
Tic
If(raw>peak)
peak = raw
T1 = toc
if(raw<X)
T2=toc
If(raw < trough)
Figure 7. First-stage noise filtering through = raw
T4 = toc
If(raw<Y)
T3=toc
If(raw < trough)
through = raw
T4 = toc
If(raw>Y)
T5 = toc
wavePattern = (peak through T1 T2 T3 T4 T5)
NetI= wavePattern

By collecting the time of points, peak and trough of blinking

and noise, a training dataset can be formed to train a neural
network to classify the blinking and noise.
A Levenberg-Marquardt backpropagation neural network
function is used.[24, 25] According to the matlab documen-
tation, it is the fastest backpropagation algorithm and is
Figure 8. Wave pattern of a blinking wave highly recommended as a first-choice supervised algorithm,
Published by Sciedu Press 13
http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

although it does require more memory than other algo- if(isempty(blink_1))

rithms.[26] In this part, the neural network with 7 inputs, blink_1=wavePattern
4 hidden layer size is used. 75% of the dataset is used as t1=clock
training, while 15% is used as testing and 15% is used as elseif(isempty(blink_2))
blink_2=wavePattern
validation.
t2=clock
The error rate of the training data, testing data and validating time = time between t1 and t2
data are 3.59%, 3.22% and 4.97%, respectively, which is a t1=clock
great success in classifying the blinking and noise. else
blink_1=blink_2;
t2=clock;
11. I DENTIFY THE COMMANDS
blink_time_1=etime(t2,t1);
After classifying the blinking and noise, the blinking signal t1=clock;
need to be further analyses to identify: NetII=(blink_1 blink_2 time)

(1) Whether the user is sending a robot controlling com-

mand, or the blinking are just natural blinking of hu- 12. P ERSONAL AND UNIVERSAL NEURAL
man. NETWORK
(2) If the blinking is belonging to a robot controlling com-
In order to enhance the accuracy, user can build their per-
mand, which command does the blinking representing.
sonal neural networking by doing the training process. In
the training process, the blinking patterns of user will be
To solve the first problem, the time between two blinking can
captured and save as a training data set.
be taken as a consideration to identify whether the user is
sending a robot controlling command or not. Typically, the Three stages of training are required to build a personal neu-
time between two natural blinking is greater than 1 second. ral network. The LED board is set up on the Arduino robot
While the robot controlling command requires the user blink to guide the user to have the training.
twice or more rapidly. This differentiation is used to identify In the first stage, the LED will on for 20-30 seconds. Users
the natural blinking and the robot controlling command in are required to avoid blinking in this period. The aim of
this project. this stage is to collect the noise. At the next stage, users are
To solve the second problem, the wave pattern and the time required to blink only when the LED is on. The LED will
between two blinking can be taken as a consideration to iden- light up and down continuously with 1 second separation.
tify which controlling command is sent by the user. Another Which means the user need to blink once within 1 second
neural network is created to solve these two problems. Firstly, when the LED is on and keep opening their eyes for 1 second
if the wave pattern is classified as blinking, the wave pattern when the LED is off. The aim of this stage is to collect
collected in the first neural network will be used again, but the blinking pattern of the user’s natural blinking and also
this time, two continues wave will be recorded which means to capture some noise. In the final stage, the left LED and
we will have two datasets from the first neural network. Sim- right LED will light up one by one with 1 second separation.
ply name that as blink_1 and blink_2. The time between two Users are required to blink twice rapidly when the left LED
blinking will also be recorded. is on and blink twice in a bit slower speed when the right
LED is on. The aim of this stage is to collect the blinking
Input of neural network II: [blink_1 blink_2 time]. Output
pattern of different controlling command.
of neural network II: different controlling commands.
A universal neural network was also provided for the user
A Levenberg-Marquardt backpropagation neural network
so that they can control the robot without any training proce-
function is used. The neural network with 15 inputs, 8 hid-
dure. Compare to the personal neural network, the accuracy
den layer size is used. 75% of the dataset is used as training,
of the universal one is lower, so it is not recommended for a
while 15% is used as testing and 15% is used as validation.
long-term usage.
The error rate of the training data, testing data and validating
The universal neural network is formed by other users.
data are 2.09%, 3.02% and 2.66%, respectively, which is a
Which means, when a user trained a personal neural net-
great success in identifying different commands.
work, the data set in that personal neural network will be
Below are the pseudocode showing the wave pattern identifi- reused with other users’ personal neural network to form
cation process. a universal one. Therefore, the more the user trains their
14 ISSN 2577-7742 E-ISSN 2577-7769
http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

personal neural network, the universal neural network should By adopting the modified method, users are able to get closer
be more accurate in terms of classifying human blinking and to the obstacle as they may want. And users can control the
noise. Also, the universal neural network can be a reference robot by changing the direction even one or more sensors
for building a personal neural network as it already has a have an unexpected error. Therefore, both limitations are
good classification of human blinking and noise. solved.
By including autonomous terrain detection, the motion can
13. AUTONOMOUS CONTROLLING FUNC -
be modified and prevent the robot falling from the stair. If
TION the command from eye blinking is received, the robot will
Because of the safety issue, 5 ultrasonic sensors and 3 in- follow this command. Otherwise, maintain the motion in the
frared sensors are added in the robot. The ultrasonic sensors last time step and run terrain detection.
in the front part of the robot will detect the obstacles. Each
The same method will be used as mentioned above. By
sensor will echo an ultrasonic wave with a time delay to
adopting the modified method, users are able to get closer to
avoid the wave-conflict problem. The infrared sensors are
the stair as they may want. And users can control the robot
installed at the bottom part of the robot, they are used to
by changing the direction even one or more sensors have an
detect the distance between the robot body and the ground to
unexpected error.
prevent falling from the stair (see Figure 9).
By including autonomous obstacle avoidance, the motion can 14. T ESTING
be modified and require less controlling command even the Accuracy and safety are the most important part of this
road has lots of obstacles. If the command from eye blinking project. In order to test the accuracy of the robot the fol-
is received, the robot will follow this command. Otherwise, lowing methods are used, and each method will be tested for
maintain the motion in the last time step and run obstacle three rounds.
avoidance. In terms of performance, the car will become
smoother in motion. During the test, the number of correct command means the
motion of the robot match the command sent by the user; the
number of the wrong command means either: 1) The motion
does not match the command. 2) The command is seen as
noise. 3) Noise is seen as a command.
The first test is focused on the autonomous terrain detection
(see Figure 10). During the testing, the robot should detect
the terrain and avoid falling from the stair. The testing crite-
ria are to count the times of falling from the stair. In this test,
a higher-level ground will be used to simulate the stair.

Figure 9. Ultrasonic sensors and infrared sensors

However, there are some limitations when adopting this

method. First, it is impossible when user wants to get closer
to the obstacle. Second, the robot will be totally out of
control if one or more sensor has some unpredictable errors.
Therefore, this method has been modified to enhance the per-
formance. If the robot is moving forward, it will include the
autonomous obstacle avoidance as mentioned before. How- Figure 10. Autonomous terrain detection
ever, when the robot is turning left or right, the autonomous
obstacle avoidance function will not be used. The testing results are listed in Table 1.
Published by Sciedu Press 15
http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

Table 1. Testing results of autonomous terrain detection

Number of times trying to
Round fall from the high-level Results Accuracy
ground
1 2 Both avoided 100%
2 2 Both avoided 100%
3 2 Both avoided 100%

During the test, the robot can avoid falling from stair suc-
cessfully. When the sensors detected the distance between
the robot body and the ground is too large, it will go back
and turn to avoid falling from stairs.
The next test is based on a simple rectangular map (see Figure
11). During the testing, the robot should follow the rectangle
drawn on the floor. The testing criteria are to count the times Figure 12. Test on an irregular map
of incorrect command received. In this test, 5 checkpoints
are labeled on the ground. The robot should reach each The next test is based on an irregular map (see Figure 12)
checkpoint in the order of: red, orange, yellow, green, and During the testing, the robot should follow the irregular path
finally stop at blue. on the floor. The testing criteria is to count the times of
incorrect command received. In this test, 5 checkpoints are
labeled on the ground. The robot should reach each check-
point in the order of: red, orange, yellow, green, and finally
stop at blue.
The testing results are listed in Table 3.

Table 3. Testing results of running on an irregular map

The number of
Round Correct Wrong Accuracy
commands sent
1 22 20 2 90.9%
2 26 23 3 88.4%
3 20 18 2 90%

Same with the previous test, the robot can pass through all
the checkpoints successfully. The accuracy is around 90%.
Figure 11. Test on a simple map

The testing results are listed in Table 2.

Table 2. Testing results of running on a simple map

The number of
Round Correct Wrong Accuracy
commands sent
1 20 18 2 90%
2 19 18 1 94.7%
3 15 15 0 100%

During the test, the robot can pass through all the checkpoints
successfully. Even sometime the robot may not recognize the
user’s command correctly, the user can send the command
again to avoid the robot being derailed or directly stop the
robot by blinking three times and more rapid. The stop com-
mand is the most sensitive and accurate command so that it
can use to prevent the risk happen. Figure 13. Static obstacle avoidance

16 ISSN 2577-7742 E-ISSN 2577-7769

http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

The next test is focused on the static obstacle avoidance func- Table 5. Testing results of dynamic obstacle avoidance
tion (see Figure 13). During the testing, the robot should Number of times
reach all checkpoints and avoid the obstacle automatically, Round of placing the Impacted Avoided Accuracy
even the user is not sending any command. The testing cri- obstacles
1 22 1 16 94.1%
teria is to count the times of incorrect command received.
And counting the times of hitting the obstacle. In this test,
5 checkpoints are labeled on the ground. The robot should During the test, the robot can avoid almost all dynamic obsta-
reach each checkpoint in the order of: red, orange, yellow, cles successfully. The robot only impacted when the speed
green, and finally stop at blue. Several static obstacles are of the dynamic obstacle is faster than the speed of the robot.
placed on the path, the robot should avoid it automatically.
The testing results are listed in Table 4. 15. E VALUATION
In terms of mobility, this project provided a method that
Table 4. Testing results of static obstacle avoidance
The number
no any map or path need to be predefined. Although some
Obsta- other systems provided a simple controlling method, the us-
Round of commands Correct Wrong Accuracy
cle hit
sent age of the wheelchair is limited to a specific environment
1 24 20 4 83.3% 0 because the system require the predefined paths.[17] If the
2 25 22 3 88% 0
environment changed, a new map is required to load into the
3 26 21 5 80.7% 0
system. In this project, the robot can move anywhere like a
real wheelchair. The user can blink three times or more to
Although the accuracy becomes lower in this test, the average
stop or start the robot. And blink twice to start turning or
accuracy can keep in around 85%, which is acceptable. One
stop turning. This is a simple controlling method that allows
possible reason of the accuracy drop is that we need to find
robots to move in any direction and not to rely on any prede-
a good path to hit the obstacle, so we can test the obstacle
fined path. However, it is important to ensure the Bluetooth
avoidance function during the test. This abnormal motion
connection between different devices must be established
may cause some confusion in terms of robot control.
and the strength should be stable.
In terms of accuracy, this project provided a robot control-
ling method with around 85% accuracy on average, which is
an acceptable performance. During the test, we found that
the accuracy is relatively high in level 1 and level 2. The
accuracy can reach 100%. However, start from level 4, the
accuracy becomes lower. One possible reason behind is that
we need to find a good path to hit the obstacle so that we
can test the obstacle avoidance function. This abnormal mo-
tion may cause some confusion to the robot controller. And
this situation may not happen in the real life as we will not
want to hit the obstacle by using the wheelchair. Although
the accuracy is getting lower from level 4, this method still
provided 85% accuracy on average.
In terms of safety, this project provided an immediate com-
mand to stop the robot to avoid the risk. Also, obstacle
avoidance and autonomous terrain detection are included to
Figure 14. Dynamic obstacle avoidance
enhance the safety. Some other similar systems require users
to select a command to stop while the command selection
The final testing is focused on the dynamic obstacle avoid- time may take up to 7 second. Therefore, compare to other
ance function (see Figure 14). During the test, the robot similar system, this project has a better performance in terms
should avoid the dynamic obstacle automatically, even the of safety as it provided an immediate command to stop the
user is not sending any command. The testing criteria is to robot.[9, 17, 18]
count the times of hitting the obstacle.
In terms of cost-effective, all the controlling command of
The testing results are listed in Table 5. the robot are in real time. The user does not need to wait be-
Published by Sciedu Press 17
http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

fore sending any command. Also, all the commands are just user and it is not required to predefine any map or path. An
simple blinking which can be sent by the user immediately. accurate and natural controlling method is provided, and the
Some other similar systems require users to spend a long user can stop the robot any time immediately to avoid risks
time to select a controlling command. And the commands or danger.
are difficult to perform, for example, performing the motor
This project used the eye blinking as the robot controlling
imagery, keeping in a high attention and performing stress
method as the eye blinking will cause a significant pulse in
blinking.[17–19] This project provided a set of simple and
the EEG signal. By using the neural network to classify the
effective command for the user to control the robot. Blinking
blinking signal and the noise, user can send the command to
three times or more means start or stop the robot, blinking
control the robot by blinking twice in a short period of time.
twice means start or stop turning. These two simple com-
mands can perform rapidly, so that the user can control the Autonomous obstacle avoidance and autonomous terrain de-
robot in real time. Therefore, compare to other similar sys- tection are used to reduce the frequency of sending the con-
tem, the method used in this project is more cost-effective as trolling commands, and avoid the risks and danger immedi-
it provided an immediate command to stop the robot. ately. This project has been evaluated by driving the robot
in different places to test whether it can follow the expected
In terms of simplicity, this project provided a natural con-
path, avoid the obstacles, and stop on a specific position. The
trolling method that does not require unnatural blinking (e.g.
accuracy is around 85%, which is acceptable, and the robot
left blinking, right blinking, strong blinking, long blinking).
can arrive all the checkpoint, avoid all the obstacles and stop
The user can blink twice to turn and blink three times or
at a specific point accurately.
more to stop. Also, as autonomous obstacle avoidance and
autonomous terrain detection are included, the user can send
less command to avoid the jerky blinking. Some other similar 17. F UTURE WORK
systems require users to avoid the obstacles manually. Con-
The future work will be implementing this method to a real
sidering the daily-life situation, there must be some static and
wheelchair and take experiments with some paralysis patients
dynamic obstacle in the street, the users may need to send a
like ALS. It is important to ensure that the patients can also
lot of command if they are required to avoid the obstacles
be able to use our mind wave-controlled wheelchair system
manually. Therefore, the controlling method implemented
satisfactorily. As mentioned by a survey,[27] the wheelchair
in this project has included the autonomous obstacle avoid-
functions for the ALS patients is different to the normal mo-
ance and autonomous terrain detection function to reduce the
torized wheelchair. When implementing this method to a real
frequency of sending the command.
wheelchair for the ALS patients, collecting the user experi-
Overall, all the objectives of this project are met, which is a ence will be a very important step to make the wheelchair
huge success in the development of mind-wave controlled success.
robot.
Moreover, in terms of scalability, current system can be fur-
ther modified to be more scalable and add more module to
16. C ONCLUSION
it. For example, it can be further extended to a wheelchair
This project has provided a new method for paralyzed pa-
with an arm[28] or combining other BCI application to this
tients to control the wheelchair using a low-cost brainwave-
system.[29]
reading headset. This project has implemented an Arduino
robot to simulate a brainwave-controlled wheelchair for par- To achieve higher mobility and lower cost, instead of a lap-
alyzed patients with an improved controlling method. The top, a powerful single-board computer like raspberry pi can
robot can move freely in anywhere under the control of the be used to implement and train the neural network.[30]

R EFERENCES Series. 2010: 3-19.

[1] Puviarasi R, Ramalingam M, Chinnavan E. Self assistive technology [3] Siswoyo A, Arief Z, Sulistijono IA. Application of Artificial Neu-
for disabled people – voice controlled wheel chair and home automa- ral Networks in Modeling Direction Wheelchairs Using Neurosky
tion system. IAES International Journal of Robotics and Automation Mindset Mobile (EEG) Device. EMITTER International Journal of
(IJRA). 2014; 3(1). Engineering Technology. 2017; 5(1).
[2] Tan D, Nijholt A. Brain-Computer Interfaces and Human-Computer [4] Susan K. Mind over Matter: a $199 Headset Controls Objects via
Interaction. Brain-Computer Interfaces Human-Computer Interaction Brain Waves. (NeuroSky MindSet). IEEE Spectrum. 2010; 47(5): 23.

18 ISSN 2577-7742 E-ISSN 2577-7769

 http://ijrc.sciedupress.com International Journal of Robotics and Control 2018, Vol. 1, No. 1

[5] McFarland DJ, Wolpaw JR. Brain-computer interfaces for communi- [19] Stephygraph LR, Arunkumar N, Venkatraman V. Wireless mobile
cation and control. Communications of the ACM. 2011; 54(5): 60-66. robot control through human machine interface using brain signals.
PMid:21984822. https://doi.org/10.1145/1941487.194150 2015 International Conference on Smart Technologies and Manage-
6 ment for Computing, Communication, Controls, Energy and Materi-
[6] He B. Neural Engineering. Boston (MA): Kluwer Academic/Plenum als (ICSTM); 2015.
Publishers, 2005. [20] “Bluetooth Module HC-05”, 2017. [Internet]. Available from:
[7] Girase P, Deshmukh M. Mindwave Device Wheelchair Control. Inter- https://wiki.eprolabs.com/index.php?title=Bluetoot
national Journal of Science and Research (IJSR). 2016; 5(6): 2172- h_Module_HC-05 [Accessed: 19- Nov- 2017].
2176. https://doi.org/10.21275/v5i6.NOV164722 [21] L298 Dual H-Bridge Motor Driver. 2017. [Internet]. Available
[8] Stamps K, Hamam Y. Towards Inexpensive BCI Control for from: http://www.robotshop.com/media/files/pdf/data
Wheelchair Navigation in the Enabled Environment – A Hardware sheet-mot103b1m.pdf [Accessed: 19-Nov-2017].
Survey. Brain Informatics Lecture Notes in Computer Science. 2010: [22] Nguyen HT, Trung N, Toi V, et al. An autoregressive neural network
336-345. for recognition of eye commands in an EEG-controlled wheelchair.
[9] Arzak M, Sunarya U, Hadiyoso S. Design and Implementation of 2013 International Conference on Advanced Technologies for Com-
Wheelchair Controller Based Electroencephalogram Signal using munications (ATC 2013), Ho Chi Minh City. 2013. p. 333-338.
Microcontroller. International Journal of Electrical and Computer
[23] Ning B, Li MJ, Liu T, et al. Human Brain Control of Electric
Engineering (IJECE). 2016; 6(6): 2878. https://doi.org/10.1
Wheelchair with Eye-Blink Electrooculogram Signal. Intelligent
1591/ijece.v6i6.11452
Robotics and Applications Lecture Notes in Computer Science. 2012:
[10] Ghorpade S, Patil A. Mindwave-A New Way to Detect an Eye Blink. 579-5882.
IJARCCE. 2015: 82-84.
[24] Levenberg-Marquardt (trainlm): Backpropagation (Neural Network
[11] Yasui Y. A Brainwave Signal Measurement and Data Process-
Toolbox). [Internet]. Available from: http://matlab.izmiran
ing Technique for Daily Life Applications. Journal of Physiolog-
.ru/help/toolbox/nnet/backpr12.html [Accessed: 05-Jan-
ical Anthropology. 2009; 28(3): 145-150. PMid:19483376. https:
2018].
//doi.org/10.2114/jpa2.28.145
[25] Yu H, Wilamowski BM. LevenbergMarquardt Training Industrial
[12] Tjust A. Extraocular Muscles in Amyotrophic Lateral Sclerosis. PhD
Electronics Handbook, vol. 5 Intelligent Systems, 2nd Edition, chap-
dissertation, Umeå; 2017.
ter 12, pp. 12-1 to 12-15, CRC Press 2011.
[13] Udayashankar A, Kowshik A, Chandramouli S, et al. Assistance for
the Paralyzed Using Eye Blink Detection. 2012 Fourth International [26] Documentation, MATLAB & Simulink. [Internet]. Available
Conference on Digital Home. 2012: 104-108. from: https://www.mathworks.com/help/nnet/ref/trainl
m.html [Accessed: 10-Jan-2018]
[14] Ayudhya C. A Method for Real-Time Eye Blink Detection and Its
Application. IEEE Computer Society Conf. on Computer Vision and [27] Trail M, Nelson N, Van JN, et al. Wheelchair use by patients with
Pattern Recognition (CVPR); 2017. amyotrophic lateral sclerosis: A survey of user characteristics and
[15] Aramideh M, Eekhof J, Bour L, et al. Electromyography and recovery selection preferences. Archives of Physical Medicine and Rehabilita-
of the blink reflex in involuntary eyelid closure: a comparative study. tion. 2001; 82(1): 98-102. PMid:11239293. https://doi.org/10
Journal of Neurology, Neurosurgery & Psychiatry. 1995; 59(6): 662. .1053/apmr.2001.18062
https://doi.org/10.1136/jnnp.59.6.662-b [28] Bousseta R, Ouakouak IE, Gharbi M, et al. EEG Based Brain Com-
[16] NeuroSky. Exercise Equipment for Your Mind. Exercise Equipment puter Interface for Controlling a Robot Arm Movement Through
for Your Mind. Thought. Irbm. 2018; 39(2): 129-135. https://doi.org/10.101
[17] Rebsamen B, Burdet E, Guan C, et al. Controlling a wheelchair using 6/j.irbm.2018.02.001
a BCI with low information transfer rate. 2007 IEEE 10th Interna- [29] Sanjana M. Brain Computer Interface and its Applications – A Re-
tional Conference on Rehabilitation Robotics. 2007: 1003-1008. view. 2017 Jun.; 8(5).
[18] Gandhi V, Prasad G, Coyle D, et al. EEG-Based Mobile Robot Con- [30] Kucukyildiz G, Ocak H, Karakaya S, et al. Design and Implementa-
trol Through an Adaptive Brain–Robot Interface. IEEE Transactions tion of a Multi Sensor Based Brain Computer Interface for a Robotic
on Systems, Man, and Cybernetics: Systems. 2014; 44(9): 1278- Wheelchair. Journal of Intelligent & Robotic Systems. 2017; 87(2):
1285. https://doi.org/10.1109/TSMC.2014.2313317 247-263. https://doi.org/10.1007/s10846-017-0477-x

Published by Sciedu Press 19

View publication stats