AnEye TrackingBasedWirelessControlSystem
AnEye TrackingBasedWirelessControlSystem
AnEye TrackingBasedWirelessControlSystem
net/publication/250003251
CITATIONS READS
9 3,225
3 authors:
Yim Fun Hu
University of Bradford
149 PUBLICATIONS 2,746 CITATIONS
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
Sensors and IoT in Modern Healthcare Delivery and Applications View project
All content following this page was uploaded by Suraj Verma on 28 May 2014.
ABSTRACT
The power of vision is taken one step ahead with the introduction of sophisticated eye-tracking and gaze-
tracking techniques which track the movement of the eye and the gaze location to control various applications.
This paper describes in detail the low-cost hardware development and the software architecture for a real-time
eye-tracking based wireless control system using the open-source image processing framework of AForge.NET.
The system developed has been tested in the field of remote robotic navigation, using the Lego NXT Mindstorm
robot, and wireless home automation, using the X10 transmission protocol. The system designed uses a USB
camera to detect the eye movements and transmit control signals over a wireless channel. This provides a highly
useful alternative control system for a person suffering from quadriplegia (i.e. full body paralysis). Test results
on system response time and error percentage show the performance of the developed eye tracking systems.
Keywords: Eye tracking, wireless control, low-cost headgear, AForge.NET framework, assistive technology.
1. INTRODUCTION
Several people around the world suffer from several physical disabilities that prohibit them from leading a
normal life. Several conditions like Amyotrophic Lateral Sclerosis (ALS), cerebral palsy, traumatic brain
injury, or stroke may result in loss of muscle movements in the body [1], thus rendering the person
paralyzed. However, in several cases not all parts of the body are paralyzed and the person may have
limited voluntary movements of the head, eyes or tongue and may even be able to blink, wink or smile [2].
Assistive technology systems, that make use of these available movements, can be developed to aid people
suffering from these conditions not only communicate with other people, but also control various electronic
devices [3]. For people suffering from quadriplegia, one of the only reliable and convenient sources of
voluntary movement that may be used by an assistive technology system is the movement of the eye [4].
Hence, the person‘s eye movements may be used to control computers for browsing the internet, simple
word processing or even paint like application. In this paper we present a real-time low-cost eye-tracking
based wireless control system called IRIS – Intelligent Recognition for Interactive Systems. The IRIS
system can be used to remotely control a wireless robot and can also be used for basic home automation by
wirelessly controlling the lights. In the future, the IRIS system aims to provide a single control system to
help people suffering from disabilities to continue with their day-to-day activity, helping them to move
around without being dependant on somebody and also control various household appliances.
The current commercial eye tracking system is the Tobii X60 & X120 eye tracking system which can
record eye movements irrespective of any surface or environments such as TVs, projection screens,
physical objects such as brochures, magazines, products and even shopping shelves [5]. The eye tracking
system is placed near the environment under study and the user, without wearing a head-mounted gear, can
look anywhere within that environment and the system analyzes the user‘s eye movements and determines
where or what the user is looking at. This system proves to be highly accurate and allows a large degree of
head movement and distance from the eye tracking system. However, they are very expensive and hence
there is a need for a low-cost eye tracking device. A light-weight eye tracking system called the ‗View
Pointer‘ detects eye movements towards ubiquitous computers embedded in real world objects [6]. The
View Pointer uses IR tags that are embedded onto each object. The IR tags emit infrared light set at a
unique frequency whose reflection on the cornea of the user‘s eye determines what object the user is
looking at. The View Pointer uses a light-weight USB camera, as a video fed to the system, to detect the
frequency of the infrared light reflected on the cornea of the eye and determines which object the user is
looking at depending on the frequency recorded by the system. However, this form of detection cannot be
used in robotic navigation as the distance between the user and the robot, embedded with the IR tag
programmed to transmit the desired frequency, varies. It can be used for home automation, but the only
drawback is that the user cannot remotely control home appliances from a large distance as the user has to
be close to the object embedded with the IR tag.
686
26th International Conference on CAD/CAM, Robotics and Factories of the Future 2011
26th-28th July 2011, Kuala Lumpur, Malaysia
The rest of the paper is organized as follows. Section 2 introduces some of the key eye tracking
mechanisms and systems. The hardware architecture and the head-mounted gear are described in Section 3.
Section 4 describes in detail the software architecture and how the Aforge.NET framework was adopted to
develop the eye tracking software. Two applications developed using the IRIS control system is discussed
in Section 5. Some performance results of the IRIS system are presented in Section 6. Finally, the
conclusion is discussed in Section 7.
The process adopted to track the movement of the eye is termed as eye tracking [7]. Eye tracking uses
devices, called as ‗Eye Trackers‘, that are responsible for tracking the movement of the eye and act as the
front end device which feeds the input (i.e. video frames of eye movement) to the system for further
processing. There are various metrics used to track the movement of the eye such as fixations, gaze
duration, point of gaze/area of interest, scan path and hot gaze plots [7, 8, 9]. An early technique used to
detect the movement of the eye was with search coils [10] which involved attaching a plaster of paris ring
directly to the cornea and through mechanical linkages to a recording pen for measurements in a varied
electromagnetic field.
One commonly used eye tracking mechanism is the Electro-OculoGraphy (EOG) [11, 12] method which is
a process of tracking the movement of the eye based on recording the Corneal-Retinal Potential (CRP), the
potential difference existing between the cornea and the retina during eye-movements. The EOG signal
ranges from 0.05 to 3.5 mV in humans and is linearly proportional to eye displacement which generates a
change of the order of around 20 μV in readings for each degree of eye movement [9].
Another commonly used eye tracking mechanism is Video-OculoGraphy (VOG). This method has been
proven to be a highly effective non-invasive technology for evaluating eye movements [13]. It is a method
in which the movement of the eye is recorded with the help of digital cameras, in particular infra-red
cameras [14]. Image processing filters are applied to the images collected from the video source and the
movement of the eye is determined from these images. The IRIS system described in this paper also uses a
form of VOG eye tracking mechanism.
However, even though EOG based systems are more reliable than VOG based systems, they require
additional hardware like A/D converter, amplifier, filters and samplers to accurately process the analogue
signals before feeding it to the control system. It is here that Video-Oculography plays an important role
since it uses only a light-weight portable USB video camera to provide images for the detection process.
The complex process required to determine the direction of the eye movement is controlled by the eye
tracking software, thereby reducing the size and cost of the eye tracking system.
Some of the applications that incorporate eye-tracking and eye-gaze technology are in the following areas:
Field of media and marketing research: to create a comprehensive study on how people
understand, use and respond to web pages [15]
Cognitive psychology: to study mental processes including how people think, perceive, remember
and learn [15]
Human Computer Interaction (HCI): to understand human perception of virtual environments and
computer games [15]
Indoor/outdoor navigation: to produce eye controlled wheelchairs that aid in navigating the
wheelchair through different environments [11, 12, 16]
The Attentive Response Technology (ART) [17] has employed eye tracking techniques in home automation
using the X10 protocol [18]. The system provides an interface for the physically challenged people to
control their household appliances like televisions, lighting, heating and cooling. The ART system uses two
cameras, one to monitor the eye movements and the second to monitor the environment. Depending on
what device the user looks at, the system maps the real-time image of that device with the pre-known
devices, which are saved in an image database, to search for a match. The main disadvantage of this system
is that the image database has to be updated if there are any changes around the house. However, the IRIS
control system enables the user to control home automation wirelessly in real-time by just looking at the
options on the user interface and is independent of the position of the object or surrounding.
687
26th International Conference on CAD/CAM, Robotics and Factories of the Future 2011
26th-28th July 2011, Kuala Lumpur, Malaysia
The IRIS control system uses Video-Oculography eye tracking mechanism to detect the movement of the
eye with the help of a low-cost video camera and passes navigation commands over a Bluetooth channel to
an NXT Lego Mindstorm robot and home automation unit via the X10 wireless transmitter. The top-level
system architecture for the two applications developed, robotic navigation and home automation is shown
in Figure 1.
The IRIS control system consists of a USB camera and infrared LEDs. Micro IR wireless cameras which
are available in the market are expensive and have a low frame rate compared to the inexpensive, high
frame rate (60-100 fps) PS3 eye camera which can be easily modified into an IR camera [19]. The eye
tracker developed consists of the PS3 eye-camera that captures real time video frames of the eye
movements and sends these video frames to the IRIS control system. The eye tracker also consists of an
array of Infrared Light Emitting Diodes (IR LEDs) which are housed close to the eye. The main function of
the IR LED is to illuminate infrared light around the eye which helps in pupil detection. This process of
detecting the pupil is known as corneal-reflection/pupil-centre method [20]. In doing so, the image received
consists of two points of interest, the bright pupil reflection and the corneal reflection, as shown in Figure 2.
In this paper we consider the bright pupil reflection as the object of interest and track the movement of the
pupil which in turns corresponds to different eye movements.
688
26th International Conference on CAD/CAM, Robotics and Factories of the Future 2011
26th-28th July 2011, Kuala Lumpur, Malaysia
button-controlled power source for the IR LED array as shown in Figure 4. Figure 5 shows the developed
IRIS eye tracker.
Figure 4: Assembly of the Eye Tracker Unit Figure 5: Developed Eye Tracker
Once the eye movements have been detected by the IRIS control system, it analyses the movements and
determines the corresponding control command to be transmitted over a wireless channel to the respective
application device (i.e. NXT robot or the X10 lamp device). The control system in this case is a HP Pavilion
Laptop running on Windows Vista, Home Edition, which acts as an interface between the eye tracker and
the application devices. The IRIS control system uses two forms of wireless communication protocols, i.e.
Bluetooth 2.0 for robotic navigation using the HP Integrated Bluetooth Module and X10 for home
automation using the X10 transmitter unit.
A robot that was designed and assembled using the LEGO Mindstorm NXT programmable robotics kit is
shown in Figure 6. It consists of the NXT Intelligent Brick [22], which is a 32-bit programmable micro-
processor that enables communication with the IRIS control system over a Bluetooth channel. Thus,
depending on the command received from the IRIS control system the NXT-brick instructs the respective
LEGO servo motors to rotate in the desired direction, speed and turn ratio, thereby navigating the robot.
The IRIS system is a sophisticated system that requires the integration of both hardware and appropriate
software. Visual C# programming language under the .NET framework of Microsoft Visual Studio, 2008,
was used to develop the IRIS eye tracking control software. It uses libraries from AForge.NET, which is an
open source C# framework designed for developers and researchers [23]. The framework is used in the field
of computer vision for image processing and robotics that comprises of a set of libraries of which the
following are used in the software development of the IRIS control system:
AForge.Imaging
AForge.Imaging.Filters
689
26th International Conference on CAD/CAM, Robotics and Factories of the Future 2011
26th-28th July 2011, Kuala Lumpur, Malaysia
AForge.Video.DirectShow
AForge.Robotics.Lego
The software for the IRIS control system can be broken down into several software modules which support
the respective applications such as robotic navigation and home automation. The video capturing module
retrieves the real-time video frame from the PS3 eye camera and provides the frame to the image processing
module. The image processing module is responsible for applying different image filters on the received
video frame to facilitate the pupil detection stage. Three different image processing filters are used in
sequence. First the greyscale filter is used to convert the colour image into a greyscale image. Second the
invert filter is used to invert the image since the colour filter in the final step does not detect the colour
black. Hence, the pupil that appears black in colour, prior to this stage will be converted to white and the
remaining part of the image is converted to black, thus removing all unwanted objects from the video
frame. The final filter used is the colour filter which filters the pixels from the image that are not within the
specified RGB range (i.e. set the colour white or fine tune by varying the RGB values until you receive a
clear image of only the pupil). The general IRIS software architecture is shown in Figure 7.
The calibration module calculates the four co-ordinates of the rectangle blob drawn around the object of
interest (pupil). These co-ordinates are used in the program to further map the image co-ordinates on to the
main screen and use them as reference points in various applications. The calibration process is performed
in order to get the boundary co-ordinates of the screen with respect to the image co-ordinates and then map
the image co-ordinates onto the corresponding points of the main screen. Figure 8 shows the calibration
points on the screen which the user has to look at in order to complete the calibration process.
690
26th International Conference on CAD/CAM, Robotics and Factories of the Future 2011
26th-28th July 2011, Kuala Lumpur, Malaysia
x11, y11, x21, y21 = previous x & y coordinate; x1, y1, x2, y2 = current x & y coordinate
On acquiring the corner coordinates (x1, y1) and (x2, y2) we can plot the remaining two coordinates (x2,
y1) and (x1, y2) which correspond to the main screen coordinates (a2, b1) and (a1, b2) respectively as
shown in figure 8.
Thus, for any image coordinate (x, y) the system will map it to the corresponding main screen coordinate (a,
b). A high value of (x2 – x1) and (y2 – y1) signifies more system accuracy and higher degree of calibration.
A summary of all the classes, methods and fields used from AForge.NET framework for the software
development is as shown in Table 1.
The IRIS eye tracking control system developed was used to test the performance of the system in remote
robotic navigation and wireless home automation applications.
691
26th International Conference on CAD/CAM, Robotics and Factories of the Future 2011
26th-28th July 2011, Kuala Lumpur, Malaysia
The first application developed to test the IRIS eye tracking based control system is the Robotic Navigation
Application. In this application the IRIS control system remotely controls a robot by passing commands to
the robot over a wireless channel. According to the point on the screen the user is looking at the
corresponding navigation commands are determined. The user looks at the left side of the screen to make
the robot move left, right side of the screen to make the robot move right and looks at the centre of the
screen to keep the robot moving forward. The bottom-centre of the screen provides an emergency stop
signal that completely halts the robot. The GUI developed using Visual C# for this application is shown in
Figure 9.
Figure 10: Code Snippet - Left Navigate Figure 11: Code Snippet - Right Navigate
692
26th International Conference on CAD/CAM, Robotics and Factories of the Future 2011
26th-28th July 2011, Kuala Lumpur, Malaysia
Figure 12: Code Snippet - Forward Navigate Figure 13: Code Snippet – Stop
5.2 Home Automation
The second application developed to test the working of the IRIS control system is the Wireless Home
Automation Application. The main aim of this application is to provide control over household lighting for
the physically challenged people by tracking the movement of their eye and providing control signals to the
lighting unit. This concept is achieved with the help of the X10 transmitter and the X10 lamp device which
acts as a wireless relay switch. The user looks left to turn OFF the light and looks right to turn ON the light
with the help of the GUI developed using C# shown in Figure 14. The code snippet for home automation
written in C# is shown in Figure 15 where the threshold point corresponding to ‗newtrackx‘ is calculated
during the calibration process. If the user looks at a point on the screen that corresponds to a value trackx >
(newtrackx + offset), the light turns OFF and if the user looks at a point on the screen that corresponds to a
trackx < (newtrackx – offset), the light turns ON. Using the X10-CM17A class, the command
X10HouseCode.A selects the X10 lamp device which is set to ‗A‘ on channel ‗1‘ and the command
X10Command.TurnOff/TurnOn performs the necessary turn OFF/ON operation.
6. EXPERIMENTAL RESULTS
The two applications developed were used for evaluating the performance of the IRIS control system. The
system response time and the system error percentage were measured for each application. The response
time of the system is the time elapsed once the control signal has been activated and the corresponding
operation has been executed. Figure 16 shows the system response time for 16 different commands. The
average system response time of the 16 commands executed was measured to be 2.95 seconds.
693
26th International Conference on CAD/CAM, Robotics and Factories of the Future 2011
26th-28th July 2011, Kuala Lumpur, Malaysia
The main objective of calculating the system error percentage is to determine whether the command
transmitted is the command executed. The experiment transmits a series of 20 consecutive ON/OFF
commands using the X10 transmitter which are recorded as the transmitted commands and after the
response time of 1.65 seconds elapses, the command transmitted is cross checked with the command
executed. If the command transmitted and the command executed differs then an error is registered. The red
dots in Figure 17 are commands showing an error which has a response time of over 1.65 seconds. Hence
the control system does not execute this command and the next transmitted command gets executed. The
error percentage of the IRIS control system is calculated as 15%, shown in Table 3. Note that the X10
transmitter/receiver device has no delay in transmission unlike the NXT robot.
694
26th International Conference on CAD/CAM, Robotics and Factories of the Future 2011
26th-28th July 2011, Kuala Lumpur, Malaysia
7. CONCLUSION
The AForge.NET framework used to develop the IRIS control system is a useful tool in software
development since it is an open source framework with the available library files for image processing and
the integration of the NXT robot and the X10 transmitter. However, the sequence of filters chosen using the
AForge.NET framework proved to produce a processing delay since the image is being processed
individually by the three filters, thus, making the IRIS control system prone to errors. Hence, through this
paper, the performance of a low-cost real-time eye tracking system has been tested using the AForge.NET
framework in the field of wireless robotic remote navigation and wireless home automation. The test results
yielded an average response time of 1.65 seconds along with a system error percentage of 15%. The IRIS
control system can be further extended to applications in robotic wheelchair control, computer mouse
control and gaming, mobile phone applications such as making phone calls and sending/receiving text
messages. With the rapid accretion and extensive research in the field of Human Computer Interaction and
Assistive Technology, eye tracking will soon be as common and easily available as mobile phones or
computers.
8. REFERENCES
[8] M. Betke, J. Gips, and P. Fleming, ―The camera mouse: Visual tracking of body features to provide
computer access for people with severe disabilities‖, IEEE Transactions on Neural Systems and
Rehabilitation Engineering, vol. 10:1, pp 1–10, March, 2002.
[9] J.J. Magee, M.R. Scott, B.N. Waber, and M. Betke, ―Eyekeys: A real-time vision interface based on
gaze detection from a low-grade video camera‖, Proceedings of the IEEE Workshop on Real-Time
Vision for Human-Computer Interaction, Washington D.C., July 2004.
[10] N. Garay, I. Cearreta, J.M. Lopez and I. Fajardo, ―Assistive Technology and Effective Mediation‖, An
Interdisciplinary Journal on Humans in ICT Environments, vol. 2:1, pp 55-83, April 2006.
[11] S. Azam, A. Khan and M.S.H. Khiyal, ―Design and Implementation of a Human Computer Interface
Tracking System based on Multiple Eye Features‖, Journal of Theoretical and Applied Information
Technology, vol. 9:2, November, 2009.
[12] Tobii X60 & X120 Eye Trackers. Internet:
[13] http://www.tobii.com/scientific_research/products_services/eye_tracking_hardware/tobii_x120_eye_tr
acker.aspx [Accessed: 10 May 2011]
[14] J.D. Smith, R. Vertegaal and C. Sohn, ―View Pointer: lightweight calibration-free eye tracking for
ubiquitous hands-free deixis‖, Proceedings of the 18th annual ACM symposium on User interface
software and technology, Seattle, WA, USA, October 23-26, 2005.
[15] A.T. Duchowski, Eye Tracking Methodology – Theory and Practice, 1st edition. Great Britain:
Springer-Verlag London Limited, 2003.
[16] M. Schiessl, S. Duda, A. Tholke and R. Fischer, ―Eye tracking and its application in usability and
media research‖, Eye Square, Berlin.
[17] Jacob, R. J. K. and K. S. Karn, ―Eye Tracking in Human-Computer Interaction and Usability Research:
Ready to Deliver the Promises‖, The Mind's eye: Cognitive and Applied Aspects of Eye Movement
Research, J. Hyona, R. Radach, and H. Deubel, Editors. Elsevier Science: Amsterdam. pp. 573--605,
2003.
[18] P.J. Murphy, A.L. Duncan, A.J. Glennie, P.C. Knox, ―The Effect of Scleral Search Coil Lens wear on
the Eye‖, Br J Ophthalmol, vol. 85, pp 332–335, 2001.
[19] R. Barea, L. Boquete, M. Mazo, and E. L´opez, ―System for Assisted Mobility using Eye Movements
based on Electrooculography,‖ IEEE Transactions in Neural System and Rehabilitation Engineering,
vol. 10, pp. 209–218, 2002.
[20] R. Barea, L.Boquete, L.M Bergasa, E. Lopez & M. Mazo, ―Electro-Oculography Guidance of a
wheelchair using eye movement‘s codification‖, International journal of Robotic Research, 2003.
[21] E. Vitte, A. Sémont, ―Assessment of Vestibular Function by Videonystagmoscopy‖, J Vestib Res, vol.
5(4), pp 1-7, 1995.
[22] Gans, E. Richard, ―Video-Oculography: A new diagnostic technology for vestibular patients‖, Hearing
Journal, Vol. 54(5), pp 40, 42, May 2001.
[23] C. Ehmke and S. Wilson, (2007) ―Identifying web usability problems from eye-tracking data‖,
Proceedings of the 21st British HCI Group Annual Conference on People and Computers, vol. 1, pp
119–128, 2007.
[24] H. Yanco, ―Wheelesley: A robotic wheelchair system: Indoor Navigation and User Interface,‖ in
Assistive Technology and Artificial Intelligence—Application in Robotics, User Interfaces and Natural
Language Processing, Eds. Heidelberg, Germany: Springer, pp. 256–286, 1998.
695
26th International Conference on CAD/CAM, Robotics and Factories of the Future 2011
26th-28th July 2011, Kuala Lumpur, Malaysia
[25] D. Bonino and F. Corno. Dogont, ―An ontology modeling for intelligent domotic environments‖. In 7th
International Semantic Web Conference, 2008.
[26] ―X10 Transmission Theory‖, Internet: http://www.x10.com/support/technology1.htm , [April 28, 2011]
[27] ―The-EyeWriter.pdf‖, Internet: http://fffff.at/eyewriter/The-EyeWriter.pdf [April 28, 2011].
[28] Goldberg, H. J., & Wichansky, A. M. (2003). Eye tracking in usability evaluation: A practitioner‘s
guide. In J. Hyönä, R. Radach, & H. Deubel (Eds.), The mind's eye: Cognitive and applied aspects of
eye movement research (pp. 493-516). Amsterdam: Elsevier.
[29] J.S. Babcock and J.B. Pelz, ―Building a lightweight eyetracking headgear‖, Rochester Institute of
Technology.
[30] Lego Mindstorm. Internet: http://shop.lego.com/Product/?p=9841 [Accessed: 2nd August 2010]
[31] ―AForge.Net: Framework‘s Documentation‖, Internet:
[32] http://www.aforgenet.com/framework/documentation.html [April 28, 2011].
696