Signature Redacted
Signature Redacted
Signature Redacted
by
at the
June 2016
Signature redacted
Author:
Department of Mechanical Engineering
May 6, 2016
Signature redacted
Accepted By:
Anette Hosoi
________EINSTITUTEProfessor of Mechanical Engineering
MASSACHES NSTITUTE Undergraduate Officer
OF TECHNOLOGY
I
JUL 0 8201
LIBRARIES
A Robotic Hand that Utilizes Ergonomic Evaluation as
Feedback to Improve Human Robot Collaboration in
Soldering Applications
by
Abstract
People never seem to have enough hands. There are many tools that aim to address
this challenge, ranging from the ubiquitous benchtop vise to the "helping hands"
commonly used for soldering. However, these tools do not measure up to their human
counterparts. They cannot adjust the position or orientation of the workpiece to suit
a particular task which can cause workers to maintain unhealthy postures that are
detrimental to their long-term health. This thesis addresses this shortcoming with a
robotic arm that utilizes a gripper to grasp and hold a workpiece during a soldering
task. The robot uses a Microsoft Kinect sensor to continuously analyze the posture
of the human worker and calculate a score based on the RULA (Rapid Upper Limb
Assessment), an objective measure used in the ergonomics field to evaluate ergonomic
working postures. The robot adjusts the workpiece in order to optimize the RULA
score using an adaptive simulated annealing algorithm to balance the exploration
and exploitation phases of the optimization process. Initial testing indicates that the
robot can consistently find positions which improve the RULA ranking by 24.6% of
the measured range. This project demonstrates that human robot collaboration can
be improved by utilizing sensors to evaluate the needs of a human partner and adjust
the robot behavior accordingly.
Thesis Supervisor: H. Harry Asada
Title: Ford Professor of Engineering
3
4
Acknowledgements
I would like to first thank my mother, and all of my siblings for always believing in
me and encouraging me, first to attend college, and then to transfer to MIT. Without
all of your love and support, critiquing of countless resumes, and emergency editing
of application essays, I would never be where I am today.
I would also like to express my appreciation to Professor Asada for his mentorship
and guidance over the past two years during which I was a part of his research group.
Professor Asada sparked my first interest in robotics when I was a student in his class,
and then provided me the opportunity to conduct the research for this project, which
ultimately led to my decision to continue in this field in graduate school. I would
also like to thank Faye Wu who first introduced me to the nuts and bolts of robotics
research during our work on her Supernumerary Robotic Fingers, and then continued
to provide advice and direction throughout this thesis project. Our weekly meetings
were invaluable not only for this work, but also for providing me a framework for
conducting research upon which I will surely rely in the future.
I am grateful to Aaron Ross whose expertise and assistance was critical for the
ergonomics aspect of this work. Thank you to Ayelet Lobel for being at my side
throughout my years at MIT for always being there to help and support me, even
when my work seemed to leave little time for anyone else, and for reading this thesis
(in its entirety!) and providing insightful and valuable comments.
Finally, I'd like to extend my heartfelt gratitude to all of the amazing faculty, staff,
administration, and students, of the Massachusetts Institute of Technology. Thank
you for welcoming me with open arms when I transferred in my sophomore year, for
providing me with an environment of innovation and invention beyond my wildest
expectations, and for allowing me to join a community of some of the kindest and
most supportive people that I have ever met. I Have Truly Found Paradise.
5
6
Contents
1 Introduction 13
1.1 Background and Prior Work . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Kinematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2.3 Workspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2.5 Gripper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
7
3.2 Kinect Sensor . . . . . . . . . . . . . . . . . . . . . . . .3 31
5 System Architecture 39
5.1 Putting it all Together . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.2 The Windows PC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.3 The Arduino Microcontroller . . . . . . . . . . . . . . . . . . . . . . . 40
5.4 The ROS Computer . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
Bibliography 48
8
List of Figures
hum an workers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.1 The robotic arm. A serial linkage composed of a uiversal joint at the
shoulder, revolute joints at the elbow and wrist, and a gripper at the
2.5 The BNC connector circled in red was added to the soldering station
u n it. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.1 A hardware schematic showing how the various sensors and actuators
com municated. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
9
5.2 ROS node graph. Each ellipse represents a ROS node or module while
the rectangles show the inputs and outputs. . . . . . . . . . . . . . . 42
10
List of Tables
mended in [11. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.1 Tuned parameter values for the adaptive simulated annealing optimiza-
tion algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
11
12
Chapter 1
Introduction
Robots have become popular in the manufacturing field for their ability to perform
repetitive or hazardous tasks with precision and reliability. However, these robots are
typically unable to collaborate with human workers in close range as they lack the
human ability to infer the intentions of others. Figure 1.1 shows a line of industrial
robots at work in a BMW production plant.- The conspicuous gates prevent human
workers from wandering into the robot workspace and suffering injury. More recently,
innovations in controls using series elastic actuators has produced a number of indus-
trial robots that are intended to be safe to work near humans [2]. One example is the
Baxter robot made by Rethink Robotics (Figure 1.2). It aims to provide manufactur-
ing facilities more flexibility by allowing Baxter to work near humans [3]. However,
Baxter is intended to be trained through demonstration to complete repetitive tasks.
It does not actively collaborate with human workers during the task execution.
This thesis examines soldering as a case study for human-robot collaboration. Hu-
mans are not well suited to soldering alone, which usually involves holding multiple
13
CMt
LANI
AA
-ANN
;P0
Figure 1.2: The Baxter robot created l) vRethink Robotics can safely
operate near huiman workers[3]. Its series elastic actuators and tor(lue
control allow it to stop iupon making unexpected contact with human
without causing injry. 15
14
components in precise relative positions while handling a dangerously hot soldering
iron. On the other hand, while machines for soldering components do exist, they
typically require a substantial amount of time for programming and set up. Con-
sequently, humans are still required to perform soldering tasks when the number of
parts needed is not large enough to justify setting up an autonomous system. In this
thesis, a method is proposed to allow a robot to collaborate with a worker to com-
plete a soldering task. The robot monitors the worker's posture in order to choose a
suitable position for the workpiece without any explicit direction from the worker.
15
performance metrics such as speed at which the human is working or the quality of
of the finished parts. Allowing a robot to infer the quality of its job performance in
real-time and utilize this feedback to adjust its actions makes for a more collaborative
and successful partnership between the human and the robot. This could combine
the benefits of the precision, reliability, and safety of a robot with the flexibility of a
human operator allowing humans and robots to collaborate on a variety of tasks and
thus improve both the productivity and safety of human workers.
The remainder of this thesis describes the project using a bottom-up approach. The
individual components described first, followed by the system level architecture, and
finally the results and the conclusions that were reached.
In Chapter 2 the design and construction of robotic device is described along
with the analysis and motivations behind the various design choices. In Chapter
3 the method by which the robot quantifies and evaluates the human posture is
discussed along with the setup of the Kinect sensor which provides the data. In
Chapter 4 the simulated annealing optimization algorithm is described. The reason
this algorithm was chosen is discussed and the various design choices used in this
particular implementation are explained. Finally, in Chapter 5 we transition to the
system level architecture. The hardware communication setup including how the
various computational components, sensors, and actuators communicate is described.
Additonally, the inputs, outputs, and functionality of the various software modules are
presented. Chapter 6 gives the results of some tests that were performed to evaluate
the functionality of the system and we wrap up with Chapter 7 which provides some
suggestions for future work and areas for improvement.
16
Chapter 2
The robotic arm was designed to be as simple as possible while still achieving the
minimum required functionality. The arm contains five degrees of freedom (DOF).
Four are used to choose the position and orientation of the end-effector, while the
last actuates the gripper. Figure 2.1 shows how the serial linkage was constructed
in a manner that is similar to that of a human arm: two DOF for a universal joint
at the shoulder, one DOF for the elbow and the last allowing the wrist to rotate.
While future iterations could certainly increase the flexibility of the robotic arm by
adding two more revolute joints at or near the wrist, which would increase the DOF
to six and allow for an arbitrary position and orientation of the end-effector, for this
project, it was determined that four kinematic DOF would suffice. These allow the
end-effector to be placed at any position in the workspace using the first three DOF
while the wrist rotation is primarily used to choose which surface of the workpiece is
facing up.
Each of the first three DOF was actuated using a Robotis Dynamixel servo. The
17
Figure 2.1: The robotic arm. A serial linkage composed of a uiversal
joint at the shoulder , revolute joints at the elbow and wrist , and a gripper
at the end-effector.
18
first two actuators in the shoulder joint were the MX-106 model which can supply a
maximum torque of 10 N.m. These are mounted so that their axes of rotation crossed
perpendicularly at a single point providing a universal joint. The center of rotation
of this joint is also on the longitudinal axis of the upper arm link which simplifies the
kinematics. Since these actuators are fully supported by the structure, their weight
was not a concern. For the elbow joint, the weight of the actuator must be borne
by the shoulder joint with a moment arm equal to the length of the upper-arm link.
Therefore, a smaller actuator, the MX-64 model weighing 153g, was used in order to
decrease the load. The final two actuators, one at the wrist, and one on the gripper
were both Hitech HS-422 servos. These servos are much smaller, weighing only 45g.
Although they can only provide 0.3 N-m of torque, the torque requirements of these
joints are small because the load at the end-effector is transmitted by the structure
to the elbow and shoulder joints.
2.2 Kinematics
The robotic arm described above can be parameterized with two parameters, L 1
and L 2 , the lengths of the upper-arm and forearm respectively. Figure 2.2 shows a
schematic view of the kinematics of the robotic arm. The forward kinematics solution
was calculated to in order to determine the position of the end effector p = (x, y, z)T
measured with respect to a fixed frame whose origin is at the axis of rotation of the
shoulder joint as a function of the three joint angles q = (01, 02, 0 3 )T.
19
L
-
01L
01
20
X cos 02 cos 01, cos (02 + 03) cos 01
L,
p zY sin 02, sin (92 + 93)
L2
cos 2 sin 01, cos (92 + 03) sin 01
The jacobian was calculated in order to find the workspace of the robot parameterized
by the link lengths. This aided in designing the links to achieve the desired workspace.
The Jacobian was found from the previous forward kinematics relationship. For
brevity, let
Ci = cos (0) , Cij = cos ( 1 +93
)
Si = sin (0j) , Sij = sin (i +93
)
then
-L 1 C 2 S1 - L2C23S1, -L 1 S2 C - L23C1, -L 2S 23 C1
dp
Jdq 01 L 1 C 2 + L2 C23 L2C23
,
93 = n7r, n E Z
Thus the reachable workspace for the robot is independent of 01 and 92 (it is
spherically symmetric) and is singular when 93 is either fully extended or fully folded
21
back which encloses the space between two spherical shells with radii L1 + L 2 and
L, - L 2 respectively.
2.2.3 Workspace
As seen from the jacobian the robot has a workspace enclosed between two spherical
shells. However, for the purpose of collaborating with a human worker, the required
interaction space was chosen as a rectangular region centered in front of the person.
Figure 2.3 shows the bounds of the interaction space which ensured that the robotic
arm would not invade the personal space of the worker, collide with the work table,
or move too far to the sides to be within human reach. The link lengths were chosen
to be L, = L2 = 23 cm in order to ensure that the required interaction space was
fully contained within the workspace of the arm. These lengths are 2 cm longer than
strictly required for reach in order to keep the interaction space free from singularities.
The inverse kinematics solution was required in order to control the position of the
gripper in cartesian space. It is expect that the worker would react to changes in
this space, for example to prefer certain regions of cartesian space for a particular
task. Therefore, all of the control and optimization of the robot positioning was
done in cartesian coordinates and the inverse kinematics was used as the last step
to transform a desired destination point into a joint configuration command. The
inverse kinematics was calculated from the geometry by first calculating the following
intermediate values:
L= Vx
2
+ y2 + z 2
22
Figure 2.3: Designing the robot workspace. The red-shaded rectangular
region is the space in which the human will interact with the robot when
seated at the work table. The robot links were designed to ensure that
this space is fully enclosed within the spherical workspace of the robotic
arm.
23
a = tan-
2
b = bcos- _ (Li1 - Li+ L
)
+
C = 7r - Cos-
_ (L2 1 +LL2 - L 2
2LL 2
Then the inverse kinemat ics has two physically unique solutions. The elbow up
solution is:
(1 (tan-' (2)
q 902 a+b
0~\3} -c
01 tan-' ()
)
q= 02 a- b
(3) C
The entire interaction space is accessible with the elbow-up solution alone, whereas
the elbow-down solution could collide with the worktable. Therefore, the elbow-up
solution was used exclusively throughout this project. Furthermore, the solutions
which involve, for example, rotating 01 by 1800 are physically equivalent to the pre-
vious solutions because the links are symmetric so they were not considered.
2.2.5 Gripper
A Lynxmotion parallel gripper as shown in Figure 2.4 was attached at the end-effector
of the robot arm. This gripper was chosen because the parallel motion of the fingers
is ideal for grasping the flat circuit boards used for soldering. Furthermore, the rack-
and-pinion mechanism reduces the back-drive load on the servo. Finally, the gripper
24
Figure 2.4: A Lynxnotion piarallel gripper was attached at the end-
effector of the robotic arm for grasping the workpiece.
also includes aln attachnient for the rotating wrist servo which allows illounlting both
the wrist rotation and gripper actuators at, a single gripper attachnient point.
A Parallax Ultrasonic Distance sensor was nlited on the gripper in order to sense
the presence of a workpiece. This sensor provides non-contact neasnreiment fron 2 (Iii
to 3ni bY ineasnring tine between pings. The distance is calculated using d = 2vt
where the c is the distance to the reflecting object. c = 343 m is the speed of souind in
25
air, and t is the duration between the sending of a ping, and its detection. In order to
convert this measurement into a binary value indicating the presence of a workpiece,
a threshold was used. Since the distance to the gripper was 3 cm from the sensor,
a threshold of 5 cm was found to be large enough to guarantee detection, while still
small enough to avoid false positives.
The solder stand sensor is a binary sensor for determining if the soldering iron is
currently in the soldering stand. In order to detect this information, the soldering
stand was connected to an Arduino microcontroller board through a 10 kQ resistor
pulled up to 5.0V. In order to meet Electrostatic Discharge safety requirements, sol-
dering irons typically have the tip of the iron connected to ground. This provided a
convenient method for detecting the location of the iron without requiring the attach-
ment of any additional sensors. When the soldering iron contacted the conductive
aluminum stand, it grounds the stand and thus pulls the Arduino pin down. This
information was then transmitted to the main robot computer over the serial connec-
tion. Since the wire brush used to clean the solder tip was also conductive, cleaning
the tip could also be detected using this sensor. This provided the robot with a way
of detecting that the worker was taking a break to clean the soldering iron between
joints, which was used by the robot as a signal that it was a good opportunity to
adjust the position if necessary. The only caveat was that the soldering stand used
had a protective coating which was not conductive and thus had to be removed at the
junction between parts in order to ensure a reliable electrical connection throughout.
26
2.3.3 Temperature Sensor
One of the additional applications explored was the ability for the robot to assist
with soldering by feeding solder wire through a tube directly onto the tip of the hot
iron. Since it would be undesirable for the robot to attempt to feed solder onto a
cold iron, a temperature sensor was used to allow the robot to know when the iron
was at a hot enough temperature to begin tinning the tip in preparation for soldering
a joint. In order to determine the temperature of the soldering iron tip, the voltage
across a K-type thermocouple was measured since the soldering iron already had such
a thermocouple embedded in the iron tip in order to keep the temperature at the
desired setting. The terminals for the K-type thermocouple were simply connected
in parallel to the main soldering unit control board, and a separate amplifier was
used to increase this voltage from the millivolt range at which it operates to the
Arduino range of 0 -5 V. Figure 2.5 shows the soldering station with the added BNC
connector circled in red. This connector allowed the robot to determine the soldering
iron temperature in parallel with the soldering unit without the need for an additional
temperature sensor.
27
Figure 2.5: The BNC connector circled in redi was added to the sol-
dering station unit. It was connected in piarallel to the existing K-ty'pe
thermnocouple einibedded in the tip of the soldering iron. This allowed the
teiperature of the soldering iron to lbe neasnred without the need for an
additional sensor.
28
Chapter 3
In order to evaluate the posture of the worker to detect positions that are healthy
and comfortable, the robot continuously performs a Rapid Upper Limb Assessment
(RULA), a standard assessment in the ergonomics industry[1]. The RULA assigns a
score between 1 and 7 based on the positions of the upper limbs of a worker during
a task. While there exists a more general form of assessment known as the Rapid
Entire Body Assessment (REBA) 111], which includes more joints, the RULA was
used in this case because the worker is presumed to be sitting at a workbench and
thus only the upper limbs are affected.
The RULA assesses two primary areas of upper limb ergonomic health: Group A
includes the arms and wrists while Group B includes the neck and trunk. The scores
in these two groups are first computed individually based on the relative joint angles.
Figure 3.1 shows, for example, how the subscore for the upper arm is calculated. The
lowest score of 1 is assigned to joint angles which involve minimal ergonomic risk with
increasing scores indidcating higher risk positions. Additional criteria are also taken
29
+1 +2
+~
+2
+~ Q 45-90'
901-
+3 +4
20* 200 200 20-450
Figure 3.1: Assessing a BULA subscore for the upper arm 112]. The
suibscore for the upper arm is calculated by measuring the joint angle in
the sagittal plane. The lowest score of 1 is assigned to the joint position
with minimal ergonomic risk.
into account such as a heavy load or repeated motions. Once the score in the two
groups is computed, they are combined to form a single holistic score in the range
1-7. Table 3.1 gives the meaning for the scores as described in [fl.
For this project, the complete RULA score was not measured because some joint
angles such as the wrists proved difficult to obtain visually do to occlusion by the
soldering tool, Additionally, some of the other measurements were not applicable
such as the loading force which we assume is small for soldering applications. The
parameters that were measured included: neck angle, trunk angle, right and left upper
arm angles, right and left elbow angles, as well as right and left, shoulder positions with
respect to the base of the neck. These measurements were used to compute a subset
of the RULA score which ranged from 1-4. Since this subset has a different range
than the complete RULA score it is used only for comparing observed postures on a
relative basis and cannot be used to infer the the true total score. Therefore, while
we cannot determine based on the subset measured into which range in Table 3.1 a
given posture would fall, we can compare two postures and determine which one has
a lower overall score and by improving the components of the score which the robot,
could observe, infer that this would increase the total score as well. Furthermore,
subjective testing showed that more comfortable postures tended to get better scores
30
RULA Score Range Required Action
as measured in this subset, while less comfortable postures that required excessive
leaning or twisting also received higher (worse) scores in the measured subset of the
RULA.
In order to measure the required joint angles to compute the RULA score, a Microsoft
Kinect sensor was used to monitor the human worker. The viability of using a Kinect
sensor to estimate ergonomic information was investigated in [9, 10 they find that
the ideal location for the Kinect is at an angle to the subject between 200 - 45'.
As shown in Figure 3.2 the Kinect was placed at a distance of 1.5 m in front of the
worker's position and at an angle of 20'. The angle also helped ensure the sensor
obtained an unobstructed view of the worker's upper body despite the robotic arm
mounted directly in front of the worker. The transform between the camera frame
and the base frame for the robotic arm baseT*ineet was measured in order to transform
31
1.5m
20
Figure 3.2: Position of the Kinect Sensor with respect to the worker.
32
Chapter 4
Workpiece Position
Using the RULA score for worker's posture obtained in real time as described in
the previous chapter, the problem of finding the optimal position for the workpiece
can be reduced to a constrained optimization problem where the robot must find
the position that minimizes the objective function f(x) which gives the RULA score
over the parameter x = (x, y, z)T which represents the position of the end-effector in
Cartesian coordinates. Formally:
where I and u are vectors that define the lower and upper limits of the interaction space
in which the robot and human operate. The values were x = (-0.2, -0.2, 0.05)T m
33
and x = (0.05, 0.2, 0.3)T m as shown by the red shaed region in Figure 2.3.
There are many choices of algorithms for finding solutions to optimization prob-
lems such as this and a detailed treatment of the subject is beyond the scope of this
thesis. However, there are certain features unique to this problem which motivated
the particular solution used here. One key point is the expense of sampling the ob-
jective function. While many algorithms assume hundreds or thousands of points can
be sampled in order to find a very accurate solution, in this problem, obtaining each
sample requires that the human worker complete a task at a given position and that
the RULA score for the posture during that time be calculated. Thus, it is infeasible
to require a large number of samples before converging to an optimal solution. This
also highlights another concern that is especially important for this application: the
tradeoff between exploration and exploitation. Once some information is known, the
benefit of exploiting that knowledge to increase the likelihood of choosing a good
position for the next task must be balanced with the possibility that exploring new
areas of the parameter space might lead to the discovery of an even better solution.
One final consideration is our prior knowledge of the qualities of the objective func-
tion. Since the human has many more degrees of freedom than the three dimensional
space of the parameter, we expect there to be many possible human postures for a
given workpiece position. Furthermore, the posture could switch between solutions
at discrete points such as changing from an elbow-down to elbow-out posture once
the joint limit is reached. Therefore, there is no expectation that the function will
be smooth, or even continuous which makes it more difficult to use gradient based
methods.
34
4.2 The Simulated Annealing Algorithm
eters
For the simulated annealing algorithm to work for this application a number of choices
and parameters had to be chosen. The acceptance function used is a commonly used
function known as the Metropolis acceptance function which is defined as follows:
False, othewise
where f is the previous value of the objective function. fi 1 is the candidate value at
the new sample point, T is the current temperature, and rand () is a random variable
drawn from a uniform disribution on the interval [0, 1].
35
Set parameters
Accept? N
es
No Better than
revu optim?
Ml
vious 0es
r <
Update Optimal Value
Exploration No
limit reached?
tYes
End
36
4.3.2 The Adaptive Step Function
For the adaptive step function, the step size was applied to each of the three coordinate
axes independently allowing the step to shrink faster along axes which were moving
quickly and more slowly along axes that did not have many accepted samples 1131.
The step size is as a diagonal matrix D with with the current step sizes for each axis
along the diagonal entries. A new sample point is generated as follows:
xj+1 = xi + Dv
Dj+1 = (1 - a) Di + awR
where a is a parameter that determines how strongly the successful sample should
affect the step size, R is a diagonal matrix where the diagonal elements are taken
from the vector xj+ 1 - xi, and w is a scaling factor appropriate for R. The initial
step size Do was set with the diagonal values taken from the vector #3(x - x) such
that f is a constant that gives the proportion of the initial step size to the limits of
the interaction space defined previously in Section 4.1.
The cooling schedule determines the rate at which the algorithm settles on an optimal
value, if it cools too quickly, the algorithm could get stuck and converge to a local
minimum, while if it cools too slowly, it effectively becomes a random search which
37
Parameter Symbol Value
Initial Temperature TO 1
Step Size Rate a 0.1
Step Size Adjustment Ratio w 1.5
Cooling Schedule Ratio r 0.9
Initial Step Size Ratio / 0.75
Table 4.1: Tuned parameter values for the adaptive simulated annealing
optimization algorithm.
Ti = rT
In order to choose approprate values for the parameters described above, simulation
was used to test the algorithm on various functions. There is always a tradeoff between
the quality of the result and the number of iterations for which the algorithm runs.
Since each iteration required the completion of a soldering task, twenty was chosen as
the maxium allowable number of iterations for the alogrithm to converge. With this
goal, the parameters were tuned in sumlation to find the values that would give the
testing phase consisiting of five runs with twenty samples in each run was then used
to further fine tune the parameters. The resulting tuned values for these parameters
38
Chapter 5
System Architecture
The hardware schematic in Figure 5.1 shows how the various system components
communicated. There are three computational units: 1) The windows PC used ex-
clusively for managing the Microsoft Kinect sensor 2) The Arduino microcontroller
which performed low-level management of the various sensors and the wrist actua-
tors, and 3) The main Ubuntu PC running the Robot Operating System (ROS). The
The Windows PC serves as the processing unit for the Microsoft Kinect. The Kinect
sensor is compatible with the Natural User Interface (NUI) library provided by Mi-
crosoft which is only compatible with a PC running Microsoft Windows. Since ROS
is only compatible with UNIX based operating systems, this was the primary motiva-
tion for using two computers. However, since the Kinect sensor collects a very large
39
Proximity Temperature Solder Stand Kinect
Sensor Sensor Sensor Sensor
Arduino Windows PC
Microcontroller
Serial
TcP
(LAN)
ROS Computer
PWM PWM
Seil Serial
SSerial
Figure 5.1: A hardware schemiatic showinig how the variouis senisors anid
actulators conuniiiicated. The three comipu-tational units are shown as
ellipses while the sensors and actuators are rectangular. The directed
arrows dlenote communication and are labelled with the protocol used.
amount of data at a high rate, and processing this data to segment and detect human
joints in the images can be very processor intensive, there is a benefit to using a ded-
icated PC for this task. The resulting joint angles are streamed over the network as
a simple array of numbers which has very little overhead in comparison to the image
segmentation task.
particular, it monitors the voltage of the solder holder to detect the location of the
soldering iron, and sends a status update to the ROS computer over a USB serial
the analog voltage of the output of the K-Type thermocouple amiplifier. The voltage is
40
read using the built-in analog-to-digital converter of the Arduino Nano, converted to
degrees Celsius, and sent to the ROS computer at a rate of 10 Hz. Next, the proximity
sensor discussed above is also managed by the Arduino, which sends out pings lasting
5 pts, times the delay, calculates the distance measurement, and then transmitts it
to the ROS computer at a rate of 10 Hz. Finally, the Arduino is also responsible
for driving the two small HS-422 servos in the robot wrist. Upon receiving a joint
command from the ROS computer, the Arduino generates a Pulse Width Modulation
(PWM) signal to drive the servo to the desired position.
The ROS computer serves as the main controller for the robot. Figure 5.2 shows how
the various modules (ROS nodes) interact in order to achieve the desired behavior.
The sensors attached to the arduino provide their respective measurements directly
to the main Controller node. The Robot Joint State Publisher node polls the Dy-
namixel motors at a rate of 20 Hz to obtain joint angle measurements and makes that
information available to the Controller node as well. Next, the Human Skeleton State
Publisher listens over a TCP Socket for human sekeleton updates from the Windows
PC. Once an update arrives, it publishes a vector of Human Skeleton Joint Positions.
The RULA Score Evaluation node takes these skeleton joint positions as input and
evaluates a RULA score from these as described in Section 3.1. The RULA Time
Average Calculation node maintains an average RULA score over the course of a task
evaluation. Each time a task is begun, the average is reset in order to independently
measure the time average for a given robot position. Finally, this average score is pro-
vided to the controller which performs the adaptive simulated annealing optimization
in order to select the optimal choice for the robot position.
41
Solder Holder Dynamixel
Sensor State Kinect Sensor
Motors
Proximity
Sensor Human
Distance Robot Joint Skeleton
Positions Joint Positions
RULA Score
Evaluation
Controller
RULA Real
Time Score
Cartesian
Space Goal
RULA Average
Score
Configuration
Space Goal
Robot Joint
Controller
Figure 5.2: ROS node graph. Each ellipse represents a ROS node or
imodule while the rectangles show the inputs and outputs.
42
Chapter 6
In order to test the system performance using a typical application, the task of solder-
ing a 20 pin header to a circuit board was performed for a total of 11 runs. Each run
was independent in that no information was saved between runs. However, within
each run, the state was saved after each joint was completed allowing the system to
optimize the position over the course of twenty samples. Twenty samples was cho-
sen as a quantity small enough that it would not be overly cumbersome to require
a worker to perform twenty joints as the robot attempted to find the optimal posi-
tion while still being large enough to reasonably expect the optimization algorithm
1. The robot chooses a random position in the workspace and waits with an open
2. The worker hands the circuit board to the robot which grasps it with the gripper.
3. The worker places a 20 pin header into the top of the circuit board.
43
4. The worker removes the solder from the holder which signals the robot to rotate
the circuit board to have the solder side face up.
5. The worker completes a joint by placing the iron on the pin and dispensing
molten solder.
6. The worker cleans the tip of the solder or places it back into the solder holder,
which signals the robot that it is an opportune time to consider a change of
position.
7. The robot measures the human's posture during the course of the previous task,
as well as its current position in the workspace and adds this sample point to
the optimization algorithm.
9. The robot moves to the new location if necessary and resets the posture score
to begin a new measurement.
10. The process is repeated until the header is complete at which point the data for
the posture score at every sample point is saved and the run is complete.
6.2 Results
Figure 6.1 shows the results of the 11 test runs as each of the colored lines. The
thick black line is the average RULA score at each sample number. As expected, the
individual runs behave differently with some finding optimal scores relatively quickly
while others explore for a longer number of samples. Furthermore, the average RULA
44
3.5
-- Average of 11 Test Runs
Q)
3-
1.5
5 10 15 20
Sample Points
Figure 6.1: Results of eleven test runs. The RULA score is shown on
the vertical axis while the horizontal axis denotes sample number. The
thick black line shows the average of the all eleven runs at each sample
point. This average decreases over the course of each run even though the
individual runs take different paths toward the lower RULA scores. This
is expected due to the random nature of the adaptive simulated annealing
optimization algorithm used.
score continues to decline as the number of samples increases. Notice that during the
first four samples, the average does increase a small amount. However at the beginning
that it isn't strictly decreasing in the earliest stages. The total change of the average,
amongst the 11 test runs was 0.74 RULA points. Since the subset of the RULA score
measured had a range of 3 RULA points (from 1 - 4), this yields a decrease of 24.7%
45
6.3 Discussion
The goal of this project was to build a robotic gripper that could better assist a
human partner through real-time feedback by evaluating the posture of the human
and assimilating that information into its decision making for the future. The previous
results demonstrate that such a strategy can sucessfully lead to an increase in the
quality of the assistance the robot provides without any explicit direction from the
human. It is anticipated that this concept could be applied to additional tasks where,
as long as a method is devised to allow the robot to evaluate a quality score of its
performance thus far, it could learn to increase that score using this method.
46
Chapter 7
One of the key components of this work was measuring the quality of a human worker's
posture and using it to evaluate the RULA score. This was accomplished solely using
the Microsoft Kinect sensor. While limiting the posture evaluation to this method
had the benefit of minimal interference with the worker, there were several drawbacks.
Firstly, only a subset of the full RULA score could be measured reliably due to the
visual occlusion that occurs at the wrist joints when a tool is used. Furthermore, even
those joints which are not blocked by the tool, could sometimes become occluded by
other limbs. Future work could incorporate additional sensors to solve the occlusion
problem. These could include additional cameras that view from another angle or
even other types of sensors such as Inertial Measurement Units or Strain sensors
that could be used in conjuction with, or in place of, the visual method. Secondly,
while the RULA is a standard score for ergonomic health, it could also be valuable to
measure more subjective metrics of the robot's performance by asking the human for
from an ergonomics perspective, it would also be beneficial for the robot to favor
actions that engender a positive feeling for its performance in the mind of the human
47
worker.
Additionally, while this project tested the specific task of soldering a printed circuit
board, future work could test this method in a variety of other scenarios including:
Theses are just some of the examples where a robotic arm with a standard gripper
could assist a human worker. If the robot is equipped with specialty tools, the pos-
sibilities are truly endless. For each task, the quantity measured would also differ.
For example, rather than measuring worker posture, the time taken for completion
or the quality of a finished workpiece could be measured. The robot would then use
this feedback to choose optimal strategies that maximize the score in those areas.
Finally, more testing is needed especially on untrained workers to quantify the
robot's performance amongst users who do not have the same level of experience
with robots as that of robotics researchers. A full series of experiments in which users
with comparable skill levels to a factory worker are compared when working with
the robot to a control group completing the same tasks alone is necessary in order
to quantify how much such a robot could assist human workers in the completion of
real-world tasks.
48
Bibliography
[1] L. McAtamney and E. N. Corlett, "RULA: A survey method for the investigation
of work related upper limb disorders," Applied Ergonomics, vol. 24, no. 2, pp. 91-
99, 1993.
teraction and Cooperative Robots', vol. 1, no. 1524, pp. 399-406, 1995.
14] BMW PressClub, "Bmw i3 production bmw plant leipzig: Cfrp body shop,"
2013. https://www.press.bmwgroup.com/global/photo/detail/P90127374/bmw-
i3-production-bmw-plant-leipzig-cfrp-body-shop-09-2013.
2007.
49
[7] R. Boboc, "Natural Interaction with an Assistive Humanoid Robot.," Applied
Mechanics & ... , 2015.
191 P. Plantard, E. Auvinet, A.-S. Pierres, and F. Multon, "Pose Estimation with
a Kinect for Ergonomic Studies: Evaluation of the Accuracy Using a Virtual
Mannequin," Sensors, vol. 15, no. 1, pp. 1785-1803, 2015.
[131 F. Busetti, "Simulated annealing overview," JP Morgan, Italy, no. 1, pp. 1-10,
2003.
50