Adaptive Industrial Robot Control For Designers: September 2017
Adaptive Industrial Robot Control For Designers: September 2017
See discussions, stats, and author profiles for this publication at: https://www.researchgate.
net/publication/319944935
Adaptive Industrial Robot Control for Designers
Conference Paper · September 2017
CITATIONS READS
2 948
3 authors:
Shani Sharif Varun Agrawal
Georgia Institute of Technology Georgia Institute of Technology
12 PUBLICATIONS 31 CITATIONS 4 PUBLICATIONS 69 CITATIONS
SEE PROFILE SEE PROFILE
Larry Sweet
Georgia Institute of Technology
24 PUBLICATIONS 774 CITATIONS
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
BIM for Masonry View project
All content following this page was uploaded by Shani Sharif on 20 September 2017.
The user has requested enhancement of the downloaded file.
Adaptive Industrial Robot Control for Designers
Shani Sharif1 , Varun Agrawal2 , Larry Sweet3
1,2,3
Georgia Institute of Technology
1,2
{shani|varunagrawal}@gatech.edu 3 larry.sweet@cc.gatech.edu
In this research, we present a system to allow designers to adaptively control an
industrial robot from within a 3D modeling environment, for the purpose of real
time feedback with respect to visual imagery of the object as well as robot pose
during the fabrication process. Our work uses the Kuka industrial robots due to
their capability in fabrication and programmability, and the Rhino 3D modeling
software with the Grasshopper plugin which allows for visual programming for
designers. A Microsoft Kinect sensor is used to provide visual feedback of the
part during the fabrication process. We present the methodology used to develop
the system, explaining various design and architecture choices made to allow for
easy use of our system, while ensuring our system is open to further extension. We
also show qualitative results of the fabrication process performed using our
system in order to validate that our proposed system improves the interaction and
collaboration between designer and robot when performing the task, in contrast
to the iterative process that is generally followed.
Keywords: Human-robot collaboration, Robotic fabrication, Adaptive control,
Feedback
INTRODUCTION inally developed for industrial mass production of
This research proposes a framework to enhance the components in factories. Robot path planning and
collaboration process between human designers and programming is completely engineered for a spe-
industrial robots. Recent technological advance- cific production process with predictable outcomes.
ments have led to the development of the new gen- The main criteria for these industrial robot systems
eration of industrial robots. Compared to earlier gen- is functional accuracy and repeatability over an ex-
erations, these robots are much more affordable, ac- tended operation time period. The developed and
curate, and highly flexible multi-purpose manipula- confirmed robot tool-path with these software sys-
tors. All these aspects have made these robots opti- tems runs for several months on a robot for the mass-
mal tools for creative and mass-customized architec- production of a unique product. On the other hand,
tural fabrication processes. Over the last decade, ar- the creative design and fabrication processes in ar-
chitects have adopted industrial robots for additive, chitecture are highly exploratory, meaning that ar-
subtractive and deformative fabrication processes. chitects need to solve complexities of new forms, ma-
However, the robot control and motion pro- terials, fabrication processes, and unpredictable con-
gramming software tools for these robots are all orig- struction environment. The solution for these cre-
FABRICATION - ROBOTICS - Volume 2 - eCAADe 35 | 151
Figure 1
Geometric
modeling, and
pobot path
planning and
simulation in visual
programming
environment
Figure 2
Failed attempts in
fabrication of the
module before the
final successful
results. Problem
with wrong
ative endeavors relies on a reciprocal investigation As the conventional methods of robot control and material placing,
between conceptual, digital and materials forms in motion programming were not developed based on singularities and
order to find the best possible solution for the design the needs and skills of designers, researchers in these unanticipated robot
and fabrication problem in hand. Consequently, for field have focused on the development of more flex- path optimization
the processes that use digital design and fabrication ible and intuitive robot control and programming
tools, it is essential that tools of these processes facil- tools. These new software tools have acquired graph-
itate the reciprocal design and fabrication develop- ical programming editors such as Grasshopper. Dif-
ment process. ferent plug-ins such as Kuka|prc (Braumann and Brell-
However, using the existing industrial robot con- Cokcan 2011), HAL (Schwartz 2013), and Scorpion
trol system for architectural fabrication processes re- (Elashry and Glynn 2014) for programming and kine-
quires that the designer have a comprehensive view matic simulation of industrial robots such as KUKA,
of the fabrication and machining process, and embed ABB, and Universal Robots have been developed for
all the required considerations in the digital model, Grasshopper and Dynamo. The plug-ins for graphi-
before the start of the fabrication phase (Sharif and cal robot programming provide the option for archi-
Gentry 2015). This is a one-directional workflow (Fig- tectural designers to program and simulate industrial
ure 1), in which the designer has to predict mate- robots directly out of the parametric modelling envi-
rial state, tool selection, fixture positioning and robot ronment based on the geometric parameters of their
motion planning based on prior experience (Figure designs. These tools provide interactive design and
2). This established workflow of robotic technology robot programming/simulation in the initial stages of
that performs adequately and effectively in produc- the process. However, most of these tools result in
tion manufacturing does not provide much room for a static robot control data file that has to be trans-
any interactive creative design/fabrication activity. ferred from the personal PC equipped with the robot
There is a high cost and time penalty for re-work in programming tool to the robot computer. After the
this process. robot path generation and export of the file, the con-
152 | eCAADe 35 - FABRICATION - ROBOTICS - Volume 2
nection between the design model and the robot sor systems (camera, and 3D or laser scanner) that en-
movement path is completely disconnected. By the able the accessible programming for defining robotic
start of the robotic fabrication process, the designer tool-paths. Robots such as LBR iwaa from Kuka,
would have no control over the process of fabrication Yumi from ABB, UR10 and UR5 from Universal Robots,
in real-time. In this process, the final physical proto- and Baxter and Saywer from Rethink Robotics are
type outcomes are still highly dependent on the ar- developed based on the principals of human-robot
chitect’s predictive capability of design, fabrication, interaction. These robots are usually lightweight
analytical and process models. and desktop size robots compared to their industrial
Consequently, this research proposes an inte- counterparts. And their major feature is that these
grated framework that transforms the current one- robots can perform in close proximity to, and collabo-
directional workflow of design-to-fabrication process ration with human workers (Shepherd and Buchstab
into a comprehensive closed-loop workflow. By us- 2014).
ing the proposed system, designers can monitor the The controller kernels of these robots is by de-
fabrication process and control the robot in real-time. sign modular and open interface that enables object-
The developed system provides the functionality for oriented programming of complex robot kinemat-
the users to modify the programmed robot tool-path ics. In addition, these interfaces make external con-
in real-time, due to material tolerances or dynamic or trol of the robots and integration of sensors sys-
unstructured environment conditions that vary from tems relatively straightforward. Taking advantage
the expected state. The robot tool-path may also be of these integrated features, researchers have devel-
required based on the designer’s decision for a differ- oped interactive robotic fabrication application us-
ent course of action during the fabrication process to ing interactive robots. Research by Elshary and Glynn
achieve the desired design and fabrication outcome. presents a developed robot control plug-in for Uni-
versal Robots in the Grasshopper environment. Scor-
BACKGROUND pion takes advantage of Java and Python object ori-
Robot manufactures produce robots for two major ented programming languages and their existing li-
target users, manufacturing industries, and research braries for programming and real-time visualization
applications and developments. While the mechan- of the robot tool-path and configuration (Elashry and
ical and hardware requirements of robots for both Glynn 2014). In another research project, researchers
these applications is similar, the software and control have developed a system for on-site robot program-
requirements are quite different. While the manu- ming that takes advantage of the embedded force
facturing industry requires higher safety features and torque sensors of Kuka LBR iiwa robot to facilitate
easy to use operational interfaces, researchers, in- human-robot collaboration and manage material tol-
cluding design researcher, prefer higher control over erances during the fabrication process (Stumm et al.
the robot. In the last few years, robot manufacturers 2016).
are investing more on the development of research While the collaborative robots provide great fea-
robots, also known as collaborative robots. tures for researchers, these robots have lower pay-
Collaborative robotics is a new research and de- loads. Robot payload is the maximum weight that
velopment approach in robotic industry. By design, a robot can pick up or manipulate. Robot payload
collaborative robots are equipped with communica- is specifically important for architectural fabrication
tion interfaces that provide accessible external con- processes, as the robots needs to handle heavy con-
trol to these robots. These robots are provided with struction materials such as concrete, metal or stone,
new interfaces and internal sensor (motor torques and apply high force for processes such as milling
and joint torque sensors) and access to external sen- or lifting and assembly. While collaborative robots
FABRICATION - ROBOTICS - Volume 2 - eCAADe 35 | 153
have about 5-10 Kg payloads, industrial robots - with lege, Norway (Sanfilippo et al. 2015). JOpenShow-
medium payloads from 50 to 150 Kg and high pay- Var is a custom designed program which connects
loads from 250 to 600 Kg - are more suitable for ar- to the Kuka controller from a remote computer via
chitectural construction processes. However, as dis- TCP/IP, without using Kuka software packages, such
cussed above these robots are not equipped with RSI (Robot Sensor Interface) or Ethernet.XML. This
flexible programming and control interfaces. In order system uses KUKAVARPROXY, a server developed in
to make high payload industrial robots suitable for re- Visual Basic to implement the Kuka CrossComm in-
search or design fabrication applications, it is neces- terface which allows for the interaction with the real-
sary to develop custom communication interfaces to time control process of the robot. JOpenShowVar
provide required flexibility for the control of robots in which is written in Java runs as a client on a remote
such applications. computer connected with the Kuka controller via
TCP/IP. The Java based platform allows for high-level
ADAPTIVE CONTROL INTERFACES FOR IN- programming and use of third party libraries. How-
DUSTRIAL ROBOTS ever, as discussed by the authors, when accessed
Focusing on Kuka industrial robots, these robots with TCP/IP communication to the KUKAVARPROXY
are programmed with Kuka’s own programming lan- server, it creates unavoidable delays in the real-time
guage, KRL (Kuka Robot Language) (2003). KRL is a access to the robot’s data, thus it can only be used
text-based language that contains tool and machine for soft real-time applications. In addition, as this sys-
movement commands as well as data type declara- tem is based on Java, it makes if difficult to work with
tions, conditional clauses and interaction with tools C#.NET components in Grasshopper and Rhino .NET
and sensors via digital or analog Input/Output (I/O) SDK (Software Developers Kit) that is the target appli-
operations. KRL programming is usually performed cation for creative industry.
offline via an external computer with software tools Another interface that is currently available for
such as KUKA.Sim Pro, or Kuka|PRC with a visual pro- Kuka robots is mxAutomation which allows for real-
gramming interface. While KRL features are adequate time communication with Kuka industrial robots
for industrial applications, it has limited capabilities (Munz, Braumann, and Brell-Cokcan 2016, Braumann
for adaptive control: no support for advanced math- and Brell-Cokcan 2015). The mxAutomation soft-
ematical operations or including third party libraries ware package, which has been created in collabora-
to extend KRL with user defined methods or object tion with Siemens, has two main parts, a server pro-
(Sanfilippo et al. 2015). In addition, in order to use gram that runs on Kuka robot controller (KRC), and a
external input devices such as sensors, it is neces- robot control program with a client library that runs
sary to install supplementary software packages such on a remote computer. These two parts of the sys-
as Kuka.RobotSensorInterface (RSI) or Kuka.Ethernet tem are connected via either field buses EtherCAT or
KRL XML. UDP-Ethernet. The authors have developed a custom
Various research groups have investigated creat- client software that connects the mxAutomation li-
ing custom communication interfaces to act as mid- brary with Robots in Architecture’ s KUKA|prc frame-
dleware between user programs (such as CAD soft- work which runs in the Rhino/Grasshopper environ-
ware tools in architecture) and robot controllers to ment. This would allow for the exchange of data be-
make the interface of industrial robots suitable for re- tween the remote computer with KUKA|prc and the
search or design fabrication applications. An open- robot. While the system offers a high-level program-
source communication interface for Kuka industrial ming interface for the user with promising applica-
robots, OpenShowVar, is a Java based cross-platform tions, it requires the use of the mxAutomation soft-
developed by researchers at Aalesund University Col- ware package.
154 | eCAADe 35 - FABRICATION - ROBOTICS - Volume 2
METHOD The design of our system is based on the following
This research proposes a framework for human-robot choices:
interaction that has two main elements: 1) An adap- • Target user: This system is intended for use
tive robot control system based on sensor feedback, in design and fabrication processes, where ar-
and 2) a design-fabrication library. The main advan- chitects and designers are the target users. As
tage of our proposed framework compared to other a result, it is important for the system to sync
discussed research efforts on the development of and run on CAD applications such as Rhino/-
adaptive robot controls is the use of real-time feed- Grasshopper or Autodesk Dynamo for visual
back from a scanning system, as well as the read- programming purposes.
/write of data from/to the design-fabrication library. • Speed: Although Microsoft Windows com-
This paper discusses the first element of the puters do not provide hard real-time commu-
frame work, the adaptive robot control system. The nication, it is desired to minimize the lag in
system architecture diagram in Figure 3 illustrates the the communication between the robot con-
high-level view of our developed system and its ma- trol system and remote controlling computer
jor components. The adaptive control system uses as much as possible. Industrial robots also
the following hardware and software elements as the have real-time constraints, hence maintaining
testbed for the framework, hardware: Industrial Kuka the speed of the application is imperative.
robotic arms with either KRC2 or KRC4 operating sys- • Native packages: This system employs Kuka’s
tem, and Microsoft Kinect as the 3D scanner, Soft- own developed software package, Kuka RSI
ware: Rhino 3D, Grasshopper, KUKA|prc robot pro- (Real-time Sensor Interface), for real-time
gramming plug-in for Grasshopper, Kuka RSI (Robot communication to ensure system compatibil-
Sensor Interface), and Kinect Fusion Library. The only ity.
pre-requisite is that the user has a solid understand- • Flexibility: This is accomplished by providing
ing of computer networking and the KRL program- a structure and system architecture that can
ming language. offer the future possibility of including third
party libraries.
Figure 3
The proposed
real-time control
system architecture
for Kuka industrial
robots. The server
runs as client on a
remote computer.
The server interacts
locally with
Rhino/Grasshopper
and communicates
with Kuka KRC
remotely via TCI/IP.
Feedback data from
Kinect 3D scanner is
received by the
Grasshopper.
FABRICATION - ROBOTICS - Volume 2 - eCAADe 35 | 155
Figure 4
Kuka RSI Server and
sensor feedback
system structure
Our overall architecture consists of three high level cate between each other. The Robot server is the
modules - a RSI-Grasshopper module, a Kinect- server that connects to the KUKA robot over UDP and
Grasshopper module and the KUKA Robot Sensor In- always responds to the robot in the 12-millisecond
terface (RSI) server (Figure 4). We outline the design time limit in order to maintain the hard real-time con-
of each module below. straint and keep the connection active. This also al-
lows the Read and Write servers to perform long run-
KUKA RSI Server ning operations independently and not violate the
KUKA robots allow their real-time control via the response time constraint. The Robot server checks
KUKA.RobotSensorInterface or RSI from an external for any new input at each cycle before transferring
PC over an Ethernet connection. To enable this, the the input or a standard response without corrections
user needs to write a UDP (User Datagram Protocol) to the robot while always updating the new robot
based network server on an external PC, in a pro- configuration in its internal data structures. The Read
gramming language of their choice, and provide the Server reads the RSI data from the Robot Server’s in-
Internet Protocol address of the server to the robot ternal structures and provides it to the user in JSON
via the RSI configuration XML. This allows for bidi- format for display or logging. The Write Server ac-
rectional communication between the robot and the cepts input from the user in the form of a JSON of per-
server, allowing for the robot motion to be corrected axis corrections and transforms this input to a JSON
via XML-based instructions. We developed the RSI format which is then sent to the Robot Server to en-
Server in the Python programming language due code into valid XML and then send to the robot. We
to its ease of experimentation and abundance of li- use JSON to communicate between the three sub-
braries for network operations. Our implementation servers due to its ease of use with Python and many
supports both the KUKA KRC2 and KRC4 robots. other high-level languages (such as Matlab or C#) and
Our RSI Server spawns three sub-servers - a relatively lower memory requirements compared to
Robot Server, a Read Server, and a Write Server. XML. All communication between the 3 sub-servers
These are essentially sub-processes that communi- is done using inter-process messaging queues.
156 | eCAADe 35 - FABRICATION - ROBOTICS - Volume 2
To ensure safety of the robot and to not violate its ner to generate a real time point cloud of the ob-
torque correction limits while performing the robot ject as feedback to the user. Using the Kinect Fu-
corrections via the RSI, we chunk all the correc- sion Library in the C# programming language, we de-
tions into smaller corrections of 2 millimeters or less, veloped the plugin for Grasshopper that can bring
and generate the appropriate number of UDP pack- the Fusion depth stream into Grasshopper, thus pro-
ets which are sent in a batch. This also provides viding a 3D point cloud of the actual physical en-
smoother path corrections for the robot allowing for vironment of the part under fabrication inside the
better feedback to the user. The chunk size is a con- Grasshopper environment. Using this 3D view, the
figurable variable in our program, thus allowing for user would be able to move the robot Tool Center
either very slow and small motions or large rapid mo- Point (TCP) to an actual point in the physical space,
tions as may be desired. via the RSI-Grasshopper module, and see the visual
From the KRL programming end, we create the feedback from the Kinect window to guide the robot.
RSI object and set the correction limits to the approxi- As future steps for the completion of this system,
mate boundaries of the workspace in which the robot we intend to develop a structure that by overlapping
will operate. This ensures the KRL program does not the generated mesh from the Kinect point cloud data
give an error due to limit violations. The RSI object is on top of the existing CAD model, it would be possi-
enabled when the program is run, allowing it to com- ble to measure the deviations between the expected
municate to the RSI server. 3D and actual physical forms.
Figure 5 RSI-Grasshopper Module
Developed To allow for Rhino 3D to transfer user updates to the
grasshopper plugin: robot, we developed a RSI-Grasshopper module us-
server connection, ing the Grasshopper plugin (Figure 5). This module
data read, and data allows the integration of Rhino 3D with the RSI Server
write by linking the Rhino 3D UI elements with the data
in the RSI Server via TCP connections over Ethernet.
The Read and Write sub-servers in the RSI server pro-
vide the external interface to our RSI server and al-
lows us to send the robot corrections and receive the
RSI data. In this module, we create the TCP requests
to transfer the data to the Write Server and to read
Figure 6
the robot pose from the Read Server and update the
Grasshopper and
Rhino 3D user interface, thus allowing for real time
Kinect Fusion
control and updates for the user. Using the devel-
oped server and added module in Grasshopper to-
gether, we can receive and see the robot’s actual po-
sition in real time, and send corrections for the robot
tool path in real time.
Kinect-Grasshopper Module
To allow for visual feedback of the object we are fab-
ricating, we integrated a Kinect sensor feedback into
Rhino 3D by developing a Kinect-Grasshopper plu-
gin module (Figure 6). The Kinect acts as a 3D scan-
FABRICATION - ROBOTICS - Volume 2 - eCAADe 35 | 157
CONCLUSIONS AND FUTURE STEPS and the Collaborative and Advanced Robotic Manu-
In this research, we have elaborated upon the first facturing Lab at the Institute for Robotics and Intelli-
part of a real-time framework for robotic fabrica- gent Machines (IRIM), School of Interactive Comput-
tion, particularly the integration between Rhino/- ing both at the Georgia Institute of Technology. We
Grasshopper CAD modeling and visual programming would like to thank Sonia Chernova for her helpful
environment, and an industrial robot to allow for in- and insightful discussion and comments.
stant feedback during the fabrication process. We
have described the architecture of our system that REFERENCES
allows for maintaining the real-time constraints re- Braumann, J and Brell-Cokcan, S 2011 ’Parametric robot
quired by the robot as well as providing efficiency control’, Integrated CAD/CAM for Architectural Design,
for the end user to complete their task without dam- Proceedings of ACADIA 2011, p. 242
aging the robot or its surroundings. Visual feedback Braumann, J and Brell-Cokcan, S 2015 ’Adaptive Robot
both in terms of robot data as well as 3D depth data Control’, Proceedings of the 33rd eCAADe, Vienna,
Austria
from a Microsoft Kinect with the KinectFusion library
Elashry, K and Glynn, R 2014 ’An Approach to Automated
provides more comprehensive information during Construction Using Adaptive Programing’, Robotic
the whole process. Fabrication in Architecture, Art and Design, pp. 51-66
As the next step of this project for the devel- Munz, H, Braumann, J and Brell-Cokcan, S 2016 ’Di-
opment of a comprehensive human-robot interac- rect Robot Control with mxAutomation: A New Ap-
tion framework, we will focus on the development proach to Simple Software Integration of Robots
in Production Machinery, Automation Systems, and
of a design-fabrication library that would utilize the
New Parametric Environments’, Robotic Fabrication
developed adaptive robot control system for stor- in Architecture, Art and Design, pp. 440-447
ing various design and fabrication information in its KUKA Roboter, GmbH 2003, Expert Programming: KUKA
database. In the conceptual design stage, this li- System Software (KSS), KUKA Roboter GmbH
brary will assist the designers with the decision mak- Sanfilippo, F, Hatledal, LI, Fago, H, Fago, M and Pettersen,
ing process on the adequacy of the design choices, KY 2015, ’Controlling Kuka industrial robots: Flexible
communication interface JOpenShowVar’, Robotics
detailing, material selection for the selected robotic
& Automation Magazine, IEEE, 22(4), pp. 96-109
production techniques, and end-effectors. This de- Schwartz, T 2013 ’HAL’, Rob| Arch 2012, pp. 92-101
sign library will be instantiated incrementally based Sharif, S and Gentry, TR 2015 ’Design Cognition Shift
on both experimentation and human expert knowl- from Craftsman to Digital Maker’, Proceedings of
edge. The database will grow over time with data CAADRIA 2015, Daegu, Korea, pp. 683-692
from continued experimentation by multiple users, Shepherd, S and Buchstab, A 2014 ’Kuka robots on-site’,
Robotic Fabrication in Architecture, Art and Design
which will be enabled by the system described here.
2014, pp. 373-380
In addition, a user study on Human-Robot Interac- Stumm, S, Braumann, J, von Hilchen, M and Brell-Cokcan,
tion aspects of the system will be performed. This S 2016 ’On-Site Robotic Construction Assistance for
study will compare scenarios for an identical fabrica- Assembly Using A-Priori Knowledge and Human-
tion process with the assistance of the adoptive robot Robot Collaboration’, International Conference on
control system or without it, by measuring aspects Robotics in Alpe-Adria Danube Region
such as time, success rate, number of successful at-
tempts, and quality of the final product.
ACKNOWLEDGMENT
This project is a joint collaboration between the Dig-
ital Fabrication Lab, as part of School of Architecture,