applied

sciences
Article
Real-Time Remote Maintenance Support Based on
Augmented Reality (AR)
Dimitris Mourtzis * , Vasileios Siatras and John Angelopoulos
Laboratory for Manufacturing Systems and Automation, Department of Mechanical Engineering and
Aeronautics, University of Patras, 26504 Patras, Greece; siatras@lms.mech.upatras.gr (V.S.);
angelopoulos@lms.mech.upatras.gr (J.A.)
* Correspondence: mourtzis@lms.mech.upatras.gr; Tel.: +30-2610-910-160




Received: 31 January 2020; Accepted: 5 March 2020; Published: 8 March 2020


Abstract: In the realm of the current industrial revolution, interesting innovations as well as new
techniques are constantly being introduced by offering fertile ground for further investigation
and improvement in the industrial engineering domain. More specifically, cutting-edge digital
technologies in the field of Extended Reality (XR) have become mainstream including Augmented
Reality (AR). Furthermore, Cloud Computing has enabled the provision of high-quality services,
especially in the controversial field of maintenance. However, since modern machines are becoming
more complex, maintenance must be carried out from experienced and well-trained personnel,
while overseas support is timely and financially costly. Although AR is a back-bone technology
facilitating the development of robust maintenance support tools, they are limited to the provision
of predefined scenarios, covering only a limited number of scenarios. This research work aims to
address this emerging challenge with the design and development of a framework, for the support of
remote maintenance and repair operation based on AR, by creating suitable communication channels
between the shop-floor technicians and the expert engineers who are utilizing real-time feedback
from the operator’s field of view. The applicability of the developed framework is tested in vitro in
a lab-based machine shop and in a real-life industrial scenario.

Keywords: augmented reality; maintenance; real-time

1. Introduction
Equipment maintenance is one of the key elements consisting of manufacturing systems. In their
research work, Mourtzis et al. in Reference [1] highlight the importance equipment maintenance as
part of the production lifecycle, which reaches 60%–70% of the total production cost. Therefore, being
able to forecast machine maintenance operations and perform them in a short time period, can lead to
successful troubleshooting and simultaneously increase machine tools availability. Additionally, since the
replacement of damaged components can be as high as 70% of the total maintenance cost [1], it is one of
the top priorities among manufacturing firms to discover alternative policies for cutting maintenance costs
as a means of increasing their revenues. As a result, the overall production performance is optimized [2].
Besides the failure forecast, a major issue identified is the completion of accurate and error-free
maintenance operations, and, thus, the reassurance of fully functional machines in the earliest time
possible. Based on that concept, a considerable amount of research effort has been put on the
design and development of real-time maintenance support tools and applications based on mobile
devices to prevent unnecessary errors from happening [3–6]. With the evolution of Information
and Communication Technologies (ICT) as well as the massive break out of augmented reality (AR),
the research community extended that approach by exploiting the advantages of AR for data projection
during the maintenance operations [6–11]. Generally, under the framework of Industry 4.0 in an

Appl. Sci. 2020, 10, 1855; doi:10.3390/app10051855 www.mdpi.com/journal/applsci

Appl. Sci. 2020, 10, 1855 2 of 16

attempt to bridge the physical world with virtual worlds, new technologies have emerged, such as
Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR) [12]. These digital technologies
lie under the umbrella term Extended Reality (XR). The term XR is often used by practitioners and
scientists in order to describe the three immersive technologies mentioned above [13].
However, most of the research works available on the web cover basic aspects of Maintenance
and Repair Operations (MRO) and provide flexible and accurate troubleshooting [3–6]. The constant
innovation of cutting-edge technologies unveils new opportunities for remote maintenance support.
Therefore, there is an apparent need for the development of real-time AR frameworks for the support
of MRO.
More specifically, the proposed research deals with the design and development of a novel
framework for real-time remote maintenance support based on AR. Inspired by the concepts of
Cloud manufacturing and Product Service Systems (PSS), the framework was built in a cloud-enabled
environment aimed at bridging the different working groups involved in the production lifecycle.
Considering that companies seek increased operational availability, also known as availability (A) of
their machines, engineers are constantly developing new maintenance policies that can ensure decreased
Mean Time To Repair (MTTR) and increased Actual Machining Time (AMT). The following formula
reflects the correlation between the above-mentioned variables and characteristics of actual machines.

ROTBF = MTBF − AMT (1)

where ROTBF is the Remaining Operating Time Between Failures. MTBF is the Mean Time Between
Failures. AMT is the Actual Machining Time.

MTBF
A= (2)
MTTR + MTBF

where A is the machine availability. MTTR is the Mean Time To Repair.

From Equation (2), it is clear that, in order to increase the operational availability of the machine,
represented by A in Equation (2), there are two possible alternatives. The first alternative is to
increase the numerator, i.e., the MTBF, and the second alternative is to decrease the denominator, i.e.,
(MTTR + MTBF). However, under the scope of this research work, the aim is to develop a framework
that will facilitate the maintenance process of machine tools and, by extension, it will result in the
minimization of MTTR. Therefore, if MTTR is decreased, then the denominator of Equation (2) will
also decrease and, thereby, the operational availability of the machine tool will eventually increase.
It is stressed out that the above-mentioned indexes are the cornerstones of maintenance contracts
offered either as a service by maintenance contractors or OEMs (Original Equipment Manufacturers)
as part of their sales program. Although, a maintenance plan may seem attractive due to the low
MTTR offered by the maintenance contractor, e.g., 24 h. The Mean Time To Recovery is another index
that is highly affecting the quality of the service, as it directly affects Operational Availability of the
machine. The Mean Time To Recovery is an aspect of technical maintenance, dependent on a variety of
factors, such as part lead times, administrative delays, and technician transportation delays. Aimed at
minimizing the MTTR and eliminating the factors mentioned above, this research work is dedicated to
the modeling, development, and practical implementation of a real-time, AR-based framework for
remote maintenance support.
The reminder of the paper is structured as follows. In Section 2, an extensive literature review
is carried out, focusing on topics relevant to technical maintenance and AR. Then, in Section 3,
the proposed system architecture is presented and, in continuation in Section 4, the steps leading to
the practical implementation of the framework are discussed. Section 5 deals with the discussion of
a real-life industrial scenario used for validating the developed framework. Lastly, the paper concludes
in Section 6, arguing the results and the future work of this research work.
Appl. Sci. 2020, 10, 1855 3 of 16

2. Literature Review

2.1. Literature Review Methodology

In order to conduct a substantial literature review, a research methodology has been adopted
by Reference [14], restructured so as to meet the requirements of the current research work. For the
research, the most well renowned publication databases, namely Google Scholar, ResearchGate,
ScienceDirect, Scopus, and Web of Science, were used. a total of 34 research papers has been tracked
and examined. The investigation of the available literature was performed in two levels based on the
year of publication. More specifically, publications older than a decade have been examined in order to
track down the evolution of AR throughout past years and the newer publications were examined so
as to conclude on the current trends in the field of Remote Maintenance Support and, by extension,
to compare similar frameworks for the proposed framework.

2.2. Maintenance as Part of the Modern Manufacturing System

As stated in the previous paragraphs, it is clear that maintenance besides being a crucial part of
the production line, as it ensures the continuous and flawless operation of the physical assets, is also
a very complex process. Therefore, there is a considerable amount of research work trying to cover all
of its aspects. However, maintenance activities can be further categorized regarding the status of the
physical assets. Concretely, the most common types of technical maintenance found are: (i) Corrective
Maintenance, (ii) Preventive Maintenance, (iii) Risked-based Maintenance, and (iv) Condition-based
Maintenance [15]. Throughout recent decades, many types of technical maintenance policies have been
developed in an aim to cover the majority of the industrial needs. However, with the big diversity in
the general policies that firms adopt, there is no optimal solution regarding the equipment maintenance
plan. With that said, the maintenance policy adopted a company, and is highly dependent on the
general business model and, by extension, to the general goals to be achieved. As a result, all of
the above-mentioned policies are equally important for the research community and the industrial
domain as well. Regardless, the scope of this research work is to present the design and development
of a framework that supports field technicians mainly in corrective scenarios, where a malfunction has
already occurred and, in many cases, standard procedures cannot be applied. Therefore, the research
work of this paper could be classified as an enhanced corrective maintenance policy, which is based on
the provision of service for on-demand guidance.
The current industrial era is characterized by the immense decentralization of the manufacturing
networks. Thus, the actual manufacturing plant may be in a different region than the firm’s headquarters,
which makes communication of the two departments more difficult [3,16]. However, the ongoing
technological advances in the ICT have enabled the remote monitoring, remote operation, and the
remote maintenance of the modern manufacturing systems [17,18]. Modernized maintenance under
the term of Smart Maintenance have occurred over the last decade.

2.3. Augmented Reality Based Remote Maintenance

The concept of Digital Reality has been under the research scope during recent decades. With the
advent of the fourth industrial revolution, such cutting-edge digital technologies have met an increasing
advance, as a result of the technological advances in the ICT. More specifically, Augmented Reality (AR)
is a very popular digital technology, which offers the advantage of enhancing the user’s perception
by augmenting their physical environment with computer-generated information. In the field of AR,
Azuma is considered a pioneer, with the first survey on AR being published two decades ago [19],
concluding that AR back then was a very immature technology. a huge potential for future development
was expected. In a more recent publication [20], the challenges that must be overcome by researchers
in order to make AR a mainstream form of media are discussed. Among others, the most important
challenge the research and development community have to overcome, is the creation of highly
intuitive AR applications and tools.
Appl. Sci. 2020, 10, 1855 4 of 16

In the available literature, there can be found many publications covering the AR-based aspect of
maintenance. The majority of the works is focused on the remote assistance of technical maintenance
operations through the various AR-based frameworks. a typical application of remote maintenance
support is presented in Reference [21], where the technicians are capable of performing maintenance
operations in robotic arm manipulators with the use of AR-projected instructions. Another unique AR
framework for the support of MRO is presented in Reference [22]. In Reference [23], the authors cover
another aspect of AR-based remote maintenance under the framework of Product Service Systems
(PSS). Therefore, the AR tools developed can facilitate manufacturers to add value to their line of
products by transitioning to the PSS philosophy.
In the research work presented in Reference [4], the authors have carried out research on the recent
trends around the Smart Maintenance paradigm by investigating recent publications as well as by
extracting useful empirical knowledge. In Reference [5], a collaborative remote maintenance framework
has been suggested. Although the research work is inspiring and of good quality, the framework
implementation is based on technologies and equipment that currently can be considered as obsolete,
as the technician has limited mobility since a PC is required by their side so as to visualize the AR
instructions. Moreover, the instructions are based mostly on the projection of textual information. Lastly,
in this research work, the use of frame markers is not suitable for the maintenance of bigger and more
complex machines. Therefore, what is needed is more robust and more compact framework enhancing the
technician’s mobility when working on the machine. In a more recent paper, a similar remote maintenance
support system based on AR is presented [24]. It is remarkable that the authors have put a considerable
amount of effort on providing a mobile tool, based on the utilization of tablet PC. However, similar to
the previous work presented, the use of frame markers is inevitable, which can highly affect the overall
performance of the tool and, by extension, the performance of the shop-floor/field technician.
Although there is a continuous development of the AR-based tools proposed by the research
community during the last two decades, there is a lack of added functionality. The above-mentioned
development as well as the practical implementation of these frameworks in real-life industrial
scenarios and the in-vitro experimentation in laboratory environments has led to the conclusion that
AR has to treated as a back-bone technology. By extension, in the near future, the development of AR
tools has to be further enhanced by the addition of more functionalities. a representative example is
the framework presented in Reference [25], where an adaptive AR framework for machine operation
as well as technical maintenance is presented. The adaptivity of the framework is based on the skill
level of the technicians. Similarly, in Reference [26], an innovative framework based on a wrist-haptic
tracking methodology is introduced.

2.4. Maintenance under the Framework of Industry 4.0

The modern manufacturing world is undergoing a digitalization phase under the framework
of Industry 4.0. During the current industrial revolution, digital technologies have tremendously
advanced, which enables the improvement of other scientific fields, including that of technical
maintenance. With the recent development in the ICT, and the integration of Internet of Things (IoT),
the manufacturing domain has passed in a whole new phase by relying on the utilization of data, i.e.,
Big Data Analytics. For the successful completion of MRO besides the technical knowledge/expertise
of the stuff carrying out the operations, it is crucial that all the needed components are acquired.
Therefore, companies are maintaining an inventory of components. However, under the framework of
Industry 4.0, and with the utilization of digital technologies, the existing inventory practices can be
further improved through proper classification of the MRO components while additive manufacturing
can facilitate in reducing lead times to produce several components [6–10,26–28].
For Industry 4.0 applications, two key objectives are to ensure maximum uptime throughout the
production chain, and to increase productivity while reducing production costs. In Reference [29],
a systematic approach to analyze the strengths and weaknesses of current open-source Big Data and
stream processing technologies and assess their use for Industry 4.0 use cases is presented. In addition,
Appl. Sci. 2020, 10, 1855 5 of 16

the model of Industry 4.0 is gradually being implemented in worldwide development, distribution,
and marketing chains. As such, the design and integration of a stainless-steel predictive maintenance
system is presented in Reference [29]. This case study uses data from machinery involved in producing
Appl. Sci. 2020, 10, x FOR PEER REVIEW
high-quality steel sheets. Following the predictive maintenance field, the use of real-time 5detection of 16

and prediction algorithms regarding future failures has significantly benefited from the technological
producing high-quality steel sheets. Following the predictive maintenance field, the use of real-time
advances of Industry
detection 4.0 era. algorithms
and prediction There’s also growingfuture
regarding interest in decision-making
failures has significantlyalgorithms triggered
benefited from the by
predictions of failure over the past few years. a literature review on decision
technological advances of Industry 4.0 era. There’s also growing interest in decision-making making in predictive
maintenance
algorithms in the context
triggered by of smart manufacturing
predictions of failure overis done
the pastin References
few [30,31]. review
years. A literature Additionally, a review
on decision
making inmaintenance
on predictive predictive maintenance in the context
as an integrated of smartplatform
predictive manufacturing is done in References
for production [30,31]. on
systems, focusing
Additionally,
maintenance a review on
approaches, predictive
methods, andmaintenance as an integrated
tools is described predictive
in Reference [32].platform for production
systems, focusing on maintenance approaches, methods, and
Another significant issue is the estimation of the maintenance time for a new tools is described in Reference [32]. project,
maintenance
Another significant issue is the estimation of the maintenance time for a new maintenance
which is among the main maintenance offerings, is based solely on the experience and knowledge of the
project, which is among the main maintenance offerings, is based solely on the experience and
engineer. The research work presented in Reference [33] proposes a framework for knowledge-based
knowledge of the engineer. The research work presented in Reference [33] proposes a framework for
calculation of maintenance
knowledge-based time based
calculation on Key Performance
of maintenance time based on Indicators (KPI) monitoring
Key Performance Indicators to support
(KPI)
the capture and reuse of information in maintenance activities as well as to improve
monitoring to support the capture and reuse of information in maintenance activities as well as to the performance
of theimprove
given maintenance
the performance PSS. Next,
of the givenanmaintenance
extendable PSS.and Next,
reusable scheduling
an extendable andapproach in the context
reusable scheduling
of predictive
approachmaintenance,
in the context which supports
of predictive multiple which
maintenance, heterogeneous inputs and
supports multiple outputs, isinputs
heterogeneous described
and outputs,
in Reference [33]. isIndescribed
accordance in Reference [33]. In accordance
with the current with the current
schedule, predictive schedule,indicators
maintenance predictivefrom
maintenance
the tracked indicators
equipment are from
usedthe fortracked
schedulingequipment are used for
maintenance schedulingamaintenance
operations. web serviceoperations.
architecture is
A web service architecture is implemented to accommodate
implemented to accommodate highly different use scenarios such as suppliers highly different useofscenarios such as
equipment.
suppliers of equipment.
3. System Architecture
3. System Architecture
3.1. General System Architecture
3.1. General System Architecture
As discussed in the previous paragraphs, the scope of this research work is focused on the design
As discussed in the previous paragraphs, the scope of this research work is focused on the design
and development of a real-time, remote maintenance assistance framework based on AR by enabling the
and development of a real-time, remote maintenance assistance framework based on AR by enabling
creation of new communication
the creation of new communicationchannelschannels
betweenbetween
the expert
theengineers and theand
expert engineers shop-floor technicians.
the shop-floor
Concretely, thereConcretely,
technicians. is an opportunity toopportunity
there is an eliminate the need for the
to eliminate preparing
need forAR scenes,AR
preparing which considerably
scenes, which
decreases the timedecreases
considerably and efforttheneeded
time and foreffort
creating suchfor
needed content.
creatingMoreover, certain
such content. limitations
Moreover, can be
certain
limitations
overcome can engineers,
as expert be overcome as expert
through theengineers,
use of thethrough
proposedtheframework,
use of the proposed
are able framework, are
to guide shop-floor
able toinguide
technicians shop-floor
less likely MROtechnicians
scenarios. inIn less
orderlikely MRO
to meet thescenarios. In order to
above-mentioned meet the
technical above-
requirements,
mentioned technical requirements, the methodology adopted by aiming at addressing
the methodology adopted by aiming at addressing the identified scientific gaps will be presented in the identified
scientific gaps will be presented in detail in the following paragraphs. The general architecture
detail in the following paragraphs. The general architecture flowchart is depicted in Figure 1.
flowchart is depicted in Figure 1.

Figure
Figure 1.1.Flowchart
Flowchart of
of the
the proposed
proposedsystem.
system.

The sequence of actions begins as long as a malfunctioning machine tool has been noticed. Then,
the shop-floor technician who is carrying a device, a Head Mounted Display (HMD), contacts the
Appl. Sci. 2020, 10, 1855 6 of 16

Appl. Sci. 2020, 10, x FOR PEER REVIEW 6 of 16

The sequence of actions begins as long as a malfunctioning machine tool has been noticed.
Then, the shop-floor technician who is carrying a device, a Head Mounted Display (HMD), contacts the
expert engineer for assistance. It is stressed out that the coexistence of the two involved parties under
expert engineer for assistance. It is stressed out that the coexistence of the two involved parties under
the same premises is not required. More specifically, through the use of dedicated communication
the same premises is not required. More specifically, through the use of dedicated communication
protocols, the shop-floor technicians can be remotely connected to the technical support department
protocols, the shop-floor technicians can be remotely connected to the technical support department
of the machine tool manufacturer. As long as the connection between the two parties has been
of the machine tool manufacturer. As long as the connection between the two parties has been
established, a live video stream from the shop-floor technicians’ device is broadcast to the expert
established, a live video stream from the shop-floor technicians’ device is broadcast to the expert
engineer. This process can be realized as a live teleconference between the two parties. Therefore,
engineer. This process can be realized as a live teleconference between the two parties. Therefore,
while the video stream is broadcast to the expert engineer, the shop-floor technician is capable of
while the video stream is broadcast to the expert engineer, the shop-floor technician is capable of
making annotations so as to ease the engineer to get a better understanding of the machine tool
making annotations so as to ease the engineer to get a better understanding of the machine tool
malfunction. As soon as the engineer is fully aware of the malfunction, they can create on-demand
malfunction. As soon as the engineer is fully aware of the malfunction, they can create on-demand
AR content, which will be projected in the shop-floor technician’s FOV (Field Of View). It is stressed
AR content, which will be projected in the shop-floor technician’s FOV (Field Of View). It is stressed
out that, during the live-video casting from the technician, the device is also capable to 3D scan the
out that, during the live-video casting from the technician, the device is also capable to 3D scan the
space around the technician. As such, the algorithm running in the background is fully aware of the
space around the technician. As such, the algorithm running in the background is fully aware of the
user’s surroundings. Therefore, the expert engineer can register 3D content in the technician’s FOV
user’s surroundings. Therefore, the expert engineer can register 3D content in the technician’s FOV
more easily and more successfully. The framework architecture can be realized through a platform,
more easily and more successfully. The framework architecture can be realized through a platform,
where the shop-floor technicians can connect and request assistance from expert engineers in real-
where the shop-floor technicians can connect and request assistance from expert engineers in real-time.
time. From the expert engineer point-of-view, the platform serves as a tool enabling the expert to
From the expert engineer point-of-view, the platform serves as a tool enabling the expert to guide the
guide the shop-floor technician with the use of simple 3D tools. Therefore, it is of great importance
shop-floor technician with the use of simple 3D tools. Therefore, it is of great importance to initially
to initially present the platform architecture. In Figure 2, the architecture of the proposed framework
present the platform architecture. In Figure 2, the architecture of the proposed framework is depicted.
is depicted. From the figure, the structure of the Cloud Platform is visible and consists of services and
From the figure, the structure of the Cloud Platform is visible and consists of services and the toolbox.
the toolbox.

Figure 2. Proposed system architecture and steps sequence.

Figure 2. Proposed system architecture and steps sequence.

The most
The most important
important component
component of of the
the platform
platform architecture
architecture is is the
the online
online connection
connection andand user
user
interconnection framework.
interconnection framework. More More specifically,
specifically, the
the platform
platform actsacts as
as aa cloud
cloud platform,
platform, where
where users,
users,
either expert engineers or field/shop-floor technicians, can connect to and, most importantly, theythey
either expert engineers or field/shop-floor technicians, can connect to and, most importantly, can
can exchange
exchange digital
digital information.
information. It is Itstressed
is stressed
out out
thatthat there
there are are no limitations
no limitations in the
in the type type
andand volume
volume of
of the exchanged information. Additionally, the user experience can be
the exchanged information. Additionally, the user experience can be further improved by the support further improved by the
support
of of live teleconference
live teleconference of the involvedof the actors.
involved actors.toInenhance
In order order to theenhance the communication
communication experience
experience between the involved parties, the framework supports
between the involved parties, the framework supports a teleconference feature. When a teleconference feature. When
this this
feature
feature
is is enabled,
enabled, then thethen
user the user is of
is capable capable
havingofa having a live
live video feedvideo
fromfeed from the
the expert expert projected
engineer, engineer,
projected in his FoV. As such, it is expected that the communication between
in his FoV. As such, it is expected that the communication between the two parties is facilitated, as the two parties is
facilitated,
the as the
technician cantechnician
perform MRO can perform MRO simultaneously.
simultaneously. As far as theAs far as the information
information exchange is exchange
concerned, is
concerned, the platform is connected with a cloud database. In the database, there
the platform is connected with a cloud database. In the database, there is dedicated space where certain is dedicated space
where
3D certain 3D
geometries are geometries
saved. These are saved. These
geometries geometries
are imported in are imported
the AR in the
scene only byAR
thescene
expertonly by the
engineers.
expert
The engineers.
collection The collection
of these tools canofbethese toolsas
realized cana be realized
digital as a digital
toolbox toolbox
of the most of the most
common toolscommon
used in
tools used in MRO. However, due to increasing machine tool complexity, specialty tools might be
needed. As such, the expert engineer is capable of adding a digital copy of the specialty tool through
the “Create tool” Graphical User Interface (GUI).
Appl. Sci. 2020, 10, 1855 7 of 16

MRO. However, due to increasing machine tool complexity, specialty tools might be needed. As such,
the expert engineer is capable of adding a digital copy of the specialty tool through the “Create tool”
Graphical
Appl. User
Sci. 2020, 10, xInterface
FOR PEER (GUI).
REVIEW 7 of 16
In Figure 3, the GUI for joining an online maintenance session is depicted. On the top left corner
of theInGUI,
Figure the3,user
the GUI
through for joining an online
the provided maintenance
dialog can selectsession
a sessionis depicted.
to join. InOn thethe
case topsession
left cornerhas
of the GUI, the user through the provided dialog can select a session to join.
reached its end, then the user has to leave the session. On the lower left corner, the provided buttons In case the session has
reached
assist theits end, then
master the user
engineer has to from
to retrieve leavethetheCloud
session. On thethe
platform lower left corner,
required tools bytheperforming
provided buttons
a “drag
assist the master engineer to retrieve from the Cloud platform the required
and drop” action to position them in the technician’s FOV. However, if the desired tool does not exist tools by performing a
“drag and drop” action to position them in the technician’s FOV. However,
in the Cloud toolbox, then the engineer is capable of uploading a 3D geometry. More specifically, in if the desired tool does
not
Figureexist in the
3, the Cloud
“plus” icontoolbox,
when hit then the engineer
triggers the “Create is capable of uploading
Tool” function, so thatathe 3Dengineer
geometry. addsMorethe
specifically, in Figure 3, the “plus” icon when hit triggers the “Create
geometry in a suitable filetype along with relevant info and a description of the tool. The filetypes Tool” function, so that the
engineer
accepted by adds thethe geometry
Cloud Database in ahave
suitable
beenfiletype
limited to along with relevant
the filetypes acceptedinfobyandtheaARdescription of the
tool. Therefore,
tool.
sinceThethe filetypes
framework accepted by the in
is developed Cloud Database
the Unity 3D gamehave engine,
been limited to the filetypes
the supported modelaccepted
file formatsby thefor
AR tool. Therefore, since the framework is developed in the Unity 3D game
3D geometries are, .fbx, .dae (Collada), .3ds, .dxf, .obj. Then the algorithm running in the background engine, the supported
model file formats
is responsible for 3D geometries
for uploading the geometry are, on
.fbx,
the.dae
Cloud(Collada),
Database .3ds,
as .dxf,
well as.obj.
to Then
createthe
a newalgorithm
record
running
in the SQL Database. The SQL database is created and used in an ontological manner, so well
in the background is responsible for uploading the geometry on the Cloud Database as that
as to create a new record
a smart-searching algorithmin thecan SQL Database. The
automatically SQL database
generate is created
suggestions to theand used induring
engineer an ontological
the AR
manner, so thatphase.
scene creation a smart-searching
In Figure 4, the algorithm
set of thecan automatically
initial tools usedgenerate suggestions
for populating to theToolbox
the Cloud engineer is
during the AR scene creation phase. In Figure 4, the set
presented. For this manner, the most common technical tools have been utilized. of the initial tools used for populating the
Cloud Toolbox is presented. For this manner, the most common technical tools have been utilized.

Figure 3. GUI for online maintenance session.

Figure 3. GUI for online maintenance session.

Figure 4. 3D CAD geometries of tools for AR instructions.

Appl. Sci. 2020, 10, 1855 8 of 16

Figure 3. GUI for online maintenance session.

Figure 4. 3D CAD geometries of tools for AR instructions.

Figure 4. 3D CAD geometries of tools for AR instructions.

3.2. Cloud Platform

The proposed remote maintenance service is developed as a cloud-based platform. The Cloud
platform enables the communication of the shop-floor/field technicians and the expert engineer as well
as the distribution of information between the involved parties. The Cloud platform is comprised
of two domains: (i) the communication domain and (ii) the data storage domain. As discussed in
the previous paragraphs, the communication domain is used for matching shop floor technicians for
expert engineers and for establishing the communication channels between them. On the other hand,
the data storage domain in essence is the Cloud Database where all the data are saved in SQL tables as
well as the 3D geometries uploaded.

3.3. Remote Maintenance Sessions

As long as the shop-floor/field technicians identify a malfunctioning machine, then, with the use
of a mobile device, they use the cloud platform so as to communicate with an expert engineer. In the
cloud platform, this form of communication is realized through the creation of online sessions. For each
session created, a communication domain is created. Upon the establishment of the communication
domain, the two parties are capable of bilaterally exchanging information.

4. Implementation
As far as software is concerned, the first step for the development of the framework is the
development of the cloud platform. From a hardware point of view, the architecture of the framework
has been developed so that a variety of platforms can be supported including PCs, HMDs, and handheld
devices. In its current form, the framework has been developed so that the technician uses a Microsoft
Hololens HMD and the expert engineer operates from a desktop PC.
For the platform, a remote server is utilized. The server has a dedicated domain for the storage
of files, where 3D geometries are also uploaded. The supported filetypes for the 3D geometries are
.fbx, .dae (Collada), .3ds, .dxf, .obj, and .skp. Besides the 3D geometries, the storage domain supports
any form of files, providing enough degrees of freedom to the engineers in exchanging files with the
technicians during a remote maintenance session.
As far as the development of the software is concerned, the Unity 3D game engine has been utilized.
Regarding the AR application for the technicians, Unity 3D in conjunction with the Vuforia and MRTK
(Mixed Reality Tool Kit) libraries were used. The Vuforia library is useful for the creation and handling
of the AR content, whereas the MRTK library supports advanced user interaction functionalities for
 any form of files, providing enough degrees of freedom to the engineers in exchanging files with the
technicians during a remote maintenance session.
As far as the development of the software is concerned, the Unity 3D game engine has been
utilized. Regarding the AR application for the technicians, Unity 3D in conjunction with the Vuforia
and Sci.
Appl. MRTK (Mixed
2020, 10, 1855 Reality Tool Kit) libraries were used. The Vuforia library is useful for the creation
9 of 16
and handling of the AR content, whereas the MRTK library supports advanced user interaction
functionalities for the Microsoft Hololens HMD. Regarding the expert engineer application for the
the Microsoft Hololens HMD. Regarding the expert engineer application for the remote support,
remote support, a Windows Form Application has been developed. In order to enable the
a Windows Form Application has been developed. In order to enable the communication of the two
communication of the two applications, the integrated Unity UNet API (Application Programming
applications, the integrated Unity UNet API (Application Programming Interface) was used. Since the
Interface) was used. Since the above-mentioned API is targeted for the creation of multiplayer games,
above-mentioned API is targeted for the creation of multiplayer games, several functionalities had
several functionalities had to be adjusted so as to align with the requirements of the framework. UNet
to be adjusted so as to align with the requirements of the framework. UNet can be realized through
can be realized through different layers. In Figure 5, the functionalities per layer of the UNet are
different layers. In Figure 5, the functionalities per layer of the UNet are presented.
presented.

Figure5.5.Functionalities
Figure Functionalitiesper
perLayer
Layerof
ofUnity
Unity3D
3DUNet
UNetAPI.
API.

As far as the communication module is concerned and, more specifically, the teleconference
functionality, a variety of solutions has been examined, including Skype, Agora.io [34], GIGA Video
Streamer [35], and the development of a custom teleconference tool. Among the available solutions,
the latter was selected, i.e., the development of a custom teleconference interface. To begin with,
the integration of Skype, although it is an appealing solution, and the quality of the service itself is
sufficient. It required that it was run as an external application on the Microsoft HoloLens HMD, which
had a negative effect on the performance of the app, as well as being unpleasant for the end-users to
switch over to two different applications including one for the communication and one for the AR
remote support. On the other hand, the Agora.io and the GIGA Video Streamer are very capable
SDKs (Software Development Kit) that could be integrated in the proposed framework. However,
these two SDKs are not free. Therefore, the development of a custom module based on the UNet API,
discussed above, was the only choice satisfying the framework’s requirements. Concretely, from the
expert engineer side, the video stream from the camera and the microphone of the PC was captured
and transmitted to the Microsoft Hololens HMD. Form the technician side, a pop-up window appears
where the technician can view the expert engineer and, through the embedded speakers of the HMD,
to hear the expert’s voice.
From a hardware point of view, for the on-site technician, a Microsoft HoloLens [36] device has
been utilized in order to take advantage of its four environment-understanding cameras and the mixed
reality capture in terms of sensors, whereas, for the development of the framework and the validation
experiments, a laptop PC has been utilized. More specifically, the laptop PC is equipped with an Intel
core i7 processor clocked at 2.20 GHz, 8 GB DDR4 RAM, and a NVIDIA GeForce 1060 GPU with 6 GB
dedicated memory.

5. Case Study and Results

The applicability of the proposed framework is tested and validated in-vitro in a laboratory-based
machine shop as well as in a real-life industrial scenario. The industrial scenario involves the machine
tools used in an existing machine-shop. Taking into consideration the technological level of the
machines, this use case is ideal since the machine shop consists of machines coming from different
technological eras. Thus, it is of great importance to examine the actual impact of the proposed
framework in varying machine tools. In Figure 6, the validation of the framework in the real-life
 5. Case Study and Results
The applicability of the proposed framework is tested and validated in-vitro in a laboratory-
based machine shop as well as in a real-life industrial scenario. The industrial scenario involves the
machine
Appl. Sci. tools
2020, 10, 1855used in an existing machine-shop. Taking into consideration the technological level of 10 of 16
the machines, this use case is ideal since the machine shop consists of machines coming from different
technological eras. Thus, it is of great importance to examine the actual impact of the proposed
framework
scenario in varyingMore
are presented. machine tools. In in
specifically, Figure 6, the
Figure 6a,validation of the framework
the malfunctioning machinein the real-lifealong
is visible
with scenario are presented.
the shop-floor More
technician whospecifically,
is wearing in the
Figure 6a, the malfunctioning
HoloLens HMD in order machine is visible
to perform along
the maintenance
with the shop-floor technician who is wearing the HoloLens HMD in order to
operations, as instructed by the expert engineer. Similarly, in Figure 6b, another engineer is testingperform the the
maintenance operations, as instructed by the expert engineer. Similarly, in Figure 6b, another
developed framework. It is stressed out that, in the background, the expert engineers using the laptop
engineer is testing the developed framework. It is stressed out that, in the background, the expert
PC in order to communicate with the technician are visible.
engineers using the laptop PC in order to communicate with the technician are visible.

Figure 6. (a) Validation of the proposed framework by shop-floor technician; (b) Another technician
Figure (a) framework.
6. the
tests Validation of the proposed framework by shop-floor technician; (b) Another technician
tests the framework.

Similarly,
Appl. Sci. 2020,in
10,Figure
x FOR PEER 7, REVIEW
the practical implementation and validation of the framework in the
10 of 16
lab-based scenario is presented. Concretely, in Figure 7a, the expert engineer, from his desktop,
Similarly,
can view the user’sinFoV,Figure 7, thecontains
which practicalthe
implementation
malfunctioning and machine
validationtool.
of theAsframework in the
can be seen, thelab-
spatial
based scenario is presented. Concretely, in Figure 7a, the expert engineer, from his desktop, can view
recognition algorithm of the Microsoft HoloLens device has recognized the technician’s physical
the user’s FoV, which contains the malfunctioning machine tool. As can be seen, the spatial
environment. Thus, the engineer based on that feature can place the 3D augmentations on the user’s
recognition algorithm of the Microsoft HoloLens device has recognized the technician’s physical
physical environment.
environment. Thus,On thethe otherbased
engineer hand,on inthat
Figure 7b, can
feature the place
shop-floor
the 3Dtechnician
augmentationsis presented. What is
on the user’s
worthphysical
noticing environment. On the other hand, in Figure 7b, the shop-floor technician is presented. Whathead
is that, in the top right corner of that figure, a snapshot from the HoloLens
up display
is worthisnoticing
presented.is that,The shop
in the top floor technician,
right corner of that after
figure,requesting assistance
a snapshot from from the
the HoloLens headengineer,
up
the connection
display is has been established
presented. and technician,
The shop floor a video teleconference has assistance
after requesting begun. Therefore,
from the a communication
engineer, the
connection
window has opened has been established
in the and aFoV,
technician’s video teleconference
where the audiohas begun.
and Therefore,
video a communication
stream from the engineer are
window has
received and displayed.opened in the technician’s FoV, where the audio and video stream from the engineer are
received and displayed.

Figure 7. (a) Expert engineer visualizes video stream from technician; (b) Technician is performing
Figure 7. (a) Expert engineer visualizes video stream from technician; (b) Technician is performing
maintenance tasks based on expert’s instructions.
maintenance tasks based on expert’s instructions.
5.1. Validation Strategy—Design of Experiments
In the following paragraphs, the steps followed for the validation of the proposed framework in
the real-life industrial scenario will be presented. Since maintenance is a very important aspect
regarding the flawless and continuous operation of manufacturing systems, the validation strategy
should be carefully designed and also should provide a quantified result of the impact. Therefore, for
each of the machine-tools examined in this process, two basic metrics will be used. According to the
guidelines given by Chryssolouris in Reference [21] regarding the mathematical programming of
Appl. Sci. 2020, 10, 1855 11 of 16

Validation Strategy—Design of Experiments

In the following paragraphs, the steps followed for the validation of the proposed framework
in the real-life industrial scenario will be presented. Since maintenance is a very important aspect
regarding the flawless and continuous operation of manufacturing systems, the validation strategy
should be carefully designed and also should provide a quantified result of the impact. Therefore,
for each of the machine-tools examined in this process, two basic metrics will be used. According to
the guidelines given by Chryssolouris in Reference [21] regarding the mathematical programming of
manufacturing systems, the operating and maintenance costs for a machine can be calculated by the
equation below.
XT N
X
Operating and Maintenance Costs = (P/F, I, t) mit xit (3)
t=1 i=1

where (P/F, I, t) ≡ Present a worthy interest factor for discrete compounding when the discount rate
is I% per period and the discount interval is t periods. mit ≡ The cost of operating and maintaining
a machine of type i during period t. Is the number of machines, on one hand, in the work center i for
period t.
In order to quantify the outcome of the validation process for the proposed framework, each of
the above-mentioned values will be calculated in order to conclude on the efficiency of the real-time
remote maintenance.
Therefore, the cost for operation and maintenance for a CNC milling machine has been calculated,
based on data derived from a real-life machine-shop. The cost of operation and maintenance has
been calculated on an annual basis for time horizon of five years, and the results are depicted in
Appl. Sci. 2020, 10, x FOR PEER REVIEW 11 of 16
Figure 8. Additionally, in Figure 8, a prediction of the cost is made for the oncoming five years based
on statistical
depictedregression,
in Figure 8. presented
Additionally,by in
the red dot
Figure 8, atrendline.
prediction It
ofis stressed
the out that,
cost is made due
for the to the non-linear
oncoming five
relation between the individual costs of operation and maintenance of machine
years based on statistical regression, presented by the red dot trendline. It is stressed out that, tools during
due to their
lifecycle [37], a 2ndrelation
the non-linear degreebetween
polynomial has beencosts
the individual calculated. The same
of operation calculations
and maintenance ofwere made
machine toolsfor the
cost estimation
during theirbased on[37],
lifecycle theaproposed
2nd degreemethodology
polynomial hasand,beensimilarly,
calculated.a The
prediction is made for
same calculations were future
made for costs.
maintenance the costAs estimation
a result,based
withonthethe proposedmethodology,
proposed methodology and, similarly, of
a reduction a prediction is made100 k
approximately
eurosfor future maintenance
is feasible in a time span costs. Asyears,
of 10 a result,
whichwith the proposed
indicates that themethodology,
framework could a reduction
eventuallyof be
approximately 100 k euros is feasible in a time span
implemented in manufacturing systems and improve their operation. of 10 years, which indicates that the framework
could eventually be implemented in manufacturing systems and improve their operation.

Figure
Figure 8. 8.Cost
Costofofoperation
operation and
andmaintenance
maintenanceanalysis.
analysis.

Besides the cost for operating and maintenance, another metric has been calculated and
compared regarding the downtime of the above-mentioned machine. In manufacturing systems, time
resources are equally important for the financial resources. Therefore, a maintenance scenario has
been investigated. For this repair scenario, the machine tool breaks down at an unpredicted time. At
that time, the production process for this machine stops and the expert engineer is called to come on-
site, inspect the machine, and repair it. The same scenario has also been investigated, but instead of
Appl. Sci. 2020, 10, 1855 12 of 16

Besides the cost for operating and maintenance, another metric has been calculated and compared
regarding the downtime of the above-mentioned machine. In manufacturing systems, time resources are
equally important for the financial resources. Therefore, a maintenance scenario has been investigated.
For this repair scenario, the machine tool breaks down at an unpredicted time. At that time, the production
process for this machine stops and the expert engineer is called to come on-site, inspect the machine,
and repair it. The same scenario has also been investigated, but instead of requesting an expert engineer
to visit the machine shop, a remote support session is established, and the repair is conducted by the
technicians available in the machine shop. In Figure 8, the distribution of time per maintenance task is
displayed for each of the scenarios examined. From Figure 6, it is undeniable that a significant amount of
time can be saved with the adoption of the proposed framework, as the time for travelling and inspection
are considerably longer, which affects the overall time the machine tool has been off the production
schedule. Through the use of Figure 9, it is intended to reflect the difference of time distribution between
the two scenarios examined during the validation process of the proposed framework. Therefore, from
Figure 9, it can be concluded that, in the first scenario, i.e., the “AS-IS-SITUATION”, the most time spent
was on inspection and travelling, as the expert engineer had to travel on site and physically inspect the
malfunctioning machine. On the contrary, in the second scenario, i.e., “PROPOSED METHODOLOGY”,
a totally different time distribution has been observed with the most time spent on activities relative to the
acquisition of spare parts for the machine repair. In addition to that, it can be concluded that the actual
repair time, which is the black portion of the pie graph, in the second scenario is bigger. This observation
can be considered normal as, in the second scenario, the maintenance operations have been carried out by
a field technician, whereas, in the first scenario, the same operations have been carried out by an expert
engineer with solid practical experience on such machines.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 12 of 16

Figure 9. Distribution of time AS-IS situation vs. Proposed Methodology.

Figure 9. Distribution of time AS-IS situation vs. Proposed Methodology.
In Figure 10, a side-by-side comparison of the specific times for the two maintenance scenarios
In Figure 10, a side-by-side comparison of the specific times for the two maintenance scenarios is
is presented. It is stressed that, with the adoption of the proposed framework, the time for inspection
presented. It is stressed
of the machine can be that, withasthe
shorter no adoption
travelling of theisproposed
time framework,
involved in the time
order to reduce MTTR.forHowever,
inspection of
the machine can be shorter as no travelling time is involved in order to reduce MTTR. However,
due to the inexperience of the technician carrying out the maintenance activities, the actual repair due to
the inexperience
operation time of isthe technician
longer, carrying
as the expert out the
engineer maintenance
had activities,checks
to perform additional the actual repair
in order operation
to ensure
time that
is longer, as the
the repair wasexpert engineer had to perform additional checks in order to ensure that the
successful.
repair was successful.

Figure 10. Time per maintenance action AS-IS situation vs. Proposed Methodology.

6. Discussion
 In Figure 10, a side-by-side comparison of the specific times for the two maintenance scenarios
is presented. It is stressed that, with the adoption of the proposed framework, the time for inspection
of the machine can be shorter as no travelling time is involved in order to reduce MTTR. However,
due to the inexperience of the technician carrying out the maintenance activities, the actual repair
Appl. Sci. 2020, 10, 1855 13 of 16
operation time is longer, as the expert engineer had to perform additional checks in order to ensure
that the repair was successful.

Figure 10. Time per maintenance action AS-IS situation vs. Proposed Methodology.
Figure 10. Time per maintenance action AS-IS situation vs. Proposed Methodology.

6. 6. Discussion
Discussion
TheThe outcomeofofthe
outcome thecurrent
currentresearch
research work,
work, is
is aa multi-sided,
multi-sided,multi-device
multi-deviceapplication,
application, enabling
enabling
the remote maintenance of complex machine tools under the framework of Industry 4.0. The term
the remote maintenance of complex machine tools under the framework of Industry 4.0. The term
multi-sided is used in order to describe that multiple operators can connect to the platform
multi-sided is used in order to describe that multiple operators can connect to the platform simultaneously
simultaneously and discuss on a malfunction that has occurred unexpectedly. On the other hand, the
and discuss on a malfunction that has occurred unexpectedly. On the other hand, the term multi-device,
term multi-device, is used in order to describe the compatibility of the developed framework in a
is used in order to describe the compatibility of the developed framework in a variety of devices. It is
variety of devices. It is stressed out that the most common devices used for AR implementations are
stressed out that the most common devices used for AR implementations are HDMs and handheld
HDMs and handheld devices, including smart-phones and tablets. However, since the scope of the
devices,
currentincluding
research wassmart-phones and on
mainly focused tablets. However,
the provision of asince the
totally scope of the
marker-less ARcurrent
solution,research
the use of was
mainly focused on the provision of a totally marker-less AR solution, the use of Microsoft’s
Microsoft’s HoloLens HMD is preferred, since the device supports spatial recognition, which, in turn, HoloLens
HMD is preferred,
has facilitated since the device
the development of thesupports spatial
framework. It is recognition, which,
stressed out that in turn,
the use of the has
HMD facilitated
is initiallythe
development of the framework. It is stressed out that the use of the HMD is
intended only for the shop-floor technician as a means to capture their surrounding and, based oninitially intended only
forthat,
the shop-floor technician
the expert engineer as a means
is capable to capture their surrounding
of placing/registering and, based
the virtual content on theonuser’s
that, physical
the expert
engineer is capable
environment. of placing/registering
The development the virtual
of the framework content oninthe
was performed user’s
a set physical stages.
of consecutive environment.
The
The development
first of the on
stage was based framework was performed
the conceptualization in asystem’s
of the set of consecutive stages.
architecture, The first stage
the discussion of thewas
modules
based on thecomprising the architecture,
conceptualization and thearchitecture,
of the system’s tools to be used for the final
the discussion of development. Then, in
the modules comprising
thestage two, the basic
architecture, and modules
the toolswere
to bedeveloped, utilizing
used for the the software described
final development. Then, in in stage
Section 4. As
two, soon
the basic
modules were developed, utilizing the software described in Section 4. As soon as the alpha version
of the framework was released, the framework has been internally tested in vitro, in the lab-based
machine shop described in Section 5. During this stage, colleague engineers have been asked to test
the functionalities of the developed application either from the side of the expert engineer or from the
side of the shop-floor technician aimed at gathering insightful feedback. Then, based on the feedback
gathered during the previous stage, the framework has been improved in certain areas, regarding the
layout of the GUIs and the functioning of the communication services. Then the beta version of the
developed framework has been prepared and tested with the industrial partner in their premises.
Throughout the development and validation stages, meaningful feedback has been gathered.
The main strengths of the proposed framework can be summarized to the minimum development
effort required by the expert engineer. More specifically, the remote maintenance frameworks are based
on the development of AR scenes, which are then implemented on an AR-ready device. Therefore,
this requires advanced development skills and a certain amount of preparation time. On the contrary
with the proposed framework, an expert engineer is capable of creating AR content on the user’s
FoV with the use of simple Drag-and-Drop actions. As such, no special skills are required and
the time for AR content preparation and implementation is eliminated. Another strength of this
framework is the ability to create online sessions. The online sessions are based on the Unity 3D
multiplayer API offering the remote connection of multiple users. As discussed in previous paragraphs,
Appl. Sci. 2020, 10, 1855 14 of 16

the developed application is compatible with most devices. Thus, there is no practical limitation
on the use of equipment. In comparison with other approaches, presented in References [3–6],
the functionality of remote connectivity combined with the broad compatibility of the app further
extends mobility of the user, i.e., the shop-floor technician in an industrial environment. More
specifically, in Reference [4], the technicians have to carry a laptop computer along with a USB web
camera in order to establish communication with the expert engineer and proceed with the maintenance
operations. The frameworks presented in the above-mentioned publications are using frame markers
for the registration of the content on the physical environment. With that said, the performance of the
system relies heavily on the quality of the camera used for the detection of the marker. The user’s
mobility is limited, as the camera has to maintain visibility of the marker at all times. Otherwise,
the augmentations cannot be registered and updated as the user moves around. In addition to that,
the proposed framework, since it utilizes newer equipment with increased computational power,
the use of more vivid/realistic virtual objects is feasible. Not only that, but in previous approaches,
the augmentations are limited to the display of simple schematics, e.g., arrows, text boxes, or indicate
the component of interest via a wireframe model. On the contrary, the current approach can recognize
the objects by the utilization of Microsoft HoloLens.

7. Conclusions
In this research work, the most recent and relevant publications regarding remote maintenance
support have been investigated. Through the literature review process, it became apparent that,
although there are numerous approaches contributing in the field of remote maintenance support, there
is a small amount of research work done on the provision of real-time support. Therefore, an opportunity
for a new research work has been identified. Concretely, the design and development of a framework
for the real-time remote maintenance and repair operations based on AR has been presented in detail.
The framework aimed at creating interactive, intuitive, and collaborative communication channels
between the shop-floor technicians and the expert engineers. Furthermore, the developed application
was tested in a real-life industrial scenario in order to verify that all the functional requirements are
met. The results indicate that the present research work has managed to successfully address the
functional requirements discussed during the modelling phase of the problem and has succeeded in
delivering a zero-time content authoring AR tool, which further minimizes the MTTR. However, there
is fertile ground for future improvement. As discussed in the literature review section, one of the key
elements consisting of an AR application is the content authorization methodology. Towards that end,
future work could focus on integrating the developed framework to an automated methodology for
content authoring, which could considerably reduce the effort and the skills needed from engineers in
creating AR content.

Author Contributions: All authors participated in the modelling of the research project. More specifically, D.M.
Supervisor, V.S. Development and methodology, J.A. Research, conceptualization, and writing. All authors have
read and agreed to the published version of the manuscript.
Funding: This research received no external funding.
Acknowledgments: The H2020 EC funded project “Balancing Human and automation levels for the manufacturing
workplaces of the future—MANUWORK” (GA No: 723711) partially supported this work.
Conflicts of Interest: The authors declare no conflict of interest.
Appl. Sci. 2020, 10, 1855 15 of 16

References
1. Mourtzis, D.; Vlachou, E.; Milas, N.; Xanthopoulos, N. a Cloud-based Approach for Maintenance of Machine
Tools and Equipment Based on Shop-floor Monitoring. In Proceedings of the 48th CIRP Conference on
Manufacturing Systems, Ischia, Italy, 24–26 June 2015; pp. 655–660. [CrossRef]
2. Predictive Maintenance: Taking Pro-Active Measures Based on Advanced Data Analytics to Predict and
Avoid Machine Failure. Available online: https://www2.deloitte.com/content/dam/Deloitte/de/Documents/
deloitte-analytics/Deloitte_Predictive-Maintenance_PositionPaper.pdf (accessed on 2 January 2020).
3. Wang, W.; Peter, W.T.; Lee, J. Remote machine maintenance system through Internet and mobile
communication. Int. J. Adv. Manuf. Technol. 2007, 31, 783–789. [CrossRef]
4. Wang, J.; Feng, Y.; Zeng, C.; Li, S. An augmented reality based system for remote collaborative maintenance
instruction of complex products. In Proceedings of the 2014 IEEE International Conference on Automation
Science and Engineering (CASE), Taipei, Taiwan, 18–22 August 2014; pp. 309–314. [CrossRef]
5. Masoni, R.; Ferrise, F.; Bordegoni, M.; Gattullo, M.; Uva, A.E.; Fiorentino, M.; Carrabba, E.; Di Donato, M.
Supporting remote maintenance in industry 4.0 through augmented reality. Procedia Manuf. 2017, 11,
1296–1302. [CrossRef]
6. Ong, S.K.; Zhu, J. a novel maintenance system for equipment serviceability improvement. CIRP Ann. 2013,
62, 39–42. [CrossRef]
7. Mourtzis, D.; Vlachou, A.; Zogopoulos, V. Cloud-based augmented reality remote maintenance through
shop-floor monitoring: a product-service system approach. J. Manuf. Sci. Eng. 2017, 139, 061011. [CrossRef]
8. Mourtzis, D.; Zogopoulos, V.; Katagis, I.; Lagios, P. Augmented Reality based Visualization of CAM
Instructions towards Industry 4.0 paradigm: a CNC Bending Machine case study. In Proceedings of the 28th
CIRP Design Conference 2018, Nantes, France, 23–25 May 2018; pp. 368–373. [CrossRef]
9. Cardoso, S.F.L.; Mariano, Q.M.C.F.; Zorzal, R.E. a Survey of Industrial Augmented Reality. Comput. Ind. Eng.
2020, 139, 106159. [CrossRef]
10. Palmarini, R.; Erkoyuncu, A.J.; Roy, R.; Torabmostaedi, H. a systematic review of augmented reality
applications in maintenance. Robot. Comput. Integr. Manuf. 2018, 49, 215–228. [CrossRef]
11. Fernández del Amo, I.; Erkoyuncu, A.J.; Roy, R.; Palmarini, R.; Onoufriou, D. a systematic review of
Augmented Reality content-related techniques for knowledge transfer in maintenance applications. Comput.
Ind. 2018, 103, 47–71. [CrossRef]
12. Berglund, F.Å.; Gong, L.; Li, D. Testing and validating Extended Reality (xR) technologies in manufacturing.
In Proceedings of the 8th Swedish Production Symposium (SPS 2018), Stokholm, Sweeden, 16–18 May 2018;
pp. 31–38. [CrossRef]
13. What Is Extended Reality Technology? a Simple Explanation for Anyone. Available
online: https://www.forbes.com/sites/bernardmarr/2019/08/12/what-is-extended-reality-technology-a-
simple-explanation-for-anyone/#69b6421f7249 (accessed on 16 February 2020).
14. Mourtzis, D. Simulation in the design and operation of manufacturing systems: State of the art and new
trends. Int. J. Prod. Res. 2019, 1–23. [CrossRef]
15. 4 Types of Maintenance Strategy, Which one to Choose? Available online: https://new.abb.com/medium-
voltage/service/maintenance/feature-articles/4-types-of-maintenance-strategy-which-one-to-choose
(accessed on 2 January 2020).
16. Mourtzis, D.; Doukas, M. Design and Planning of Manufacturing Networks for Mass Customisation and
Personalisation: Challenges and Outlook. In Proceedings of the 2nd CIRP Robust Manufacturing Conference
(RoMac 2014), Bremen, Germany, 7–9 July 2014; pp. 1–13. [CrossRef]
17. Xun, X. From cloud computing to cloud manufacturing. Robot. Comput. Integr. Manuf. 2012, 28, 75–86.
[CrossRef]
18. Mourtzis, D.; Vlachou, A. a cloud-based cyber-physical system for adaptive shop-floor scheduling and
condition-based maintenance. J. Manuf. Syst. 2018, 47, 179–198. [CrossRef]
19. Azuma, T.R. a Survey of Augmented Reality. Presence Teleoperators Virtual Environ. 1997, 6, 355–385.
[CrossRef]
20. Azuma, T.R. The Most Important Challenge Facing Augmenting reality. Presence Teleoperators Virtual Environ.
2017, 25, 234–238. [CrossRef]
Appl. Sci. 2020, 10, 1855 16 of 16

21. Mourtzis, D.; Zogopoulos, B.; Vlachou, E. Augmented Reality Application to Support Remote Maintenance
as a Service in the Robotics Industry. In Proceedings of the 50th CIRP conference on Manufacturing Systems,
Taichung City, Taiwan, 3–5 May 2017; pp. 46–51. [CrossRef]
22. Henderson, S.; Feiner, S. Exploring the Benefits of Augmented Reality Documentation for Maintenance and
Repair. IEEE Trans. Vis. Comput. Graph. 2011, 17, 1355–1368. [CrossRef] [PubMed]
23. Mourtzis, D.; Angelopoulos, J.; Boli, N. Maintenance assistance application of Engineering to Order
manufacturing equipment: a Product Service System (PSS) approach. In Proceedings of the 16th IFAC
Symposium on Information Control Problems in Manufacturing INCOM 2018, Bergamo, Italy, 11–13 June
2018; pp. 217–222. [CrossRef]
24. Mourtzis, D.; Xanthi, F.; Zogopoulos, V. An Adaptive Framework for Augmented Reality Instructions
Considering Workforce Skill. In Proceedings of the 52nd CIRP Conference on Manufacturing Systems (CMS),
Ljubljana, Slovenia, 12–14 June 2019; pp. 363–368. [CrossRef]
25. Siew, Y.C.; Ong, K.S.; Nee, C.Y.A. a practical augmented reality-assisted maintenance system framework for
adaptive user support. Robot. Comput. Integr. Manuf. 2019, 59, 115–129. [CrossRef]
26. Chryssolouris, G. Manufacturing Systems: Theory and Practice, 2nd ed.; Springer: New York, NY, USA, 2006.
27. Chen, J.; Gusikhin, O.; Finkenstaedt, W.; Liu, Y. Maintenance, Repair, and Operations Parts Inventory
Management in the Era of Industry 4.0. In Proceedings of the 9th IFAC Conference on Manufacturing
Modelling, Management and Control MIM 2019, Berlin, Germany, 28–30 August 2019; pp. 171–176. [CrossRef]
28. Bokrantz, J.; Skoogh, A.; Berlin, C.; Wuest, T.; Stahre, J. Smart Maintenance: An empirically grounded
conceptualization. Int. J. Prod. Econ. 2019, 107534. [CrossRef]
29. Sahal, R.; Breslin, G.J.; Ali, I.M. Big data and stream processing platforms for Industry 4.0 requirements
mapping for a predictive maintenance use case. J. Manuf. Syst. 2020, 54, 138–151. [CrossRef]
30. Ruiz-Sarmiento, J.R.; Monroy, J.; Moreno, F.A.; Galindo, C.; Bonelo, J.M.; Gonzalez-Jimenez, J. a predictive
model for the maintenance of industrial machinery in the context of industry 4.0. Eng. Appl. Artif. Intell.
2020, 87, 103289. [CrossRef]
31. Bousdekis, A.; Lepenioti, K.; Apostolou, D.; Mentzas, G. Decision Making in Predictive Maintenance:
Literature Review and Research Agenda for Industry 4.0. IFAC-PapersOnLine 2019, 52, 607–612. [CrossRef]
32. Efthymiou, K.; Papakostas, N.; Mourtzis, D.; Chryssolouris, G. On a Predictive Maintenance Platform for
Production Systems. Procedia CIRP 2012, 3, 221–226. [CrossRef]
33. Nikolakis, N.; Papavasileiou, A.; Dimoulas, K.; Bourmpouchakis, K.; Makris, S. On a versatile scheduling
concept of maintenance activities for increased availability of production resources. Procedia CIRP 2018, 78,
172–177. [CrossRef]
34. Agora.io. Available online: https://www.agora.io/en/ (accessed on 26 February 2020).
35. GIGA Video Streamer. Available online: https://assetstore.unity.com/packages/tools/video/giga-video-
streamer-125456?_ga=2.75845011.740357347.1582886552-1421986738.1578039987 (accessed on 26 February
2020).
36. Microsoft HoloLens Specification. Available online: https://www.windowscentral.com/hololens-hardware-
specs (accessed on 26 February 2020).
37. Bengtsson, M.; Kurdve, M. Machining Equipment Life Cycle Costing Model with Dynamic Maintenance
Cost. In Proceedings of the 23rd CIRP Conference on Life Cycle Engineering, Berlin, Germany, 22–24 May
2016; pp. 102–107. [CrossRef]

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).