MicroRCA: Root Cause Localization of Performance

Issues in Microservices
Li Wu, Johan Tordsson, Erik Elmroth, Odej Kao

To cite this version:

Li Wu, Johan Tordsson, Erik Elmroth, Odej Kao. MicroRCA: Root Cause Localization of Performance
Issues in Microservices. IEEE/IFIP Network Operations and Management Symposium (NOMS), Apr
2020, Budapest, Hungary. �hal-02441640�

HAL Id: hal-02441640

https://hal.inria.fr/hal-02441640
Submitted on 16 Jan 2020

HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est

archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents
entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non,
lished or not. The documents may come from émanant des établissements d’enseignement et de
teaching and research institutions in France or recherche français ou étrangers, des laboratoires
abroad, or from public or private research centers. publics ou privés.
 MicroRCA: Root Cause Localization of
Performance Issues in Microservices
Li Wu∗† , Johan Tordsson∗‡ , Erik Elmroth∗‡ , Odej Kao†
∗ Elastisys,Umeå, Sweden, Email: {li.wu, johan.tordsson, erik.elmroth}@elastisys.com
† Complex and Distributed IT-Systems Group, TU Berlin, Berlin, Germany, Email: odej.kao@tu-berlin.de
‡ Department of Computing Science, Umeå University, Umeå, Sweden

Abstract—Software architecture is undergoing a transition stacks for their services. However, performance anomalies
from monolithic architectures to microservices to achieve re- manifest differently for different technology stacks, making
silience, agility and scalability in software development. However, it hard to detect performance issues and locate root causes;
with microservices it is difficult to diagnose performance issues
due to technology heterogeneity, large number of microservices, 4) frequent updates: microservices are frequently updated to
and frequent updates to both software features and infrastruc- meet customers’ requirements, (e.g., Netflix updates thousands
ture. This paper presents MicroRCA, a system to locate root of times per day [7]). This highly dynamic environment
causes of performance issues in microservices. MicroRCA infers aggravates the difficulty in root cause localization.
root causes in real time by correlating application performance
symptoms with corresponding system resource utilization, with-
To date, many studies have been conducted on root cause
out any application instrumentation. The root cause localization diagnostics in distributed systems, clouds and microservices.
is based on an attributed graph that model anomaly propagation These either require the application to be instrumented
across services and machines. Our experimental evaluation where (e.g., [8], [9]) or numerous metrics to be analyzed (e.g.,
common anomalies are injected to a microservice benchmark [5], [10]). A third class of approaches [11]–[13] avoid these
running in a Kubernetes cluster shows that MicroRCA locates
root causes well, with 89% precision and 97% mean average
limitations by building a causality graph and inferring the
precision, outperforming several state-of-the-art methods. causes along the graph based on application-level metrics.
Index Terms—root cause analysis, performance degradation, With this approach, potential root causes are commonly ranked
microservices through the correlation between back-end services and front-
end services. However, this may fail to identify faulty services
I. I NTRODUCTION that have little or no impact on front-end services.
More and more applications are using microservices archi- In this paper, we propose a new system, MicroRCA1 , to
tectures (MSA) in domains such as internet of things (IoT) [1], locate root causes of performance issues in microservices.
mobile and cloud [2], to build large-scale systems that are MicroRCA is an application-agnostic system designed for
more resilient, robust and better adapted to dynamic customer container-based microservices environments. It collects ap-
requirements. With MSA, an application is decomposed into plication and system levels metrics continuously and detects
self-contained and independently deployable services with anomaly on SLO (Service Level Objective) metrics. Once an
lightweight intercommunication [3]. anomaly is detected, MicroRCA constructs an attributed graph
To operate microservices reliably and with high uptime, with services and hosts to model the anomaly propagation
performance issues must be detected quickly and their root among services. This graph does not only include the service
causes pinpointed. However, it is difficult to achieve this call paths but also include services collocated on the same
in microservices systems due to the following challenges: (virtual) machines. MicroRCA correlates anomaly symptoms
1) complex dependencies: the number of services can often of communicating services with relevant resource utilization
be hundreds, or thousands, (e.g., Uber has deployed 4000 to infer the potential abnormal services and ranks the potential
microservices [4]). Consequently, the dependencies among root causes. With the correlation of service anomalies and
services are much more complex than for traditional dis- resource utilization, MicroRCA can identify abnormal non-
tributed systems. A performance degradation from one service compute intensive services that have non-obvious service
can propagate widely and cause multiple alarms, making anomaly symptoms, and mitigate the effect of false alarms to
it difficult to locate the root causes; 2) numerous metrics: root cause localization. We evaluate MicroRCA by injecting
the number of monitoring metrics available is very high. various anomalies to the Sock-shop2 microservice benchmark
According to [5], Netflix exposes 2 million metrics and Uber deployed on Kubernetes running in Google Cloud Engine
exposes 500 million metrics. It would cause a significant (GCE)3 . The results show that MicroRCA achieves a good
overhead if all these metrics were to be used for performance
issue diagnosis; 3) heterogeneous services: technology het- 1 MicroRCA stands for Microservices Root Cause Analysis
erogeneity [6], one key benefit of MSA, enables development 2 Sock-shop - https://microservices-demo.github.io/
teams to use different programming languages and technology 3 Google Cloud Engine - https://cloud.google.com/compute/
diagnosis result, with 89% in precision and 97% in mean MonitorRank [13], Microscope [12] and CloudRanger [11]
average precision (MAP), which is higher than several state- identify root causes based on application level metrics only.
of-the-art methods. MonitorRank considers internal and external factors, and
In summary, our contributions are threefold: proposes a pseudo-anomaly clustering algorithm to classify
• We propose an attributed graph with service and host external factors, then traverses the provided service call graph
nodes to model the anomaly propagation in container- with a random walk algorithm to identify anomalous services.
based microservices environments. Our approach is Microscope considers communicating and non-communicating
purely based on metrics collected at application and dependencies between services and constructs a service causal-
system levels and requires no application instrumentation. ity graph to represent these two types of dependencies. Next,
• We provide a method to identify the anomalous services it traverses the constructed graph from the front-end service
by correlating service performance symptoms with cor- to find the root cause candidates and ranks them based
responding resource utilization, which adapts well to the on the metrics similarity between candidate and front-end
heterogeneity of microservices. service. CloudRanger constructs an impact graph with causal
• We evaluate MicroRCA by locating root causes from analysis and proposes a second-order random walk algorithm
different types of faults and different kinds of faulty to locate root causes. All of these approaches achieve a good
services. On average of 95 test scenarios, it achieves 13% performance in identifying faulty services that impact a front-
precision improvement over the baseline methods. end service. However these methods commonly fail to identify
The remainder of this paper is organized as follows. Related root causes from backend services of the type that scarcely
work is discussed in Section II. Section III gives an overview impact front-end services. Similar to these works, we also use
of the MicroRCA system. Root cause localization procedures a graph model, and localize root causes with an algorithm
are detailed in Section IV. Section V describes the experimen- similar to random walk. However, we correlate anomalous
tal evaluation and Section VI concludes the paper. performance symptoms with relevant resource utilization to
comprehensively represent services anomaly, which improves
II. R ELATED W ORK the precision of root cause localization.
In recent years, many solutions have been proposed to
identify root causes in distributed systems, clouds and mi- III. S YSTEM OVERVIEW
croservices. Log-based approaches [14]–[17] build problem In this section, we briefly introduce MicroRCA and its main
detection and identification models based on logs parsing. components.
Even though log-based approaches can discovery more in-
formational causes, they are hard to work in real time and A. MicroRCA Overview
require abnormal information to be hidden in logs. Similarly, Figure 1 shows the overview of MicroRCA. Data collection
trace-based approaches [8], [9], [18]–[23] gather information module collects metrics from application and system levels.
through complete tracing of the execution paths, then identify The application level metrics continuously, in particular the
root causes through analyzing the deviation of latencies along response times of microservices, are used to detect perfor-
the paths, e.g., based on machine learning. These approaches mance issues, and metrics from both levels are used to locate
are very useful to debug distributed systems. However, it is a root causes. Once anomalies are detected, the cause analysis
daunting task for developers to understand source code well engine constructs an attributed graph G with service and host
enough to instrument tracing code. nodes to represent the anomaly propagation paths. Next, the
In addition, there are many metrics-based approaches [5], engine extracts an anomalous subgraph SG based on detected
[9]–[13], [24], [25], as well as this work. These use metrics anomalies and infers which service is the most likely to cause
from applications and/or additional infrastructure levels to the anomalies.
construct a causality graph that is used to infer root causes.
Seer [9] and Loud [10] use multiple-level metrics to identify B. Data Collection
root causes. Seer identifies the faulty services and which MicroRCA is designed to be application-agnostic. It collects
resource, like CPU overhead, that causes the service perfor- application and system levels metrics from a service mesh [26]
mance degradation. It is a proactive method which applies and monitoring system separately and stores them in a time
deep learning on a massive amount of data to identify root series database. In container-based microservices, system-level
causes. This approach requires instrumentation of source code metrics include container and host resource utilization, as
to get metrics, meanwhile, its performance may decrease illustrated by container and host in ”Model Overview” in
when microservices are frequently updated. Loud identifies Figure 1. Application level metrics include response times
any faulty components that generate metrics. It uses the between two communicating services, etc.
causality graph of KPI metrics from the anomaly detection
system directly and locates the faulty components by means C. Anomaly Detection
of different graph centrality algorithms. However, it requires Anomaly detection is the starting point of root cause
anomaly detection to be performed on all gathered metrics, localization. MicroRCA leverages the unsupervised learning
which would cause a significant overhead. algorithm Distance-Based online clustering BIRCH [27] as the
 among all communicating services and located hosts. Fig-
Data Collection
ure 2(b) shows an example of a constructed attributed graph
from the initial microservices environment.

B. Anomalous Subgraph Extraction

Cause Analysis Engine
After constructing the attributed graph, we proceed to ex-
anomalies Attributed Anomalous
Anomaly
Graph
G
Subgraph
SG Faulty Service tract the anomalous subgraph based on detected anomalies and
Detection Localization
Construction Extraction assign anomaly-related attributes to the nodes. An anomalous
subgraph SG is a connected subggraph that represents the
RT: 45ms
container{cpu,
anomaly propagation through services and hosts. Note that
s1 s2
mem, we use the response times between communicating services,
…} service node
which are the edges between service nodes in the attributed
host{cpu: 75%, graph as the anomaly detection targets. Once the response time
h1 h2 mem: 62%,
host node
Model Overview …}
of an edge is determined to be an anomaly, we consider the
edge as an anomalous edge and the origin of the edge as an
Figure 1. Overview of MicroRCA main components and root cause localiza- anomalous node.
tion workflow. To obtain the anomalous subgraph, we first extract the
anomaly detection method. We use the slow response time of anomalous service nodes. In Figure 2(a), the response times
a microservice as the definition of an anomaly. between (s1 , s2 ), (s2 , s3 ), (s2 , s4 ) are anomalous. Therefore,
we extract the anomalous edges and take the origins of the
D. Cause Analysis Engine edges (s2 , s3 , s4 in the example), as the anomalous service
Once anomalies are detected, the cause analysis engine nodes. Figure 2(c) depicts the anomalous edges as dashed
starts to locate root causes. The engine is composed of three lines and anomalous services nodes as shadowed nodes. Next,
main procedures: attributed graph construction, anomalous we compute the average of anomalous response times for
subgraph extraction, and faulty service localization. The en- each anomalous service node, denoted as rt a, to a node
gine constructs an attributed graph to represent the anomaly attribute. Finally, we add nodes and edges that are connected
propagation through services and hosts. Next, it extracts the to anomalous service nodes. Figure 2(c) shows the extracted
anomalous subgraph and locates the faulty services with a anomalous subgraph, where adjacent services nodes s1 and s5
graph centrality algorithm named Personalized PageRank [28]. and adjacent host nodes h1 and h2 probably are impacted.
IV. ROOT C AUSE L OCALIZATION C. Faulty Services Localization
In this section, we describe the three procedures to iden-
Once an anomalous subgraph is extracted, we start to
tify root causes, namely attributed graph construction (Sec-
locate the most likely faulty services. We firstly compute the
tion IV-A), anomalous subgraph extraction (Section IV-B) and
similarity between connected nodes by weighing the subgraph,
faulty services localization (Section IV-C).
then assign anomaly scores to anomalous service nodes, and
A. Attributed Graph Construction finally locate root causes based on a graph centrality algorithm.
MicroRCA constructs an attributed graph to represent the Anomalous Subgraph Weighing: Edge weights repre-
anomaly propagation in microservices environments, which is sent similarity between pairs of nodes. Similar to previous
based on the observation that anomalies propagate not only work [10]–[13], we use the Pearson correlation function to
among services along the service call paths but also to services measure similarity, henceforth denoted as corr(i, j). In par-
collocated on the same (virtual) machines [29], [30]. ticular, weights are computed in following three ways:
Our attributed graph consists of a set of services nodes S = • Weight of an anomalous edge wa . From anomalous sub-
{s1 , s2 , ..., sk } and host nodes H = {h1 , h2 , ..., hm } as shown graph extraction (Section IV-B), we get a list of anoma-
in Figure 1. For each service node si , we add edges to all other lous edges. To each anomalous edge, we assign a constant
service sj it communicates with, and all hosts hk it runs on. value α ∈ (0, 1], which denote the anomaly detection
We discover the graph nodes and their dynamic relationships confidence. If the real anomalies are correctly detected,
by enumerating and parsing the metrics monitored at applica- we would set α higher, and vice versa. In Figure 2(d),
tion and system levels. The graph nodes are interconnected as weights w(s1 ,s2 ) , w(s2 ,s3 ) , w(s2 ,s4 ) are assigned α.
follows. When service si sends requests to service sj , we add • Weight between an anomalous service node and a normal
a directed edge from si to sj . If a containerized service si is service node wn . The weight is assigned the correlation
allocated in host hj , we add a directed edge from si to hj . between two response times: the response time between
To construct the attributed graph, we use the metrics of an anomalous service node and a normal service node,
a certain time frame before the anomaly was detected. We and the average anomalous response time rt a of an
assume all microservices run reliably during this time frame anomalous service node. In Figure 2(d), the weight
and the collected metrics can precisely provide relationships w(s1 ,s3 ) between the normal service node s1 and the
 0 i) response times between 1 i) parse metrics 2 i) extract anomalous service nodes 3 i) compute weights (w) 4 i) use personalized pagerank to rank
(s1, s2), (s2, s3), (s2, s4), are ii) build attributed graph model ii) extract adjacent nodes of anomalous ii) compute anomaly score(AS) ii) remove host nodes from the
detected as anomalous service nodes for anomalous service nodes ranked list
ii) microservices and hosts s1 s1 s1
they are running on
wa
wn wh 1. s4
s1 h1 h2 h1 h2 h1 h2
s2 s2 s2 wh s2 2. s2
AS2 3. s3
wh
…
h1 s3 h2 s3 s4
s3 s4 s3 s4 AS3
AS4
wn
s4
s5 s5
s5 s5

(a) Initial Microservices Environment (b) Attributed Graph (G) (c) Anomalous Subgraph (SG) (d) Weighted and Scored Anomalous Subgraph (e) A Ranked List of Root Causes

service node host node anomalous service node anomalous edge

Figure 2. MicroRCA root cause localization procedures.

anomalous node s3 is the correlation between the re- Algorithm 1: Anomalous Subgraph Weighing
sponse time rt(s1 ,s3 ) between s1 and s3 , and the response Input: Anomalous subgraph SG, anomalous edges,
time rt(s2 ,s3 ) which is the average anomalous response anomalous nodes, anomaly response time rt a,
time of s3 . host metrics Uh , response times of edges rt
• Weight between an anomalous service node and a nor- Output: Weighted SG
mal host node wh . We use the maximum correlation
1 for node vj in anomaly nodes do
coefficient between the average anomalous response time
2 for edge eij in in-edges of vj do
rt a of an anomalous service node, and the host resource
3 if rt(i,j) in anomalous edges then
utilization metrics Uh , including CPU, memory, I/O, net-
4 Assign α to wij ;
work, to represent the similarity between service anomaly
5 else
symptoms and the corresponding resource utilization.
6 Assign corr(rt a(j), rt(i,j) ) to wij
Given that the response time of both normal and abnormal
7 end
services have strong correlations with the host resource
8 end
utilization, we take the average of in-edge weights win
h 9 for edge ejk in out-edges of vj do
as a factor of the similarity. Thus, wij between service
10 if node vk is service node then
node i and host node j can be formulated as:
11 Assign corr(rt a(j), rt(j,k) ) to wjk ;
h else
wij = max corr(uk , rt a(i)) · wI (i) (1) 12
k:uk ∈Uh (j)
13 Assign
To summarize, the weight wij between node i and node avg(win (j)) × max(corr(rt a(j), Uh (k)))
j is given by Equation 2, where rt a(i) denotes the average to wjk
anomalous response time of anomalous service node i. Fig- 14 end
ure 2(d) shows these three types of weights along the edges. 15 end
 16 end

 α, if rt(i,j) is anomalous, 17 return Weighted SG

(i,j) , rt a(j)), if i ∈ S and normal,
corr(rt
wij = (2)


 corr(rt(i,j) , rt a(i)), if j ∈ S and normal,

as per Equation 1 if j ∈ H and normal. and its container resource utilization Uc . To summarize, given
anomalous service node sj , the anomaly score AS(sj ) is
The procedure of anomalous subgraph weighing is pre- defined as:
sented in Algorithm 1. In this algorithm, we iterate over the
anomalous nodes and compute the weights for in-edges in lines AS(sj ) = w(sj ) · max corr(uk , rt a(sj )) (3)
k:uk ∈Uc (sj )
L1-L8 and for out-edges in lines L9-L15.
Assigning Service Anomaly Score: We calculate anomaly Localizing Faulty Services: We locate the faulty services
scores for anomalous service nodes and assign them to a node from the anomalous subgraph with a graph centrality algo-
attribute, denoted as AS ∈ [0, 1]. rithm - Personalized PageRank [28], which proved a good
To quantify the anomaly, we take an average weight of performance in capturing anomaly propagation in previous
service node w(sj ) that indicates the impact to linked nodes. work [10], [11], [13], [29]. In Personalized PageRank, the
Furthermore, In container-based microservices, we assume Personalized PageRank vector (PPV) v is regarded as the root
container resource utilization is correlated with service per- cause score for each node. To compute PPV, we first define
w
formance. We thus complement the service anomaly with P as the transition probability matrix, where Pij = P ijwij if
j
the maximum correlation coefficient between the average node i links to node j, and Pij = 0 otherwise. A preference
anomalous response time rt a of the anomalous service node, vector u denotes the preference of nodes, which we assign
 Table I Table II
H ARDWARE AND SOFTWARE CONFIGURATION USED IN EXPERIMENTS . R EQUEST RATES SENT TO MICROSERVICES .

Hardware Configuration Microservices front-end catalogue user carts orders

Component Master node Worker node(x4) Workload generator
Concurrent users 100 100 100 100 100
Operating System Container-Optimized OS Container-Optimized OS 18.04.2 LTS
vCPU(s) 1 4 6 Request rate(/s) 200 300 50 50 20
Memory(GB) 3.75 15 12
Software Version Table III
Kubernetes Istio Prometheus Node-exporter T HE DETAIL OF INJECTED FAULTS .
1.14.1 1.1.5 2.3.1 v0.15.2
Microservices front-end catalogue user carts orders payment shipping
Latency(ms) 200 200 200 200 200 200 200
the value of anomaly scores of the nodes in our method. CPU Hog(vcpu*%) - 2*95 2*95 2*95 2*95 2*99 2*95
This way, the anomaly-related service nodes are visited more Memory Leak(vm*MB) - 2*1024 2*1024 1*2048 1*2048 2*1024 2*1024

frequently when the random teleportation occurs. Formally, we limit the CPU resource to 1vCPU and memory to 1GB.
the personalized PageRank equation [28] is defined as: The replication factor of each microservice is set with 1.
v = (1 − c)P v + cu (4) Workload Generator: We develop a workload generator
using Locust4 , a distributed, open-source load testing tool that
where c ∈ (0, 1) is the teleportation probability, indicating that simulates concurrent users in an application. The workload is
each step jumps back to a random node with probability c, selected to reflect real user behavior, e.g., more requests are
and with probability 1 − c continues forth along the graph. sent to the entry points front-end and catalogue, and fewer
Typically c = 0.15 [28]. After ranking, we removed the to the shopping carts, user and orders services. We distribute
host nodes from the ranked list as MicroRCA is designed requests to front-end, orders, catalogue, user, carts with five
to locate faulty services. As the link between service nodes locust slaves, and provision 500 users that in total generate
in the anomalous subgraph represents the service call-callee about 600 queries per second to sock-shop. The request rate
relationship, we need to reverse the edges before running the of each microservice is listed in Table II.
localization algorithm. We give an example of the ranked list Data Collection: We use the istio5 service mesh, which in
of root causes in Figure 2(e). term uses Prometheus6 , to collect service-level and container-
level metrics and node-exporter7 to collect host-level metrics.
V. E XPERIMENTAL E VALUATION Prometheus is configured to collect metrics every 5 seconds
In this section, we present the experimental setup, exper- and sends the collected data to MicroRCA. In service-level,
imental results, a comparison with state-of-the-art methods, we collect response time between each pair of services. In both
and discuss the characteristics of our approach. container-level and host-level, we collect CPU usage, memory
usage, and the size of total sent bytes.
A. Experimental Setup Faults Injection: Our method is applicable to any type
Testbed: We evaluate MicroRCA in a testbed established of anomaly that manifests itself as increased microservice
in Google Cloud Engine (GCE) where we set up a Kubernetes response time. In this evaluation, we inject three types of
cluster, deploy the monitoring system and service mesh, and faults commonly used in the evaluation of the state-of-the-
run the benchmark named Sock-shop2 . There are four worker art approaches [10], [12], [31] to sock-shop microservices to
nodes and one master node in the cluster, three of the worker simulate the performance issue. (1) Latency, we use the tc 8
nodes are dedicated to microservices and one for data collec- to delay the network packets; (2) CPU hog, we use stress-ng
9
tion. In addition, one server outside of the cluster runs the , a tool to load and stress compute system, to exhaust CPU
workload generator. Table I describes the detail configuration resources. As microservice payment is non-compute intensive
of the hardware and software in the testbed. whose CPU usage is only 50mHz, we exhaust its CPU heavily
Benchmark: Sock-shop2 is a microservice demo applica- with 99% usage. (3) Memory leak: we use stress-ng to allocate
tion that simulates an e-commerce website that sells socks. memory continuously. As microservice carts and orders are
It is a widely used microservice benchmark designed to CPU and memory intensive services, and memory leak causes
aid demonstration and testing of microservices and cloud- CPU overhead [27], we only provision 1 virtual machine. The
native technologies. Sock-shop consists of 13 microservices, details of the injected faults are described in Table III.
which are implemented in heterogeneous technologies and To inject performance issues in microservices, we customize
intercommunicate using REST over HTTP. In particular, front- the existing sock-shop docker images with installing above
end serves as the entry point for user requests; catalogue faults injection tools. Each fault lasts 1 minute. To increase
provides a sock catalogue and product information; carts holds
4 Locust - https://locust.io/
shopping carts; user provides the user authentication and store
5 Istio - https://istio.io/
user accounts, including paymenet cards and addresses; orders 6 Prometheus - https://prometheus.io/
places orders from carts after user log-in through the user 7 Node-exporter - https://github.com/prometheus/node xporter
e
service, then process the payment and shipping from the pay- 8 tc - https://linux.die.net/man/8/tc

ment and shipping services separately. To each microservice, 9 stress-ng - https://kernel.ubuntu.com/ cking/stress-ng/
the generality, we repeat the injection process 5 times for each Table IV
fault. This produces a total of 95 experiment cases. P ERFORMANCE OF M ICRO RCA.
Baseline Methods: We compare MicroRCA with some microservics front-end orders catalogue user carts shipping payment average
baseline methods as follows: name
Latency
• Random selection(RS): Random selection is a way that an PR@1 1.0 1.0 1.0 0.6 1.0 0.6 1.0 0.89
operating team use without specific domain knowledge PR@3 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
MAP 1.0 1.0 1.0 0.91 1.0 0.89 1.0 0.97
of the system. Every time, the operating team randomly CPU Hog
selects one microservice from the uninvestigated mi- PR@1 - 1.0 1.0 1.0 0.8 1.0 0.6 0.9
PR@3 - 1.0 1.0 1.0 1.0 1.0 1.0 1.0
croservices to investigate until they find the root cause. MAP - 1.0 1.0 1.0 0.94 1.0 0.89 0.97
• MonitorRank [13]: MonitorRank uses a customized ran- Memory Leak
PR@1 - 1.0 0.8 1.0 1.0 0.8 0.8 0.9
dom walk algorithm to identify the root cause. It is similar
PR@3 - 1.0 1.0 1.0 1.0 1.0 1.0 1.0
with personalized PageRank[28] we adopt in MicroRCA. MAP - 1.0 0.97 1.0 1.0 0.94 0.94 0.975
As we only consider the internal faults in microservices,
we do not implement the pseudo-anomaly clustering The average precision of shipping and payment that is lower
algorithm to identify the external factors. To implement than other services is likely for three reasons. First, payment is
MonitorRank, we use their customized random walker al- a non-compute intensive service, even though we exhaust the
gorithm to identify root causes in the extracted anomalous resource of CPU and memory, its response time is scarcely
subgraph . impacted. Second, there are few requests to shipping and
• Microscope [12]: Microscope is another graph-based
payment and they do not request any other services, which
method to identify faulty services in microservices en- makes the response time increase less obviously than other
vironment and it also takes sock-shop as the benchmark. faulty services. Third, in order to detect anomalies in their
To implement Microscope, we construct the services experimental cases, we use a small threshold in anomaly
causality graph with paring the service-level metrics, then detection module, which causes more false alarms.
use their cause inference, which traverses the causality Table IV demonstrates the performance of MicroRCA in
graph with the detected anomalous services nodes, to different types of faults and microservices. It shows that
locate root causes. MicroRCA can achieve almost 90% in terms of PR@1 and
effectively locate all root causes in the top three faulty services.
The anomaly detection methods in MonitorRank and Mi-
croscope fail to detect anomalies, especially for non-obvious C. Comparisons
anomalies, like payment service. As we in our experiments
aim to compare the root cause localization performance, we To evaluate the performance of MicroRCA further, we apply
thus use the same results from our anomaly detection module. it and the baseline methods on all experimental cases.
Evaluation Metrics: To quantify the performance of each We compare the overall performance of all methods and
algorithm on a set of anomalies A, we use following metrics: their performances in identifying different types of faults.
Table V shows the performance, in terms of PR@1, PR@3
• Precision at top k denotes the probability that the top k
and MAP, for all methods. We can observe that MicroRCA
results given by an algorithm include the real root cause,
outperforms the baseline methods in overall. In particular,
denoted as P R@k. A higher P R@k score, especially
MicroRCA achieves a precision of 89% and MAP of 97%,
for small values of k, represents the algorithm correctly
which are at least 13% and 15% higher than the baseline
identifies the root cause, Let R[i] be the rank of each
methods separately. In general, Microscope performs well in
cause and vrc be the set of root causes. More formally,
CPU hog and memory leak when the anomalies are detected
P R@k is defined on a set of given anomalies A as:
P correctly but worse in fault latency when more false alarms
1 X i<k (R[i] ∈ vr c ) are detected. However, MicroRCA performs well in all types
P R@k = (5)
|A| (min(k, |vr c |)) of faults, and achieves average improvement of 24% in MAP
a∈A
comparing to MonitorRank and Microscope.
• Mean Average Precision (MAP) quantifies the overall Next, we compare the performance of each method on
performance of an algorithm, where N is the number of different microservices. Figure 4 shows the comparison results,
microservices: in terms of PR@1, PR@3 and MAP, on different services. We
1 X X can see that MonitorRank performs well in identify dominating
M AP = P R@k. (6)
|A| nodes which have large degrees, like microservice orders.
a∈A 1≤k≤N
However it fails to identify leaf nodes, like microservice pay-
B. Experimental Results ment. This is because MonitorRank calculates the similarity
Figure 3 shows the results of our proposed root cause based on the correlation between front-end services and back-
localization method for different types of faults and sock-shop end services. The anomaly of microservice payment decreases
microservices. We observe that all services achieve a MAP the correlation during propagation and thus the root cause
in the range of 80%-100% and an average PR@1 over 80% localization fails. On the contrary, Microscope performs better
except shipping and payment. in identifying leaf nodes, such as the microservice payment,
 Latency
Precision MAP CPU Hog
Memory Leak
1.0 1.0
0.8 0.8
0.6 0.6
0.4 0.4
0.2 0.2
0.0 0.0
front-end orders catalogue user carts shippingpayment front-end orders catalogue user carts shippingpayment
Figure 3. The result of PR@1 and MAP.

Table V Table VI
P ERFORMANCE OF EACH ALGORITHM . T HE OVERHEAD OF M ICRO RCA.

Metric RS MonitorRank Microscope MicroRCA Improvement to Improvement to Modules Cost

MonitorRank (%) Microscope(%)
Overall
Data collection 0.6 vCPU and 1511MB RAM
PR@1 0.21 0.41 0.79 0.89 118 13.3 Anomaly Detection 0.01s (8 cores)
PR@3 0.46 0.65 0.86 1.0 53.2 15.9 Attributed Graph Construction 3.3s (8 cores)
MAP 0.58 0.73 0.85 0.97 33.7 14.7
Root Cause Localization 0.03 (8 cores)
Latency
PR@1 0.17 0.23 0.66 0.89 287 34.8
PR@3 0.46 0.66 0.71 1.0 51.5 40.8
MicroRCA identifies the root cause correctly both in scenarios
MAP 0.58 0.73 0.7 0.97 32.9 38.6 with high and low F1-score.In particular, MicroRCA identifies
CPU Hog
PR@1 0.23 0.6 0.87 0.9 50 3.5
13 out of 18 faults in top 1 when F1-score is less than 0.4,
PR@3 0.5 0.67 0.93 1.0 49.3 7.5 which is higher than the other two methods.
MAP 0.61 0.77 0.92 0.97 26 5.4
Memory Leak
PR@1 0.23 0.43 0.87 0.9 109 3.5
D. Discussion
PR@3 0.37 0.63 0.97 1.0 58.7 3
Here we discuss the overhead and sensitivity of our pro-
MAP 0.57 0.68 0.95 0.98 44.1 3.2
posed MicroRCA system.
but worse in identify dominating node, like microservice Overhead: The overhead of MicroRCA on the microser-
orders. This is because Microscope traverses the graph based vices system is mostly caused by the data collection module,
on the detected anomalies and put the anomalous child nodes which collects the application-level and system-level metrics
into a list of potential causes, which makes it fail to identify the continuously. Table VI shows the overhead of data collection
dominating nodes when alarms are reported from child nodes. and the execution time of modules in MicroRCA. We can
Comparing to MonitorRank and Microscope, MicroRCA gen- see that the execution time of MicroRCA is short enough to
erally has a good performance for all the services, no matter run in real-time given a data collection interval of 5 seconds.
dominating nodes or leaf nodes. However the overhead of the data collection module is a
Finally, we compare the root cause localization performance little high. In the future, we would like to explore lightweight
of each method against the anomaly detection results to eval- monitoring tools to reduce this overhead.
uate their sensitivity to the performance of anomaly detection. Sensitivity: To evaluate the sensitivity of MicroRCA to
We compute the F1-score of each experimental case and the anomaly detection confidence(α), which is assigned to
count the number of different ranks of all experiment cases the anomalous edges in weighing anomalous subgraph (sec-
to different F1-scores. F1-score is computed as the harmonic tion IV-C). We analyze the performance of MicroRCA with
mean of precision and recall: 2 × precision×recall
precision+recall . In our different values of α. Figure 6 shows the performance of
experiment, we only inject one fault to one microservice, MicroRCA, in terms of overall PR@k(k=1,2,3) and MAP
which means only one root cause in a case. Thus in each case, of different types of faults over all the test cases, when α
the recall is 1 if the root cause is in the ranked list, otherwise 0. ranges from 0.15 to 1.0. We can see that the performance of
Precision is the number of detected root causes divided by the MicroRCA changes with the weight assigned to the anomalous
number of detected anomalies. As RS(random selection) is not nodes. PR@1 increases to the maximum when α = 0.55 and
related to anomaly detection results, we analyze MonitorRank, drops when α > 0.7; PR@3 is always 1 when α is less
Microscope and our MicroRCA only. than 0.55, and drops after that. MAP of all type of faults are
Figure 5 shows the number of different ranks against F1- relatively stable. and keeps over 96%. In this case, we choose
score for MonotorRank, Microscope and MicroRCA sepa- 0.55 as the weight of anomalous edges where the PR@1 and
rately. To show the results compactly, we set rank with 10, PR@3 are the maximum. In practise, as different anomaly
which is higher than the total number of microservices, when detection methods have different performances, we need to
the root cause is failed to locate. We can see all of them have tune the anomaly detection confidence(α) according to the
a good performance when the faulty services are correctly de- anomaly detection module.
tected. However, when false alarms are frequent, Microscope Furthermore, we analyze the sensitivity to anomaly detec-
cannot identify root causes in some cases; MonitorRank can tion results with different thresholds in anomaly detection
identify root causes, but root causes are always low-ranked; module. Figure 7 shows the MAP of MicroRCA and other
 PR@1 PR@3 MAP
user user user
catalogue catalogue catalogue

carts 1.0 carts 1.0 carts 1.0

0.8 0.8 0.8
0.6 0.6 0.6
0.4 0.4 0.4
0.2 0.2 0.2
front-end front-end front-end

RS
MonitorRank
orders orders orders Microscope
MicroRCA
payment payment payment
shipping shipping shipping

Figure 4. The comparison results of PR@1,PR@3 and MAP for different microservices.

10
MonitorRank Microscope 10
MicroRCA
10 Number
9 9 9 of cases
8 8 8
7 7 7 1
6 6 6
Rank

Rank

Rank
5 5 5 5
4 4 4
3 3 3 10
2 2 2
1 1 1 20
0 0 0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
F1-score F1-score F1-score
Figure 5. Root cause rank against anomaly detection F1-Score.

1.05 1.0
0.9
1.00
0.8
Performance

0.95 0.7
MAP

0.90 0.6
0.5 F1-score
0.85 PR@1 MAP:Latency MonitorRank
PR@2 MAP:CPU Hog 0.4 Microscope
PR@3 MAP:Memory Leak MicroRCA
0.80 0.3
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.02 0.04
0.06 0.08 0.10
threshold
Figure 6. Performance against anomaly detection confidence(α). Figure 7. Performance against anomaly detection threshold.

two baseline methods in solid lines and F1-score in dash line

when threshold ranges from 0.01 to 0.1. Note that we set α with a microservice benchmark show that MicroRCA achieves
to the constant 0.55 when MicroRCA regularly performs well. 89% in precision and 97% in mean average precision (MAP).
We can see that the three methods share the similar pattern In particular, our method improves the precision in identifying
with F1-score, and MicroRCA performs well in a range from root causes from both dominating services with large degrees
0.02 to 0.065, which is wider than other two methods. As all and leaf services with non-obvious anomaly symptoms, where
three methods performs well when threshold is 0.045, we use the state-of-the-art methods fall short.
it in the anomaly detection module for all above analysis. As the involved tools (Kubernetes, Istio, etc) are not de-
signed to operate with sub-second monitoring intervals, this
VI. C ONCLUSION AND F UTURE W ORK adds a limitation to how fast anomalies can be detected
In this paper, we proposed a new application-agnostic with our approach, rendering our system unable to identify
system, MicroRCA, for root cause localization of performance anomalous services for which the anomalies propagate faster
anomalies in container-based microservices. It constructs an than our monitoring interval. In the future, we would like to
attributed graph model and correlates service anomalous per- improve our system to overcome this limication. Furthermore,
formance symptoms with corresponding resource utilization to we would like to study the automatic remediation methods for
infer the anomalous microservices. Experimental evaluations performance degradations in microservices environments.
 ACKNOWLEDGMENT [16] I. Weber, C. Li, L. Bass, X. Xu, and L. Zhu, “Discover-
ing and visualizing operations processes with pod-discovery
This work is part of the FogGuru project which has received
and pod-viz”, in 2015 45th Annual IEEE/IFIP International
funding from the European Union’s Horizon 2020 research and Conference on Dependable Systems and Networks, 2015,
innovation programme under the Marie Skłodowska-Curie grant pp. 537–544.
agreement No 765452. The information and views set out in this [17] Jiaqi Tan, Xinghao Pan, E. Marinelli, S. Kavulya, R.
publication are those of the author(s) and do not necessarily reflect the Gandhi, and P. Narasimhan, “Kahuna: Problem diagnosis for
mapreduce-based cloud computing environments”, in 2010
official opinion of the European Union. Neither the European Union
IEEE Network Operations and Management Symposium -
institutions and bodies nor any person acting on their behalf may be NOMS 2010, 2010, pp. 112–119.
held responsible for the use which may be made of the information [18] X. Zhou, X. Peng, T. Xie, J. Sun, C. Ji, W. Li, and D.
contained therein. Ding, “Fault analysis and debugging of microservice systems:
Industrial survey, benchmark system, and empirical study”,
R EFERENCES IEEE Transactions on Software Engineering, pp. 1–1, 2018.
[1] B. Butzin, F. Golatowski, and D. Timmermann, “Microser- [19] M. Chow, D. Meisner, J. Flinn, D. Peek, and T. F. Wenisch,
vices approach for the internet of things”, in 2016 IEEE “The mystery machine: End-to-end performance analysis of
21st International Conference on Emerging Technologies and large-scale internet services”, in OSDI’14, 2014, pp. 217–231.
Factory Automation (ETFA), 2016, pp. 1–6. [20] H. Mi, H. Wang, Y. Zhou, M. R. Lyu, and H. Cai, “Toward
[2] P. D. Francesco, I. Malavolta, and P. Lago, “Research on fine-grained, unsupervised, scalable performance diagnosis for
architecting microservices: Trends, focus, and potential for production cloud computing systems”, IEEE Transactions on
industrial adoption”, in 2017 IEEE International Conference Parallel and Distributed Systems, vol. 24, no. 6, pp. 1245–
on Software Architecture (ICSA), 2017, pp. 21–30. 1255, 2013.
[3] S. Newman, Building Microservices. 2015. [21] M. Chen, A. X. Zheng, J. Lloyd, M. I. Jordan, and E. Brewer,
[4] How Uber Is Monitoring 4,000 Microservices with Its Open “Failure diagnosis using decision trees”, in International Con-
Sourced Prometheus Platform, (accessed: 03.07.2019). ference on Autonomic Computing, 2004. Proceedings., 2004,
[5] J. Thalheim, A. Rodrigues, I. E. Akkus, P. Bhatotia, R. Chen, pp. 36–43.
B. Viswanath, L. Jiao, and C. Fetzer, “Sieve: Actionable [22] P. Barham, A. Donnelly, R. Isaacs, and R. Mortier, “Using
insights from monitored metrics in distributed systems”, in magpie for request extraction and workload modelling”, in
Middleware ’17, 2017, pp. 14–27. OSDI’04, 2004, pp. 18–18.
[6] N. Dragoni, S. Giallorenzo, A. L. Lafuente, M. Mazzara, [23] M. Y. Chen, E. Kiciman, E. Fratkin, A. Fox, and E. Brewer,
F. Montesi, R. Mustafin, and L. Safina, “Microservices: Yes- “Pinpoint: Problem determination in large, dynamic internet
terday, today, and tomorrow”, Book: Present and Ulterior services”, in Proceedings International Conference on De-
Software Engineering, pp. 195–216, 2017. arXiv: 1606.04036. pendable Systems and Networks, 2002, pp. 595–604.
[7] Why Netflix, Amazon, and Apple Care About Microservices, [24] P. Chen, Y. Qi, and D. Hou, “Causeinfer: Automated end-to-
(accessed: 03.07.2019). end performance diagnosis with hierarchical causality graph
[8] R. Fonseca, G. Porter, R. H. Katz, S. Shenker, and I. Sto- in cloud environment”, IEEE Transactions on Services Com-
ica, “X-trace: A pervasive network tracing framework”, in puting, vol. 12, no. 2, pp. 214–230, 2019.
NSDI’07, 2007, pp. 20–20. [25] B. Sharma, P. Jayachandran, A. Verma, and C. R. Das,
[9] Y. Gan, Y. Zhang, K. Hu, D. Cheng, Y. He, M. Pancholi, and “Cloudpd: Problem determination and diagnosis in shared dy-
C. Delimitrou, “Seer: Leveraging big data to navigate the com- namic clouds”, in 2013 43rd Annual IEEE/IFIP International
plexity of performance debugging in cloud microservices”, in Conference on Dependable Systems and Networks (DSN),
ASPLOS ’19, 2019, pp. 19–33. 2013, pp. 1–12.
[10] L. Mariani, C. Monni, M. Pezzé, O. Riganelli, and R. Xin, [26] What’s a service mesh? And why do I need one?, (accessed:
“Localizing faults in cloud systems”, in ICST, 2018, pp. 262– 27.04.2019).
273. [27] A. Gulenko, F. Schmidt, A. Acker, M. Wallschläger, O. Kao,
[11] P. Wang, J. Xu, M. Ma, W. Lin, D. Pan, Y. Wang, and P. and F. Liu, “Detecting anomalous behavior of black-box ser-
Chen, “Cloudranger: Root cause identification for cloud native vices modeled with distance-based online clustering”, in 2018
systems”, in CCGRID, 2018, pp. 492–502. IEEE 11th International Conference on Cloud Computing
[12] J. Lin, P. Chen, and Z. Zheng, “Microscope: Pinpoint per- (CLOUD), 2018, pp. 912–915.
formance issues with causal graphs in micro-service environ- [28] G. Jeh and J. Widom, “Scaling personalized web search”, in
ments”, in Service-Oriented Computing, C. Pahl, M. Vukovic, WWW ’03, 2003, pp. 271–279.
J. Yin, and Q. Yu, Eds., 2018, pp. 3–20. [29] J. Weng, J. H. Wang, J. Yang, and Y. Yang, “Root cause
[13] M. Kim, R. Sumbaly, and S. Shah, “Root cause detection in analysis of anomalies of multitier services in public clouds”,
a service-oriented architecture”, SIGMETRICS Perform. Eval. IEEE/ACM Transactions on Networking, vol. 26, no. 4,
Rev., vol. 41, no. 1, pp. 93–104, 2013. pp. 1646–1659, 2018.
[14] J. Xu, P. Chen, L. Yang, F. Meng, and P. Wang, “Logdc: [30] O. Ibidunmoye, F. Hernández-Rodriguez, and E. Elmroth,
Problem diagnosis for declartively-deployed cloud applica- “Performance anomaly detection and bottleneck identifica-
tions with log”, in 2017 IEEE 14th International Conference tion”, ACM Comput. Surv., vol. 48, no. 1, 4:1–4:35, 2015.
on e-Business Engineering (ICEBE), 2017, pp. 282–287. [31] H. Kang, H. Chen, and G. Jiang, “Peerwatch: A fault detection
[15] T. Jia, P. Chen, L. Yang, Y. Li, F. Meng, and J. Xu, “An and diagnosis tool for virtualized consolidation systems”, in
approach for anomaly diagnosis based on hybrid graph model ICAC ’10, 2010, pp. 119–128.
with logs for distributed services”, in 2017 IEEE International
Conference on Web Services (ICWS), 2017, pp. 25–32.