APM - Performance Management
APM - Performance Management
APM - Performance Management
IT 2014-311
Audit Title:
17 Dec2014
Client:
Lead
Auditor:
Assistant
Auditor:
50
50
Background
Performance Management is performed to ensure network availability, acceptable network
quality, and great experience for subscribers, while avoiding additional CAPEX. Performance
management involves monitoring the network service activity in order to adjust or improve
network performance and traffic management. Network Performance is a continuous process
that is required as the network evolves due to subscriber growth, network enhancement or
coverage.
Network performance is the target evaluation of the network based on design goals versus
the performance achieved, while Network optimization is performed to improve the
performance of the network with existing resources. The dynamic nature of mobile network
caused by network expansion, subscriber growth and new service launch creates multiple
challenges for operators to know exactly where the performance of the network stands
compared to the design objectives. However, the optimization goal is to better utilize
existing network resources and solve potential problems to identify possible solutions for
future planning. Through Network Optimization, the service quality and resources usage of
the network are greatly improved to achieve a balance between coverage, capacity and
quality.
Performance management evaluates and report upon the behavior of network equipment
and the effectiveness of the network. Therefore, a continuous and real-time review of
network performance assists inidentifying and eliminating problems before impacting
services and customers.
ENIDCn xaht e ircon.ufaMtTivlroeNnc aMh gtlweCnor ailAgkc euPryiteVsrOSAPf woperNi,mtecfPoahSwnrti moc&gernkDCEsaPoncterg ifNnoEePr mMg ain gec r in g M a n a g e r M rM. Jru. Sd wa min e Dr iMg .o M a h a y n i
MMO afa nnicaageger mr m e ne tn t
Transmission Operations
NSN
2G, 3G & 4G
2G, 3G & 4G
Huawei
2G, 3G & 4G
2G, 3G & 4G
2G, 3G & 4G
2G & 3G
Alcatel
2G, 3G & 4G
-
Ericsson
2G, 3G & 4G
-
Samsung
4G
4G
Figure 3AVendor Distribution of BSC/RNC & MME nodes for 2G, 3G & 4G
CENTRAL
EAST
BSC-2G
NORTH
RNC-3G
MME-4G
WEST/SOUTH
Core
Network
Nodes
Ericsson
Huawei
MSS
MGW
STP
MSC
HLR
HSS
GMSC
TMGW
TMSC
ITSC
DSA
RMS
Total
28
26
10
8
2
7
5
5
8
99
7
9
2
2
2
1
2
25
Oberthur
Technologies
4
4
28
9 10
Ericsson
7
4
2
2
Huawei
8
2
22
Oberthur Technologies
7
1
5
2
Roamware
Roamware
2
2
Ericsson
3
4
3
10
Huawei
10
10
2
4
2
28
10
8
6 3
4
2
0
10
Ericsson
32
Huawei
10
Servic
e
Service KQI
Availability
Connect time
Successful call attempts
Failed call attempts
Signaling
PDP utilization
PDP success rate
Service KPI
MTBF, MTBR
Loss of Service
Voice Quality
Voice Delay
Voice clarity
Data browsing
Paging
Service
Target
KQI
>99.99%
Hurdle
(Acceptable Value)
>99.9%
KQI
>99%
>98%
Weight
1.0%
1.0%
The service availability shows target key performance indicators for links.
Network Switching Subsystems Voice Service
1
3
3
4
5
6
KPI
Service
Target
KQI
KPI
KPI
>= 99.9%
>=98.0%
Hurdle
(Acceptable
Value)
>= 99.8%
>=97.0%
>=98.0%
>=97.0%
2.0%
KPI
KQI
>=96.0%
>=95.0%
1.0%
>=98.5%
>=97.5%
1.0%
KPI
>=93.0%
>=91.0%
1.0%
Weight
2.0%
2.0%
The Network Switching Subsystem (NSS) is the main part of the Mobile Switching Center
(MSC) that performs the switching of calls between mobile and fixed network users.
11
1
2
3
KPI
Service
Target
KQI
KPI
KPI
>=97.0%
>=98.0%
>=92.0%
Hurdle
(Acceptable
Value)
>=96.0%
>=97.0%
>=91.0%
Weight
1.0%
1.0%
1.0%
Circuit Switching Fallback is a functionfor combining Long Term Evolution (LTE/4G) and
Circuit Switching (CS)services likeVoice over LTE and 3G Generation voice calls.
Access & Roaming Network Level
KPI
1
2
3
4
5
Service
KQI
KQI
Target
>=92.0
%
>=96.0
%
Hurdle
(Acceptable
Value)
Weigh
t
>=90.0%
1.0%
>=95.0%
1.0%
KQI
>=96.0
%
>=95.0%
1.0%
KQI
>=92.0
%
>=90.0%
1.0%
KQI
>=98.0
%
>=96.0%
1.0%
Access and roaming shows the key performance indicators of international roaming which
is the ability to automatically make and receive voice calls, send and receive data, or
access other services when travelling outside the geographical coverage area.
Radio Access Network Level
KPI
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Servic
e
KQI
KQI
KQI
KPI
KPI
KPI
KPI
KPI
KPI
KQI
KQI
KQI
KQI
KPI
Target
Hurdle
(Acceptable value)
Weight
>=99
>=99%
>=98.5
<=0.45%
<=0.45%
>= 98%
<=2%
>=0.8%
>=98.5
>=98.5%
>=97.5
>=0.5%
>=0.5%
>=97.5%
<=3%
<=1%
2%
2%
1%
2%
2%
1%
Reporting
1%
>=99%
>=98.5%
2.5%
>=99%
>=98.5%
2.5%
>=99%
>=98.5%
1.5%
>=97.5%
99.00%
>=98.5%
>=96.5%
98.50%
>=98%
2.0%
1.5%
Reporting
12
KPI
1
2
3
4
5
6
7
8
9
10
Servic
e
Target
KQI
KQI
KPI
KPI
KQI
KQI
KQI
KQI
KQI
KQI
>98%
>98%
<0.6%
<0.6%
>98%
>2Mbs
>98%
>99%
>3Mbs
<0.2%
Percentage of cells
meeting the
threshold
Rural
City Nodes
Node
s
90%
70%
85%
75%
55%
40%
55%
35%
87%
75%
75%
60%
85%
70%
94%
91%
50%
45%
95%
90%
Weight
1%
1%
1%
1%
1%
1%
1%
1%
1%
1%
Network consists of different cells and each cell transmits signals to and receives signals
from mobile stations.
GPRS Level
Service Level
1
2
3
4
5
6
7
8
9
Successful PS Attachment 2G
Successful PS Attachment 3G
Successful PDP context activation 2G
Successful PDP context activation 3G
Inter RAU 3G
PDP cut off ratio
Successful PS Attachment 4G
Successful SAE Bearer context
Activation 4G
PDP context SR (Top 20 operator)
Service
KQI
KQI
KQI
KQI
KPI
KQI
KQI
KQI
KQI
Hurdle
(Acceptable Value)
>=98.0%
>=95.0%
>=99.0%
>=98.0%
>=99.0%
>=98.0%
>=99.9%
>=99.0%
>=99.0%
>=98.0%
<=0.0001
<=0.00015
>=99.0%
>=98.0%
Target
Weight
1.0%
1.0%
1.0%
1.0%
1.0%
1.0%
1.0%
>=99%
>=98.0%
1.0%
>=90%
>=88.0%
1.0%
General Packet Radio Service (GPRS) is a packet oriented mobile data service on the 2G and
3G cellular communication system's global system for mobile communications (GSM).
Tools
Network Performance Management team uses various tools to extract the reports and
measure KPIs for RAN, Core & IP networks. Below are the tools that are used by various
vendors for measuring the network performance.
Vendor
Ericsson
Ericsson
Ericsson
Huawei
Huawei
Huawei
Huawei
NSN
NSN
ALU
ALU
Samsung
Samsung
Anritsu
Anritsu
Cacti
InfoVista
Tool
ENIQ Business Object client
ENIQ Portal
Ericsson Dashboards
Huawei M2000
Huawei U2000
Huawei PRS
Huawei MOS5200
Nokia NetAct
Nokia Dashboards
2G NPO; 3G NPO and LTE NPO
ALU Dashboards
LSM- Radio (Jeddah)
Samsung Dashboards
MasterClaw Business Objects
MasterClaw Dashboards
Cacti traffic graphs
InfoVista portal
Tools Description
Used to monitor Radio access, PS Core and CS core
platforms for monitoring.
Used to monitor Transmission Platforms.
Used to monitor Radio and PS Core platforms.
Used to monitor Access and CS Core platforms.
13
Lebara Dashboards
Weekly
Monthly
Monthly
Utilization
Core
Monthly
Gb Utilization Report
2G REPORTS
Daily
2G CS Daily Radio Network
Performance Report of Huawei on
Node Level.
2G CS Daily Radio Network
Performance Report of Huawei on
Regional Level.
2G PS Daily Radio Network
Performance Report of Huawei on
Node Level.
2G PS Daily Radio Network
Performance Report of Huawei on
Regional Level.
Weekly
Monthly
Monthly TT Report
14
on
Network
__
__
__
Weekly
Monthly
Monthly Report & Monthly TT
Tracking
Utilization
&
congestion
report-central & south
3G REPORTS
Daily
3G Daily Analysis Report
Top 10 Worst
sleeping cells
Cells
Weekly 3G report
and
Network
topology
performance KPIs report
Monthly
and
Weekly Task
15
Budget
During initial interactions it was noted that Mobily Performance team is operating on
outsourced model through Managed Network Services (MNS) contracts and they dont
manage any budget on their own. However, they use performance services, human
resources and tools from vendors having MNS contracts.
Managed Network Service contracts are depicted in the following table:
Vendor
Contract ID
Alcatel-Lucent
318/2013/AlcatelLucent
Ericsson
318/2013/Ericsson
Huawei
318/2013/Huawei
Nokia Siemens
Networks
318/2013/Nokia
Siemens Networks
ZTE
318/2013/ZTE
Begin
Date
Apr201
4
Apr201
4
Apr201
4
Apr201
4
Apr201
4
Expiry
Date
LOI Value
Mar2015
SAR 37,794,000
Mar2017
Mar2019
Mar2019
Mar2019
SAR
653,084,724
SAR
817,153,585
SAR
408,720,000
SAR
532,577,862
Per Year *
SAR 37,794,000
SAR 217,694,908
SAR 163,430,717
SAR 81,744,000
SAR
106,515,572.4
SAMSUNG 4G
(LTE-TDD,
Jan201
SAR
243/2010/SAMSUNG
Dec2015
SAR 62,497,440
Western
4
124,994,880
region)
*Data related to cost of performance has been requested and will be analyzed during fieldwork
16
Audit Objectives
The objective of this audit is to review and evaluate the adequacy and effectiveness of
process, procedure, and technical controls covering key areas related to Network
Performance.The audit will address the following audit objectives:
1. Adequate Governance
Performance Function
Practices
Pertaining
to
the
Network
To review defined roles & responsibilities of selected staff and their duties are
adequately segregated.
To assess and review the availability of skilled and experienced resources for network
performance.
To review the process of considering internal and external factors in the network
performance strategy for long term achievement and alignment with Mobilys vision.
communicated,
3. Budget Management
To assess that adequate controls are in place for preparation, dimensioning, allocation &
management of budget for applicable MNS contracts pertaining to network
performance.
To review the alignment of network performance with business objectives and how
performance plays a role in reduction of CAPEX.
4. Vendor Management
To assess and review Management Network System contracts against the approved
KPIs, SLAs & Penalties.
To assess that process is in place to keep KPI formulae definition uniform across all
vendors.
To assess that adequate controls are in place for vendor resiliency and fair selection.
17
To assess that approved and acceptable target levels of Key Performance Indicators and
Key Quality Indicators are in place, monitored and escalated in timely manner.
To validate that the performance team have effective processes in place to provide the
required reports to relevant teams for actions in timely manner for better network
availability.
To assess the validation of formulae and effectiveness of reports that are monitored and
reviewed by performance team to provide network view for accurate future planning.
To follow up with concerned teams on reported degradations and previous audit results
to ensure that proper steps have been taken to remedy the issues and confirm it
through updated performance reports.
To assess that end to end Key Performance Indicators (KPI) and Key Quality Indicators
(KQI) are aligned throughout the call set up and the packet data flow.
To ensure all performance reports for Transmission network are addressing the required
KPIs and the Quality of Service (QoS) of the transmission network.
To validate that the Radio Access Network, Transmission, and PS/CS CORE are working
according to the desired throughput and that identified degradation are well
communicated to the relevant teams in efficient manners.
To assess and ensure that all installed network elements are available and monitored through
Network Management System tools for proper monitoring and KPI evaluation.
To ascertain that performance of transmission links are adequate and compared to the reasonable
level such as power level are in line with Mobily business requirements and industry best practices.
To assess critical rings of Metro, Long-Haul and Back-Haul links bandwidth utilization and
redundancy are in line with Mobily Business requirements and industry best practices.
18
Audit Scope
The review will focus on Network Performance of RAN, Core & Transmission and samples will
be selected for the years 2013 & 2014. The main areas that will be reviewed during Network
Performance audit are:
Audit Areas
Governance
Strategy
Budget Management
Vendor Management
19
AUDIT AREAS
NO. OF
HOURS
RISK
Rating
(H/M/L
)
1.
Audit planning
90
2.
Governance
30
3.
Strategy
30
4.
Budget Management
30
5.
Vendor Management
40
6.
Adherence to
Performance
Principles
50
7.
Effectiveness of
Performance and
Quality Management
50
8.
RAN, Transmission&
Core Performance
50
9.
Audit Reporting
30
JUSTIFICATION
First time audit requires meeting with
various teams across Mobily.
Policies and procedures pertaining to
network performance features are
documented communicated and are
adhered.
Review if strategy is properly
communicated to performance team
and complied in routine performance
circle.
Review the accuracy of budgeting,
allocation,
revenue
losses
and
financial
discipline
on
vendor
dealings.
Review overall vendor contracts and
assess the KQIs, SLAs, Outages and
penalties.
Review
and
analyze
effective
performance
utilization
as
per
requirements and growth.
Review effectiveness of current
performance& quality to analyze if
these processes can be further
improved.
Review effectiveness of RAN,
Transmission& Core Performance
400
50
*Hours mentioned above are SME Hours only, these hours does not include project
management and team member hours.
20
REPORTING
The audit opening meeting will be held on__________________with___________________.
The report will be addressed to Chief Audit Executive. Prior to the issuance of the
final report, significant finding will be communicated to the client and their
comments will be incorporated into the audit report.
21