Skip to content

Commit 965f127

Browse files
author
Ole John Aske
committed
Bug#31790329 PERFORMANCE REGRESSION DUE TO TOO MUCH RELEASE OF NDB API OBJECTS
A performance regression was introduced by WL#8351, where too many NDB API objects were released to the memory allocator, instead of being kept in the free-list. The overhead of the memort allocator being used too much, mainly resulted in an increased latency being observed for the transactions at high load. Maximum TPS was also somewhat affected. Root cause seems to be that the NDB API object usage statistic introduced by WL#8351, made assumption about the object usage being normal distributed over the transactions being executed in each Ndb context. That might not be the case, resulting in the object recycling mechanism in WL#8351 not behaving as intended. When instrumenting some customer usecases the observed behavior could typically be a majority (~99%) of transactions using 1-2 NDB API objects, then a few outliers using a lot more of the NDB API objects. As all objects usage was sampled with the same 'priority', these small usages dominated the usage statistics, preventing the larger transactions to be served from the free-list of API objects. As we want even these larger transactions to be served from the free-list, we should focus on sampling these in the statistcs. Smaller usage periode inbetween should be ignored until it is likely that a more permanent change in behaviour has occured. This patch implements logic for ignoring such intermediate low usage periode, and just sample the larger usage peaks - from comments in patch: + * update_stats() is called whenever a new local peak of 'm_used_cnt' + * objects has been observed. + * + * The high usage peaks are most interesting as we want to scale the + * free-list to accomodate these - The smaller peaks inbetween are mostly + * considered as 'noise' in this statistics. Which may cause a too low + * usage statistics to be collected, such that the high usage peaks could + * not be served from the free-list. + * + * In order to implement this we use a combination of statistics and + * heuristics. Heuristics is based on observing free-list behavior of + * an instrumented version of this code. + * + * 1) A 'high peak' is any peak value above or equal to the current + * sampled mean value. -> Added to the statistics immediately . + * 2) A sampled peak value of 2 or less is considered as 'noise' and + * just ignored. + * 3) Other peak values, less than the current mean: + * These are observed over a periode of such smaller peaks, and their + * max value collected in 'm_sample_max'. When the windows size has expired, + * the 'm_sample_max' value is sampled. + * Intn with this heuristic is that temporary reduced usage of objects + * should be ignored, but longer term changes should be acounted for. + * + * When we have taken a valid sample, we use the statistics to calculate the + * 95% percentile for max objects in use of 'class T'. Reviewed by: Frazer Clement <frazer.clement@oracle.com>
1 parent 20703c7 commit 965f127

File tree

3 files changed

+67
-8
lines changed

3 files changed

+67
-8
lines changed

‎storage/ndb/include/util/stat_utils.hpp

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
Copyright (c) 2016, 2020, Oracle and/or its affiliates. All rights reserved.
2+
Copyright (c) 2016, 2021, Oracle and/or its affiliates.
33
44
This program is free software; you can redistribute it and/or modify
55
it under the terms of the GNU General Public License, version 2.0,
@@ -99,7 +99,7 @@ class NdbStatistics
9999
/* Add 'sample' as 'simple moving average' */
100100
m_noOfSamples++;
101101
m_mean += (delta / m_noOfSamples);
102-
m_sumSquare += (delta * (sample - m_mean));
102+
m_sumSquare += fabs(delta * (sample - m_mean));
103103
}
104104
}
105105

‎storage/ndb/src/ndbapi/NdbImpl.hpp

Lines changed: 64 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
Copyright (c) 2003, 2018, Oracle and/or its affiliates. All rights reserved.
2+
Copyright (c) 2003, 2021, Oracle and/or its affiliates.
33
44
This program is free software; you can redistribute it and/or modify
55
it under the terms of the GNU General Public License, version 2.0,
@@ -65,13 +65,64 @@ struct Ndb_free_list_t
6565
Ndb_free_list_t& operator=(const Ndb_free_list_t&);
6666

6767
/**
68-
* Based on a serie of sampled max. values for m_used_cnt;
69-
* calculate the 95% percentile for max objects in use of 'class T'.
68+
* update_stats() is called whenever a new local peak of 'm_used_cnt'
69+
* objects has been observed.
70+
*
71+
* The high usage peaks are most interesting as we want to scale the
72+
* free-list to accomodate these - The smaller peaks inbetween are mostly
73+
* considered as 'noise' in this statistics. Which may cause a too low
74+
* usage statistics to be collected, such that the high usage peaks could
75+
* not be served from the free-list.
76+
*
77+
* In order to implement this we use a combination of statistics and
78+
* heuristics. Heuristics is based on observing free-list behavior of
79+
* an instrumented version of this code.
80+
*
81+
* 1) A 'high peak' is any peak value above or equal to the current
82+
* sampled mean value. -> Added to the statistics immediately .
83+
* 2) A sampled peak value of 2 or less is considered as 'noise' and
84+
* just ignored.
85+
* 3) Other peak values, less than the current mean:
86+
* These are observed over a periode of such smaller peaks, and their
87+
* max value collected in 'm_sample_max'. When the windows size has expired,
88+
* the 'm_sample_max' value is sampled.
89+
* Intention with this heuristic is that temporary reduced usage of objects
90+
* should be ignored, but longer term changes should be acounted for.
91+
*
92+
* When we have taken a valid sample, we use the statistics to calculate the
93+
* 95% percentile for max objects in use of 'class T'.
7094
*/
7195
void update_stats()
7296
{
73-
m_stats.update(m_used_cnt);
74-
m_estm_max_used = (Uint32)(m_stats.getMean() + (2 * m_stats.getStdDev()));
97+
const Uint32 mean = m_stats.getMean();
98+
if (m_used_cnt >= mean)
99+
// 1) A high-peak value, sample it
100+
m_stats.update(m_used_cnt);
101+
else if (m_used_cnt <= 2)
102+
// 2) Ignore very low sampled values, is 'noise'
103+
return;
104+
else
105+
{
106+
// 3) A local peak, less than current 'mean'
107+
if (m_sample_max < m_used_cnt)
108+
m_sample_max = m_used_cnt;
109+
110+
// Use a decay function of current 'mean' to decide how many small samples
111+
// we may ignore - Smaller samples are ignored for a longer time.
112+
const Uint32 max_skipped = (mean*5) / m_used_cnt;
113+
m_samples_skipped++;
114+
if (m_samples_skipped < max_skipped && m_samples_skipped < 10)
115+
return;
116+
117+
// Expired low-value observation periode, sample max value seen.
118+
m_stats.update(m_sample_max);
119+
}
120+
m_sample_max = 0;
121+
m_samples_skipped = 0;
122+
123+
// Calculate upper 95% percentile from sampled values
124+
const double upper = m_stats.getMean() + (2 * m_stats.getStdDev());
125+
m_estm_max_used = (Uint32)(upper+0.999);
75126
}
76127

77128
/** Shrink m_free_list such that m_used_cnt+'free' <= 'm_estm_max_used' */
@@ -83,6 +134,12 @@ struct Ndb_free_list_t
83134
/** Last operation allocated, or grabbed a free object */
84135
bool m_is_growing;
85136

137+
/** Number of consecuitive 'low-peak' values skipped */
138+
Uint32 m_samples_skipped;
139+
140+
/** Max sample value seen in the 'm_samples_skipped' periode */
141+
Uint32 m_sample_max;
142+
86143
/** Statistics of peaks in number of obj 'T' in use */
87144
NdbStatistics m_stats;
88145

@@ -399,6 +456,8 @@ Ndb_free_list_t<T>::Ndb_free_list_t()
399456
m_free_cnt(0),
400457
m_free_list(NULL),
401458
m_is_growing(false),
459+
m_samples_skipped(0),
460+
m_sample_max(0),
402461
m_stats(),
403462
m_estm_max_used(0)
404463
{}

‎storage/ndb/test/ndbapi/testNdbApi.cpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -306,7 +306,7 @@ int runTestMaxOperations(NDBT_Context* ctx, NDBT_Step* step){
306306
}
307307

308308
maxOpsLimit = 100;
309-
Uint32 coolDownLoops = 25;
309+
Uint32 coolDownLoops = 250;
310310
while (coolDownLoops-- > 0){
311311
int errors = 0;
312312
const int maxErrors = 5;

0 commit comments

Comments
 (0)