COMPARABILITY OF

METHODS AND ANALYSERS

Nora Nikolac
Zagreb, October 15th EFLM Continuing Postgraduate Course in Clinical
24-25, 2015 Chemistry and Laboratory
 When?
2

Introducing new method or analyzer

Multiple analytical systems in laboratory

Using services of another laboratory

 Why?
3

 To increase patient safety

 To assure that method change is not going to
influence laboratory result for the patient.
 How?
4

 Experimental procedures following protocols

 CLSI EP09-A3: Measurement procedure comparison
and bias estimation using patient samples

1. Number of samples
2. Measurement range
3. Time of analysis
4. Data analysis
5. Data interpretation
 1. Number of samples
5

 Min: 40 samples
 Optimal: 100 samples

 To
identify unexpected errors from sample
matrix or interferences
 Measurements in duplicate
 2. Measurement range
6

 Cover 90% of the method measurement range

Method B

Good agreement
between methods
Difference in
higher
concentration
range

Method A

Measurement range
 2. Measurement range
7

 Overlaping measurement range for both methods

Method A determined
using dilution protocol
Method A

Method B Glucose
Method B concentration
reported as LOQ
 3. Time of analysis
8

 Measurements done within 2 hours

 Not for: glucose, lactate, ammonia,
blood gass testing...

 Measurements done over 5 days

 Better over longer period of time

 Collecting samples over period of time (first method)

and analyzing in batch using second method
 4. Analyzing results
9

Several statistical aproaches:

 Correlation

 Paired test for difference

 Linear regression

 Deming regression
 Passing-Bablok regresion

 Bland-Altman analysis
 4. Analyzing results
10

 Comparison of two methods for direct bilirubin

concentration measurement
Summary data
Method 1 Method 2
N=40 N=40
Analyzer, method Architect (Abbott) AU 680 (Beckman Coulter)
Diazo method DPD method
Min-Max 2.7-232.3 5.5-273.4
Mean ± SD 65.5 ± 67.9 82.4 ± 83.6
Median (IQR) 38.5 (7.9-127.8) 42.4 (11.1-158.2)
P (normality) 0.059 0.036
 4.1 Correlation
11

 Spearman coefficient of correlation

r (95% CI) =
0.97 (0.95-0.98)

Excellant
correlation
 What is the meaning of this result?
12

 Methods are significantly associated

 Linear relation between methods
 ↑ of Method A associated with ↑ of Method B
 Nothing about amount of increase!

Same correlation coefficient!

 4.2 Significance of difference
13

 Wilcoxon test (normality failed)

P < 0.001

Significant
difference
between methods
 What is the meaning of this result?
14

 Calculating differences for each pair of measurement

 Comparing number of negative and positive
differences
 If there is no difference between methods, number of
differences is equal

More measurements were higher using Method 2

 4.3 Linear regression
15

Equation to describe relationship between

High correlation methods
Linear relationship Determine proportional and constant
error
Deming regression
Passing and Bablok regression

Bilić-Zulle L. Comparison of methods: Passing and Bablok regression. Biochem Med (Zagreb) 2011;21:49-52.
 Linear regression
16

95% confidence
y= x intervals
Method B
y

Regression equation
y = a + bx
tg (α) = b Regression equation
α
y = a (95% CI) +
Intercept = a b (95% CI) x

Method A x
 Constant and proportional error
17

Regression equation
y = a (95% CI) + b (95% CI) x
Method B
y y= x Excluding 1
Excluding 0
Proportional
Constant error
error

tg (α) = b
α

Intercept = a

Method A x
 Deming regression
18

 Includes analytical
variability of both
methods (CV)
 Assumes that errors
are independent and
normally distributed
 Both methods prone to
errors
y = 1.74 (-1.77 to 5.24) + 1.23 (1.16 to 1.30) x
No constant error Proportional error
 Passing-Bablok regression
19

 Non-parametric method
 No assumptions about distributions of samples
 No assumptions about distributions of errors
 Not sensitive to outliers
 Why don’t we recalculate results?
20

Direct bilirubin (Method 2) =

1.23 x Direct bilirubin (Method 1)

Direct bilirubin (Method 1) =

Direct bilirubin (Method 2) / 1.23
 Residual analysis
21

 How well data fit to the regression model

>0 >0

0 x 0 x

<0 <0
Y––F(x)
Y F(X) Y – F(X)
y y
 Residual analysis
22

Differences between measured and calculated values

 4.3 Bland-Altman analysis
23

 Graphical method to compare two measurements

technique
 Analyzing differences between measurement pairs

Giavarina D. Understanding Bland Altman analysis. Biochem Med (Zagreb) 2015;25(2):141-51.

 Mountain plot
24

More positive
differences

Normal distribution
of differences
 4.3 Bland-Altman analysis
25

 Plotting differences against:

 Mean of two methods (no reference method)
 One method (reference method)
+1.96 s Limits of
agreement
95% CI
method 2 – method 1

Mean
difference
95% CI 0
Limits of
-1.96 s agreement

Mean of method 1 and method 2

 LoA and mean difference
26

Absolute units
Wide LoA =
Constant bias
poor agreement
+1.96 s Limits of
agreement
95% CI Including 0 =
method 2 – method 1

Mean no constant bias

difference Excluding 0 =
95% CI 0 Constant bias
Limits of
-1.96 s agreement

Mean of method 1 and method 2

Narrow LoA =
good agreement
Percentage (%)
Proportional bias
 Bland-Altman analysis
27

Plotting against mean difference Plotting against % difference

No constant bias Proportional bias

 5. Data interpretation
28

Statistical significance Clinical significance

Comparing values
with predefined
acceptance criteia
 Method comparison
29

 Important laboratory procedure for verification

 Included into validation protocols for new reagents
 Comparison with the reference method
 Comparison with different manufacturers

 Comparison with same manufacturer

 Results are presented in manufacturers declarations

 Can we rely on manufacturers declarations?
30

Correlation for
 Comparing 7 insert sheets for glucose concentration measurement
determination of
agreement
Intercept Slope
Manufacturer N Unit r
(95% CI) (95% CI)

A 102 mg/dL 0.9993 -4.54 (?-?) 1.06 (?-?)

B 117 mmol/L 0.998 -0.081 (?-?) No BA (?-?)
1.037 analysis
C 75 mmol/L 1.000 0.179 (?-?) 0.996 (?-?)
D 43 mg/dL 0.9977 -2.6 (?-?) 1.084 (?-?)
E ? mg/dL 0.999 0.68 (?-?) 0.99 (?-?)
F 40 mg/dL 0.98 -3.14 (?-?) 0.98 (?-?)
G 60 mmol/L 0.998 No 95% CI(?-?)
0.08 for 1.008 (?-?)
evaluation of
bias
 To conclude
31

Clinically relevant
criteria

Interpretation Laboratory
of results methods

Number of samples
Linear regression Measurement range
analysis Data Verification Time of analysis
Bland-Altman plot analysis procedure Data analysis
Data interpretation
Take a home massage

Comparability of methods and analyzers

 Coefficient of correlation doesn’t allow conclusions about
comparability of methods, but only about linear association between
them, even when it is very high (close to 1)
 Regression equation: Y = 0.67 (-0.15-1.32) + 1.09 (1.03-1.22) x is
an example of proportional bias between methods (95% CI for
slope not including 1) without constant bias between methods (95%
CI for intercept including 0)
 Regression equation for glucose concentration: Y = 0.07 (0.01-0.13)
+ 1.15 (0.85-1.23) x (mmol/L) is an example of statistically
significant, but clinically non-significant constant bias. Value of 0.07
(0.01-0.13) mmol/L glucose is lower than conventional analytical
performance of the test