Iso-Cd - 16140-5 (E)

ISO/CD 16140-5
ISO/TC 34/SC 9/WG 3
Secretariat: NEN
2016-02-08
Microbiology of the food chain — Method validation — Part 5:

Protocol for factorial interlaboratory validation for non-
proprietary methods
Committee Draft
Warning for WDs and CDs
This document is not an ISO International Standard. It is distributed for review and comment. It is subject to
change without notice and may not be referred to as an International Standard.
Recipients of this draft are invited to submit, with their comments, notification of any relevant patent rights of
which they are aware and to provide supporting documentation.
ISO/CD 16140-5
© ISO 2016
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or
utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or
posting on the internet or an intranet, without prior written permission. Permission can be requested
from either ISO at the address below or ISO's member body in the country of the requester.
ISO copyright office

Case postale 56  CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.org
Web www.iso.org
Published in Switzerland.
2 © ISO 2016– All rights reserved

ISO/CD 16140-5
Contents
Foreword 5
Introduction 6
1 Scope 7
2 Normative references ...............................................................................................................................................8
3 Terms and definitions ..............................................................................................................................................8
4 General principles for the validation of non-proprietary methods.........................................................9
4.1 Validation against a reference method ....................................................................................................................... 9
4.2 Validation without reference method ......................................................................................................................... 9
5 Qualitative methods - Technical protocol for factorial interlaboratory validation ....................... 10
5.1 In-house validation study .............................................................................................................................................. 10
5.1.1 General considerations ........................................................................................................................................... 10
5.2 Interlaboratory study ...................................................................................................................................................... 10
5.2.1 General considerations........................................................................................................................................... 10
5.2.2 Measurement protocol ........................................................................................................................................... 10
5.2.3 Experimental design ................................................................................................................................................ 12
5.3 Calculations and summary of data ............................................................................................................................. 13
5.4 Interpretation ..................................................................................................................................................................... 14
6 Quantitative methods - Technical protocol for factorial interlaboratory validation..................... 14
6.1 In-house validation study .............................................................................................................................................. 14
6.2 Interlaboratory study ...................................................................................................................................................... 15
6.2.3 Measurement protocol ........................................................................................................................................... 15
6.2.3 Experimental design ................................................................................................................................................ 16
6.3 Calculations and summary of data ............................................................................................................................. 17
6.4 Precision study without a reference method ........................................................................................................ 19
Annex A (normative) List of factors for factorial study design .................................................................. 20
Annex B (informative) Example of a factorial interlaboratory study for a quantitative method . 22
B.1 Study design ........................................................................................................................................................................ 22
B.2 Calculations and summary of data ............................................................................................................................. 24
B.2.1 Summary of the results of laboratories 1 to 5 of the interlaboratory study .................................... 24
B.2.2 Precision data............................................................................................................................................................. 25
B.2.3 Accuracy profile ........................................................................................................................................................ 27
Annex C (informative) Example of a factorial interlaboratory study for a qualitative method .... 31
Bibliography 32
© ISO 2016 – All rights reserved 3

ISO/CD 16140-5
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work. ISO
collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of any
patent rights identified during the development of the document will be in the Introduction and/or on
the ISO list of patent declarations received (www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
The committee responsible for this document is ISO/TC 34, Food products, Subcommittee SC 9,
Microbiology (Working Group WG 3, Method validation).
ISO 16140 consists of the following parts, under the general title Microbiology of the food chain — Method
validation:
 Part 1: Vocabulary
 Part 2: Protocol for the validation of alternative (proprietary) methods against a reference method
 Part 3: Protocol for the verification of reference and validated alternative methods implemented in a
single laboratory (to be published)
 Part 4: Protocol for single-laboratory (in-house) method validation (to be published)
 Part 5: Protocol for factorial interlaboratory method validation (to be published)
 Part 6: Protocol for the validation of alternative (proprietary) methods for microbiological
confirmation and typing (to be published)

ISO/CD 16140-5
Introduction
This part of ISO 16140 provides a protocol that addresses the special case that an interlaboratory
validation of a method shall be conducted, but the number of 8 laboratories required by ISO 16140-2
cannot be reached. The protocol allows a method validation based on only 4 laboratories. It applies for
example in situations in which the protocols of ISO 16140-2 require too much time, for example in specific
situations when the rapid availability of a validated method is urgently needed. The protocol in this part
of ISO 16140 also addresses the problem of method validation of highly specialized methods, for which
only few laboratories might be available for a validation study.
In order to reduce the number of required laboratories to 4 while maintaining the generalizability of the
validation to all competent laboratories, the protocol is based on a factorial experimental design. In the
factorial design, several factors such as technician or culture medium are altered simultaneously and the
method is used in a range of different factor settings. This approach allows a more detailed analysis of
the precision parameters of the method while at the same time requiring only a reduced number of
laboratories and tests.

ISO/CD 16140-5
Microbiology of the food chain — Method validation — Part 5:

Protocol for factorial interlaboratory validation of non-
proprietary methods
1 Scope
This part of ISO 16140 specifies the general principle and the technical protocol (based on orthogonal,
factorial studies) for the validation of non-proprietary methods for microbiology of the food chain.
This part of ISO 16140 is applicable to the validation of non-proprietary methods used in the analysis
(detection or quantification) of microorganisms in
 products intended for human consumption,
 products intended for animal feeding,
 environmental samples in the area of food and feed production, handling, and
 samples from the primary production stage.
This part of ISO 16140 is in particular applicable to bacteria and fungi. Some clauses may be applicable
to other (micro)organisms or their metabolites, to be determined on a case-by-case-basis.
It can be applied only for methods that are fully specified with regard to all relevant parameters
(including tolerances on temperatures and specifications on culture media) and which have already been
optimized.
An interlaboratory study, according to ISO 16140-2 (proprietary methods), requires at least 8

laboratories. ISO 16140-5 is intended to be used for interlaboratory studies comprising 4-7 laboratories
for non-proprietary methods. Table 1 provides an overview of the different protocols.
Table 1 — Overview of different validation protocols described in ISO 16140
Number of laboratories With reference method Without reference method

1 Part 4 of ISO 16140: factorial or conventional Part 4 of ISO 16140: factorial or conventional
4 to 7 Part 5 of ISO 16140: for non-proprietary Part 5 of ISO 16140: for non-proprietary
methods only methods only
8 or more Part 2 of ISO 16140 Not available
(choose single-laboratory validation)
This part of ISO 16140 uses a modified protocol based on orthogonal, factorial studies. By selection of
suitable influencing factors (e.g. technician, culture media, sample preparation, temperature, duration) a
high certainty of the determined method validation parameters is obtained, so that the number of
required collaborating laboratories can be reduced up to a minimum of 4.

ISO/CD 16140-5
2 Normative references
The following documents, in whole or in part, are normatively referenced in this document and are
indispensable for its application. For dated references, only the edition cited applies. For undated
references, the latest edition of the referenced document (including any amendments) applies.
ISO 7218, Microbiology of food and animal feeding stuffs — General requirements and guidance for
microbiological examinations
ISO 16140-1:—1), Microbiology of the food chain — Method validation — Part 1: Vocabulary
ISO 16140-2:—1), Microbiology of the food chain — Method validation — Part 2: Protocol for the validation
of alternative (proprietary) methods against a reference method
3 Terms and definitions

For the purposes of this document, the terms and definitions given in ISO 16140-1 and the following
definitions apply.
3.1
block
a group of settings which have to be conducted in parallel or in a short time interval, and which are used
for the same samples
EXAMPLE Two settings:

Technician 1 + culture media 2 + temperature 1 + incubator 1
AND
Technician 2 + culture media 1 + temperature 2 + incubator 2
3.2
factor
qualitative or quantitative parameter within the method that can be varied at two or more levels within
the limits of the specified method
EXAMPLE Technician.
Note to entry In this part of ISO 16140, only those factors are considered that are in line with the prescription of
the method.
3.3
factor level
value of the factors within the experimental design
EXAMPLE Technician 1, technician 2, etcetera.
Note to entry In this part of ISO 16140, each factor is varied at two factor levels 1 and 2.
3.4
1) To be published. (Revision of ISO 16140:2003)

ISO/CD 16140-5
setting
combination of factor levels
EXAMPLE Technician 1 + culture media 2 + temperature 1 + etcetera.
Note to entry These conditions can be described by the combination of factor levels of factors varied within the
study.
4 General principles for the validation of non-proprietary methods

General concepts and considerations given in ISO 16140-2 shall be used, if not explicitly excluded. The
validation can be conducted against a reference method or by means of appropriate reference materials.
This validation protocol can be used in different ways:
 If 4 laboratories can be considered a ‘random sample’ of independent and competent laboratories

from different organizations, the test method can be considered as being validated in the sense that
accurate and precise measurements are to be expected from any competent laboratory.
 If 4 laboratories can be considered a ‘random sample’ of independent and competent laboratories

from one organization, the test method can be considered as being validated in the sense that
accurate and precise measurements are to be expected from any competent laboratory in this
organization.
4.1 Validation against a reference method
The validation protocol comprises two phases:
 an in-house validation study of the non-proprietary method against the reference method carried
out in the organizing laboratory;
 an interlaboratory study of the non-proprietary method against the reference method carried out in
different laboratories.
The technical protocol for performing the in-house validation study and the interlaboratory study are
given in clause 5 and 6, depending upon whether the test method is qualitative or quantitative in nature.
The selected factors for the factorial interlaboratory study should be relevant to both the reference and
the alternative method.
4.2 Validation without reference method
The validation protocol comprises two phases:
 an in-house validation study of the non-proprietary method based on reference samples carried out
in the organizing laboratory;
 an interlaboratory study of the non-proprietary method based on reference samples carried out in
different laboratories.
The technical protocol for performing the in-house validation study and the interlaboratory study for
quantitative test methods are given in clause 6.

ISO/CD 16140-5
Qualitative test methods cannot be validated without a reference method according to this part of the
International Standard.
5 Qualitative methods - Technical protocol for factorial interlaboratory validation

5.1 In-house validation study
5.1.1 General considerations
The in-house validation study is the part of the validation process that is performed in the organizing
laboratory. It can be conducted according to the conventional approach or the factorial approach
described in part 4 of this International Standard.
An in-house validation study can be used in order to demonstrate the performance of the method for the
laboratory that conducted the study. In the framework of general method validation, it is the first step
and is needed for assessing the performance of the method across a large number of food types and
samples, whereas in the interlaboratory study the performance of the method is assessed across a larger
number of laboratories.
NOTE In this part of ISO 16140, the words ‘category’, ‘item’, ‘matrix’ and ‘type’ are combined with ‘food’ to
improve the readability of this document. However, the word ‘food’ is interchangeable with ‘feed’ and the other
areas of the food chain as mentioned in the Scope of ISO 16140-5.
5.2 Interlaboratory study
The aim of the interlaboratory study is to compare the performance of the reference method to the
alternative method in terms of the RLOD by different laboratories, using identical samples examined
under reproducibility conditions and to compare these results with pre-set criteria for the acceptable
difference between the reference method and the alternative method. The interlaboratory study is
planned by the organizing laboratory.
5.2.2 Measurement protocol
Four or more independent laboratories are required to conduct the measurement series. These
laboratories shall be representative for the population of competent laboratories and shall belong to a
minimum of 3 different organizations but preferably 4 different organizations, excluding the organizing
laboratory. Technicians, involved in the preparation of the samples used in the interlaboratory study,
shall not take part in the testing of those samples within the interlaboratory study.
NOTE 1 Laboratories in different locations, but belonging to one company or institute, are accepted as different
organizations.
NOTE 2 If the 4 laboratories are independent and representative for the population of competent laboratories
within one organization, the outcome of the validation study applies only to laboratories within this organization.
If possible, more than 4 laboratories should participate, so that at least 8 technicians remain even if for
some reason, certain data cannot be used.
The organizing laboratory is preparing the samples and a data sheet for the recording of all experimental
data and critical experimental conditions used by each laboratory. It is necessary for each laboratory to

ISO/CD 16140-5
demonstrate its competence in the use of the alternative and the reference method according to ISO 7218
prior to participating in the study.
Technicians, involved in the preparation of the samples used in the interlaboratory study, shall not take
part in the testing of the samples of the interlaboratory study.
The protocol is the following:
 In cases where different enrichment protocols for the alternative method exist, a challenging
enrichment protocol shall be selected, e.g. the protocol having the shortest incubation time or the
most selective enrichment conditions. The selected food type shall be relevant for the chosen
enrichment protocol. This relevant food type, which is used to prepare the test samples, should
contain a natural background microflora.
 The food shall be inoculated with the target organism. The protocol for inoculation of the food
samples shall be appropriate for the selected food. Samples shall be prepared by the organizing
laboratory to ensure homogeneity between and within samples using matrix preparation protocols
contained in Annex B and Annex C of ISO 16140-2.
 At least three different levels of contamination per food item shall be used: a negative control (L0)
and two levels (L1 and L2). Level L1 should be between the of the reference method and the
of the alternative method.
 For each food item, for each test setting and for each collaborating laboratory, 4 replicates shall be
conducted at level L1 and only 1 test at the two other levels L0 and L2.
 8 different test settings are considered by each collaborating laboratory according to the factorial,
orthogonal design described below. In total 8 test settings have to be conducted by each laboratory
for both test methods and for both the lower and the upper level of contamination. In total a minimum
of 96 results (1  4  1 replicates × 8 settings × 2 methods) per laboratory.
 All samples should be blind-coded to ensure that the analysts are not aware of their level of
contamination.
 For tests which give paired results (paired result occurs when the primary enrichment is the same
for the alternative and reference method), one sample is required to obtain a result for the alternative
and the reference method. For tests which give unpaired results (unpaired result occurs when the
alternative and reference methods start from different primary enrichments), different test portions
from the same sample are required. One test portion is analysed by the alternative method and
another test portion by the reference method.
 The data are reported in two tables, giving the results from the reference method and from the
alternative method before and after confirmation of the results. If the results for alternative and
reference methods have been obtained from the same initial enrichment broth (paired data), then
the confirmation of the reference method also confirms the alternative method. In cases when the
reference method gives a negative result and the alternative method gives a positive result, then
confirmation of the positive result is required. If the results for alternative and reference methods
have been obtained from different enrichments (unpaired data), then all enrichments obtained with
the alternative method shall be taken forward for confirmation. Confirmation pathways are
described in clause 5.1.3.3 of ISO 16140-2.
 The organizing laboratory can indicate that broths, plates, and/or isolates shall be retained for a
certain period of time to be able to confirm results obtained by a laboratory, if needed.

ISO/CD 16140-5
 The analysis of samples shall be performed by each laboratory at the stipulated date.
To predict accuracy and precision in the single laboratory under routine conditions, relevant method
factors that are difficult to control shall be selected and varied systematically, for example personnel,
media and incubation conditions. The choice of these factors and factor levels is crucial to the reliability
of the test result. Four relevant method factors shall be varied simultaneously, each on 2 levels.
For cultivable microorganisms, the following factors ‘technicians’ and ‘culture media’ have the greatest
significance and shall be included in all studies:
 tests shall be conducted in the single laboratory by 2 technicians independently;
 culture media (use media from 2 different manufacturers if available, or pre-prepared versus
prepared from dehydrated media, otherwise use 2 different batches [lots]; if available, each
laboratory should use manufacturers/batches that are different from the ones used in the other
laboratories).
Three further factors have to be selected for use in the study. Decision on the most suitable additional
factors for the particular study should be based on expert knowledge. The following list gives examples
for the selection of further factors and factor levels:
 sample preparation if appropriate (e.g. thawing of frozen samples at room temperature = 1 or at 4 °C

= 2);
 different incubators (use two different incubators; if only one incubator is available, vary incubation
conditions at two levels, e.g. 1 = shortest time and lowest temperature permitted by the method
tolerance, and 2 = longest time and highest temperature);
Optimal conditions are specified in each method, e.g. 37 °C, 24 h, and these will give the best results.
However, ranges around these, due to inaccuracies of the instrument but which provide still acceptable
condition provide acceptable conditions (e.g. 37 °C ± 1 °C, 24 h ± 1 h), are permitted and the design should
test the extremes of these. Other influences such as stress conditions might also be taken into account.
If, according to expert knowledge, other factors are considered to be more relevant with regard to the
result of the method under study, these may be varied on two levels within the experimental design. Other
factors might be stress factors or sample storage conditions. A non-comprehensive grouped list of
potential factors is provided in Annex A. Depending on the method, one factor from each of the most
relevant groups shall be selected.
Factors and factor levels shall reflect the normal variation within routine laboratories. Factors are studied
simultaneously using the study design described in the following clause.
5.2.3 Experimental design
Each setting is a combination of levels of 5 method factors. At each setting each of the 4 laboratories
conducts 1  4  1 replicates at levels L0, L1 and L2 and for both test methods. Eight different settings
(1 to 8) shall be considered. The study design is described in Table 2. Numbers 1 and 2 represent levels
of respective factors.

ISO/CD 16140-5
Table 2 — Study design for culture methods to be conducted by each of at least 4 laboratories
Setting Factor 1 Factor 2 Factor 3 Factor 4 Factor 5
(technician) (culture medium) (thawing process) (incubator) (background flora)
1 1 1 1 1
2 2 2 2 2
1
3 1 1 2 2
4 2 2 1 1
5 1 2 1 2
6 2 1 2 1
2
7 1 2 2 1
8 2 1 1 2
With 4 laboratories, 3 levels of contamination and 2 test methods (reference and alternative) the
minimum number of individual binary tests is (1  4  1) × 8 × 4 × 2 = 384.
The organizing laboratory, using all recorded data, shall determine which results are suitable for use in
analysing the data. The organizing laboratory shall examine the raw data and other information
requested in the data sheet to ascertain that all laboratories have performed the analyses according to
both the alternative and reference methods as written. When there is evidence that results might be
obtained under inappropriate conditions and/or the methods have not been followed strictly, these or all
results from the laboratory are excluded for further analysis.
5.3 Calculations and summary of data
The results obtained by the individual laboratories in the interlaboratory study are summarized in Table
3 and Table 4.
Table 3 — Positive results by the reference method
Contamination level
Laboratory L0 L1 L2
Laboratory 1 /8 /32 /8
Etcetera /8 /32 /8
Total
Table 4 — Positive results (before and after confirmation) by the alternative method
Contamination level
L0 L1 L2
Laboratory Presumptive Confirmed Presumptive Confirmed Presumptive Confirmed
Laboratory 1 /8 /8 /32 /32 /8 /8
Laboratory 2 /8 /8 /32 /32 /8 /8
Laboratory 3 /8 /8 /32 /32 /8 /8
Etcetera /8 /8 /32 /32 /8 /8
Total
Subsequent calculations are carried out according to ISO 16140-2, 5.2.3. These calculations have to be
conducted across all factor settings 1 to 8 but also separately for each subgroup of factor settings which
belongs to 1 out of the 5 factors and 1 out of the 2 factor levels, as displayed in Table 5.

ISO/CD 16140-5
Table 5 — Summary
Sensitivity alternative method (SE(alt)) (%)
Relative specificity alternative method (%)

Sensitivity reference method (SE(ref)) (%)
Relative specificity reference method (%)

Negative Agreement (NA)
Relative trueness RT (%)

Positive Agreement (PA)
Negative Deviation (ND)

Positive Deviation (PD)
% False Positive (FPR)

False Positive (FP)
Sum (N)
Factor levels Settings
All settings 1 to 8
Culture medium 1 1,3,5,7
Culture medium 2 2,4,6,8
Thawing process 1 1,3,6,8
Thawing process 2 2,4,5,7
Incubator 1 1,4,5,8
Incubator 2 2,3,6,7
Background flora 1 1,4,6,7
Background flora 2 2,3,5,8
5.4 Interpretation
Interpretation of results shall be carried out according to ISO 16140-2, 5.2.4. Requirements shall also be
fulfilled for the factor settings displayed in Table 5.
Additionally an evaluation should be made for the differences of the relative level of detection (RLOD)
between laboratories, taking into account factorial effects. A calculation method can be found in Uhlig
and Gowik[2].
6 Quantitative methods - Technical protocol for factorial interlaboratory

validation
6.1 In-house validation study
The in-house validation study is the part of the validation process that is performed in the organizing
laboratory. It can be conducted according to the conventional approach or the factorial approach
described in part 4 of this International Standard.
An in-house validation study can be used in order to demonstrate the performance of the method for the
laboratory that conducted the study. In the framework of general method validation, it is the first step
and is needed for assessing the performance of the method across a large number of food types and
samples, whereas in the interlaboratory study the performance of the method is assessed across a larger
number of laboratories.

ISO/CD 16140-5
By applying a factorial experimental design, the factorial interlaboratory validation does not require a
full in-house validation in accordance with ISO 16140-4, because most relevant method validation
parameters can be extracted from the interlaboratory study itself. The only exception is the
determination of inclusivity and exclusivity parameters. For this, an inclusivity and exclusivity study
according to ISO 16140-2, 5.1.5, shall be conducted.
6.2 Interlaboratory study
The aim of the interlaboratory study is to compare the performance (accuracy and precision) of the
reference method to the alternative method by different laboratories, using identical samples examined
under reproducibility conditions and to compare these results with pre-set criteria for the acceptable
difference between the reference method and the alternative method. Wherever possible the study
conditions should reflect the normal variation between laboratories. The interlaboratory study is
organized by the organizing laboratory.
6.2.3 Measurement protocol
Four or more independent laboratories are required to conduct the measurement series. These
laboratories shall be representative for the population of competent laboratories and shall come from 4
different locations and belonging to a minimum of 3 different organizations, excluding the organizing
laboratory, if the outcome of the validation study shall be applied for the population of competent
laboratories. Laboratories in different locations, but belonging to one company or institute, are not
accepted as different organizations.
NOTE If the 4 laboratories are independent and representative for the population of competent laboratories
within one organization, the outcome of the validation study applies only to laboratories within this organization.
For the statistical analysis altogether 8 technicians should be available. Therefore, it is recommended to
conduct the measurement series with more than 4 laboratories – in case that some data cannot be used.
The organizing laboratory is preparing the samples and a data sheet for the recording of all experimental
data and critical experimental conditions used by each laboratory. It is necessary for each laboratory to
demonstrate its competence in the use of the alternative and the reference method prior to participating
in the study proper.
Technicians, involved in the preparation of the samples used in the interlaboratory study, shall not take
part in the testing of those samples within the interlaboratory study.
The protocol is the following:
 A relevant food type is used to prepare the test samples. The food should contain a natural
background microflora.
 The selected food type may be inoculated with the target organism. The protocol for inoculation of
the samples shall be appropriate for the selected food type. Samples shall be prepared to ensure
homogeneity between samples using matrix preparation protocols contained in Annex B and C of ISO
16140-2. In general, liquid samples (compared to solid samples) give greater assurance to obtain
homogeneity. The samples shall be shown to be homogeneous by the organizing laboratory.
Homogeneity tests and criteria for acceptance are described in ISO/TS 22117[1].

ISO/CD 16140-5
 At least three different levels of contamination shall be used. The analyte concentrations should be
chosen to cover at least the lower, middle, and upper levels of the entire range of the alternative
method. A negative control level should be included in addition.
 Analyses are carried out by each collaborating laboratory according to the factorial, orthogonal
design described below for each enumeration method at the three levels of contamination. All
samples should be blind-coded to ensure that the analysts are not aware of their level of
contamination.
 The analysis of samples shall be performed in each laboratory at the stipulated date.
The selection of method factors shall be performed as described in 5.2.2. Factors are studied
simultaneously using the study design described in the following clause.
6.2.3 Experimental design
Each setting is a combination of levels of 5 method factors. At each setting, each of the 4 laboratories
conducts one single measurement at each level L1, L2 and L3 and for both test methods. Eight different
settings (1 to 8) shall be considered. The study design is described in Table 6 and requires 4 × 3 × 8 × 2 =
192 individual tests. If no reference method is available, the minimum number of individual tests is 4 × 3
× 8 = 96. The study design for culture methods is to be conducted by each of at least 4 laboratories.
Numbers 1 and 2 represent levels of respective factors. Only one replicate is required for each setting.
Table 6 — Study design for culture methods

(technician) (culture medium) (thawing process) (incubator) (background flora)
1 1 1 1 1
2 2 2 2 2
1
3 1 1 2 2
4 2 2 1 1
5 1 2 1 2
6 2 1 2 1
2
7 1 2 2 1
8 2 1 1 2
The organizing laboratory, using all recorded data, shall determine which results are suitable for use in
analysing the data. It is necessary for each laboratory to demonstrate its competence in the use of the
alternative and the reference method according to ISO 7218 prior to participating in the study. The
organizing laboratory shall examine the raw data and other information requested in the data sheet to
ascertain that all laboratories have performed the analyses according to both the alternative and
reference methods as written. When there is evidence that results might be obtained under inappropriate
conditions and/or the methods have not been followed strictly, these or all results from the laboratory
are excluded for further analysis. No outlier tests are performed on the selected data.
Annex B shows an example of a factorial interlaboratory study for a quantitative method.

ISO/CD 16140-5
6.3 Calculations and summary of data
The log transformed test results of the different laboratory for both the reference and alternative method
are presented in Table 7. Calculations are performed as a sequence of operations starting with the log-
transformation of all test results. The following notation is used: i refers to the level; and q is the number
of levels (1 ≤ i ≤ q); j refers to the setting (1 ≤ j ≤ 8); k refers to the laboratory, and p is the number of
laboratories (1 ≤ k ≤ p). Note the data as follows:
xijk, the log transformed test result on level i for setting j of laboratory k using the reference
method (if available);
yijk, the log transformed test result on level i for setting j of laboratory k using the alternative
method.
In Table 7, each square represents a single test result. Results of additional laboratories (p>4) can be
added.
Table 7 — Summary of the results of laboratories 1 to 4 of the interlaboratory study
Setting/method (R = Reference method, A = Alternative method)

1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8
Laboratory Level R A R A R A R A R A R A R A R A
1 Blank
2 Blank
3 Blank
4 Blank
1 Low
2 Low
3 Low
4 Low
1 Medium
2 Medium
3 Medium
4 Medium
1 High
2 High
3 High
4 High
Precision figures can be derived by means of Restricted Maximum Likelihood (REML) using a random
effect model with random factors 'Laboratory', 'Laboratory times technician', 'Laboratory times culture
medium', 'Laboratory times thawing process', 'Laboratory times incubation temperature', 'Laboratory
times reading'. The calculation can be conducted with software packages like R. Details on the statistical
method can be found in Uhlig and Gowik[3].
An alternative relatively simple calculation method for the precision data is described in the following
steps. This method is statistically less efficient and should be used only if the REML method is not
available.

ISO/CD 16140-5
Step 1: If a reference method is available, calculate for each level i of contamination Xi as the median of
the log transformed counts obtained with the reference method, xijk. These medians are called reference
values of the samples used in the validation study: Xi = median(xijk).
If no reference method is available, use log transformed counts obtained with the method to be validated,
Yi = median(yijk).
Step 2: Calculate, for each level i (using yijk), the repeatability standard deviation sri as follows:
𝑝 𝑝
2 1 1
𝑠𝑟𝑖 = ∑(𝑦𝑖1𝑘 + 𝑦𝑖2𝑘 − 𝑦𝑖3𝑘 − 𝑦𝑖4𝑘 )2 + ∑(𝑦𝑖5𝑘 + 𝑦𝑖6𝑘 − 𝑦𝑖7𝑘 − 𝑦𝑖8𝑘 )2
8𝑝 8𝑝
𝑘=1 𝑘=1
Step 3: Calculate, for each level i (using yijk), the intermediate standard deviation sAi as follows:
2 2 2 2 2 2 2
𝑠𝐴𝑖 = 𝑠𝑟𝑖 + 𝑠1𝑖 + 𝑠2𝑖 + 𝑠3𝑖 + 𝑠4𝑖 + 𝑠5𝑖 ,
with
𝑝
2 1 1 2
𝑠1𝑖 = ∑(𝑦𝑖1𝑘 + 𝑦𝑖2𝑘 + 𝑦𝑖3𝑘 + 𝑦𝑖4𝑘 − 𝑦𝑖5𝑘 − 𝑦𝑖6𝑘 − 𝑦𝑖7𝑘 − 𝑦𝑖8𝑘 )2 − 𝑠
64𝑝 8𝑝 𝑟𝑖
𝑘=1
𝑝
2 1 1 2
𝑠2𝑖 = ∑(𝑦𝑖1𝑘 − 𝑦𝑖2𝑘 + 𝑦𝑖3𝑘 − 𝑦𝑖4𝑘 + 𝑦𝑖5𝑘 − 𝑦𝑖6𝑘 + 𝑦𝑖7𝑘 − 𝑦𝑖8𝑘 )2 − 𝑠
𝑘=1
𝑝
2 1 1 2
𝑠3𝑖 = ∑(𝑦𝑖1𝑘 − 𝑦𝑖2𝑘 + 𝑦𝑖3𝑘 − 𝑦𝑖4𝑘 − 𝑦𝑖5𝑘 + 𝑦𝑖6𝑘 − 𝑦𝑖7𝑘 + 𝑦𝑖8𝑘 )2 − 𝑠
𝑘=1
𝑝
2 1 1 2
𝑠4𝑖 = ∑(𝑦𝑖1𝑘 − 𝑦𝑖2𝑘 − 𝑦𝑖3𝑘 + 𝑦𝑖4𝑘 + 𝑦𝑖5𝑘 − 𝑦𝑖6𝑘 − 𝑦𝑖7𝑘 + 𝑦𝑖8𝑘 )2 − 𝑠
𝑘=1
𝑝
2 1 1 2
𝑠5𝑖 = ∑(𝑦𝑖1𝑘 − 𝑦𝑖2𝑘 − 𝑦𝑖3𝑘 + 𝑦𝑖4𝑘 − 𝑦𝑖5𝑘 + 𝑦𝑖6𝑘 + 𝑦𝑖7𝑘 − 𝑦𝑖8𝑘 )2 − 𝑠
𝑘=1
Step 4: Calculate, for each level i, the residual laboratory standard deviation sBi as follows:
1 1 1
2
𝑠𝐵𝑖 = ∑𝑝 (𝑦̅ 2
− 𝑦̅𝑖 )2 − 𝑠𝑟𝑖 2
− (𝑠1𝑖 2
+ 𝑠2𝑖 2
+ 𝑠3𝑖 2
+ 𝑠4𝑖 2
+ 𝑠5𝑖 ),
𝑝−1 𝑘=1 𝑖𝑘 8 2
with the laboratory mean values:

1
y̅ik = 8 ∑8j=1 yijk
and the overall mean:
1 𝑝
𝑦̅𝑖 = 𝑝 ∑𝑘=1 𝑦̅𝑖𝑘 .
Step 5: Calculate, for each level i, the reproducibility standard deviation sRi as follows:

ISO/CD 16140-5
2 2
𝑠𝑅𝑖 = √𝑠𝐴𝑖 + 𝑠𝐵𝑖
The accuracy profile for comparing the alternative method with the reference method can be calculated
according to ISO 16140-2, 6.1.3: the β-ETI is calculated from the median Xi of the reference values and the
reproducibility standard deviation from the above Step 5. The necessary intermediate parameters G and
H are based in the repeatability standard deviation of the above Step 1 and the residual laboratory
standard deviation of the above Step 4. The β-ETI then defines the upper and lower limit for the
acceptability limits.
Interpretation of results is also carried out in accordance with ISO 16140-2, 6.1.3: the alternative method
is regarded as equivalent to the reference method when the values for the β -ETI fall within the
acceptability limits for all levels of contamination. If any of the values falls outside these limits, an
adjustment is performed according to ISO 16140-2, 6.1.3, based on the reproducibility standard deviation
of the reference method as calculated in the above Step 5 for the reference test results.
6.4 Precision study without a reference method
Factorial interlaboratory study can be conducted without a reference method. Calculation of precision
data for the alternative method is not depending on a reference method and can be conducted according
to clause 6.3.
For calculating the accuracy profile, samples have to be artificially inoculated such that the reference
concentration of the contamination is known. The results of the reference methods are replaced by the
corresponding reference concentration in the calculations outlined above.

ISO/CD 16140-5
Annex A (normative) List of factors for factorial study design

The following list gives examples of factors that can be varied in a factorial study design. In general,
factors shall be selected that are expected to have a noticeable effect on the end result. The selection of
factors should cover all major areas in the application of the method, such as the culture medium,
cultivation, sample preparation, sample storage, equipment and personnel.
 Cultivation:
 Supplier of non-selective/primary broth
 In-house versus ready-made non-selective/primary broth
 Supplier of selective/secondary broth
 In-house versus ready-made non-selective/secondary broth
 Supplier of agar plate
 In-house versus ready-made agar plate
 Storage time of broths or plates after incubation before subsequent subculturing or counting (as
long as specified within the procedure of the method)
 Age of the medium (within the expiry period of the medium)
 Batches of media (e.g. media for Enterobacteriaceae containing bile salts)
 Time of incubation of different steps (e.g. 22 h versus 26 h when 24 h ± 2 h is allowable)
 Background flora (at least two levels of background flora per item, e.g. fresh and best-before)
 Background flora storage (refrigerate/store samples)
 Amount (add different amounts artificially)
 Interference organisms
 Sample preparation
 Stomach, blender, or vortex
 Resuscitation procedure: e.g. thawing of frozen samples (1) at room temperature or (2) at 4 °C
 Immediate use or freezing/thawing of samples
 Transport conditions: storage or transport of samples
 Equipment and personnel
 Technician
 Different pipetting systems

ISO/CD 16140-5
 Different incubators (use two different incubators, e.g. different 35° air incubators, air incubator
versus water bath incubation; if only one incubator is available, vary incubation conditions at
two levels, e.g. level 1 = shortest time and lowest temperature permitted by the method
tolerance, and level 2 = longest time and highest temperature)
 Streak plates versus spread plates
 Stress factors (if applicable)
 Low/normal temperature
 High/normal temperature
 Chemical stress (e.g. smoking and curing)
 pH-stress
 High pressure stress
 Low-dose radiation/no radiation (e-beam)
 Dry/non-dry
NOTE A setting is a specific combination of assignments for all factors. Two examples of a setting for the factors
in Table A.1 are:
 Technician 2; no storage; in-house agar plates; incubator 1
 Technician 2; refrigerated storage; ready-made agar plates; incubator 1
Table A.1 — Examples of factors and their settings
Factor Level 1 Level 2

Technician Technician 1 Technician 2
Storage of inoculated samples None Hold refrigerated 2 days
(increases background levels)
Preparation of agar plates Made in-house Purchased ready-made
Incubator Incubator 1 Incubator 2

ISO/CD 16140-5
Annex B (informative) Example of a factorial interlaboratory study for a

quantitative method
B.1 Study design
Eight test portions of naturally contaminated instant non-fat dry milk powder (NFMP) with 3 different
levels were shipped to 5 laboratories located in 3 countries (United States, Canada, France). Two
technicians from each laboratory performed the reference method: Standard Methods for the
Examination of Dairy Products[4], and an alternative method: 3M™ Petrifilm™ Rapid Aerobic Count Plates,
for the aerobic plate count analysis of the NFMP. Incubation times were for the reference method and the
alternative method 48 h ± 3 h and 72 h ± 3 h, respectively.
Five influencing factors were evaluated in the study:
 Technician
 Diluent
 Incubation time
 Incubator
 Plating pipette
See Table B.1 for a detailed summary of the factors to be evaluated.
Table B.1 — Experimental design

(technician) (diluent) (incubation time) (incubator) (plating pipette)
alternative reference
1 1 Butterfield’s Phosphate buffer 45 hours 69 hours Incubator A 1,0 mL Serological
(Pre-Made)
2 1 Butterfield’s Phosphate buffer 51 hours 75 hours Incubator B 1000 µL Micropipette
(Prepared from Dehydrated Media)
3 1 Butterfield’s Phosphate buffer 45 hours 69 hours Incubator B 1000 µL Micropipette
(Pre-Made)
4 1 Butterfield’s Phosphate buffer 51 hours 75 hours Incubator A 1,0 mL Serological
5 2 Butterfield’s Phosphate buffer 51 hours 75 hours Incubator A 1000 µL Micropipette
(Pre-Made)
6 2 Butterfield’s Phosphate buffer 45 hours 69 hours Incubator B 1,0 mL Serological
7 2 Butterfield’s Phosphate buffer 51 hours 75 hours Incubator B 1,0 mL Serological
(Pre-Made)
8 2 Butterfield’s Phosphate buffer 45 hours 69 hours Incubator A 1000 µL Micropipette
Each laboratory received 8 samples with three levels of contamination for a total of samples.
All 24 samples were evaluated in a paired study design, signifying that the same replicates were used for
both the alternative and the reference method. Test portions from each contamination level were

ISO/CD 16140-5
randomized and blind-coded for analysis by the organizing laboratory. See Table B.2 below for a
summary of the evaluation. The variations in technicians, diluent, incubation time, incubators and plating
pipettes according to the experimental design summarized in Table B.1 were applied to both the
alternative and the reference method.
Table B.2 — Overview of study design
Matrix/test portion size Target contamination level # Test portions Alternative method Reference method
NFMP 11g Low 8 2 duplicate plates 2 duplicate plates
Medium 8 (from single lab (from single lab
replicate) replicate)
High 8 incubated incubated
at 32 °C ± 1 °C at 32 °C ± 1 °C
for 48 h ± 3 h a for 72 h ± 3 h
a With variations through different settings.
Each laboratory prepared and analysed all test portions. All test portions were randomized and blind-
coded with a 3-digit reference number, followed by the test setting number. For example, 125-1 or 145-
3. Test portions were prepared so that the analyst did not know the concentration level. Samples were
clearly identified with the factor levels that needed to be adhered to for each sample (i.e. enrichment
broth, incubation time). Detailed instructions were provided for the analysis, including a flowchart for
each of the two technicians (Table B.3 and Table B.4).
Table B.3 — Instructions for Technician 1
For Each Contamination Level

Prepared by Technician 1
Test Setting - 1 Test Setting - 2 Test Setting - 3 Test Setting - 4
Dehydrated Dehydrated
Pre- Made Dilution Buffer Pre- Made Dilution Buffer
Dilution Buffer Dilution Buffer
1 mL Serological Pipette 1000 uL Micropipette 1000 uL Micropipette 1 mL Serological Pipette
Incubator A Incubator B Incubator B Incubator A
Incubate Incubate Incubate Incubate Incubate Incubate Incubate Incubate

Plates of Plates of Plates of Plates of Plates of Plates of Plates of Plates of
Alternative Reference Alternative Reference Alternative Reference Alternative Reference
Method for Method for Method for Method for Method for Method for Method for Method for
45 hours 69 hours 51 hours 75 hours 45 hours 69 hours 51 hours 75 hours
[Comment of WG 3-Secretariat: the table shown above will be changed in the ISO-template after the
Committee Draft-ballot]

ISO/CD 16140-5
Table B.4 — Instructions for Technician 2
For Each Contamination Level

Prepared by Technician 2
Test Setting - 5 Test Setting - 6 Test Setting - 7 Test Setting - 8
Dehydrated Dehydrated
Pre- Made Dilution Buffer Pre- Made Dilution Buffer
Dilution Buffer Dilution Buffer
1000 uL Micropipette 1 mL Serological Pipette 1 mL Serological Pipette 1000 uL Micropipette
Incubator A Incubator B Incubator B Incubator A
Incubate Incubate Incubate Incubate Incubate Incubate Incubate Incubate

Plates of Plates of Plates of Plates of Plates of Plates of Plates of Plates of
Alternative Reference Alternative Reference Alternative Reference Alternative Reference
Method for Method for Method for Method for Method for Method for Method for Method for
51 hours 75 hours 45 hours 69 hours 51 hours 75 hours 45 hours 69 hours
[Comment of WG 3-Secretariat: the table shown above will be changed in the ISO-template after the
Committee Draft-ballot]
B.2 Calculations and summary of data
B.2.1 Summary of the results of laboratories 1 to 5 of the interlaboratory study
Upon receipt of the data, it was verified that that all laboratories performed the analyses according to the
methods as indicated in the study outline. No outlier test was performed on the data, and no data point
was excluded.
All data points (mean values of the count results of the two duplicate plates) were log10 transformed: the
results are displayed in Table B.5.

ISO/CD 16140-5
Table B.5 — Results of interlaboratory study
Setting/method (R = reference, A = alternative method)

1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8
Laboratory Level R A R A R A R A R A R A R A R A
1 Low 2,08 2,11 2,15 2,15 2,04 2,15 2,11 2,20 1,90 2,00 2,04 2,08 2,00 2,00 2,00 2,08
2 Low 2,52 2,41 2,41 2,38 2,46 2,41 2,59 2,51 2,34 2,45 2,32 2,30 2,49 2,53 2,45 2,46
3 Low 2,69 2,36 2,73 2,38 2,66 2,45 2,53 2,53 2,04 2,53 2,62 2,45 2,49 2,45 2,20 2,48
4 Low 2,62 2,40 2,62 2,40 2,58 2,60 2,72 2,45 2,64 2,54 2,78 2,51 2,72 2,41 2,91 2,66
5 Low 2,15 2,45 2,57 2,45 2,41 2,60 2,84 2,85 2,26 2,45 2,64 2,45 2,34 2,43 2,75 1,90
1 Medium 3,28 3,26 3,20 3,20 3,36 3,18 3,08 3,15 3,26 3,08 3,08 3,15 3,11 3,08 3,38 3,38
2 Medium 2,97 2,91 2,94 2,91 2,91 2,93 2,99 2,90 3,08 2,94 2,99 2,97 2,98 2,94 2,99 2,99
3 Medium 2,83 2,93 2,91 3,11 2,89 2,88 2,81 2,66 2,38 2,96 2,82 2,95 2,78 2,90 2,85 3,04
4 Medium 2,93 2,98 2,96 2,97 2,94 2,86 3,04 3,08 2,98 3,04 2,97 2,99 3,00 3,00 3,08 3,08
5 Medium 2,58 2,91 2,70 2,92 2,65 2,79 2,73 2,91 2,66 2,97 2,59 2,97 2,70 2,96 2,85 3,04
1 High 3,94 3,79 4,08 3,70 4,08 3,79 3,66 3,43 3,95 3,61 3,89 3,58 4,15 3,72 3,98 3,74
2 High 4,34 4,20 4,15 4,08 4,38 4,08 4,08 3,89 4,34 4,08 4,49 4,18 4,34 4,11 4,30 4,08
3 High 4,26 4,23 4,56 4,20 4,56 4,36 4,58 4,40 4,00 3,86 4,08 4,00 4,08 3,80 4,04 4,15
4 High 4,11 3,89 4,52 4,11 4,18 4,04 4,26 4,08 4,49 4,30 4,28 4,08 4,18 4,00 4,26 4,08
5 High 3,73 3,70 3,80 3,89 3,80 3,58 3,81 3,66 3,60 3,68 3,92 3,97 3,69 3,62 3,97 3,79
B.2.2 Precision data
Calculation of precision data was conducted according to the steps described in 6.3.
Step 1: Median of the reference values Xi and the values of the alternative method Yi.
Reference method Alternative method

Low Medium High Low Medium High
𝑋1 𝑋2 𝑋3 𝑌1 𝑌2 𝑌3
2,491 2,952 4,097 2,447 2,971 3,984
2
Step 2: Repeatability variance 𝑠𝑟𝑖 sri for each level i (using yijk)
Alternative method
Low Medium High
2 2 2
𝑠𝑟1 𝑠𝑟2 𝑠𝑟3
0,020850 0,008751 0,015915
2
Step 3: Intermediate variance 𝑠𝐴𝑖 for each level i (using yijk)
Alternative method
Low Medium High
2 2 2
𝑠𝐴1 𝑠𝐴2 𝑠𝐴3
0,029348 0,012188 0,028170
with

ISO/CD 16140-5
Alternative method
Factor Low Medium High
𝑖 = 1 𝑖 = 2 𝑖 = 3
Technician s1i² 0,004384 0,000820 0,006292
Diluent s2i² 0,000011 0,000374 0,002970
Incubation time s3i² 0,002102 0,000645 0,001945
Incubator s4i² 0,000007 0,000705 0,000306
Plating pipette s5i² 0,001994 0,000893 0,000743
2
Step 4: Residual laboratory variance 𝑠𝐵𝑖 for each level i
Alternative method
Low Medium High
2 2 2
𝑠𝐵1 𝑠𝐵2 𝑠𝐵3
0,019794 0,008760 0,038709
with
Alternative method
Low Medium High
𝑖 = 1 𝑖 = 2 𝑖 = 3
Laboratory mean 𝑦̅𝑖1 2,096 3,183 3,671
values 𝑦̅𝑖2 2,432 2,939 4,088
𝑦̅𝑖3 2,453 2,932 4,125
𝑦̅𝑖4 2,497 3,001 4,073
𝑦̅𝑖5 2,447 2,934 3,737
Overall mean 𝑦̅𝑖 2,385 2,998 3,939
2
Step 5: Reproducibility variance 𝑠𝑅𝑖 for each level i
Alternative method
Low Medium High
2 2 2
𝑠𝑅1 𝑠𝑅2 𝑠𝑅3
0,049142 0,020948 0,066880

ISO/CD 16140-5
Summary
Reference method Alternative method

Low Medium High Low Medium High
Median log10 2,491 2,952 4,097 2,447 2,971 3,984
Repeatability s.d. 𝑠𝑟 0,114 0,093 0,110 0,144 0,094 0,126
𝑠1 0,080 0,036 0,102 0,066 0,029 0,079
𝑠2 0,096 0,034 0,057 0,003 0,019 0,054
𝑠3 0,017 0,036 0,056 0,046 0,025 0,044
𝑠4 0,058 0,034 0,043 0,003 0,027 0,018
𝑠5 0,038 0,038 0,040 0,045 0,030 0,027
Intermediate s.d. 𝑠𝐴 0,184 0,123 0,180 0,171 0,110 0,168
Residual laboratory s.d. 𝑠𝐵 0,215 0,197 0,205 0,141 0,094 0,197
Reproducibility s.d. 𝑠𝑅
0,283 0,232 0,273 0,222 0,145 0,259
Pooled average 0,263 0,214

reproducibility s.d.
NOTE For reasons of readability, the index i is not used in this overview.
B.2.3 Accuracy profile
Calculation of the accuracy profile was conducted according to the steps described in
ISO/FDIS 16140-2:2014, 6.2.3:
Step 1: Average of the reference values Xi for each level i
According to ISO 16140-5, 6.3, as average of the reference values the respective median will be
used.
Reference method
Low Medium High
𝑋1 𝑋2 𝑋3
2,491 2,952 4,097
2
Step 2: Reproducibility variance 𝑠𝑅𝑖 for each level i
Taken from Step 5 (ISO 16140-5, 6.3) calculated above.
Alternative method
Low Medium High
2 2 2
𝑠𝑅1 𝑠𝑅2 𝑠𝑅3
0,049142 0,020948 0,066880
Step 3: Global average 𝑦̅𝑖 of measurements made with the alternative method
Alternative method
Low Medium High
𝑦̅1 𝑦̅2 𝑦̅3
2,385 2,998 3,939

ISO/CD 16140-5
Step 4: Absolute bias 𝑦̅𝑖 − 𝑋𝑖
Bias
Low Medium High
𝑦̅1 − 𝑋1 𝑦̅2 − 𝑋2 𝑦̅3 − 𝑋2
-0,106 0,046 -0,158
Step 5 and Step 6: Limits of the 𝛽-ETI

Low Medium High
𝑖 = 1 𝑖 = 2 𝑖 = 3
𝑛 8 8 8
𝑝 5 5 5
2 0,0209 0,0088 0,0159
𝑠𝑟𝑖
2 2 2
𝑠𝐿𝑖 = √𝑠𝑅𝑖 − 𝑠𝑟𝑖
0,0283 0,0122 0,0510
2
𝑠𝑅𝑖 0,0491 0,0209 0,0669
𝐻 1,357 1,394 3,202
𝐺 0,446 0,444 0,397
2
𝑠𝑇𝐼𝑖 0,235 0,154 0,278
𝑣 9,731 9,573 6,330
𝑘𝑀𝑖 1,383 1,383 1,440
𝑈𝑖 2,710 3,210 4,339
Limits of 𝛽-ETI
𝐿𝑖 2,060 2,785 3,538
Step 7: Graphical representation of computed results
Low Medium High

𝑖 = 1 𝑖 = 2 𝑖 = 3
𝐿𝑖 − 𝑋𝑖 -0,432 -0,167 -0,559
𝑈𝑖 − 𝑋𝑖 0,219 0,258 0,243
1.0
0.8
0.6
Log10(RAC) - Log10(SMDEP)
Accuracy: difference
0.4
0.2
0.0
-0.2
-0.4
-0.6
-0.8
-1.0
2.2 2.4 2.6 2.8 3.0 3.2 3.4 3.6 3.8 4.0 4.2 4.4
SMEPD reference value (Log10)

ISO/CD 16140-5
Step 8: Additional evaluation procedure
For the high target contamination level, the lower β-ETI falls below 0,5. Hence, an additional
evaluation procedure is followed.
Pooled average reproducibility standard deviation of the reference method:
𝑠𝑅,𝑟𝑒𝑓 = 0,263
Step 9: New acceptability limits
𝐴𝐿𝑠 = 3,3 ∙ 𝑠𝑅,𝑟𝑒𝑓 = 0,869
Summary
Low Medium High

Step
𝑖 = 1 𝑖 = 2 𝑖 = 3
1 𝑋𝑖 2,491 2,952 4,097
2
2 𝑠𝑅𝑖 0,0491 0,0209 0,0669
3 𝑦̅𝑖 2,385 2,998 3,939
4 𝑦̅𝑖 − 𝑋𝑖 -0,106 0,046 -0,158
𝑛 8 8 8
𝑝 5 5 5
2 0,0209 0,0088 0,0159
𝑠𝑟𝑖
2 2 2
𝑠𝐿𝑖 = √𝑠𝑅𝑖 − 𝑠𝑟𝑖
0,0283 0,0122 0,0510
𝐻 1,357 1,394 3,202
5/6
𝐺 0,446 0,444 0,397
2
𝑠𝑇𝐼𝑖 0,235 0,154 0,278
𝑣 9,731 9,573 6,330
𝑘𝑀𝑖 1,383 1,383 1,440
𝑈𝑖 2,710 3,210 4,339
Limits of 𝛽-ETI
𝐿𝑖 2,060 2,785 3,538
Li − Xi -0,432 -0,167 -0,559
7
Ui − X i 0,219 0,258 0,243
8 sR,ref 0,263
9 ALs 0,869

ISO/CD 16140-5
1.0
0.8
0.6
Log10(RAC) - Log10(SMEDP)
Accuracy: difference
0.4
0.2
0.0
-0.2
-0.4
-0.6
-0.8
-1.0
2.2 2.4 2.6 2.8 3.0 3.2 3.4 3.6 3.8 4.0 4.2 4.4
SMEDP reference value (Log10)
𝑈𝑖 − 𝑋𝑖 ≤ 𝐴𝐿𝑠 and 𝐿𝑖 − 𝑋𝑖 ≥ −𝐴𝐿𝑠 for all 𝑖 in the accuracy profile. Therefore the alternative method is
accepted as being equivalent to the reference method.

ISO/CD 16140-5
Annex C (informative) Example of a factorial interlaboratory study for a

qualitative method
[Comment of the project leaders: after the Committee Draft-ballot, an example for a factorial
interlaboratory study for a qualitative method will be added.]

ISO/CD 16140-5
Bibliography
[1] ISO/TS 22117, Microbiology of food and animal feeding stuffs — Specific requirements and guidance
for proficiency testing by interlaboratory comparison
[2] Uhlig, S. and Gowik, P. (2016a), Efficient estimation of reproducibility of LOD and RLOD by means
of GLMM.
[3] Uhlig, S. and Gowik, P. (2016b), Efficient estimation of in-house reproducibility and reproducibility
standard deviation in factorial validation studies.
[4] Wehr, H. Michael and Frank, Joseph F. (2004), Standard Methods for the Examination of Dairy
Products, 17th Edition, American Public Health Association.

Iso-Cd - 16140-5 (E)

Uploaded by

Copyright:

Available Formats

Iso-Cd - 16140-5 (E)

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Iso-Cd - 16140-5 (E)

Uploaded by

Copyright:

Available Formats

ISO/CD 16140-5

ISO/TC 34/SC 9/WG 3

Microbiology of the food chain — Method validation — Part 5:

ISO copyright office

2 © ISO 2016– All rights reserved

© ISO 2016 – All rights reserved 3

 Part 4: Protocol for single-laboratory (in-house) method validation (to be published)

 Part 5: Protocol for factorial interlaboratory method validation (to be published)

© ISO 2016 – All rights reserved 5

6 © ISO 2016– All rights reserved

Microbiology of the food chain — Method validation — Part 5:

 products intended for human consumption,

 products intended for animal feeding,

 samples from the primary production stage.

An interlaboratory study, according to ISO 16140-2 (proprietary methods), requires at least 8

Table 1 — Overview of different validation protocols described in ISO 16140

Number of laboratories With reference method Without reference method

© ISO 2016 – All rights reserved 7

3 Terms and definitions

EXAMPLE Two settings:

EXAMPLE Technician 1, technician 2, etcetera.

1) To be published. (Revision of ISO 16140:2003)

8 © ISO 2016– All rights reserved

EXAMPLE Technician 1 + culture media 2 + temperature 1 + etcetera.

4 General principles for the validation of non-proprietary methods

This validation protocol can be used in different ways:

 If 4 laboratories can be considered a ‘random sample’ of independent and competent laboratories

 If 4 laboratories can be considered a ‘random sample’ of independent and competent laboratories

4.1 Validation against a reference method

The validation protocol comprises two phases:

4.2 Validation without reference method

The validation protocol comprises two phases:

© ISO 2016 – All rights reserved 9

5 Qualitative methods - Technical protocol for factorial interlaboratory validation

5.1.1 General considerations

5.2 Interlaboratory study

5.2.1 General considerations

5.2.2 Measurement protocol

10 © ISO 2016– All rights reserved

The protocol is the following:

© ISO 2016 – All rights reserved 11

 tests shall be conducted in the single laboratory by 2 technicians independently;

 sample preparation if appropriate (e.g. thawing of frozen samples at room temperature = 1 or at 4 °C

5.2.3 Experimental design

12 © ISO 2016– All rights reserved

5.3 Calculations and summary of data

Table 3 — Positive results by the reference method

© ISO 2016 – All rights reserved 13

Sensitivity alternative method (SE(alt)) (%)

Relative specificity alternative method (%)

Relative specificity reference method (%)

Relative trueness RT (%)

Negative Deviation (ND)

% False Positive (FPR)

6 Quantitative methods - Technical protocol for factorial interlaboratory

6.1.1 General considerations

14 © ISO 2016– All rights reserved

6.2 Interlaboratory study

6.2.1 General considerations