7.

5-03
ITTC – Recommended -01-01
Procedures and Guidelines Page 1 of 13
Uncertainty Analysis in CFD Effective Date Revision
Verification and Validation 2017 03
Methodology and Procedures

ITTC Quality System Manual

Recommended Procedures and Guidelines

Procedure

Uncertainty Analysis in CFD Verification and Validation,

Methodology and Procedures

7.5 Process Control

7.5-03 CFD

7.5-03-01 General

7.5-03-01-01 Uncertainty Analysis in CFD Verification and Validation, Methodology and Pro-
cedures

Updated by Approved

Resistance Committee of 28th ITTC 28th ITTC 2017

Date 02/2017 Date 09/2017

 7.5-03
ITTC – Recommended -01-01
Procedures and Guidelines Page 2 of 13
Uncertainty Analysis in CFD Effective Date Revision
Verification and Validation 2017 03
Methodology and Procedures

Table of Contents

4.5 Estimating Errors and Uncertainties

1. PURPOSE OF PROCEDURE.............. 3
using a Least Squares Root approach
2. INTRODUCTION ................................. 3 8
4.6 Estimating Errors and Uncertainties
3. VERIFICATION AND VALIDATION for Point Variables ............................ 9
METHODOLOGY ................................ 4
4.7 Discussion of Fundamental and
4. VERIFICATION PROCEDURES....... 5 Practical Issues ................................ 10
4.1 Convergence Studies ......................... 5 5. VALIDATION PROCEDURES ........ 11
4.2 Generalized Richardson 5.1 Interpretation of the Results of a
Extrapolation ..................................... 6 Validation Effort.............................. 11
4.3 Estimating Errors and Uncertainties 5.2 Use of Corrected vs. Uncorrected
with Correction Factor...................... 7 Simulation Results ........................... 12
4.4 Estimating Uncertainties with
Factors of Safety ................................ 8 6. REFERENCES .................................... 12
 7.5-03
ITTC – Recommended -01-01
Procedures and Guidelines Page 3 of 13
Uncertainty Analysis in CFD Effective Date Revision
Verification and Validation 2017 03
Methodology and Procedures

Uncertainty Analysis in CFD, Verification and Validation

Methodology and Procedures

Revision 02 QM Procedure 7.5-03-01-01

1. PURPOSE OF PROCEDURE “Uncertainty Analysis in CFD, Verification and
Validation Methodology and Procedures” is a
Provide methodology and procedures for es- minor update of Revision 01, in which the latest
timating the uncertainty in a simulation result. revisions to the correction factor approach have
been incorporated. These modifications have
been discussed in Section 4.3.
2. INTRODUCTION
Revision 03 of this procedure is a minor up-
Revision 01 was a revision of QM Proce-
date of Revision 02, in which the references to
dures 4.9-04-01-01 “Uncertainty Analysis in
the ISO document have been updated to the lat-
CFD, Uncertainty Assessment Methodology”
est issue of the JCGM GUM, and further details
and 4.9-04-01-02 “Uncertainty Analysis in
of the Least Squares Root method for estimating
CFD, Guidelines for RANS Codes,” which were
error have been added.
prepared and recommended by 22nd Resistance
Committee and adopted as interim procedures. The symbols used in the procedure have
The QM Procedures were largely based on the been checked against the list of symbols in An-
methodology and procedures of Stern et al. nex J of the JCGM document “Evaluation of
(1999) [most recently Stern et al. (2001) and measurement data – Guide to the expression of
Wilson et al. (2001)] and Coleman and Stern uncertainty in Measurement” or GUM (JCGM,
(1997). Valuable experience was also gained at 2008), proposed by the Quality Systems Group
Gothenburg 2000 A Workshop on Numerical of the 28th ITTC to be used as a reference. The
Ship Hydrodynamics (Larsson et al., 2000) GUM is entirely and specifically intended for
where present QM Procedures were recom- uncertainty estimation for measurements and
mended and used. therefore most of the symbols are not related to
the recommended procedure at hand. The sym-
Revision 01 QM Procedure 7.5-03-01-01
bols most closely connected with this procedure
“Uncertainty Analysis in CFD, Verification and
are the ones for uncertainty. In the standards
Validation Methodology and Procedures” was
document the uppercase symbol ( U ) is used for
updated for clarity of presentation and expanded
expanded uncertainty of an estimate that defines
discussion of verification procedures and imple-
mentation based on three years experience, as an interval y − U < Y < y + U having a high
discussed in Section 7 of 23rd ITTC RC Report. level of confidence and is equal to the coverage
In particular, verification procedures were ex- factor k times the standard combined uncer-
panded to include user options of either correc- tainty. Lower case symbol ( u ) is used for stand-
tion factors or factor of safety approaches for es- ard uncertainty of an estimate that is the positive
timating numerical errors and uncertainties and square root of the estimated variance. Due to ob-
discussion was provided on fundamental and vious similarities between the use of the cover-
practical issues to aid in implementation of ver- age factor and the uncertainty estimation in this
ification procedures.
 7.5-03
ITTC – Recommended -01-01
Procedures and Guidelines Page 4 of 13
Uncertainty Analysis in CFD Effective Date Revision
Verification and Validation 2017 03
Methodology and Procedures

procedure, the use of upper case symbols for un- S C = S − δ SN

*
(1)
certainty has been retained. Therefore, also in
terms of symbols, only minor modifications Verification is defined as a process for as-
have been. These are all related to the subscripts sessing simulation numerical uncertainty U SN
used. In the GUM the different input parameters
and, when conditions permit, estimating the sign
are defined with subscript i . To be in line with
the standards document the corresponding sub- and magnitude δ SN *
of the simulation numerical
scripts have been changed. Additionally, in or- error itself and the uncertainty in that error esti-
der to improve legibility, the use of subscripts of mate U SC N . For the uncorrected simulation ap-
subscripts has been minimized and most of these proach, numerical error is decomposed into con-
have been moved into subscripts with comma tributions from iteration number δ I , grid size
separating the items.
δ G , time step δ T , and other parameters δ P ,
Present verification procedures are consid- which gives the following expression for simu-
ered best presently available and further work is lation numerical uncertainty
also recommended for improved procedures,
which once available can be incorporated. Vali- 2
U SN = U I2 + U G2 + U T2 + U P2 (2)
dation procedures were not changed. In the fol-
lowing an overview of the overall verification For the corrected simulation approach, the
and validation approach is provided, including solution is corrected to produce a numerical
methodology and procedures. Stern et al. (2001) benchmark S C and the estimated simulation nu-
should be consulted for detailed presentation
and discussions. merical error δ SN
*
and corrected uncertainty
U Sc N are given by
3. VERIFICATION AND VALIDATION
METHODOLOGY δ SN
*
= δ I* + δ G* + δ T* + δ P* (3)

The definitions of errors and uncertainties U S2c N = U I2C + U G2C + U T2C + U P2C (4)
directly follow those used in experimental un-
certainty analysis. The simulation error δS is de- Validation is defined as a process for as-
fined as the difference between a simulation re- sessing simulation modelling uncertainty U SM
sult S and the truth T and is composed of addi-
by using benchmark experimental data and,
tive modelling δSM and numerical δSN errors (i.e.,
when conditions permit, estimating the sign and
δ S = S − T = δ SM + δ SN ). For certain condi- magnitude of the modelling error δ SM itself.
tions, both the sign and magnitude of the numer-
The comparison error E is given by the differ-
ical error can be estimated as δ SN = δ SN
*
+ ε SN ence in the data D and simulation S values
where δ SN
*
is an estimate of the sign and magni-
tude of δ SN and ε SN is the error in that estimate. E = D − S = δ D − (δ SM + δ SN ) (5)
The simulation value is corrected to provide a
Modelling errors δ SM can be decomposed
numerical benchmark S C , which is defined
into modelling assumptions and use of previous
data. To determine if validation has been
 7.5-03
ITTC – Recommended -01-01
Procedures and Guidelines Page 5 of 13
Uncertainty Analysis in CFD Effective Date Revision
Verification and Validation 2017 03
Methodology and Procedures

achieved, E is compared to the validation un- Careful consideration should be given to se-
certainty U V given by lection of uniform parameter refinement ratio.
The most appropriate values for industrial CFD
U V2 = U D2 + U SN
2
(6) are not yet fully established. Small values (i.e.,
very close to one) are undesirable since solution
changes will be small and sensitivity to input pa-
If E < U V , the combination of all the errors rameter may be difficult to identify compared to
in D and S is smaller than U V and validation is iterative errors. Large values alleviate this prob-
achieved at the U V level. If U V << E , the sign lem; however, they also may be undesirable
since the finest step size may be prohibitively
and magnitude of E ≈ δ SM can be used to make small (i.e., require many steps) if the coarsest
modelling improvements. For the corrected sim- step size is designed for sufficient resolution
ulation, equations equivalent to Eqs. (5) and (6) such that similar physics are resolved for all m
are solutions. Also, similarly as for small values, so-
lution changes for the finest step size may be dif-
E = D − S C = δ D − (δ SM + ε SN ) (7) ficult to identify compared to iterative errors
since iterative convergence is more difficult for
U V2C = U E2C − U SM
2
= U D2 + U S2C N (8) small step size. Another issue is that for param-
eter refinement ratio other than ri = 2 , interpo-
lation to a common location is required to com-
4. VERIFICATION PROCEDURES pute solution changes, which introduces interpo-
lation errors. Roache (1998) discusses methods
4.1 Convergence Studies for evaluating interpolation errors. However, for
industrial CFD, ri = 2 may often be too large.
Iterative and parameter convergence studies A good alternative may be ri = 2 , as it pro-
are conducted using multiple solutions (at least
3) with systematic parameter refinement by var- vides fairly large parameter refinement ratio and
at least enables prolongation of the coarse-pa-
ying the ith input parameter ∆xi while holding
rameter solution as an initial guess for the fine-
all other parameters constant. The present work parameter solution.
assumes input parameters can be expressed such
that the finest resolution corresponds to the limit Convergence studies require a minimum of
of infinitely small parameter values. Many com- m = 3 solutions to evaluate convergence with
mon input parameters are of this form, e.g., grid respect to input parameter. Note that m = 2 is
spacing, time step, and artificial dissipation. Ad- inadequate, as it only indicates sensitivity and
ditionally, a uniform parameter refinement ratio not convergence, and that m > 3 may be re-
quired. Changes between medium-fine
ri = ∆xi , 2 / ∆xi ,1 = ∆xi ,3 / ∆xi , 2 = ∆xi ,m / ∆xi ,m −1
ε i , 21 = Sˆ i , 2 − Sˆ i ,1 and coarse-medium
between solutions is assumed for presentation ε = Sˆ − Sˆ solutions are used to define the
i , 32 i ,3 i,2
purposes, but not required. Iterative errors must convergence ratio
be accurately estimated or negligible in compar-
ison to errors due to input parameters before ac- Ri = ε i , 21 / ε i ,32 (9)
curate convergence studies can be conducted.
 7.5-03
ITTC – Recommended -01-01
Procedures and Guidelines Page 6 of 13
Uncertainty Analysis in CFD Effective Date Revision
Verification and Validation 2017 03
Methodology and Procedures

and to determine convergence condition where sum. The accuracy of the estimates depends on
Sˆ i ,1 , Sˆ i , 2 , Sˆ i ,3 correspond to solutions with fine, how many terms are retained in the expansion,
the magnitude (importance) of the higher-order
medium, and coarse input parameter, respec-
terms, and the validity of the assumptions made
tively, corrected for iterative errors. Three con-
in RE theory.
vergence conditions are possible:
With three solutions, only the leading term
(i) Monotonic convergence: 0 < Ri < 1 can be estimated, which provides one-term esti-
mates for error and order of accuracy
(ii) Oscillatory convergence: Ri < 0 (10)
ε i , 21
(iii) Divergence: Ri > 1 δ RE
*(1)
= (11)
i ,1
ri pi − 1
For condition (i), generalized Richardson ln(ε i ,32 / ε i , 21 )
extrapolation (RE) is used to estimate U i or δ i* pi = (12)
ln(ri )
and U iC . For condition (ii), uncertainties are es-
timated simply by attempting to bound the error With five solutions, two terms can be esti-
based on oscillation maximums SU and mini- mated, which provides two-term estimates for
error and orders of accuracy
mums S L , i.e., U i = (SU − S L ) . For oscilla-
1
2 ri qi ε i , 21 − ε i ,32
δ RE =
(r )( )
*( 2 )
tory convergence (ii), the solutions exhibit os-
cillations, which may be erroneously identified
i ,1
i
qi
− ri pi ri pi − 1
(13)
as condition (i) or (iii). This is apparent if one ri pi ε i , 21 − ε i ,32
−
considers evaluating convergence condition
from three points on a sinusoidal curve (Cole-
(r i
qi
)(
− ri pi ri qi − 1 )
man et al., 2001). Depending on where the three
points fall on the curve, the condition could be pi =
[( )[ (
ln ai + bi / 2 ε i , 21ε i , 43 − ε i2,32 )]]
incorrectly diagnosed as either mono-tonic con- ln (ri )
[( )[ ( )]]
vergence or divergence. Bounding the error (14)
ln ai − bi / 2 ε i , 21ε i , 43 − ε i2,32
based on oscillation maximum and minimum for qi =
condition (ii) requires more than m = 3 solu- ln (ri )
tions. For condition (iii), errors and uncertainties
cannot be estimated. where

4.2 Generalized Richardson Extrapolation ai = ε i , 21ε i ,54 − ε i ,32ε i , 43

For convergence condition (i), generalized

(
bi = −3ε i2,32ε i2, 43 + 4 ε i , 21ε i3, 43 + ε i3,32ε i ,54 )
RE is used to estimate the error δ i* due to selec- − 6ε i , 21ε i ,32ε i , 43ε i ,54 + ε i2, 21ε i2,54
tion of the ith input parameter and order-of-accu-
racy p i . The error is expanded in a power series Solutions for analytical benchmarks show
that the range of applicability for Eqs. (13) and
expansion with integer powers of ∆xi as a finite
(14) is more restrictive than that for Eqs. (11)
 7.5-03
ITTC – Recommended -01-01
Procedures and Guidelines Page 7 of 13
Uncertainty Analysis in CFD Effective Date Revision
Verification and Validation 2017 03
Methodology and Procedures

and (12) since all five solutions must be both to estimate δ RE

*
i ,1
where pi and qi are replaced
monotonically convergent and sufficiently close
with piest and qiest
to the asymptotic range to evaluate pi and qi in
Eq. (14). In general, m = 2n + 1 solutions are re-
quired to estimate the first n terms of the error Ci =
(ε /ε − r qiest ri pi − 1 )( )
(r )(r )
i , 32 i , 21 i

expansion. i
p iest
−r i
q iest
i
p iest
−1
(ε )(r )
(17)
/ε −r p iest pi
−1
+
(r )(r )
i , 32 i , 21 i i
4.3 Estimating Errors and Uncertainties p iest
−r q iest q iest
−1
i i i
with Correction Factor

The concept of correction factors is based on Eq. (16) roughly accounts for the effects of
verification studies for 1D wave equation, 2D higher-order terms by replacing pi with piest
Laplace equation, and Blasius boundary layer thereby improving the single-term estimate,
analytical benchmarks for which it is shown that while Eq. (17) more rigorously accounts for
a multiplicative correction factor is useful as a higher-order terms since it is derived from a
quantitative metric to determine proximity of two-term estimate. Both expressions for Ci only
the solutions to the asymptotic range, to account require three solutions to estimate errors using
for the effects of higher-order terms, and for es- Eq. (15). Solutions for analytical benchmarks
timating errors and uncertainties. The error is show that correction of error estimates with both
defined as expressions for Ci yields improved error esti-
mates.
 ε i , 21 
δ i*,1 = Ciδ RE
*
= Ci   (15)
 ri − 1 
i ,1 pi
Expressions for uncertainties are developed
from error estimates in Eq. (15). When solutions
where two expressions for the correction factor are far from the asymptotic range, Ci is suffi-
Ci were developed. The first is based on solu- ciently less than or greater than 1 and only the
tion of Eq. (15) for Ci with δ RE
*
i ,1
based on Eq. magnitude of the error is estimated through the
uncertainty U i . Eq. (15) is used to estimate U i
(11) but replacing pi with the improved estimate
by bounding the error δ i*,1 by the sum of the ab-
piest
solute value of the corrected estimate from RE
and the absolute value of the amount of the cor-
ri pi − 1
Ci = (16) rection
ri piest − 1
U i = ( Ci + 1 − Ci )δ RE
*
i ,1
(18)
piest is an estimate for the limiting order of ac-
curacy of the first term as spacing size goes to It is shown by Wilson and Stern (2002) that Eq.
zero and the asymptotic range is reached so that (18) is not conservative enough for Ci < 1 ,
Ci → 1 . Similarly, the second is based on a two- which motivates development of an improved
term estimate of the power series which is used estimate
 7.5-03
ITTC – Recommended -01-01
Procedures and Guidelines Page 8 of 13
Uncertainty Analysis in CFD Effective Date Revision
Verification and Validation 2017 03
Methodology and Procedures

U i = (2 1 − Ci + 1)δ RE
*
i ,1
(19) 4.4 Estimating Uncertainties with Factors
of Safety
When solutions are close to the asymptotic Alternatively, a factor of safety approach
range, Ci is close to 1 so that δ i* is estimated (Roache, 1998) can be used to define the uncer-
using Eq. (15) and U iC is estimated by tainty U i where an error estimate from RE is
multiplied by a factor of safety FS to bound sim-
U iC = 1 − Ci δ *
REi ,1 (20) ulation error

Eq. (20) has the correct form for both Ci < 1 and U i = FS δ RE
*
i ,1
(23)
Ci > 1 . It should be recognized that using the
corrected simulation approach requires in addi- where δ RE
*
i ,1
can be based on a single- or two-
tion to Ci close to 1 that one have confidence in term estimate as given by Eq. (11) or (13), re-
Eq. (15). There are many reasons for lack of spectively with either assumed or estimated or-
confidence, especially for complex three-di- der of accuracy. If order of accuracy is assumed
mensional flows. (e.g., based on theoretical values), only two or
three solutions are required for evaluation of Eq.
As pointed out by Roache (2003) Eqs. (19) (11) or (13), respectively.
and (20) have the short-coming that as C k → 1
Although not proposed by Roache (1998),
the method reverts to Richardson Extrapolation, the factor of safety approach can be used for sit-
which produces only ~50% uncertainty esti- uations where the solution is corrected with an
mate. Based on this criticism a further revision error estimate from RE as
of the uncertainty estimates have been presented
U iC = (FS − 1)δ RE
by Wilson et al. (2004). The final uncertainty es- *
(24)
timates for the uncorrected and corrected ap- i ,1

proaches respectively are given as

The exact value for factor of safety is somewhat

Ui = 
[ ]
 9.6(1 − Ci )2 + 1.1 δ RE
*
i ,1
, 1 − Ci < 0.125 ambiguous and FS = 1.25 is recommended for

 [2 1 − Ci + 1]δ REi ,1 ,
careful grid studies and 3 for cases in which only
*
1 − Ci ≥ 0.125 two grids are used and order of accuracy is as-
(21) sumed from the theoretical value pth .


U iC = 
[ ]
 2.4(1 − Ci )2 + 0.1 δ RE *
i ,1
, 1 − Ci < 0.25 4.5 Estimating Errors and Uncertainties
 [1 − Ci ]δ RE* i ,1 , 1 − Ci ≥ 0.25 using a Least Squares Root approach

(22) Where there is scatter in the numerical solu-

tions, common in complex flows with relatively
These uncertainty estimates merge smoothly coarse grids, or where the use of unstructured
with the previous uncertainty estimates and pro- grids leads to variability in the grids, the error
vide 10% factor of safety in the limit Ci = 1 . can be estimated using a Least Squares Root
method (LSR) (Eça, 2010 and Larsson, 2014).
 7.5-03
ITTC – Recommended -01-01
Procedures and Guidelines Page 9 of 13
Uncertainty Analysis in CFD Effective Date Revision
Verification and Validation 2017 03
Methodology and Procedures

This requires at least four solutions to per- based on p. For 0.5 ≤ p < 2.1, assuming a theo-
form a curve fit of retical order of accuracy equal to 2, the factor of
safety of 1.25 is applied while for all other p a
S i = S 0 + αhi
p
(25) factor of safety equal to 3 is used.

where i is the grid number from 1 to the number For oscillatory or anomalous convergence
the uncertainty is based on the data range param-
of grids and hi is the grid size ratio.
eter
The convergence condition is determined
based on the observed order of accuracy, p, such U i = 3 δ ∆M (30)
that p > 0 indicates monotonic convergence and
p < 0 indicates monotonic divergence. Oscilla- 4.6 Estimating Errors and Uncertainties
tory convergence is defined as being when the for Point Variables
solution is alternately above and below the exact
solution. Determination of the convergence ratio Ri
Since p is strongly influenced by the amount for point variables can be problematic since so-
of scatter in the solutions, such that it may be lution changes ε i , 21 and ε i ,32 can both go to zero
larger than the theoretical order of accuracy, (e.g., in regions where the solution contains an
leading to an underestimate of the error, three inflection point). In this case, the ratio becomes
alternative error estimates are provided, also ill conditioned. However, the convergence ratio
found by curve fitting. can be used in regions where the solution
changes are both non-zero (e.g., local solution
δ RE = S i − S 0 = αhi p (26) maximums or minimums).

Another approach is to use a global conver-

δ RE
02
= S i − S 0 = α 02 hi (27)
2
gence ratio Ri and order of accuracy pi , which
overcomes ill conditioning, based on the L2
δ RE
12
= S i − S 0 = α11hi + α12 hi
2
(28) norm of the solution changes, i.e.,

(Si )max − (Si )min Ri = ε i , 21 2 / ε i ,32

δ∆ =
(h ) (29) (31)
h1 − 1
M
2
ng

pi =
(
ln ε i ,32 2 / ε i , 21 2
) (32)
The error estimate is chosen based on the ob-
ln(ri )
served order of accuracy, p. For 0.5 ≤ p ≤ 2, δ RE
from Eq. (26) is used. For p > 2, δ RE
02
from Eq. where denotes a profile-averaged quantity
(27) is used, and for p < 0.5 the best fit of δ RE
02
(with ratio of solution changes based on L2
or δ RE
12
is chosen.  N 
1/ 2

norms) and ε 2 =  ∑ ε k2  denotes the L2

The numerical uncertainty is calculated us-  k =1 
ing a factor of safety approach, as in Eq. (23), norm of solution change over the N points in
but the error used and the factor of safety are
 7.5-03
ITTC – Recommended -01-01
Procedures and Guidelines Page 10 of 13
Uncertainty Analysis in CFD Effective Date Revision
Verification and Validation 2017 03
Methodology and Procedures

the region of interest. Caution should be exer- Fundamental issues include from the outset
cised when defining the convergence ratio from selection of multiple vs. single grid approaches
the ratio of the L2 norm of solution changes be- for estimating errors and uncertainties. How-
cause the oscillatory condition ( Ri < 1 ) cannot ever, as discussed in Section 7 of 23rd ITTC RC
Report, the former approach can be used to es-
be diagnosed since Ri will always be greater
tablish convergence and is relatively inexpen-
than zero. Local values of Ri at solution maxi- sive to implement and therefore recommended
mums or minimums should also be examined to at this time. For multiple-grid approaches, im-
confirm the convergence condition based on an portant fundamental issues include appropriate-
L2 norm definition. ness of power series representation [Eq. (27) of
Stern et al. (2001)] and its convergence charac-
For verification of the uncorrected solution teristics along with assumptions that pi(k ) and
Eqs. (21) or (23) are used to estimate distribu-
qi(k ) are independent of ∆xi . Also, issues con-
tions of U i at each point from the local solution
cerning definitions and nature of solutions in as-
change ε i , 21 , where pi is estimated from Eq. ymptotic vs. non-asymptotic ranges.
(32). Similarly, for the corrected solution, pi
These fundamental issues are exacerbated
is used to estimate δ i* and U iC at each point us- for practical applications along with additional
ing Eqs. (15) or (11) and (22) or (24), respec- is-sues, including selection of parameter refine-
tively. An L2 norm of point distributions of er- ment ratio, procedures for generation of multi-
rors and uncertainties are then used to assess ple systematic grids and solutions, number of
verification levels and to judge if validation has grids required and variability between grid stud-
been achieved globally. ies, selection of appropriate verification proce-
dures, and interpretation of results.
An alternate approach suggested by Hoeks-
tra et al. (2000) is to transform the spatial profile Selection of the parameter refinement ratio
to wave number space and to perform a conver- was discussed previously in Section 4.1 wherein
gence study on the amplitude distribution of the the use of uniform value ri = 2 was recom-
Fourier modes. In principle, this approach
mended; however, non-uniform and
would remove the problem of ill-conditioning of
larger/smaller values may also be appropriate
the convergence ratio, Ri . under certain circumstances. Wilson and Stern
(2002) discuss procedures for generation of
4.7 Discussion of Fundamental and Practi- multiple systematic grids and solutions. Multi-
cal Issues ple systematic grids are generated using ri = 2
and a post-processing tool in which the coarse
It should be recognized that implementation
grid is obtained by removing every other point
of verification procedures is not easy and re-
from the fine grid and the medium grid is ob-
quire both experience and interpretation of re-
tained by interpolation. Multiple solutions are
sults, especially for practical applications. How-
obtained by first obtaining a solution for the
ever, their importance cannot be overempha-
coarse grid with a uniform flow initial condition,
sized to ensure fidelity and quality of CFD solu-
which is then used as an initial condition for ob-
tions.
taining a solution on the medium grid, which is
 7.5-03
ITTC – Recommended -01-01
Procedures and Guidelines Page 11 of 13
Uncertainty Analysis in CFD Effective Date Revision
Verification and Validation 2017 03
Methodology and Procedures

then used as an initial condition for obtaining a and on the correction factor approach outside
solution on the fine grid. This procedure can be this range (i.e., 1 − Ci > 0.125 ). For the cor-
used to obtain solutions on all three grids in
rected approach, uncertainties are based on the
about 1/3 the time required to obtain only the
fine grid solution without this procedure. correction factor approach, when 1 − Ci > 0.25 .
When using correction factors an important is-
For complex flows with relatively coarse sue is selection of the best estimate for the lim-
grids, solutions may be far from the asymptotic iting order of accuracy. Theoretical values can
range such that some variables are convergent be used or values based on solutions for simpli-
while others are oscillatory or even divergent. fied geometry and conditions, in either case,
The order of accuracy and therefore correction preferably including the effects of stretched
factors and factors of safety may display large grids.
variability indicating the need for finer grids.
Clearly, more than 3 grids are required to esti- Lastly, analysis and interpretation of results
mate errors and uncertainties for such cases. Eca is important in assessing variability for order of
and Hoekstra (1999, 2000) propose a least accuracy, levels of verification, and strategies
squares approach to estimate the error by com- for reducing numerical and modelling errors and
puting the three unknown parameters from RE uncertainties; since, as already mentioned, there
when more than three grids are used and there is is limited experience and no known solutions for
variability between grid studies. practical applications in the asymptotic range
for guidance.
Both correction factor and factor of safety
verification approaches have been presented
with selection being a user option. Wilson and 5. VALIDATION PROCEDURES
Stern (2002) have shown that the factor of safety
approach is over conservative when solutions 5.1 Interpretation of the Results of a Vali-
are close to the asymptotic range and under con- dation Effort
servative when solutions are far from the as-
ymptotic range. Nonetheless some users may First, consider the approach in which the
prefer factors of safety over correction factors. simulation numerical error is taken to be sto-
An alternative is to select the more conservative chastic and thus the uncertainty U SN is esti-
uncertainty from the correction factor and factor mated. From a general perspective, if we con-
of safety approaches. For the uncorrected simu-
sider the three variables U V , E , and U reqd there
lation approach the more conservative uncer-
tainty is given as the maximum of Eqs. (21) and are six combinations (assuming none of the
(23). three variables are equal):

For the corrected simulation approach, the 1) E < UV < U reqd

more conservative uncertainty is given as the
maximum of Eqs. (22) and (24). 2) E < U reqd < UV
For FS = 1.25, uncorrected uncertainty esti-
mates are based on the factor of safety approach 3) U reqd < E < UV
when Ci is close to one (i.e., 0.875 < Ci < 1.125)
 7.5-03
ITTC – Recommended -01-01
Procedures and Guidelines Page 12 of 13
Uncertainty Analysis in CFD Effective Date Revision
Verification and Validation 2017 03
Methodology and Procedures

4) UV < E < U reqd (33) EC is a better indicator of δ SMA than is E , as-

suming that one’s confidence in using the esti-
5) UV < U reqd < E mate δ SN
*
is not misplaced.

6) U reqd < UV < E 5.2 Use of Corrected vs. Uncorrected Sim-

ulation Results
In cases 1, 2 and 3, E < U V ; validation is
The requirements for correcting the solution
achieved at the U V level; and the comparison er- are that the correction factor be close to one and
ror is below the noise level, so attempting to es- that confidence in solutions exist. Since the var-
timate δ SMA is not feasible from an uncertainty iability of the order of accuracy cannot be deter-
standpoint. In case 1, validation has been mined from solutions on three grids, confidence
achieved at a level below U reqd , so validation is is difficult to establish in this case. As a result,
caution should be exercised when correcting so-
successful from a programmatic standpoint. lutions using information from only three grids.
In cases 4, 5 and 6, U V < E , so the compar- If a validation using the corrected approach
ison error is above the noise level and using the is successful at a set condition, then if one
sign and magnitude of E to estimate δ SMA is chooses to associate that validation uncertainty
feasible from an uncertainty standpoint. If level with the simulation's prediction at a neigh-
boring condition that prediction must also be
U V << E , then E corresponds to δ SMA and the
corrected. That means enough runs are required
error from the modelling assumptions can be de- at the new condition to allow estimation of the
termined unambiguously. In case 4, validation is numerical errors and uncertainties. If this is not
successful at the E level from a programmatic done, then the comparison error E and valida-
standpoint. tion uncertainty U V corresponding to the use of
the uncorrected S and its associated (larger)
Now consider the approach in which the U SN should be the ones considered in the vali-
simulation numerical error is taken to be deter-
dation with which one wants to associate the
ministic and thus δ SN *
and the uncertainty U VC prediction at a new condition. (Whether to and
are estimated. A similar set of comparisons as how to associate an uncertainty level at a vali-
those in equation (33) can be constructed using dated condition with a prediction at a neighbor-
EC , U VC , and U reqd . Since EC can be larger or ing condition is very much unresolved and is
smaller than E , but U VC should always be less justifiably the subject of much debate at this
time.)
than U V , the results for a given corrected case
are not necessarily analogous to those for the
corresponding uncorrected case. That is, a vari- 6. REFERENCES
able can be validated in the corrected but not in
the uncorrected case, or vice versa. For cases 4, Coleman, H. W. and Stern, F., 1997, “Uncer-
tainties in CFD Code Validation”, ASME J.
5, and 6 in which U VC < EC , one can argue that Fluids Eng., Vol. 119, pp. 795-803.
 7.5-03
ITTC – Recommended -01-01
Procedures and Guidelines Page 13 of 13
Uncertainty Analysis in CFD Effective Date Revision
Verification and Validation 2017 03
Methodology and Procedures

Coleman, H, Stern, F., Di Mascio, A., Campa- Larsson, L., Stern, F., and Visonneau, M., “Nu-
gna, E. “Technical Note: The Problem with merical Ship Hydrodynamics: An assess-
Oscillatory Behavior in Grid Convergence ment of the Gothenburg 2010 Workshop”,
Studies”, ASME J. Fluids Eng., Vol. 123, Is- Springer, 2014.
sue 2, June 2001, pp. 438-439.
Roache, P.J., 1998, Verification and Validation
Eça, L., and Hoekstra, M., 1999, “On the numer- in Computational Science and Engineering,
ical verification of ship stern flow calcula- Hermosa publishers, Albuquerque, New
tions”, 1st MARNET workshop, Barcelona, Mexico.
Spain.
Roache, P.J., 2003, “Criticism of the Correction
Eça, L., and Hoekstra, M., 2000, “On the Appli- Factor Verification Method”, ASME J. Flu-
cation of Verification Procedures in Compu- ids Eng., Vol. 125, July 2003.
tational Fluid Dynamics”, 2nd MARNET
workshop, Copenhagen, 2000. Stern, F., Wilson, R.V., Coleman, H., and Pater-
son, E., 2001, “Comprehensive Approach to
Eça, L., Vaz, G., and Hoekstra, M., “Code veri- Verification and Validation of CFD Simula-
fication, solution verification and validation tions-Part 1: Methodology and Procedures”,
in RANS solvers, Proceedings of ASME 29th ASME J. Fluids Eng., Vol. 123, Dec. 2001.
International Conference, OMAE2010,
Shanghai, China. Wilson, R., Stern, F., Coleman, H., and Pater-
son, E., 2001, “Comprehensive Approach to
Hoekstra, M., Eca, L., Windt, J., and Raven, H., Verification and Validation of CFD Simula-
2000 “Viscous Flow Calculations for tions-Part2: Application for RANS Simula-
KVLCC2 AND KCS Models Using the tion of A Cargo/Container Ship”, ASME J.
PARNASSOS Code”, Proceedings Gothen- Fluids Eng., Vol. 123, Dec. 2001.
burg 2000 A Workshop on Numerical Ship
Hydrodynamics, Gothenburg, Sweden. Wilson, R. and Stern, F., 2002, “Verification
and Validation for RANS Simulation of a
JCGM, 2008, “Evaluation of measurement data Naval Surface Combatant”, Standards for
– Guide to the expression of uncertainty in CFD in the Aerospace Industry, 2002 AIAA
measurement,” JCGM 100:2008 GUM 1995 Aerospace Sciences Meeting, Reno, Ne-
with minor corrections, Joint Committee for vada, 14-17 January 2002.
Guides in Metrology, Bureau International
des Poids Mesures (BIPM), Sèvres, France. Wilson, R., Shao, J. and Stern, F., 2004, “Dis-
cussion: Criticism of the Correction Factor
Larsson, L., Stern, F., Bertram, V., 2000 Verification Method”, ASME J. Fluids Eng.,
“Gothenburg 2000 A Workshop on Numeri- Vol. 126, July 2004.
cal Ship Hydrodynamics”, Chalmers Uni-
versity of Technology, Gothenburg Sweden,
Sept.