Overview of CFD Verification & Validation Page 1 of 3

V&V Home Archive Tutorial

Overview of CFD Verification and Validation

Introduction

This page presents an overview of the process of the verification and validation of
computational fluid dynamics (CFD) simulations. The overall objective is to demonstrate the
accuracy of CFD codes so that they may be used with confidence for aerodynamic simulation
and that the results be considered credible for decision making in design.

One should first understand the distinctions between a code, simulation, and model. The
formal definitions of these terms are defined on the page entitled Glossary of Verification
and Validation Terms. Essentially, one implements a model into a computer code and then
uses the code to perform a CFD simulation which yield values used in the engineering
analysis. Verification and validation examines the errors in the code and simulation results.

Credibility is obtained by demonstrating acceptable levels of uncertainty and error. A

discussion of the uncertainties and errors in CFD simulations is provided on the page entitled
Uncertainty and Error in CFD Simulations. The levels of uncertainties and errors are
determined through verification assessment and validation assessment.

Verification assessment determines if the programming and computational implementation of

the conceptual model is correct. It examines the mathematics in the models through
comparison to exact analytical results. Verification assessment examines for computer
programming errors.

Validation assessment determines if the computational simulation agrees with physical

reality. It examines the science in the models through comparison to experimental results.

There is professional disagreement on exact procedures for verification and validation of

CFD simulations. CFD is maturing, but still an emerging technology. CFD is a complex
technology involving strongly coupled non-linear partial differential equations which attempt
to computationally model theoretical and experimental models in a discrete domain of
complex geometric shape. A detailed assessment of errors and uncertainties has to concern
itself with the three roots of CFD: theory, experiment, and computation. Further, the
application of CFD is rapidly expanding with the growth in computational resources. In this
work, we primarily follow the verification and validation guidelines established by the AIAA
[AIAA-G-077-1998]. Note that this is a guide - no standards yet exist for CFD simulation
verification and validation. Other ideas from other researchers in this discipline will also be
included. Their papers are referenced in the bibliography. Notable among them is the book
on verification and validation published by Roache.

Verification and validation are on-going activities due to the complex nature of the CFD
codes and expanding range of possible applications. Some basic verfication should be done
prior to release of a code and basic validation studies should be performed on classes of flow
features prior to use of the code for similar flows. However, as the code continues to develop,
verification and validation should continue.

Use of CFD Results

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/overview.html 2004-08-23
Overview of CFD Verification & Validation Page 2 of 3

The level of accuracy required from a CFD analysis depends on the desired use of the results.
A conceptual design effort may be content with general shock structure information, whereas
a detailed design may require accurate determination of the pressure recovery. Each quantity
to be determined generally has its own accuracy requirement. Levels of credibility may vary
according to the information required.

The application of CFD for design and analysis may be catagorized into three levels
according to increased levels of required accuracy: 1) provide qualitative information, 2)
provide incremental quantities, and 3) provide absolute quantities. This discussion follows
the ideas of Benek et al.

Provide qualitative information. CFD provides details on the entire flow field not possible
with experimental methods. This is useful in understanding on a qualitative level the
behavior of the flow field. Accuracy requirements are low.

Provide incremental quantities. Corrections to experimental observations can be provided

by CFD at a higher accuracy than existing with the CFD method. This is due to cancellation
of part of the error when taking the differences. For example, if there is a design change from
a baseline, for which the quanity is known, Pbaseline, the quantity P for the design change
can be expressed as

P = Pbaseline + dP

where dP is the increment in P corresponding to the design change. If two CFD simulations
are performed, the first with the baseline geometry and the second with the modified
geometry, then the increment in P due to the modified geometry can be estimated as,

dP = ( P + E )2 - ( P + E )1 = dPactual + dE.

The E is the error associated with the quantity P obtained from the CFD simulation. As can
be seen, the error to the increment is dE, which cancels out some of the error.

Provide absolute quantities. This level involves determining absolute values of the quantity
P and requires the highest level of accuracy. The accuracy required is usually stated as part
of the design process. The accuracy observed from the CFD simulation varies according to
the character of the quantity, and so, it is not possible to state an accuracy or error band that
applies to all quantities obtained from the CFD simulation. The verification methods
discussed below regarding a grid convergence study will provide the error band for the
calculations.

Flow Characteristics

In applying CFD for flows typical of aerospace systems, we must first understand the
characteristics of the flow. We must understand the reality upon which we will validate the
CFD code and processes.

The flow is characterized primarily by the Mach number. We are interested in analyzing
flows spanning the Mach number range from Mach 0 (static conditions) to 25 (access to
space).

The flow is characterized by high Reynolds numbers which result in regions of laminar flow
transitioning to turbulent flow. Flows along the body and inlet surfaces create boundary
layers. Adverse pressure gradient may be present for internal flows. At transonic, supersonic,

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/overview.html 2004-08-23
Overview of CFD Verification & Validation Page 3 of 3

and hypersonic speeds, shock waves are present. Under these conditions, the boundary layer
may separate.

At hypersonic Mach numbers, real gas effects may become important. This requires use of
chemistry models for calorically and thermally perfect gases, equilibrium air, and chemically
reaction of gas mixtures.

Often the geometry of the system is complex, which has to be physically modeled.

Unsteady flow may become important.

Physical Models

There are several physical models that are commonly used within CFD codes:

Spatial Dimension. The geometry of the inlet may be modelled in some cases using two-
dimensional or axisymmetric space rather than full three-dimensional.

Temporal Dimension. One may assume steady-state flow or attempt to capture the time
variations.

Navier-Stokes Equations. The Navier-Stokes equations govern the continuum flow.

Viscous and heat conduction effects are modelled. If these are removed, then inviscid flow
can be used.

Turbulence Models. Various algebraic, one-equation, and two-equation turbulence models

exist with various parameters and freestream boundary conditions. The option of wall
functions exists.

Thermodynamic and Transport Properties. Constants and relations for thermodynamic

and transport properties are generally constants, algebraic equations, or curve fits.

Air Chemistry Models. Inlet flows typically involve calorically perfect air adequately
described by the perfect gas equation of state. At higher temperatures (greater than 700K),
modeling of thermally perfect air, equilibrium air, and chemically reacting air (temperatures
greater than 2000 K) may be needed.

Flow Boundary Conditions. These include subsonic and supersonic freestream inflow and
outflow. Also inflow and outflow of plenum chambers.

Bleed / Blowing. These can be treated as boundary conditions as a mass flow or porous
boundary. Another option is to grid the slots and holes of the actual geometry.

Flow Control Devices. Flow control is important and several new technologies have
developed. Vortex generators are the primary flow control devices used in inlets. These can
be modeled or an approximation of the geometry can be gridded.

Web site curator, and responsible official for content:

John W. Slater, john.w.slater@grc.nasa.gov

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/overview.html 2004-08-23
CFD Analysis Process Page 1 of 3

V&V Home Archive Tutorial

CFD Analysis Process

The general process for performing a CFD analysis is outlined below so as to provide a
reference for understanding the various aspects of a CFD simulation. The process includes:

1. Forumulate the Flow Problem

2. Model the Geometry and Flow Domain
3. Establish the Boundary and Initial Conditions
4. Generate the Grid
5. Establish the Simulation Strategy
6. Establish the Input Parameters and Files
7. Perform the Simulation
8. Monitor the Simulation for Completion
9. Post-process the Simulation to get the Results
10. Make Comparisons of the Results
11. Repeat the Process to Examine Sensitivities
12. Document

In further detail, these steps include:

1. Formulate the Flow Problem

The first step of the analysis process is to formulate the flow problem by seeking
answers to the following questions:
{ what is the objective of the analysis?
{ what is the easiest way to obtain those objective?
{ what geometry should be included?
{ what are the freestream and/or operating conditions?
{ what dimensionality of the spatial model is required? (1D, quasi-1D, 2D,
axisymmetric, 3D)
{ what should the flow domain look like?
{ what temporal modeling is appropriate? (steady or unsteady)
{ what is the nature of the viscous flow? (inviscid, laminar, turbulent)
{ how should the gas be modeled?

2. Model the Geometry and Flow Domain

The body about which flow is to be analyzed requires modeling. This generally
involves modeling the geometry with a CAD software package. Approximations of the
geometry and simplifications may be required to allow an analysis with reasonable
effort. Concurrently, decisions are made as to the extent of the finite flow domain in
which the flow is to be simulated. Portions of the boundary of the flow domain
conicide with the surfaces of the body geometry. Other surfaces are free boundaries
over which flow enters or leaves. The geometry and flow domain are modeled in such
a manner as to provide input for the grid generation. Thus, the modeling often takes
into account the structure and topology of the grid generation.

3. Establish the Boundary and Initial Conditions

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/process.html 2004-08-23
CFD Analysis Process Page 2 of 3

Since a finite flow domain is specified, physical conditions are required on the
boundaries of the flow domain. The simulation generally starts from an initial solution
and uses an iterative method to reach a final flow field solution.

4. Generate the Grid

The flow domain is discretized into a grid. The grid generation involves defining the
structure and topology and then generating a grid on that topology. Currently all cases
involve multi-block, structured grids; however, the grid blocks may be abbuting,
contiguous, non-contiguous, and overlapping. The grid should exhibit some minimal
grid quality as defined by measures of orthogonality (especially at the boundaries),
relative grid spacing (15% to 20% stretching is considered a maximum value), grid
skewness, etc... Further the maximum spacings should be consistent with the desired
resolution of important features. The resolution of boundary layers requires the grid to
be clustered in the direction normal to the surface with the spacing of the first grid
point off the wall to be well within the laminar sublayer of the boundary layer. For
turbulent flows, the first point off the wall should exhibit a y+ value of less than 1.0.

5. Establish the Simulation Strategy

The strategy for performing the simulation involves determining such things as the use
of space-marching or time-marching, the choice of turbulence or chemistry model, and
the choice of algorithms.

6. Establish the Input Parameters and Files

A CFD codes generally requires that an input data file be created listing the values of
the input parameters consisted with the desired strategy. Further the a grid file
containing the grid and boundary condition information is generally required. The files
for the grid and initial flow solution need to be generated.

7. Perform the Simulation

The simulation is performed with various possible with options for interactive or batch
processing and distributed processing.

8. Monitor the Simulation for Completion

As the simulation proceeds, the solution is monitored to determine if a "converged"

solution has been obtained, which is iterative convergence. Further discussion can be
found on the page entitled Examining Iterative Convergence.

9. Post-Process the Simulation to get the Results

Post-Processing involves extracting the desired flow properties (thrust, lift, drag, etc...)
from the computed flowfield.

10. Make Comparisons of the Results

The computed flow properties are then compared to results from analytic,
computational, or experimental studies to establish the validity of the computed results.

11. Repeat the Process to Examine Sensitivities

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/process.html 2004-08-23
CFD Analysis Process Page 3 of 3

The sensitivity of the computed results should be examined to understand the possible
differences in the accuracy of results and / or performance of the computation with
respect to such things as:
{ dimensionality
{ flow conditions
{ initial conditions
{ marching strategy
{ algorithms
{ grid topology and density
{ turbulence model
{ chemistry model
{ flux model
{ artificial viscosity
{ boundary conditions
{ computer system

Further information can be found on the pages entitled Verification Assessment and
Validation Assessment.

12. Document

Documenting the findings of an analysis involves describing each of these steps in the
process.

Web site curator, and responsible official for content:

John W. Slater, john.w.slater@grc.nasa.gov

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/process.html 2004-08-23
Uncertainty and Error in CFD Simulations Page 1 of 4

V&V Home Archive Tutorial

Uncertainty and Error in CFD Simulations

This page provides a classification of uncertainties and errors that cause CFD simulation
results to differ from their true or exact values. This discussion not only applies to the CFD
code, but other computer programs used in the analysis process such as CAD packages, grid
generators, and flow visualizers.

Defining Uncertainty and Error

Uncertainty and Error are commonly used interchangeably in everyday language. Here we
follow the definitions of the AIAA Guidlines:

Uncertainty is defined as

A potential deficiency in any phase or activity of the modeling process that is

due to the lack of knowledge." (AIAA G-077-1998)

Error is defined as

A recoqnizable deficiency in any phase or activity of modeling and simulation

that is not due to lack of knowledge. (AIAA G-077-1998)

The key phrase differentiating the definitions of uncertainty and error is lack of knowledge.

The key word in the definition of uncertainty is potential, which indicates that deficiencies
may or may not exist. Lack of knowledge has primarily to do with lack of knowledge about
the physical processes that go into building the model. Sensitivity and uncertainty analyses
can be used to better determine uncertainty. Uncertainty applies to describing deficiencies in
turbulence modeling. There is a lot about turbulence modeling that is not understood. One
approach for determining the level of uncertainty and it effect on one's analysis is to run a
number of simulations with a variety of turbulence models and see how the modeling affects
the results.

The definition for error implies that the deficiency is identifiable upon examination. Errors
can also be classified as acknowledged or unacknowledged:

Acknowledged errors (examples include round-off error and discretization

error) have procedures for identifying them and possibly removing them.
Otherwise they can remain in the code with their error estimated and listed.

Unacknowledged errors (examples include computer programming errors or

usage errors) have no set procedures for finding them and may continue within
the code or simulation.

One can differentiate between local and global errors. Local errors refer to errors at a grid
point or cell, whereas global errors refer to errors over the entire flow domain. We are
interested here in the global error of the solution that accounts for the local error at each grid
point but is more than just the sum of the local errors. Local errors are transported, advected,
and diffused throughout the grid.

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/errors.html 2004-08-23
Uncertainty and Error in CFD Simulations Page 2 of 4

The definition of error presented here is different than that an experimentalist may use, which
is "the difference between the measured value and the exact value". Experimentalist usually
define uncertainty as "the estimate of error". These definitions are inadequate for
computational simulations because the exact value is typically not known. Further these
definitions link error with uncertainty. The defintions provided in the above paragraphs are
more definite because they differentiate error and uncertainty according to what is known.

Classification of Errors

Here we provide a classification or taxonomy of error.

Acknowledged Error

1. Physical approximation error

{ Physical modeling error
{ Geometry modeling error

2. Computer round-off error

3. Iterative convergence error

4. Discretization error
{ Spatial discretization error
{ Temporal discretization error

Unacknowledged Error

1. Computer programming error

2. Usage error

Each of these types of errors are discussed below.

Physical Approximation Error

Physical modeling errors are those due to uncertainty in the formulation of the model and
deliberate simplifications of the model. These errors deal with the continuum model only.
Converting the model to discrete form for the code is discussed as part of discretization
errors. Errors in the modeling of the fluids or solids problem are concerned with the choice of
the governing equations which are solved and models for the fluid or solid properties.
Further, the issue of providing a well-posed problem can contribute to modeling errors. Often
modeling is required for turbulence quantities, transistion, and boundary conditions (bleed,
time-varying flow, surface roughness). Mehta lists sources of uncertainty in physical models
as 1) the phenomenon is not thoroughly understood; 2) parameters used in the model are
known but with some degree of uncertainty; 3) appropriate models are simplified, thus
introducing uncertainty; and 4) an experimental confirmation of the models is not possible or
is incomplete. Even when a physical process is known to a high level of accuracy, a
simplified model may be used within the CFD code for the convenience of a more efficient
computation. Physical modeling errors are examined by performing validation studies that
focus on certain models (i.e. inviscid flow, turbulent boundary layers, real-gas flows, etc...).

Computer Round-Off Error

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/errors.html 2004-08-23
Uncertainty and Error in CFD Simulations Page 3 of 4

Computer round-off errors develop with the representation of floating point numbers on the
computer and the accuracy at which numbers are stored. With advanced computer resources,
numbers are typically stored with 16, 32, or 64 bits. Round-off errors are not considered
significant when compared with other errors. If computer round-off errors are suspected of
being significant, one test is to run the code at a higher precision or on a computer known to
store floating point numbers at a higher precision. One can attempt to iterate a coarse grid
solution to a residual of machine zero; however, this may not be possible for more complex
algorithms.

Iterative Convergence Error

The iterative convergence error exists because the iterative methods used in the simulation
must have a stopping point eventually. The error scales to the variation in the solution at the
completion of the simulations.

Discretization Errors

Discretization errors are those errors that occur from the representation of the governing
flow equations and other physical models as algebraic expressions in a discrete domain of
space (finite-difference, finite-volume, finite-element) and time. The discrete spatial domain
is known as the grid or mesh. The temporal discreteness is manifested through the time step
taken. Discretization error is also known as numerical error. A consistent numerical method
will approach the continuum representation of the equations and zero discretization error as
the number of grid points increases and the size of the grid spacing tends to zero. As the
mesh is refined, the solution should become less sensitive to the grid spacing and approach
the continuum solution. This is grid convergence. Such thinking also applies to the time step.
The grid convergence study is a useful procedure for determining the level of discretization
error existing in a CFD solution. "Ordered" discretization errors are those dependent on the
grid size and vanish as the grid size approaches zero. These are the errors that are addressed
by a grid convergence study. Further details can be found on the pages entitled Examining
Spatial (Grid) Convergence and Examining Temporal Convergence.

The discretization error is of most concern to a CFD code user during an application.
Discretization errors are of major concern because they are dependent on the quality of the
grid; however, it is often difficult to precisely indicate the relationship between a quality grid
and an accurate solution prior to beginning the simulation. The level of discretization error is
dependent on grid quality. The grid should be generated with consideration of such things as
resolution, density, aspect ratio, stretching, orthogonality, grid singularities, and zonal
boundary interfaces.

The level of discretization error is dependent on the features of the flow as resolved by the
grid. Errors may develop due to representation of discontinuities (shocks, slip surfaces,
interfaces, ...) on a grid. Interpolation errors come about at zonal interfaces where the
solution of one zone is approximated on the boundary of the other zone.

The truncation error is the difference between the partial differential equation (PDE) and the
finite equation. The truncation error is a function of the grid quality and flow gradients.
Dispersive error terms causes oscillations in the solution. One fix to this is adding artificial
dissipation to decrease the size of the dispersive errors. Dissipation error terms cause a
smoothing of gradients. However, a level of dissipation comparable to the actual physical
viscosity may contaminate the solution. Boundary layers may thicken. The truncation error
terms are those of the expansion which are not used in the discretized equation. If the order
of the leading term of the truncation error is of second-order, it is known as a numerical
viscosity (dimensions of length2 / time), which is dimensions of kinematic viscosity. A

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/errors.html 2004-08-23
Uncertainty and Error in CFD Simulations Page 4 of 4

positive viscous term will indicate that errors will be damped whereas a negative viscous
term will indicate that errors will grow (unstable).

Included in the discretization error are errors due to not properly converging the solution with
respect to the iterations to the steady-state solution or within a time step. This is reffered to as
iterative convergence.

Computer Programming Errors

Programming errors are "bugs" and mistakes made in programming or writing the code.
They are the responsibility of the programmers. These type of errors are discovered by
systematically performing verification studies of subprograms of the code and the entire
code, reviewing the lines of code, and performing validation studies of the code. The
programming errors should be removed from the code prior to release.

Usage Errors

Usage errors are due to the application of the code in a less-than-accurate or improper
manner. Usage errors may actually show up as modeling and discretization errors. The user
sets the models, grid, algorithm, and inputs used in a simulation, which then establishes the
accuracy of the simulation. There may be blatant errors, such as attempting to compute a
known turbulent flow with an assumption of inviscid flow. A converged solution may be
obtained; however, the conclusions drawn from the simulation may be incorrect. The errors
may not be as evident, such as proper choice of turbulence model parameters for separated
flows with shocks. The potential for usage errors increases with an increased level of options
available in a CFD code. Usage errors are minimized through proper training and the
accumulation of experience.

The user may intentionally introduce modeling and discretization error as an attempt to
expedite the simulation at the expense of accuracy. This may be proper in the conceptual
stage of a design study where more general information is needed at less accuracy. Even in
the later stages, there may not be proper computational resources to simulate at the proper
grid density. One has to understand the level of accuracy accompanying the results.

Usage errors should be controlable through proper training and analysis.

Usage errors can exist in the CAD, grid generation, and post-processing software, in addition
to the CFD code.

Web site curator, and responsible official for content:

John W. Slater, john.w.slater@grc.nasa.gov

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/errors.html 2004-08-23
Verification Assessment Page 1 of 2

V&V Home Archive Tutorial

Verification Assessment
This page discusses Verification Assessment, which focuses on the methods for Verification
of CFD codes and simulation results.

Verification

Verification is defined as

The process of determining that a model implementation accurately represents

the developer's conceptual description of the model and the solution to the
model. (AIAA G-077-1998)

Verification assessment examines 1) if the computational models are the correct

implementation of the conceptual models, and 2) if the resulting code can be properly used
for an analysis. The strategy is to identify and quantify the errors in the model
implementation and the solution. The two aspects of verification are the verification of a
code and the verification of a calculation. The objective of verifying a code is error
evaluation, that is, finding and removing errors in the code. The objective of verifying a
calculation is error estimation, that is determining the accuracy of a calculation. Each are
discussed below.

Verification has also been described as solving the equations right. It is intended to concern
itself more with mathematics rather than engineering. It is intended to look for errors in the
programming and implementation of the models.

Roache considers two aspects of verification: 1) verification of a code and 2) verification of a

calculation. These two aspects are described below:

Verification of a code involves error evaluation, which is, looking for bugs,
incorrect implementations of conceptual models, errors in inputs, and other
errors in the code and usage. This is typically done by the developers prior to
release of the code. First, consistency checks are performed which examine basic
relationships expected in the solutions (i.e. mass conservation). Then the code is
used to simulate a suite of ``highly accurate'' verification cases. These cases
should be analytic or numeric solution to ordinary and partial differential
equations. Verification should not performed with experimental data. A grid
refinement study should be conducted to bring out potential errors. All the
options of the code should be examined. This becomes more complicated as the
number of options available within a CFD code increase. Identifying and
quantifying each type of error is important because errors can interact and cancel
each other - leading to erroneous conclusions in the validation process. One
potentially useful method of verification is comparing the results of two codes.
However, verification is not a democratic activity and one should watch for
comparing with an inaccurate code. The comparison is strengthened when the
two codes use differing numerical methods. The following paragraphs discuss
specific checks that can be performed as part of a code verification process.

Verification of a calculation involves error estimation, which is determining

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/verassess.html 2004-08-23
Verification Assessment Page 2 of 2

the accuracy of a single calculation and putting an error band on the final value.
The approach involves peformimg a grid convergence study and determine the
observed order of convergence, error bands, and grid convergence indices (GCI).

Verification Assessment Process

The process for Verification Assessment of a CFD code and / or simulation can be
summarized as:

1. Examine the Computer Programming of the Code. One of the most basic tasks of
verification assessment is the review of the computer programming or coding to check for
and identify computer programming errors or "bugs". This is done by visually checking the
coding and by computationally running subprograms using a test code. This is aided by
complete and clear documentation, both internal and external. This step is to directly detect
computer programming errors.

2. Examine Iterative Convergence. Verification assessment requires that a simulation

demonstrates iterative convergence. Further details can be page entitled Examining Iterative
Convergence.

3. Examine Consistency. One should check for consistency in the CFD solution. For
example, the flow in a duct should maintain mass conservation through the duct. Further total
pressure recovery in an inlet should stay constant or decrease through the duct.

4. Examine Spatial (Grid) Convergence. The CFD simulation results should demonstrate
spatial convergence. Further details and methods can be found on the page entitled
Examining Spatial (Grid) Convergence.

5. Examine Temporal Convergence. The CFD simulation results should demonstrate

temporal convergence. Further details and methods can be found on the page entitled
Examining Temporal Convergence.

6. Compare CFD Results to Highly Accurate Solutions The veracity of a code can be
examined by comparing the CFD simulation results to highly accurate solution to the models
used within the CFD code. This can include analytical solutions, benchmark numerical
solutions to ordinary differential equations (ODEs), and benchmark numerical solutions to
partial differential equations (PDEs).

Web site curator, and responsible official for content:

John W. Slater, john.w.slater@grc.nasa.gov

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/verassess.html 2004-08-23
Validation Assessment Page 1 of 3

V&V Home Archive Tutorial

Validation Assessment
This page discusses Validation Assessment, which focuses on the methods for the Validation
of a CFD codes for simulation of a certain type of flows.

Validation

Validation is defined as

The process of determining the degree to which a model is an accurate

representation of the real world from the perspective of the intended uses of the
model. (AIAA G-077-1998)

Validation has also been described as "solving the right equations". It is not possible to
validate the entire CFD code. One can only validate the code for a specific range of
applications for which there is experimental data. Thus one validates a model or simulation.
Applying the code to flows beyond the region of validity is termed prediction.

Validation examines if the conceptual models, computational models as implemented into

the CFD code, and computational simulation agree with real world observations. The
strategy is to indentify and quantify error and uncertainty through comparison of simulation
results with experimental data. The experiment data sets themselves will contain bias errors
and random errors which must be properly quantified and documented as part of the data set.
The accuracy required in the validation activities is dependent on the application, and so, the
validation should be flexible to allow various levels of accuracy.

The approach to Validation Assessment is to perform a systematic comparison of CFD

simulation results to experimental data from a set increasingly complex cases.

Each CFD simulation requires verification of the calculation as specified in the discussion of
Verification Assessment.

Validation Assessment Process

The process for Validation Assessment of a CFD simulation can be summarized as:

1. Examine Iterative Convergence. Validation assessment requires that a simulation

demonstrates iterative convergence. Further details can be page entitled Examining Iterative
Convergence.

2. Examine Consistency. One should check for consistency in the CFD solution. For
example, the flow in a duct should maintain mass conservation through the duct. Further total
pressure recovery in an inlet should stay constant or decrease through the duct.

3. Examine Spatial (Grid) Convergence. The CFD simulation results should demonstrate
spatial convergence. Further details and methods can be found on the page entitled
Examining Spatial (Grid) Convergence.

4. Examine Temporal Convergence. The CFD simulation results should demonstrate

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/valassess.html 2004-08-23
Validation Assessment Page 2 of 3

temporal convergence. Further details and methods can be found on the page entitled
Examining Temporal Convergence.

5. Compare CFD Results to Experimental Data. Experimental data is the observation of

the "real world" in some controlled manner. By comparing the CFD results to experimental
data, one hopes that there is a good agreement, which inreases confidence that the physical
models and the code represents the "real world" for this class of simulations. However, the
experimental data contains some level of error. This is usually related to the complexity of
the experiment. Validation assessment calls for a "building block" approach of experiments
which sets a hierarchy of experiment complexity.

6. Examine Model Uncertainties. The physical models in the CFD code contain
uncertainties due to a lack of complete understanding or knowledge of the physical
processes. One of the models with the most uncertainty is the turbulence models. The
uncertainty can be examined by running a number of simulations with the various turbulence
models and examine the affect on the results.

Building-Block Approach for Experiments

A building-block approach is followed in performing the validation assessment for a

complex system such as an aircraft inlet. The approach consists of phases involving
successively more complex flow physics, geometry, and interactions. These phases include:

Unit Problems involve simple geometry, one element of the complex flow
physics, and one relevant flow feature. An example is the measurement of a
turbulent boundary layer over a flat plate. The experiment data set contains
detailed data collected with high accuracy. The boundary conditions and initial
conditions are accurately measured.

Benchmark Cases involve fairly simple hardware representing a key feature of

the system. The flow field contains only two separate flow features of the flow
physics which are likely coupled. An example is a shock / boundary layer
interaction. The experiment data set is extensive in scope and uncertainties are
low; however, some measurements, such as, initial and boundary conditions,
may not have been collected.

Subsystem Cases involve geometry of a component of the complete system

which may have been simplified. The flow physics of the complete system may
be well represented; but the level of coupling between flow phenomena is
typically reduced. An example is a test of a subsonic diffuser for a supersonic
inlet. The exact inflow conditions may not be matched. The quality and quantity
of the experiment data set may not be as extensive as the benchmark cases.

Complete System Cases involve actual hardware and the complete flow
physics. All of the relevant flow features are present. An example is a test of a
mixed-compression inlet in the 10x10 wind tunnel at NASA Glenn. Less
detailed data is collected since the emphasis is on system evaluation.
Uncertainties on initial and boundary conditions may be large.

Requirements for Experimental Data

The experimental data likely has uncertainties and error associated with it. In comparing the
CFD simulation results to experimental data, one should discuss the experimental errors.
Plots comparing CFD results and experimental data should include a visual display of the

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/valassess.html 2004-08-23
Validation Assessment Page 3 of 3

error bars on the experimental data.

Web site curator, and responsible official for content:

John W. Slater, john.w.slater@grc.nasa.gov

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/valassess.html 2004-08-23
Examining Iterative Convergence Page 1 of 1

V&V Home Archive Tutorial

Examining Iterative Convergence

Generally, CFD methods involve some iterative scheme to arrive at the simulation results.
Here it is assumed that the iteration is with respect to time or a pseudo-temporal quantity and
some type of time step is taken at each iteration. A steady-state flow simulation involves
starting with a uniform or fabricated flow field and iterating in time until the steady-state
flow field is obtained. This is termed iterative convergence, but requires some criteria for
determining convergence. Criteria include:

Residuals. The residuals of the equations are the change in the equations over an iteration.
These are usually scaled or normalized. One usually looks for the residuals to reach a certain
level and then level-off as an indication of iterative convergence. For a time-marching,
steady-state strategy, this involves examining whether the residual has been reduced a certain
number (usually 3-4) of orders of magnitude.

Results. The CFD simulation has the objective of determining some quantity such as lift,
drag, recovery, etc... One can track the values of such engineering quantities with respect to
iteration and define iterative convergence when these quantities converge. The convergence
criteria is usually defined by acceptable error in these values. It is often the case that certain
quantities may reach convergence at a different rate than other quantities. One can check that
a monitored flow value (such as thrust, drag, or boundary layer profile) has remained
unchanged with respect to the number of iterations.

Time-Accurate Simulations. For a time-marching, time-accurate strategy, this involves

examining whether the final time has been reached with proper convergence at each time
step.

Space-Marching Simulation For a space-marching strategy, this involves examining

whether the end of the marching segment has been reached with proper convergence at each
marching step.

Web site curator, and responsible official for content:

John W. Slater, john.w.slater@grc.nasa.gov

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/iterconv.html 2004-08-23
Examining Solution Consistency Page 1 of 1

V&V Home Archive Tutorial

Examining Solution Consistency

One can evaluate convergence by checking for consistency in the flow field. The
conservation relations require a balance of fluxes through a control surface. For a closed
duct, the flow through the duct should be conserved. Low-speed flow over a closed body
should have zero drag. Other such consistency relations can be defined for specific flow
fields. These provide verification of the code since the consistency relations are usually a
statement of some analytic result.

Web site curator, and responsible official for content:

John W. Slater, john.w.slater@grc.nasa.gov

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/consistency.html 2004-08-23
Examining Spatial (Grid) Convergence Page 1 of 9

V&V Home Archive Tutorial

Examining Spatial (Grid) Convergence

Introduction

The examination of the spatial convergence of a simulation is a straight-forward method for

determining the ordered discretization error in a CFD simulation. The method involves
performing the simulation on two or more successively finer grids. The term grid
convergence study is equivalent to the commonly used term grid refinement study.

As the grid is refined (grid cells become smaller and the number of cells in the flow domain
increase) and the time step is refined (reduced) the spatial and temporal discretization errors,
respectively, should asymptotically approaches zero, excluding computer round-off error.

Methods for examining the spatial and temporal convergence of CFD simulations are
presented in the book by Roache. They are based on use of Richardson's extrapolation. A
summary of the method is presented here.

A general discussion of errors in CFD computations is available for background.

We will mostly likely want to determine the error band for the engineering quantities
obtained from the finest grid solution. However, if the CFD simulations are part of a design
study that may require tens or hundreds of simulations, we may want to use one of the
coarser grids. Thus we may also want to be able to determine the error on the coarser grid.

Grid Considerations for a Grid Convergence Study

The easiest approach for generating the series of grids is to generate a grid with what one
would consider fine grid spacing, perhaps reaching the upper limit of one's tolerance for
generating a grid or waiting for the computation on that grid to converge. Then coarser grids
can be obtained by removing every other grid line in each coordinate direction. This can be
continued to create additional levels of coarser grids. In generating the fine grid, one can
build in the n levels of coarser grids by making sure that the number of grid points in each
coordinate direction satisfies the the relation

N = 2n m + 1

where m is an integer. For example, if two levels of coarser grids are desired (i.e. fine,
medium, and coarse grids) then the number of grid points in each coordinate direction must
equal 4 m + 1. The m may be different for each coordinate direction.

The WIND code has a grid sequencing control that will solve the solution on the coarser grid
without having to change the grid input file, boundary condition settings, or the input data
file. Further, the converged solution on the coarser grid then can be used directly as the initial
solution on the finer grid. This option was originally used to speed up convergence of
solutions; however, can be used effectively for a grid convergence study.

It is not necessary to halve the number of grid points in each coordinate direction to obtain
the coarser grid. Non-integer grid refinement or coarsening can be used. This may be desired
since halfing a grid may put the solution out of the asymptotic range. Non-integer grid

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/spatconv.html 2004-08-23
Examining Spatial (Grid) Convergence Page 2 of 9

refinement or coarsening will require the generation of a new grid. It is important to maintain
the same grid generation parameters as the original grid. One approach is to select several
grid spacings as reference grid spacings. One should be the grid spacing normal to the walls.
Others may be spacings at flow boundaries, at junctures in the geometry, or at zonal
interfaces. Upon picking the ratio as which the grid is to be refined or coarsened, this same
ratio is applied to these spacings. The number of grid points are then adjusted according to
grid quality parameters, such as grid spacing ratio limits. The surface and volume grids are
then generated using the same methods as the original grid. The grid refinement ratio should
be a minimum of r >= 1.1 to allow the discretization error to be differentiated from other
error sources (iterative convergence errors, computer round-off, etc...).

Order of Grid Convergence

The order of grid convergence involves the behavior of the solution error defined as the
difference between the discrete solution and the exact solution,

where C is a constant, h is some measure of grid spacing, and p is the order of convergence.
A "second-order" solution would have p = 2.

A CFD code uses a numerical algorithm that will provide a theoretical order of convergence;
however, the boundary conditions, numerical models, and grid will reduce this order so that
the observed order of convergence will likely be lower.

Neglecting higher-order terms and taking the logarithm of both sides of the above equation
results in

The order of convergence p can be obtained from the slope of the curve of log(E) versus log
(h). If such data points are available, the slope can be read from the graph or the slope can be
computed from a least-squares fit of the data. The least-squares will likely be inaccurate if
there are only a few data points.

A more direct evaluation of p can be obtained from three solutions using a constant grid
refinement ratio r,

The order of accuracy is determined by the order of the leading term of the truncation error
and is represented with respect to the scale of the discretization, h. The local order of
accuracy is the order for the stencil representing the discretization of the equation at one
location in the grid. The global order of accuracy considers the propagation and
accumulation of errors outside the stencil. This propagation causes the global order of
accuracy to be, in general, one degree less than the local order of accuracy. The order of
accuracy of the boundary conditions can be one order of accuracy lower than the interior
order of accuracy without degrading the overall global accuracy.

Asymptotic Range of Convergence

Assessing the accuracy of code and caluculations requires that the grid is sufficiently refined
such that the solution is in the asymptotic range of convergence. The asymptotic range of

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/spatconv.html 2004-08-23
Examining Spatial (Grid) Convergence Page 3 of 9

convergence is obtained when the grid spacing is such that the various grid spacings h and
errors E result in the constancy of C,

C = E / hp

Another check of the asymptotic range will be discussed in the section on the grid
convergence index.

Richardson Extrapolation

Richardson extrapolation is a method for obtaining a higher-order estimate of the continuum

value (value at zero grid spacing) from a series of lower-order discrete values.

A simulation will yield a quantity f that can be expressed in a general form by the series
expansion

f = fh=0 + g1 h + g2 h2 + g3 h3 + ...

where h is the grid spacing and the functions g1, g2, and g3 are independent of the grid
spacing. The quantity f is considered "second-order" if g1 = 0.0. The fh=0 is the continuum
value at zero grid spacing.

If one assumes a second-order solution and has computed f on two grids of spacing h1 and h2
with h1 being the finer (smaller) spacing, then one can write two equations for the above
expansion, neglect third-order and higher terms, and solve for fh=0 to estimate the continuum
value,

where the grid refinement ratio is

r = h2 / h1

The Richardson extrapolation can be generalized for a p-th order methods and r-value of
grid ratio (which does not have to be an integer) as

Traditionally, Richardson extrapolation has been used with grid refinement ratios of r = 2.
Thus, the above equation simplifies to

In theory, the above equations for the Richardson extrapolation will provide a fourth-order
estimate of fh=0, if f1 and f2 were computed using exactly second-order methods. Otherwise,
it will be a third-order estimate. In general, we will consider fh=0 to be p+1 order accurate.
Richardson extrapolation can be applied for the solution at each grid point, or to solution
functionals, such as pressure recovery or drag. This assumes that the solution is globally

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/spatconv.html 2004-08-23
Examining Spatial (Grid) Convergence Page 4 of 9

second-order in addition to locally second-order and that the solution functionals were
computed using consistent second-order methods. Other cautions with using Richardson
extrapolation (non-conservative, amplification of round-off error, etc...) are discussed in the
book by Roache.

For our purposes we will assume f is a solution functional (i.e. pressure recovery). The fh=0
is then as an estimate of f in the limit as the grid spacing goes to zero. One use of fh=0 is to
report the value as the an improved estimate of f1 from the CFD study; however, one has to
understand the caveats mentioned above that go along with that value.

The other use of fh=0 is to obtain an estimate of the discretization error that bands f obtained
from the CFD. This use will now be examined.

The difference between f1 and fh=0 is one error estimator; however, this requires
consideration of the caveats attached to fh=0.

We will focus on using f1 and f2 to obtain an error estimate. Examining the generalized
Richardson extrapolation equation above, the second term on the right-hand side can be
considered to be an an error estimator of f1. The equation can be expressed as

A1 = E1 + O( hp+1, E12)

where A1 is the actual fractional error defined as

A1 = ( f1 - fh=0 ) / fh=0

and E1 is the estimated fractional error for f1 defined as

where the relative error is defined as

This quantity should not be used as an error estimator since it does not take into account r or
p. This may lead to an underestimation or overestimation of the error. One could make this
quantity artificially small by simply using a grid refinement ratio r close to 1.0.

The estimated fractional error E1 is an ordered error estimator and a good approximation of
the discretization error on the fine grid if f1 and f2 were obtained with good accuracy (i.e.
E1<<1). The value of E1 may be meaningless if f1 and fh=0 are zero or very small relative to
f2-f1. If such is the case, then another normalizing value should be used in place of f1.

If a large number of CFD computations are to be performed (i.e for a DOE study), one may
wish to use the coarser grid with h2. We will then want to estimate the error on the coarser
grid. The Richardson extrapolation can be expressed as

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/spatconv.html 2004-08-23
Examining Spatial (Grid) Convergence Page 5 of 9

The estimated fractional error for f2 is defined as

Richardson extrapolation is based on a Taylor series representation as indicated in Eqn. \ref

{eq:series}. If there are shocks and other discontinuities present, the Richardson
extrapolation is invalid in the region of the discontinuity. It is still felt that it applies to
solution functionals computed from the entire flow field.

Grid Convergence Index (GCI)

Roache suggests a grid convergence index GCI to provide a consistent manner in reporting
the results of grid convergence studies and perhaps provide an error band on the grid
convergence of the solution. The GCI can be computed using two levels of grid; however,
three levels are recommended in order to accurately estimate the order of convergence and to
check that the solutions are within the asymptotic range of convergence.

A consistent numerical analysis will provide a result which approaches the actual result as
the grid resolution approaches zero. Thus, the discretized equations will approach the
solution of the actual equations. One significant issue in numerical computations is what
level of grid resolution is appropriate. This is a function of the flow conditions, type of
analysis, geometry, and other variables. One is often left to start with a grid resolution and
then conduct a series of grid refinements to assess the effect of grid resolution. This is known
as a grid refinement study.

One must recognize the distinction between a numerical result which approaches an
asymptotic numerical value and one which approaches the true solution. It is hoped that as
the grid is refined and resolution improves that the computed solution will not change much
and approach an asymptotic value (i.e. the true numerical solution). There still may be error
between this asymptotic value and the true physical solution to the equations.

Roache has provided a methodology for the uniform reporting of grid refinement studies.
"The basic idea is to approximately relate the results from any grid refinement test to the
expected results from a grid doubling using a second-order method. The GCI is based upon a
grid refinement error estimator derived from the theory of generalized Richardson
Extrapolation. It is recommended for use whether or not Richardson Extrapolation is actually
used to improve the accuracy, and in some cases even if the conditions for the theory do not
strictly hold." The object is to provide a measure of uncertainty of the grid convergence.

The GCI is a measure of the percentage the computed value is away from the value of the
asymptotic numerical value. It indicates an error band on how far the solution is from the
asymptotic value. It indicates how much the solution would change with a further refinement
of the grid. A small value of GCI indicates that the computation is within the asymptotic
range.

The GCI on the fine grid is defined as

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/spatconv.html 2004-08-23
Examining Spatial (Grid) Convergence Page 6 of 9

where Fs is a factor of safety. The refinement may be spatial or in time. The factor of safety
is recommended to be Fs=3.0 for comparisons of two grids and Fs=1.25 for comparisons
over three or more grids. The higher factor of safety is recommended for reporting purposes
and is quite conservative of the actual errors.

When a design or analysis activity will involve many CFD simulations (i.e. DOE study), one
may want to use the coarser grid h2. It is then necessary to quantify the error for the coarser
grid. The GCI for the coarser grid is defined as

It is important that each grid level yield solutions that are in the asymptotic range of
convergence for the computed solution. This can be checked by observing two GCI values as
computed over three grids,

GCI23 = rp GCI12

Required Grid Resolution

If a desired accuracy level is known and results from the grid resolution study are available,
one can then estimate the grid resolution required to obtain the desired level of accuracy,

Independent Coordinate Refinement and Mixed Order Methods

The grid refinement ratio assumes that the refinement ratio r applies equally in all coordinate
directions (i,j,k) for steady-state solutions and also time t for time-dependent solutions. If
this is not the case, then the grid convergence indices can be computed for each direction
independently and then added to give the overall grid convergence index,

Effective Grid Refinement Ratio

If one generates a finer or coarser grid and is unsure of the value of grid refinement ratio to
use, one can compute an effective grid refinement ratio as

where N is the total number of grid points used for the grid and D is the dimension of the
flow domain. This effective grid refinement ratio can also be used for unstructured grids.

Example Grid Convergence Study

The following example demonstrates the application of the above procedures in conducting a
grid convergence study. The objective of the CFD analysis was to determine the pressure
recovery for an inlet. The flow field is computed on three grids, each with twice the number

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/spatconv.html 2004-08-23
Examining Spatial (Grid) Convergence Page 7 of 9

of grid points in the i and j coordinate directions as the previous grid. The number of grid
points in the k direction remains the same. Since the flow is axisymmetric in the k direction,
we consider the finer grid to be double the next coarser grid. The table below indicates the
grid information and the resulting pressure recovery computed from the solutions. Each
solution was properly converged with respect to iterations. The column indicated by
"spacing" is the spacing normalized by the spacing of the finest grid.

Grid Normalized Grid Spacing Pressure Recovery, Pr

1 1 0.97050
2 2 0.96854
3 4 0.96178

The figure below shows the plot of pressure recoveries with varying grid spacings. As the
grid spacing reduces, the pressure recoveries approach an asymptotic zero-grid spacing
value.

We determine the order of convergence observed from these results,

p = ln[ ( 0.96178 - 0.96854 ) / ( 0.96854 - 0.97050 ) ] / ln(2) = 1.786170

The theoretical order of convergence is p=2.0. The difference is most likely due to grid
stretching, grid quality, non-linearities in the solution, presence of shocks, turbulence
modeling, and perhaps other factors.

We now can apply Richardson extrapolation using the two finest grids to obtain an estimate
of the value of the pressure recovery at zero grid spacing,

Prh=0 = 0.97050 + ( 0.97050 - 0.96854 ) / ( 21.786170 - 1 )

= 0.97050 + 0.00080 = 0.97130

This value is also plotted on the figure below.

The grid convergence index for the fine grid solution can now be computed. A factor of
safety of FS=1.25 is used since three grids were used to estimage p. The GCI for grids 1 and
2 is

GCI12 = 1.25 | ( 0.97050 - 0.96854 ) / 0.97050 | / ( 21.786170 - 1 ) 100% = 0.103083%

The GCI for grids 2 and 3 is

GCI23 = 1.25 | ( 0.96854 - 0.96178 ) / 0.96854 | / ( 21.786170 - 1 ) 100% = 0.356249%

We can now check that the solutions were in the asymptotic range of convergence,

0.356249 / ( 21.786170 0.103083 ) = 1.002019

which is approximately one and indicates that the solutions are well within the asymptotic
range of convergence.

Based on this study we could say that the pressure recovery for the supersonic ramp is
estimated to be Pr = 0.97130 with an error band of 0.103% or 0.001.

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/spatconv.html 2004-08-23
Examining Spatial (Grid) Convergence Page 8 of 9

VERIFY: A Fortran program to Perform Calculations Associated with a

Grid Convergence Study

The Fortran 90 program verify.f90 was written to carry out the calculations associated with a
grid convergence study involving 3 or more grids. The program is compiled on a unix system
through the commands

f90 verify.f90 -o verify

It reads in an ASCII file (prD.do) through the standard input unit (5) that contains a list of
pairs of grid size and value of the observed quantity f.

verify < prD.do > prD.out

It assumes the values from the finest grid are listed first. The output is then written to the
standard output unit (6) prD.out. The output from the of {\tt verify} for the results of
Appendix A are

--- VERIFY: Performs verification calculations ---

Number of data sets read = 3

Grid Size Quantity

1.000000 0.970500
2.000000 0.968540
4.000000 0.961780

Order of convergence using first three finest grid

and assuming constant grid refinement (Eqn. 5.10.6.1)
Order of Convergence, p = 1.78618479

Richardson Extrapolation: Use above order of convergence

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/spatconv.html 2004-08-23
Examining Spatial (Grid) Convergence Page 9 of 9

and first and second finest grids (Eqn. 5.4.1)

Estimate to zero grid value, f_exact = 0.971300304

Grid Convergence Index on fine grids. Uses p from above.

Factor of Safety = 1.25

Grid Refinement
Step Ratio, r GCI(%)
1 2 2.000000 0.103080
2 3 2.000000 0.356244

Checking for asymptotic range using Eqn. 5.10.5.2.

A ratio of 1.0 indicates asymptotic range.

Grid Range Ratio

12 23 0.997980

--- End of VERIFY ---

Examples of Grid Converence Studies in the Archive

A grid convergence study is performed in the Supersonic Wedge case.

Examples of Grid Converence Studies in Literature

Other examples of grid convergence studies that use the procedures outlined above can be
found in the book by Roache and the paper by Steffen et al..

NPARC Alliance Policy with Respect to Grid Converence Studies

For the WIND verification and validation effort, it is suggested that the above procedures be
used when conducting and reporting results from a grid convergence study.

Web site curator, and responsible official for content:

John W. Slater, john.w.slater@grc.nasa.gov

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/spatconv.html 2004-08-23
Examining Temporal Convergence Page 1 of 1

V&V Home Archive Tutorial

Examining Temporal Convergence

Time-accurate simulations involve taking discrete time steps. One must examine the
sensitivity of the simulation results to the magnitude of the time step. The effects and
possible errors are usually related to the time filtering of various time scales existing in the
unsteady flow field.

Web site curator, and responsible official for content:

John W. Slater, john.w.slater@grc.nasa.gov

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/tempconv.html 2004-08-23
Glossary of Verification and Validation Terms Page 1 of 3

V&V Home Archive Tutorial

Glossary of Verification and Validation Terms

The following is a glossary for verification and validation. The objective is to clarify
meanings and indicate how some of the terms have specific definitions that may be different
than common usage of those terms. For example, verification and validation share the same
meaning in most dictionaries and even in common usage among technical personnel;
however, here we are concerned with attaching to them specific technical definitions that are
associated with different aspects of CFD. Most of the definitions come directly from AIAA
G-077-1998, "Guide for the Verification and Validation of Computational Fluid Dynamics
Simulations". Other definitions and discussions come from other literature cited in the
bibliography. The terms are listed alphabetically.

Calibration. Calibration in the context of CFD simulations is defined as

The process of adjusting numerical or physical modeling parameters in the

computational model for the purpose of improving agreement with experimental
data. (AIAA G-077-1998)

Calibration has more to do with improving agreement of computational results with

experimental results rather than assessing error and uncertainty. Calibration arises from
uncertainty in modeling complex physical processes. It adjusts unmeasured or poorly
characterized experimental parameters. One use of calibration may be the adjustment of
emperical constants found within a turbulence model. While calibration improves agreement
for a class of problems, it may make the code less general.

Certification. Certification of a code encompases verification and validation, but also

includes such things as documentation, quality assurance, and version control. Certification
is concerned with managerial and programmatic concerns on the use of the CFD code for
design activities. Certification is intended to put a "seal" of credibility on a production code
for use in a design project. Certification can mean that a certain standard has been satisfied
(i.e. ISO, NASA). NASA Glenn has a center level procedure LeR-P2.10.2 entitled "Software
Product Assurance" which offers some high level guidance on verification and validation of
software.

Code. A code is a set of computer instructions and data inputs and definitions. This is the
CFD code (WIND, NPARC, OVERFLOW, USM3D, ...) which may include ancillary codes
and documentation as part of the software package. A code typically has three stages related
to the level of validation completed: research, pilot, and production. The production code has
been fully validated for the intended design applications, including a system-level validation
(full inlet, inlet-aircraft integration).

Credibility. An improvement of credibility is considered to be the same as confidence

building or providing quality to the customer.

Error. Error is defined as

A recoqnizable deficiency in any phase or activity of modeling and simulation

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/glossary.html 2004-08-23
Glossary of Verification and Validation Terms Page 2 of 3

that is not due to lack of knowledge. (AIAA G-077-1998)

Further discussion of error can be found at the page entitled Uncertainty and Error in CFD
Simulations.

Grid Convergence. Grid convergence indicates that as the grid spacing is reduced, the
computed simulation results approach the continum result. Here "grid spacing" can refer to
both spatial spacing, as well as, time step for the case of unsteady, time-accurate simulations.
Further discussion can be found on the page entitled Examining Spatial (Grid) Convergence.

Iterative Convergence. Iterative convergence indicates that as the discrete equations are
iterated, the computed simulation results approach a fixed value. Further discussion can be
found on the page entitled Examining Iterative Convergence.

Model. A model is defined as

A representation of a physical system or process intended to enhance our ability

to understand, predict, or control its behavior. (AIAA G-077-1998)

A conceptual model for CFD consists of the observations, mathematical modeling data, and
mathematical (partial differential) equations that describe the physical system. It also
includes initial and boundary conditions.

The computational model is the computer program or code that implements the conceptual
model. This may be finite-difference, finite-volume, finite-element, or other type of
discretization. It includes the algorithms and iterative strategies. Parameters for the
computational model include the number of grid points, algorithm inputs, and similar
parameters.

Modeling. Modeling is defined as

The process of construction or modification of a model. (AIAA G-077-1998)

Prediction. Prediction is defined as

Use of a CFD model to foretell the state of a physical system under conditions
for which the CFD model has not been validated. (AIAA G-077-1998)

Prediction is going beyond the validation database and performing simulations of untested
systems. This of course, is a tremendous power of CFD and of use in design studies.
However, we would like to be able to estimate the accuracy of the predictions. Unfortunately,
the verification and validation processes can not formally provide these estimates. The best
one can do is point to the verification results and the most similar validation case.

Robustness. Robustness defines the ability of the numerical method to provide a solution
despite variabilities in the initial solution and control parameters. This incorporates issues of
fault tolerance. Generally, robustness is achieved at the expense of accuracy.

Simulation. A simulation is defined as

The exercise or use of a model. (That is, a model is used in a simulation). (AIAA
G-077-1998)

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/glossary.html 2004-08-23
Glossary of Verification and Validation Terms Page 3 of 3

For a CFD analysis the application or run of the CFD code is a simulation.

Uncertainty. Uncertainty is defined as

A potential deficiency in any phase or activity of the modeling process that is

due to the lack of knowledge." (AIAA G-077-1998)

Further discussion of uncertainty can be found at the page entitled Uncertainty and Error in
CFD Simulations.

Validation. Validation is defined as

The process of determining the degree to which a model is an accurate

representation of the real world from the perspective of the intended uses of the
model. (AIAA G-077-1998)

Further discussion of validation can be found at the page entitled Validation Assessment.

Verification. Verification is defined as

The process of determining that a model implementation accurately represents

the developer's conceptual description of the model and the solution to the
model. (AIAA G-077-1998)

Further discussion of verification can be found at the page entitled Verification Assessment.

Web site curator, and responsible official for content:

John W. Slater, john.w.slater@grc.nasa.gov

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/glossary.html 2004-08-23
Bibliography of CFD Verification and Validation Page 1 of 5

V&V Home Archive Tutorial

Bibliography of CFD Verification and Validation

The following is a list of references related to CFD Verification and Validation of CFD
codes. The references are grouped according to whether they are:

z Policy Statements
z Books
z Compilations and Proceedings
z Individual Papers

If anyone has any additional references they would like to have listed, please contact the
Webmaster.

Policy Statements

Roache, P.J., K. Ghia, and F. White, "Editorial Policy Statement on the Control of Numerical
Accuracy," ASME Journal of Fluids Engineering, Vol. 108,No. 1., March 1986, p. 2.

AIAA, "Editorial Policy Statement on Numerical Accuracy and Experimental Uncertainty,"

AIAA Journal, Vol. 32, January 1994, p. 3.

AIAA, "Guide for the Verification and Validation of Computational Fluid Dynamics
Simulations," AIAA G-077-1998, 1998.

ASME Editorial Board, "Journal of Heat Transfer Editorial Policy Statement on Numerical
Accuracy," ASME Journal of Heat Transfer, Vol. 116, November 1994. pp. 797-798.

Books

Anderson, D.A., Tannehill, J.C., and Pletcher, R.H. , Computational Fluid Mechanics and
Heat Transfer, McGraw-Hill Book Company, New York, 1984.

Briggs, W.L., A Multigrid Tutorial, SIAM, Philidelphia, PA, 1987.

ERCOFTAC, Best Practices Guidelines for Industrial Computational Fluid Dynamics,

Version 1.0, January 2000.

Hirsch, C. Numerical Computation of Internal and External Flows, Volume I: Fundamentals

of Numerical Discretization. New York: John Wiley & Sons, 1988.

Hirsch, C. Numerical Computation of Internal and External Flows, Volume II:

Computational Methods for Inviscid and Viscous Flows. New York: John Wiley & Sons,
1990.

Roache, P.J., Verification and Validation in Computational Science and Engineering,

Hermosa Publishers, Albuquerque, New Mexico, 1998.

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/bibliog.html 2004-08-23
Bibliography of CFD Verification and Validation Page 2 of 5

Roache, P.J., Fundamentals of Computational Fluid Dynamics, Hermosa Publishers,

Albuquerque, New Mexico, 1998.

Shyy, W., Computational Modeling for Fluid Flow and Interfacial Transport, New York:
Elsevier, 1994.

Compilations and Proceedings

AGARD, Validation of Computational Fluid Dynamics, Lisbon, Portugal, May 2-5, 1988,
NATO Advisory Group for Aeronautical Research and Development, AGARD CP 437,
December 1988.

AIAA, AIAA Journal of Spacecraft and Rockets, Vol. 27, No. 2, March-April 1990, pp. 97-
215. This issue contains 5 papers on CFD Code Validation / Verification / Certification with
the emphasis on hypersonic flight. Another section entitled CFD Code Applications has 8
papers on applying CFD codes for hypersonic flight and includes some discussion on
validation. Some of these 13 papers are listed individually below.

AIAA, AIAA Journal, Vol. 36, No. 5, May 1998, pp. 665-764. This issue contains 12 papers
in a special section entitled Credible Computational Fluid Dynamics Simulations. Some of
these 12 papers are listed individually below.

AGARD, Experimental Data Base for Computer Program Assessment, Report of the Fluid
Dynamics Panel Working Group 04, AGARD-AR-138, May 1979. This is the report that
contains the papers and data for the RAE2822 airfoil and ONERA M6 wing.

Individual Papers

Aeschliman, D.P., W.L. Oberkampf, and F.G. Blottner, "A Proposed Methodology for
Computational Fluid Dynamics Code Verification, Calibration, and Validation," Paper
presented at the 16th International Congress on Instrumentation in Aerospace Simulation
Facilities (ICIASF), July 18-21, 1995, Wright-Patterson AFB, OH 45433.

Aeschliman, D.P. and W.L. Oberkampf, "Experimental Methodology for Computational

Fluid Dynamics Code Validation," AIAA Journal, Vol. 36, No. 5, pp. 733-741.

Barber, T.J., "Role of Code Validation and Certification in the Design Environment," AIAA
Journal, Vol. 36, No. 5, pp. 752-758.

Bardina, J.E., P.G. Huang, and T.J. Coakley, "Turbulence Modeling Validation, Testing, and
Development," NASA TM 110446, April 1997.

Benek, J.A., E.M. Kraft, and R.F. Lauer, "Validation Issues for Engine - Airframe
Integration," AIAA Journal, Vol. 36, No. 5, pp. 759-764.

Blottner, F.G., "Accurate Navier-Stokes Results for the Hypersonic Flow over a Spherical
Nosetip," AIAA Journal of Spacecraft and Rockets, Vol. 27, No. 2, pp. 113-122.

Bobbitt, P.J., "The Pros and Cons of Code Validation," AIAA Paper 88-2535 (NASA TM
100657), July 1988.

Coleman, H.W. and F. Stern, "Uncertainties and CFD Validation," ASME Journal of Fluids

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/bibliog.html 2004-08-23
Bibliography of CFD Verification and Validation Page 3 of 5

Engineering, Vol. 119, December 1997, pp. 795-803.

Cosner, R.R., "Issues in Aerospace Application of CFD Analysis," AIAA Paper 94-0464,
January 1994.

Cosner, R.R., "CFD Validation Requirements for Technology Transition," AIAA Paper 95-
2227, June 1995.

Dolling, D.S, "High-Speed Turbulent Separated Flows: Consistency of Mathematical Models

and Flow Physics," AIAA Journal, Vol. 36, No. 5, pp. 725-732.

Dudek, J.C., N.J. Georgiadis, and D.A. Yoder, "Calculation of Turbulent Subsonic Diffuser
Flows Using the NPARC Navier-Stokes Code," AIAA Paper 96-0497, January 1996.

Dudek, J.C., "NPARC Validation - Subsonic Turbulent Diffusing Pipe Flow," The NPARC
Alliance, April 1996.

Dudek, J.C., "Testing Guidelines for NPARC Alliance Software Development," The NPARC
Alliance, April 1997.

Dudek, J.C., D.O. Davis, and J.W. Slater, "Validation and Verificaiton of the WIND Code for
Supersonic Diffuser Flows," AIAA Paper 2001-0224, January 2001.

Gnoffo, P.A., "CFD Validation Studies for Hypersonic Flow Prediction," AIAA Paper 2001-
1025, January 2001.

Habashi, W.G., J. Dompierre, Y. Bourgault, M. Fortin, and M.-G. Vallet, "Certifiable

Computational Fluid Dynamics Through Mesh Optimization," AIAA Journal, Vol. 36, No. 5,
pp. 703-711.

Jameson, A. and L. Martinelli, "Mesh Refinement and Modeling Errors in Flow

Simulations," AIAA Journal, Vol. 36, No. 5, May 1998, pp. 676-686.

Lewis, C.H., "Comments on the Need for CFD Code Validation," AIAA Journal of
Spacecraft and Rockets, Vol. 27, No. 2, pp. 97.

Marvin, J.G., "Perspective on Computational Fluid Dynamics Validation," AIAA Journal,

Vol. 33, No. 10, October 1995, pp. 1778-1787.

Mehta, U.B., "Computational Requirements for Hypersonic Flight Performance Estimates,"

AIAA Journal of Spacecraft and Rockets, Vol. 27, No. 2, pp. 103-112.

Mehta, U.B., "Some Aspects of Uncertainty in Computational Fluid Dynamics Results,"

Transactions of the ASME, Vol. 113, December 1991, pp. 538-543.

Mehta, U.B., "Guide to Credible Computer Simulations of Fluid Flows," AIAA Journal of
Propulsion and Power, Vol. 12, No. 5, September-October 1996, pp. 940-948. (Also AIAA
Paper 95-2225).

Mehta, U.B., "Credible Computational Fluid Dynamics Simulations," AIAA Journal, Vol.
36, No. 5, May 1998, pp. 665-667.

NPARC Alliance, "NPARC Alliance Policies and Plans", August, 1999.

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/bibliog.html 2004-08-23
Bibliography of CFD Verification and Validation Page 4 of 5

Oberkampf, W.L., "A Proposed Framework for Computational Fluid Dynamics Code
Calibration / Validation," AIAA Paper 94-2540, June 1994.

Oberkampf, W.L. and F.G. Blottner, "Issues in Computational Fluid Dynamics Code
Verification and Validation," AIAA Journal, Vol. 36, No. 5, May 1998, pp. 687-695.

Oberkampf, W.L. and T.G. Trucano, "Validation Methodology in Computational Fluid

Dynamics" AIAA Paper 2000-2549, June 2000.

Paynter, G.C. and E. Tjonneland, "Accuracy Issues in the Prediction of Supersonic Inlet
Flows," ASME 92-GT-400.

Reed, H.L., T.S. Haynes, and W.S. Saric, "Computational Fluid Dynamics Validation Issues
in Transition Modeling," AIAA Journal, Vol. 36, No. 5, pp. 742-751.

Rizzi, A. and J. Vos, "Toward Establishing Credibility in Computational Fluid Dynamic

Simulations," AIAA Journal, Vol. 36, No. 5, May 1998, pp. 668-675.

Roache, P.J., "Need for Control of Numerical Accuracy," AIAA Journal of Spacecraft and
Rockets, Vol. 27, No. 2, pp. 98-102.

Roache, P.J., "Perspective: A Method for Uniform Reporting of Grid Refinement Studies",
ASME Journal of Fluids Engineering, Vol. 116, September 1994.

Roache, P.J., "Quantification of Uncertainty in Computational Fluid Dynamics," Annual

Review of Fluid Mechanics, Vol. 29, 1997, pp. 123-160.

Roache, P.J., "Verification of Codes and Calculations," AIAA Journal, Vol. 36, No. 5, May
1998, pp. 696-702.

Slater, J.W., J.C. Dudek, and K.E. Tatum, "The NPARC Verification and Validation
Archive", ASME Paper 2000-FED-11233, June 2000.

Slater, J.W., "Verification Assessment of Flow Boundary Conditions for CFD Analysis of
Supersonic Inlet Flows", AIAA Paper 2001-3882, July 2001.

Slater, J.W., J.M. Abbott, and R.H. Cavicchi, "Validation of WIND for a Series of Inlet
Flows", AIAA Paper 2002-0669, January 2002.

Steffen, C.J. Jr., Reddy, D.R., and K.B.M.Q. Zaman, "Analysis of Flowfield from a
Rectangular Nozzle with Delta Tabs", AIAA 95-2146, June, 1995.

Tatum, K.E. and J.W. Slater, "The Validation Archive of the NPARC Alliance", AIAA Paper
99-0747, January, 1999.

Towne, C.E. and R.R. Jones, "Results and Current Status of the NPARC Alliance Validation
Effort", AIAA Paper 96-0387, January, 1996.

van Wie, D.M. and T. Rice, "Quantification of Data Uncertainties and Validation of CFD
Results in the Development of Hypersonic Airbreathing Engines," AIAA Paper 96-2028, June
1996.

Yee, H.C. and P.K. Sweby, "Aspects of Numerical Uncertainties in Time Marching to Steady-

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/bibliog.html 2004-08-23
Bibliography of CFD Verification and Validation Page 5 of 5

State Numerical Solutions," AIAA Journal, Vol. 36, No. 5, pp. 712-724.

Web site curator, and responsible official for content:

John W. Slater, john.w.slater@grc.nasa.gov

http://www.grc.nasa.gov/WWW/wind/valid/tutorial/bibliog.html 2004-08-23