SAND2016-XXXX

SAND2016-5737C

VVUQ Best Practices in Computational

Science/Engineering Problems with some
thoughts about extensions/limits to
Complex Systems Models
Laura Swiler
Sandia Laboratories
Complex Systems Working Group, June 22-23, 2016
Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed
Martin Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000.
Acknowledgements
With thanks to:
 Tim Trucano, Marty Pilch, Bill Oberkampf
 Brian Adams, Mike Eldred, Jim Stewart
 Laura McNamara
 Patrick Finley, Asmeret Bier
 Jim Kamm, Greg Weirs, Bill Rider
 …and many others

2
Some thoughts on quality of models
and assessments
 Do we have the courage to make sure that Sandia’s modeling
capabilities are not oversold?
 Expert judgment is not sufficient. Experts can be wrong and
have their own biases.
 Just because someone pays you to do something doesn’t
mean it is valid.
 My job today is to provide some background on validation as
it is performed in DOE’s Advanced Simulation and Computing
program and outline some problems with using this
framework for complex systems models.
 I will emphasize reliance on data and benchmarks, performing
uncertainty analyses and examining worst-case scenarios, and
addressing the risk of using a model for a particular situation. 3
Do you know how good (or bad) your
modeling and simulation is?
 “Perform due diligence and communicate frankly about
assumptions, approximations, and limitations affecting
simulation credibility.

Nothing against
ostriches, of course

4
From the world of physics and
engineering….

5
Prediction is hard:
 Limited physical data (observational or experimental)
 Limited simulations (high computational demands…)
 Imperfect computational models (missing physics, etc.)
 Under-resolved approximations or numerics
 Unknown model parameters and boundary conditions
 Imperfect humans
 We want to extrapolate to conditions beyond validation
regime…
Model Framework

7
Verification: Are equations solved
correctly?
 Verification: a math and computer science issue
 Software quality is the bare minimum
 Software verification tools/concepts:
 Requirements and software change and revision control
 Unit and regression tests covering intended use and features, core and edge cases, not
just line coverage (gcov/lcov)
 Static and dynamic analysis (memory, performance, red flags)
 Software quality engineering processes, design and code reviews
 Build in from the start!
• Numerical analysis tells us what properties an algorithm possesses —
symmetry, stability, conservation, convergence, etc. — and under
what conditions.
• In code verification, we test whether the implementation of the algorithm
exhibits these properties with respect to known, exact solutions.

8
Best Practices: Verification Throughout
System Design Physics

Geometric Modeling Model Equations

Meshing Discretization

Partitioning and Mapping

Optimization Time integration

and UQ Adapt
Nonlinear solve

Linear solve

Information Analysis & Validation

Improved design and understanding

What is validation?
• Validation: Are we using the right
model/equations for the intended application? A
physics/engineering/science question
• Quantitative comparison between experimental Viewgraph
data and numerical simulations. Norm

• The purpose of validation is to determine the

adequacy of a model for a particular application.
• Accounts for uncertainties and errors in both
experimental data and simulations.

“If the test data are shown in blue and the simulation data are
shown in , then all I want to see is green.”
(heard at Los Alamos, October 2005)
10
Is this model (black line) valid?

h Exp. Data
Best calculation
Validation with experimental error,
numerical and parametric uncertainty
Fine
Medium
Coarse
Extrapolated
Exp. Data

h
The Validation Hierarchy
A key construct
Example Validation Projects:
Model to Use Case
 Validation of the radiative transfer equation (RTE) used in
Sierra/Thermal-Fluid Dynamics calculations of pool-fire soot
radiation.
 Assess the validity of key model assumptions, which include: 1) gray
soot radiation and 2) the size of the soot particles is much less than
the wavelength of the radiation so that scattering is negligible.
 Validation experiments performed with lightly-sooting liquid
hydrocarbon fuels that yielded fully turbulent fires 2 m diameter.
Radiation intensities were measured, with supplemental
measurements of air flow and temperature, fuel temperature and
burn rate, and flame surface emissive power, wall heat, and flame
height and width provide a complete set of boundary condition data.

 “Fire Intensity Data for Validation of the Radiative Transfer Equation.”

Thomas Blanchat and Dann Jernigan, SAND2016-0224.
14
Example Validation Projects
 Direct electron-beam-injection experiments for validation of air-chemistry
models. SAND2016-2437C
 Development and Amputee Validation of Pressure and Shear Sensing Liner
for Prosthetic Sockets. SAND2016-1580C
 Experimental data uncertainty calibration and validation of a viscoelastic
potential energy clock model for inorganic sealing glasses. SAND2016-0856A
 Uncertainty Quantification Verification and Validation of a Thermal
Simulation Tool for Molten Salt Batteries. SAND2016-3531C
 Development of a Benchmark Series of Cask Drops for Validation of Explicit
Dynamic Finite Element Analyses. SAND2016-0757A
 Implementation and Validation of an Analytic Elastic-Plastic Contact Model
with Strain Hardening in LAMMPS. SAND2016-2745C
 Mechanical Joints: Validation Assessment for Joint Problem Using an Energy
Dissipation Model SAND2014-0864C, Validation of joint models in Salinas.
SAND2004-3654C
15
 Validation Metrics
 ASME V&V 20-2009 Standard for Verification and Validation in Computational
Fluid Dynamics and Heat Transfer:
 https://www.asme.org/products/codes-standards/v-v-20-2009-standard-
verification-validation

Term Definition Formula

T Unknown truth data
S predicted value from a simulation S=T + S
D value determined from experimental data D=T + D
E Validation comparison error E= S-D = S - D
 Error: Difference between measurement and S = model + num + input
true value
u Uncertainty: Characterization of the dispersion 𝑢 = 𝑢2 + 𝑢2 2
𝑣𝑎𝑙 𝑛𝑢𝑚 𝑖𝑛𝑝𝑢𝑡 + 𝑢𝐷
of values

model Model error due to modeling assumptions and model = E – (num + input -
approximations D).
16
Validation Metrics:
We have an interval (E uval) in which model falls:
model  [E- uval , E+ uval].

Case Validation interpretation

|E| >> uval It is likely that model ≈ E. In this case, one would want to
reduce the modeling error.
|E| ≤ uval The modeling error is within the noise level imposed by
numerical, input, and experimental uncertainties and it
will be harder to formulate model improvements.
Assume uval One can say that 95% of the population is covered within
is Gaussian the interval (E 2*uval)

17
Performing validation
 Are we ready to estimate model  [E- uval , E+ uval] ?
 What do we need? An estimate of numerical, parameter, and
experimental uncertainties
2 2
𝑢𝑣𝑎𝑙 = 𝑢𝑛𝑢𝑚 + 𝑢𝑖𝑛𝑝𝑢𝑡 + 𝑢𝐷2

 Parameter uncertainty can be estimated in two ways:

2
2 𝑚 𝑑𝑆 2 1 𝑁
𝑢𝑖𝑛𝑝𝑢𝑡 = σ𝑝=1 𝑢 OR 𝑢𝑖𝑛𝑝𝑢𝑡 = σ𝑖=1(𝑆𝑖 -S)2
𝑑𝑥𝑝 𝑥𝑝 𝑛

 Multiple extensions to multiple responses (e.g. time-varying),

various statistical methods for pooling experimental data across
different conditions, thresholds for model adequacy, etc.
18
 Other Validation Metrics
“Toward a better understanding of model validation metrics.” Y. Liu, W. Chen, P. Arendt, and H.
Huang. Journal of Mechanical Design, July 2011, Vol. 133. DOI: 10.1115/1.4004223

19
 Supporting capabilities
 Sensitivity Analysis
 Identify most important variables and their interactions
 Understand code output variations as input factors vary
 Often correlation coefficients, scatterplots, or variance-based indices
 Uncertainty Quantification
 Determine the probability distribution of code outputs, given
uncertainty in input factors
 Assess the likelihood of typical or extreme outputs given input
uncertainties: determine mean or median performance, assess
variability in model responses, find probability of failure
 Assess how close code predictions are to experimental data (validation)
or performance limits (margins)
 Calibration
 determine optimal parameter values that yield simulation results which
“best match” the experimental data in some sense
 Least-squares methods, Bayesian calibration methods 20
More on UQ/SA
 Advances in UQ methods have been tremendous BUT still
severely limited with respect to dimensionality
 Most of our UQ/SA methods deal with continuous variables,
not large numbers of discrete entities
 Sampling approaches are probably the only mainstream
viable approach currently
 Surrogate models have also advanced tremendously the past
15 years, but again, limited to a handful of variables and
responses, mainly continuous

21
 Challenge:
general coupled multi-physics
 Can we efficiently propagate UQ across scales/disciplines?
 Naively wrapping multi-physics with UQ often too costly
 Can we invert loops and perform multi-physics analysis on UQ-
enriched simulations (couple individual sampling loops based on
scalar statistics, random fields, stochastic processes)?
 Embedded methods: examining GPUs, re-structuring of sampling
loops (Eric Phipps, Sandia)
 Instead of N separate samples where each requires a full solve of
residuals/Jacobians, restructure the operations
 Form a large block nonlinear problem
 Krylov Basis Recycling
 Increased vectorization and data locality
 Implemented through Stokhos embedded UQ package

22
Lessons from the ASC Program
 DATA: Can I design and execute a validation experiment?
What benchmarks are available? Difference between data
used for calibrating the model vs. validation.
 Early emphasis on identifying and quantifying epistemic vs.
aleatory uncertainty
 More recently, issues of model form uncertainty and model
selection are of interest
 Bayesian methods are becoming more popular
 Limits of the validation hierarchy
 IT IS VERY EXPENSIVE!!!!
 Our validation statements are fairly limited
 Validation is a process
23
Frontiers of V&V/UQ
 Complexity of codes, multi-physics couplings
 Few examples of validation across the hierarchy
 UQ: Focus is on methods that minimize the number of function
evaluations while maximizing accuracy in response statistics
 Calibration: People are tacking more complicated scenarios,
want parameters estimated with uncertainties from a variety of
experimental configurations
 People are doing experiment to simulation comparisons, but
characterizing, propagating, and comparing uncertainties
remains a big issue
 DATA, DATA, DATA
 How to deal with models that are “theory rich and data poor”
vs. models that are “data rich and theory poor”?
24
Recommendations
 Focus on Data: what data was used to build the model, what
data was used to calibrate the model, what data was used to
evaluate the model, what data was used for validation? What
benchmark data sets are available?
 Use of SA/UQ/Optimization in terms of understanding the
behavior of the model, identifying worst case scenarios,
understanding the spread of possible outcomes and their
likelihood.
 Perform risk management on the use of the model: what are
the risks associated with using this model?

25
Thoughts to consider
 Hans Meir’s thoughts on validation of complex systems
models:
 Naïve practitioners in the softer sciences are pretending to “validate”
their tools based on harder-science verification practices.
 We should be enforcing a course-correction that steers the managers
and naïve practitioners away from that pseudo-science.
 Instead, we should recognize the deep uncertainty inherent in softer
science challenges, stop pretending to gloss over that uncertainty with
sophisticated modeling and validation tricks, and prevent the
inappropriate use of models for predictive/forecasting purposes.

26
Thoughts to consider
 Models don’t forecast….People Do. (Laura McNamara, SAND,
NAS)

 The challenge of evaluating computational social modeling and simulation

technologies extends far beyond verification and validation, and should
include the relationship between a simulation technology and the people
and organizations using it.
 This challenge of evaluation is not just one of usability and usefulness for
technologies, but extends to the assessment of how new modeling and
simulation technologies shape human and organizational judgment.
 The robust and systematic evaluation of organizational decision making
processes, and the role of computational modeling and simulation
technologies therein, is a critical problem for the organizations who
promote, fund, develop, and seek to use computational social science
tools, methods, and techniques in high-consequence decision making.
27
Thoughts to consider
 Sargent: three basic decision-making approaches for deciding
whether a simulation model is valid
 The model development team decides based on their testing
 The user of the model decides, based on interaction with the model
development team, ideally throughout all stages of development
 An external third party decides, based on their own testing
 Conceptual model validation is defined as determining that the theories and
assumptions underlying the conceptual model are
 Computerized model verification is defined as assuring that the computer
programming and implementation of the conceptual model are correct.
 Operational validation is defined as determining that the model’s output behavior
has a satisfactory range of accuracy for the model’s intended purpose over the
domain of the model’s intended applicability.
 Data validity is defined as ensuring that the data necessary for model building,
model evaluation and testing, and conducting the model experiments to solve the
problem are adequate and correct.
28
Thoughts to consider
 “When all models are wrong.” Saltelli and Funtowicz, IST 2014
 Checklist for the responsible development and use of models.
 Rule 1: Use models to clarify, not obscure
 Rule 2: Adopt an “assumption hunting” attitude
 Rule 3: Detect pseudo-science (the practice of ignoring or hiding the
uncertainties in model inputs in order to ensure that model outputs can be
linked to preferred policy choices. A common indicator of this kind of
pseudoscience is spurious precision.)
 Rule 4: Find sensitivity assumptions before they find you
 Rule 5: Aim for transparency
 Rule 6: Don’t just “do the sums right” but “do the right sums.”
 Rule 7: Focus the analysis.
SENSITIVITY AUDIT

29