Computational Experiments: Past, Present and Future
Computational Experiments: Past, Present and Future
Computational Experiments: Past, Present and Future
Xiao Xue*, Xiang-Ning Yu, De-Yu Zhou, Xiao Wang, Zhang-Bin Zhou, Fei-Yue Wang
Abstract: Powered by advanced information technology, more researched using the holism method [3]. 2) This repeated test
and more complex systems are exhibiting characteristics of the on the real system is not economically feasible due to the scale
Cyber-Physical-Social Systems (CPSS). Understanding the of social complex systems. 3) Many complex systems that
mechanism of CPSS is essential to our ability to control their involve social governance are constrained by legislation and
actions, reap their benefits and minimize their harms. In
often cannot be tested or reshaped, such as national security,
consideration of the cost, legal and institutional constraints on
the study of CPSS in real world, computational experiments have
military preparedness, and emergency response, and so on; 4)
emerged as a new method for quantitative analysis of CPSS. This Social complex systems are highly human-related system,
paper outlines computational experiments from several key which invlove human in many scenarios. And, the testing of
aspects, including origin, characteristics, methodological these systems may cause irreversible risks and losses, which
framework, key technologies, and some typical applications. are morally unacceptable [4].
From unstructured reality scenarios to
Finally, this paper highlights some challenges of computational structured computing scenarios
experiments to provide a roadmap for its rapid development and Social Complex
widespread application. System
In 1928, L.V.Bertalanffy first proposed the 1980-2000. Representative work includes self- In 1999, "Science" published a special issue
concept of "complexity". organization theory (chaos theory, fractal with the theme of "complex systems",
theory, critical theory ) and complex adaptive focusing on complex research in the fields of
1950-1960, representative work includes chemistry, biology, neurology, zoology,
system theory.
system theory, cybernetics, and information physical geography, climatology, and
theory. J.H.Holland proposed the concept of a complex
adaptive system. Based on this, a computer was economics.
From 1970 to 1980, representative work 2000-present. The combination of complexity
used to establish a macroscopic model of the
includes dissipative structure theory, theory and data science theory has become a
entire system, namely the Ehco model.
catastrophe theory, and synergetics. hot spot, and representative work includes
complex networks and CPSS.
Conversion
Design
Feedback selection
Design
principles Numerical
Social model
Structural generation
verification
Environment Feedback Data
Model verification Analysis of
4 computational
Abstract
Result 5 experiments
Agent model verification
Verification of Process analysis
experimental
Modeling of Abstract models Scenario What-If
1 analysis analysis
Artificial
Society
Conceptual
Actual complex
model of
Identification of system Optimization of
the system
research questions intervention strategies
intervention strategy affects the variation level of an individual and the Initialize the target
whole system. Initialize simple
Step 5: Next cycle behavior
Agent Environmental
Over time, some elites may fall behind, and some new individuals with Internal behavior test
stronger capabilities will become new elites. Finally, the evolutionary Create an agent state
structure
equilibrium of the whole system breaks and transitions into the next cycle. External Virtual environment
Create the environment state
system Perception
structure module
Decision
module
system
setting Behavior Numerical calculation
Realistic data collection, sorting, analysis, module model
and fusion
system
Physical Environment Population Temporal and spatial Scenes
Agent entity operation
Model setting setting distribution Update
Digital individual
Physical data: environmental parameters, population organization
thread parameters, temporal and spatial distribution, etc. structure Update
Confrontation
Social game
Logical data: Agent module, individual, network
organization, society
System
evolution
Run
termination
Artificial society
Fig. 8 Digital thread of computational experiment
3) Interruption mechanism artificial society (SIL, Society in the Loop). In the process of
The running process of the computational experiment is the simulation, it may affect the diffusion mode and equilibrium
result of the interaction of internal and external factors, state of society. The artificial society will respond to the
including the game between various individuals, and the game intervention and output the final results. Intervention makers
between individuals and the environment.External compare their goals and preferences with experiment results to
environmental factors are objective and uncontrollable, mainly judge the feasibility of the intervention. After repeated trial
including initial conditions and the external environment. and error, iteration, and refinement, a consensus on how to
Internal factors are controllable and adjustable, mainly intervene can be reached.
including the organization form, cooperation strategy, External input Intervention strategy
coordination mechanism between individuals, and so on.
Without the intervention of external factors, the experimental
Artificial society
system will evolve naturally, which can be used to analyze the
role of the initial setting and internal mechanism in the system Individual Organization
behavior
Individual group
evolution. If external intervention is applied to the behavior
Individual
behavior
Individual
experimental system, it can be used to evaluate and optimize behavior
...
Non-
Deterministic Random Parametric
parametric
behaviors (arrow 3); (3) These rules will be manifested as the Fig. 16 Classification of experiment verification
behavior of the agent in the agent-based model, and then the 1) Structural verification
deduction results of hypothesis phenomenon will emerge The conceptual model is an abstraction of the real system
(arrow 4). and can be implemented through programming to obtain a
Scenario 1 Scenario 2 practically executable computational model. Structural
1 verification is to verify the process of model construction so
Macro that the model can reflect the internal structural characteristics
of the real system, and ensure that the intermediate process of
2 4 model behavior is correct. In other words, the hypothetical
model is a sufficient condition to produce the behavior result.
The modeling process of artificial society includes conceptual
Agent
3 model design and computational model implementation.
the orientation to Action Therefore, the structural verification also includes conceptual
action model verification and computational model verification:
Fig. 15 What-if analysis model based on mechanism 1.1) Conceptual model verification
The mechanism-based behavior analysis is an analysis of In this stage, the consistency of conceptual models with the
the “influence-action-emergence” mechanism from macro research purposes, given assumptions, existing theories, and
phenomena to micro behaviors to macro phenomena, rather evidence is examined to determine that the simplified
than a description of the coverage law between macro processing of the model will not seriously affect the credibility
phenomena based on statistics or intelligent algorithms. As a of the model and the understanding of the important
result, it can penetrate every aspect of the evolution of characteristics of the real system. At the same time, the
artificial society and restore the mechanism of intervention. conceptual model should satisfy inherent integrity, consistency,
Therefore, the what-if analysis is the core of the computational and correctness. Whether the model is reasonable can be
experiment. To further quantify the causal relationship judged three types of methods: expert judgment, theoretical
between the two nodes, the Probabilistic Causal Model (PCM) comparison, and empirical data fitting, which are often
[94] can be used to formalize and define relevant operations. qualitative and not strictly proven. By comparing with other
By adopting a probabilistic causality model and do-calculation existing models, researchers can greatly improve the maturity
axiom system, the distribution of another variable can be of conceptual models in the following aspects: (1) Describe
concluded when one variable is intervened. In this way, the the application scope and defects of the model; (2) The
causal relationship between two variables can be quantified. mainstream theory uses the same assumptions, or other
E. Verification of experimental results researchers made similar studies and assumptions; (3) Develop
Experimental verification is used to examine whether the and analyze a set of models with the same core assumptions
model accurately maps the prototype system in its application but different additional assumptions; (4) Establish an equation
domain, to ensure that the model meets the application model with similar characteristics to the behavior of the
purpose. To enhance the confidence of the model, credible original Agent model.
1.2) Computational model verification
In this stage, it is mainly to check whether the algorithm, some emergencies, the problem of data delays is very
coding, operating conditions, etc., are consistent with the prominent. For the computational model, the better the
conceptual model. The conceptual model needs to be timeliness of the data, the better the experimental effect.
transformed into a computational model that can be run by a Relevance: In many cases, there is no data we want to
specific computer system through programming. This process study in the data resources. In other words, there is a lack of
involves a variety of factors and undetectable errors. Because direct data related to the research problems. Hence, the
the Agent model is discrete, and significant differences exist researchers will have to find alternative variables for research.
between various programming languages and computer The correlation between alternative variables and direct
systems, the programming details have an important impact on variables will affect the experimental results.
the model results. At present, the main methods used to verify 3) Result verification
the computational model are: (1) Check the consistency The result verification is mainly used to evaluate whether
between the computational model and the conceptual model; the experimental results are consistent with real system
(2) Run the same code on different computers, different behavior. Real data can be collected from the real world, but
operating systems, and different pseudo-random number the data output from the model is generated by the set model
generators; (3) Use different programming languages to operating mechanism. The data generated by the model is
implement and compare conceptual models; (4) Test the compared with the real data to infer whether it “appropriately”
behavior of the computational model in extreme cases. Code reflects reality, to achieve the purpose of result verification. In
checks are involved in any case. In principle, code checking recent years, the mainstream result verification methods
can detect all programming errors. include the indirect calibration method [98], the
2) Data verification Werker-Brenner method [99], and the historical data method
In computational experiments, the model behavior obtained [100]. However, the artificial society model is only a finite
by model assumptions and model parameters is usually sample of the population after the finite operation. The model
regarded as a prior condition for experiment research. The behavior is unstable and parameter sensitive. Therefore,
building of the model is actually equivalent to proposing a compared with the real system, the result verification of the
hypothesis, and the builder believes that this hypothesis artificial society model will face the following problems:
represents the law of the real system. By performing Q1: How to deal with parameter combination explosion and
computational experiments and observing the model behavior sensitivity analysis?
results, a causal explanation of the real system is made. The A number of model parameters will lead to combination
quality of the input data will directly affect the credibility of explosion. Even if the experimental results are consistent with
the final experimental results. Poor data input can even lead to the real data, it is difficult to determine the decisive factors
great deviations from the actual situation. It is necessary to affecting the experimental data. At the same time, the model
guarantee the data availability from the source to build an results are not stable and are largely influenced by the initial
effective model. Many researchers [96,97] have noted the conditions. At present, researchers basically adopt two
problem of data quality, including consistency, precision, methods to reduce the parameter space. The first method: at
completeness, timeliness, and entity identity. For the beginning of modeling, the parameter space is reduced by
computational experiments, the following factors need to be existing data, facts, corresponding models and theories, expert
considered: experience, and knowledge. This method can make full use of
Credibility: Data collection is actually an artificial process existing knowledge, but it is easy to introduce unverified prior
of collecting data according to propositions. Therefore, it has a information. The second method is to perform sensitivity
tendency at the beginning, and the operation process cannot be analysis on formatted features after the model is established
guaranteed to be completely true. Since there is no absolute and run, to find and further analyze sensitive parameters,
“truth”, data that strictly follows scientific principles and is thereby reducing the parameter space. This idea is very
less subjective is closer to “true” and credible. In general, convincing when using experimental design, statistical method,
government and authority data are more reliable. and Monte Carlo method. But it is difficult to implement them
Data size: In statistics, the sample size is directly related to in many cases. These two methods are often combined to
the accuracy of the inferred estimates. In the case of a given reduce space in two directions.
population, the larger the sample size, the smaller the Q2: How to compare model output and empirical data?
estimation error. On the other hand, the smaller the sample The result of running the model is the probability
size, the greater the estimation error. In estimating the distribution of multiple samples, while real-world data is only
parameters of the computational model, for the same one sample. It is difficult to test a probability distribution with
population (as in a region), larger data size can reduce the a sample. The model output can be considered “consistent”
probability of random errors and the problem of with the real data to some extent. At present, it is mainly
underrepresentation. confirmed subjectively. The results are largely restricted by
Timeliness: The operation of a complex system is a the subjective consciousness of the analysts. This judgment
dynamic process. Data trends may fluctuate sharply as must consider not only the consistency of model output but
scenarios and external interventions change. Therefore, the also the consistency of input and output. That is, the model
timeliness of data is an important factor affecting the needs to be calibrated before the consistency of the results is
experimental results, especially for some projects with strong considered.
timeliness. There is an inevitable problem in that it takes time Q3: How to validate results that have not occurred in
to collect and acquire data. In particular, in the early stages of reality?
Under normal circumstances, the computational model is
verified using the existing empirical data. Sometimes, the
results of computational experiments may not have been
produced in the real world. In this case, we need to make full
use of domain knowledge to find out what causes this
phenomenon and whether it is a trend in the real world.
Additionally, we need to decide whether the model is effective.
The results are largely restricted by the range of knowledge
and problem analysis capabilities of the analysts.
strategy that can control the epidemic situation and reduce conduct objective research on the impact of policies on the
intervention cost. real economy.
2) Virtual Taobao
C. Parallel optimization
Commodity search is the core trade on Taobao, which is
The purpose of scientific research is to provide a causal one of the largest retail platforms. The business goal of
hypothesis that can link existing facts together. If the artificial Taobao is to increase sales by optimizing the strategy of
society model represents this law, the output behavior of the displaying the page view (PV) to the customer. As the
model can be effective in the real world. Parallel optimization, feedback signal from a customer depends on a sequence of
through the establishment of an artificial social model that has PVs, it is reasonable to consider it as a multi-step decision
a homomorphic relationship with the real society, realizes the problem rather than a one-step supervised learning problem.
parallel execution and cyclic feedback of the two, thereby The engine and customers are the environments of each other.
supporting the management and control of the real complex RL solutions are good at learning sequential decisions and
system. maximizing long-term rewards. One major barrier to directly
1) Island economy applying RL in these scenarios is that current RL algorithms
Economic inequality is growing globally and is gaining commonly require a large number of interactions with the
attention because of its negative impact on economic environment, which take high physical costs, such as real
opportunities, health, and social welfare. For governments, tax money, time from days to months, bad user experience, and
policies can be used to improve social outcomes. However, even lives in medical tasks.
because of the coupling between tax and labor, taxes may
reduce productivity. Therefore, how to reduce economic
inequality while ensuring productivity remains a problem to
be solved. The lack of appropriate economic data and
experiment opportunities makes it difficult to study such
economic problems in practice. For this purpose, Salesforce
proposed a new study named “The AI Economist”. Economic
simulation through AI agents can explore tax strategies that Fig. 22 Virtual Taobao Architecture with reinforcement learning
can efficiently balance economic equality and productivity To avoid physical costs, the researchers employed
[111]. A two-level reinforcement learning framework is simulators (i.e, Virtual Taobao) for RL training. Thus, the
formed between Agent and social planner with learning and policy can be trained offline in the simulator [113]. In this
adaption functions. Fig.21 gives the core implementation work, Virtual Taobao is designed by generating customers and
framework of “The AI Economist”, which emphasizes the generating interactions. As shown in Fig.22, the
game evolution mechanisms between Agent and Planner, i.e. GAN-for-Simulating-Distribution (GAN-SD) approach is
how to use reinforcement learning to realize the joint proposed to simulate diverse customers including their request;
optimization of agent behavior and taxation strategy. the Multi-agent Adversarial Imitation Learning (MAIL)
Planner Model
Planner Actions(7x22) approach is proposed to generate interactions. After generating
Set Bracket 1
customers and interactions, Virtual Taobao is built. The
CNN MLP LSTM MLP ...
experiment results disclose that Virtual Taobao successfully
Set Bracket 7
reconstructs properties that closely parallel the real
Public Map
Public State Semi-Private State Private State
environment. Virtual Taobao can be employed to train
platform policy for maximizing revenue. And, compared with
Tax Rates Endowment
Market
Labor Skill
the traditional supervised learning approach, the training
Behavior Endowment Labor Skill strategy in Virtual Taobao achieved more than a 2%
improvement of revenue in the real environment.
Agent Actions(50)
Agent Model
3) Unmanned driving
Move
In unmanned driving, the ability of unmanned vehicles to
CNN MLP LSTM MLP Build
Planner Observations understand complex traffic scenarios and autonomously make
Agent Observations
Trade
driving decisions needs to be repeatedly tested and verified. It
Fig. 22 The core implementation framework of “The AI Economist” is one of the major challenges in the AI field. However, it is
Researchers have compared the operation results of the AI challenging to build real test scenarios (such as blizzard,
Economist and the free market (Non-taxation or redistribution), rainstorm, typhoon, etc.) due to high costs in creation,
United States federal tax plan, and tax strategies generated duplication, and iteration. Therefore, the training and testing
from the Saez framework [112]. The experiment shows that of unmanned driving strategies in the virtual world have
the AI Economist can increase the trade-off between economic become a feasible technical choice. It is not only controllable
equality and productivity by 16% compared with the tax and repeatable but also safe and effective. In order to achieve
framework proposed by Emmanuel Saez. The framework can an autonomous driving test that is infinitely close to the real
be optimized directly for any socio-economic goal, which does world in the virtual world, three levels of reduction are
not use any prior knowledge or modeling assumptions, and required: (1) Geometric reduction: 3D scene simulation and
only learners from observable data. Salesforce developers sensor simulation are required to craft the environment and
hope that the AI Economist can solve the complex problems test vehicle conditions that mirror the real world; (2) Logic
that traditional economic research cannot easily handle, and reduction: the decision making and planning process of test
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 19
vehicles should be simulated in the virtual world; (3) Physical methods of artificial society modeling: 1) Data-based
reduction: the vehicle control and body dynamics results modeling method. The complex system is regarded as a black
should be simulated. At the same time, the simulation platform box, which focuses on the relationship between input and
should satisfy the characteristics of high concurrency and output, and does not model or simulate complex processes
realize the combination of vehicle responses in all scenarios. within the system. In practice, complex systems are often
At present, parallel learning methods are increasingly used in replaced by statistical models based on data and intelligent
virtual scene generation and intelligent testing of unmanned algorithms, including Graph Neural Network (RNN), vector
driving [10,11,87,88]. machine, fuzzy method, etc. 2) Mechanism-based modeling
As shown in Fig.24, Tencent built a virtual city system method. The complex system is understood and analyzed from
parallel to the real physical world, based on a simulation an overall perspective. Following the principle of “simple
platform, a high-precision map platform, and a data cloud consistency”, the structure and function of each part of the
platform. The virtual city contains not only static system are designed and restored. In the process of building a
environmental information but also dynamic information computational model according to the bottom-up principle, the
(such as traffic and passenger flow). It can not only support accuracy and complexity of the underlying microscopic model
the development and safety verification of self-driving, but play an important role in the evolution of the entire complex
also contribute to the construction of intelligent transportation system.
and smart cities. In order to improve the utilization of road However, the complex system is characterized by diverse
data and further enrich test scenarios, a large amount of road elements, frequent changes, coupled relationships, and
data can be used to train the traffic flow Agent. As a result, multi-level/multi-stage operations. Restricted by cognitive
traffic scenes with high reality and strong interaction can be ability, the traditional mechanism modeling is difficult to use
generated for closed-loop simulation, testing efficiency to describe the operation and evolution mechanism of complex
improvement, and collection cost reduction. For example, if systems. Due to the lack of internal structure and mechanism
the tested self-driving car wants to overtake, Agent AI may be information for process units, data-driven modeling relies
used to control NPC (Non-player character) vehicles to make heavily on the quantity and quality of data samples, making it
avoidance or other game actions consistent with the real difficult to analyze and explain the system mechanism in
world. depth. Therefore, how to construct an artificial society model
for CPSS has become one of the promising research problems
in this field. Mechanism analysis is helpful to identify the
essential characteristics of the system and set up an effective
model structure. And, data-driven methods can automatically
acquire information and knowledge hidden in data. By
coupling the rule model describing individual behavior with
the statistical model, a hybrid model combining system
mechanism analysis and big data analysis is established. It not
only reduces the dependence on data but also makes up for the
situation where traditional models are unlikely to simulate
individual behavior rules. In the future, there will be ample
space to build virtual artificial society models with more
Fig. 24 Scene demonstration of Tencent unmanned driving testing system realistic behavior.
It should be noted that due to the large freedom of the
V. PROBLEMS & CHALLENGES Agent model and the lack of a standardized core model, it is
Computational experiments would cultivate human ability easy to over-promote and exaggerate experiment results, thus
of observing and understanding the world and learn to deep lead to a lot of criticism. Therefore, in the process of
thinking. However, compared with its increasingly widespread progressive modeling of complex systems, we should note: 1)
application, computational experiments have not been It is necessary to identify a simple initial model with all details
advanced as a methodology and technology. Therefore, the that can represent the core elements of the reference system. 2)
computational experiment method should solve the following The design of the model is not in a random sequence that must
three questions to transition into the toolbox of mainstream provide incremental help for the final model. 3) Throughout
researchers: 1) How to define the artificial society using big the development process, model validation is necessary, but it
data, i.e. describing intelligence; 2) How to predict the future needs to be moderate. Because the intermediate stage model is
using computational experiments, i.e. predictive intelligence; 3) not yet mature, it may be rejected due to a lack of sufficient
How to realize the feedback intervention to the real world, i.e. empirical support. 4) It is necessary to determine the final
guiding intelligence. model of the reference system, otherwise the improvement of
the model will continue indefinitely.
A. Q1: How to implement describing intelligence?
B. Q2: How to implement predictive intelligence?
By abstracting and simplifying the complex behaviors and
phenomena in real social systems, we can obtain an artificial In the complex system, each link is full of high uncertainty
society model which can run on a computer platform and triggered by individual behavior, environment, and the
satisfy logical rationality and correctness. This is the basis of implementation of the intervention strategy. As a result, the
computational experiments. Currently, there are two main analysis, formulation, and implementation of intervention
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 20
strategies are accompanied by the complexities of uncertainty. In the virtual world, extreme scene can exist and occur,
Big data technology is a summary of facts based on the which is rare in the real world. Based on the optimization
analysis of historical data. It is helpless for uncertain events theory of operations research on uncertain and multi-objective
that never happened. Therefore, how to develop scientific and conditions, the intervention strategies are constantly trained
universally accepted intervention strategies for complex and revised to obtain the expected optimization results. The
systems is a basic problem to be addressed using aim is to intervene in the current development trajectory and
computational experiments. running state from a future perspective, find out the possible
With the help of artificial society, real-world issues can be adverse effects, conflicts, and potential dangers, and provide
accurately described, analyzed and controlled in the virtual “reference”, “prediction” and “guidance” for possible
world, resulting in massive system simulation operating data. scenarios. Reinforcement learning is suitable for solving these
Based on this, we can use computational experiments to problems. Its core task is to learn a strategic function, to
deduce various scenarios that have never occurred in reality, maximize a long-term benefit and to establish a mapping
discover problems and defects in system operation, find out relationship between observed values and output behaviors.
the reasons, and put forward practical solutions or The intervention strategy continues to evolve through machine
suggestions.Various forms of analysis can be adopted for the learning methods in the loop feedback process with the test
computational experiment, including process analysis, scenario.
scenario analysis, and what-if analysis, which correspond to Further, a collaborative dynamic feedback control system
the Pearl theory of causation: association, intervention, and may be constituted between computational experiments and a
counterfactual reasoning. Process analysis focuses on real system. 1) First, by analyzing the dependence of data and
individual interactions in time and space, especially key behavior, real-world problems are abstracted into the cognitive
thresholds and key attributes. Scenario analysis can provide space. 2) Second, a model is established in the cognitive space
comprehensive methods for experimental simulation analysis. for computational experiments. The obtained results can
Also, What-if analysis can compare models using different dynamically guide or control strategy execution in the real
rule sets, such as the difference between Moore neighborhood world. 3) Third, through the game process between the
and standard von Neumann neighborhood. intervention strategy and the real world, the action space with
The public policy simulation represented by “Decision the greatest difference is explored to reduce the modeling error
Theater” is a good example of this perspective [114]. By of the artificial system and the execution error of the
presenting the “if so, then” scenario, counterfactual reasoning intervention strategy. 4) Finally, the real-world execution
can be made on the consequences of different policies. Its results, in turn, continuously update the cognitive space model
specific workflow is as follows: 1) First, collect a large in the form of dynamic data input, forming a cyclic feedback
amount of historical data on the problems to be solved and learning process. The system iterates in this way of parallel
organize them according to different dimensions; 2) Second, execution until convergence. In the continuous integration of
the problem model is established by government growing data, the training and simulation of artificial society
decision-makers and experts, and then the abstract expert will become more real and accurate. Through interactive
knowledge is transformed into an acceptable scene language feedback with the real world, it can obtain self-growth and
using computational experiments; 3) Third, the self-optimization ability.
decision-making scenarios, in reality, are simulated and
visualized to provide multiple policy scenarios and result VI. CONCLUSION
previews for policy makers; 4) Finally, in an agreed decision
In scientific research, induction is the discovery of patterns
environment, various interest groups choose a satisfactory
in empirical data, which is widely used in opinion surveys and
solution to policy issues. The utilization of water resources in
macro-law analysis; deduction is to define a set of axioms and
Phoenix City is a typical application case [115].
prove the conclusions that can be derived from these axioms.
C. Q3: How to implement guiding intelligence? Compared with deduction and induction, the computational
In complex systems, individual activities are increasingly experiment method can be regarded as the third methodology
complex, diverse, and differentiated, while intervention is of scientific research. Similar to deduction methods,
simple, identical, and average centralized control. Considering computational experiments start by defining a clear set of
the dynamics and uncertainties of the external environment, assumptions; but unlike deduction methods, computational
the intervention will lead to adaptive changes of some experiments do not prove theorems. Computational
individuals according to the surrounding environment, which experiments can produce simulation data that can be analyzed
will lead to the failure of the initial optimization method. by inductive methods; but unlike typical induction methods,
Therefore, the design of an intervention strategy is not a task simulation data comes from the emergence of rules operation
that has been accomplished in one stroke. Instead, it needs to rather than direct measurements of the real world. The purpose
be constantly adjusted and optimized according to changes in of induction is to discover patterns from data, and the purpose
the external environment. Computational experiments can of deduction is to discover the results caused by hypotheses.
build the bridge between the virtual world and the real world, Compared with them, computational experiments can verify
eliminate the unfeasible strategies by counterfactual reasoning, our intuition through systematic deduction, trial & error, and
and transfer the optimized strategies to the real space to guide estimation in the virtual world.
the operation of the real system. This paper summarizes the present situation and challenges
of computational experiment methods, including the
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 21
conceptual origin, research framework, application cases, and [8] XUE X. Computational experimental methods of complex systems
-principles, models, and cases. Beijing: Science Press, 2020.
future challenges. The framework of computational
[9] Wang Fei-Yue. Computational experimental methods and complex
experiments mainly consists of the modeling of artificial system behavior analysis and decision evaluation. Journal of System
society, construction of experiment system, experimental Simulation, 2004, 16(5): 893-897.
design, experimental analysis, and experimental verification. [10] Li L, Wang X, Wang K. Parallel testing of vehicle intelligence via
The computational experiments’ application cases can be virtual-real interaction. Science Robotics, 2019, 4(28): eaaw4106.
summarized into three categories: thought experiment, [11] Li L, Huang W L, Liu Y. Intelligence testing for autonomous vehicles:
mechanism exploration, and parallel optimization, involving A new approach. IEEE Transactions on Intelligent Vehicles, 2017,
1(2):158-166.
political, economic, commercial, and epidemiological
[12] Hu X F, Li Z Q, Yang J Y, Si G Y, Pi L, Some key issues of war gaming
dimensions, etc. The technical challenges of computational & simulation, Journal of System Simulation. 2010,22 (3):549–553 .
experiments focus on how to combine with emerging smart [13] Leigh T. Agent-based computational economics. ISU Economics
technologies, including describing intelligence, predictive Working Paper, 2003,249(1):2002-2016.
intelligence, and guiding intelligence. [14] Acevedo M F, Callicott J B, Monticino M. Models of natural and human
Computational experiments provide new perspectives for dynamics in forest landscapes: Giorgio-site and cross-cultural
the analysis and study of complex systems. Although relevant synthesis.Geoforum,2008,39(2):846-866.
research has made great progress, there are still many [15] Carley K M, Fridsma D B, Casman E. BioWar: scalable agent-based
model of bioattacks. IEEE Transactions on Systems, Man, and
problems and challenges in experimental design, experimental Cybernetics - Part A: Systems and Humans, 2006, 36(2):252-265.
analysis, and experimental verification. For example, whether [16] Huang C Y, Sun C T, Hsieh J L, Lin H. Simulating SARS: small-world
the computational experiment results match the known system epidemiological modeling and public health policy assessments. Journal
laws (correctness); whether the representation of the of Artificial Societies and Social Simulation,2004,7(4):2.
computational model is elegant, including rules, syntactic [17] Cioffi R C, Rouleau M. Mason Rebeland: An agent-based model of
politics, environment, and insurgency. International Studies Review,
structure, and similar features (conciseness); whether 2010,12(1):31-52.
computational experiments can help transform the real world [18] Mitchell W. Complexity: the emerging science at the edge of order and
(effectiveness), etc. chaos. New York: Touchstone, 1992.
In essence, computational experiments are designed to solve [19] Bertalanffy L V. General System Theory. New York, USA: Braziller,
one or more research problems defined by a realistic reference 1968.
system. Theoretically, the representation form of [20] Bertalanffy L V. Problems of Life: An Evaluation of Modern Biological
computational experiments is relatively simple, but the real and Scientific Thought. New York: Harper, 1952.
system is more complex. If the abstract degree of the [21] Norbert W. Cybernetics: Or Control and Communication in the Animal
and the Machine.Massachusetts:MIT Press, 1961.
computational system is too high, it will be difficult to reflect
[22] Shannon C E. A mathematical theory of communication. Bell System
the operation law of the real world. If the abstract degree of Technical Journal, 1948, 27(3):379-423.
the computational system is too low, it will be too complex to [23] Prigogine J. Structure, dissipation and life, theoretical physics and
build a model. A series of problems (lack of data, insufficient biology. Theoretical Physics and Biology, 1969:23-52.
resources and time, and lack of knowledge) will be [24] Thom R. Structural Stability and Morphogenesis. Boca Raton: CRC
encountered. Therefore, the development of computational Press, 2018.
experiments need to seek a balance between the flexibility of [25] Haken H P J. Synergetics. IEEE Circuits & Devices Magazine, 1977,
28(9):412-414.
modeling and the credibility of conclusion. In addition,
[26] Li T Y, Yorke J A. Period three implies chaos. The American
computational experiments need to introduce more general Mathematical Monthly, 1975, 82(10): 985-992.
and effective experimental mechanisms and analysis models to [27] Mandelbrot B B. Fractals, form, chance and dimension. The
solve the problems of analysis, design, management, control, Mathematical Gazette, 1978, 62(420): 130-132.
and synthesis of complex systems. [28] Bak P, Tang C, Wiesenfeld K. Self-organized criticality: an explanation
of noise. Physical Review Letters,1987, 59 (4): 381–384.
[29] Holland J H. Adaptation in Natural and Artificial Systems: an
Introductory Analysis with Applications to Biology, Control, and
Artificial Intelligence. New York: MIT Press, 1992.
REFERENCES
[30] Watts D J, Strogatz S H. Collective dynamics of ‘small-world’ networks.
[1] Zhou Y, Yu F R, Chen J, Kuo Y. Cyber-physical-social systems: A Nature, 1998, 393(6684): 440.
state-of-the-Art survey, challenges, and opportunities. IEEE [31] Barabási A L, Albert R. Emergence of scaling in random networks.
Communications Surveys & Tutorials, 2020, 22(1):389-425. Science, 1999, 286(5439): 509-512.
[2] Wang F Y. The emergence of intelligent enterprises: From CPS to CPSS. [32] Wolfram S. Cellular automata as models of complexity.
IEEE Intelligent Systems, 2010, 25(4):85-88. Nature,1984,311(5985):419-424.
[3] Prigogine I, Stengers I. Order Out of Chaos. New York, USA: Bantam [33] Aguilera V, Galán G, Egea G. A probabilistic extension to Conway’s
Books Inc, 1984. Game of Life. Advances in Computational Mathematics, 2019,
[4] Xue X, Wang S F, Gui B, Hou Z W. A computational experiment-based 45(4):2111-2121.
evaluation method for context-aware services in a complicated [34] Langton C G. Artificial life. in 1991 lectures in complex
environment. Information Sciences, 2016,373:269-286. systems. Addison-Wesley Reading, 1992: 189- 241.
[5] Pearl J, Mackenzie D. The Book of Why: The New Science of Cause [35] Wolfram S. Random sequence generation by cellular automata.
and Effect. New York: Basic Books, 2018. Advances in Applied Mathematics, 1986, 7(2):123-169.
[6] Boschert S, Rosen R. Mechatronic futures, Digital Twin—The [36] Teruyam , Nakao Z , Chen Y W. A boid-like example of adaptive
Simulation Aspect. Cham: Springer, 2016. 59-74. complex systems. In: Proceedings of IEEE International Conference on
[7] Wang Fei-Yue. Intelligence 5.0:parallel intelligence in parallel age. Systems. Hawai'i: IEEE, 1999.1-5.
Journal of Information, 2015, 34(6): 563-574.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 22
[37] Dorigo M, Birattari M, Stutzle T. Ant colony optimization. IEEE [63] Marathe M, Vullikanti A K S. Computational epidemiology.
Computational Intelligence Magazine, 2006, 1(4): 28-39. Communications of the ACM, 2013, 56(7): 88-96.
[38] Gilbert N, Troitzsch K G. Simulation for The Social Scientist 2nd [64] Lazer D, Pentland A, Adamic L. Social science. Computational social
Edition. Berkshire: Open University Press, 2005. science. Science, 2009, 323(5915): 721-723.
[39] Swarm main page[Online], available: http://www.swarm. org, April [65] Conte R, Gilbert N, Bonelli G, Cioffi R C, Deffuant G, Kertesz J, Loreto
3,2021. V, Moat S, Nadal J P, Sanchez A, Nowak, A. Manifesto of
[40] Repastsource[Online],available:https://repast.github.io/index.html,April computational social science. The European Physical Journal Special
1,2021. Topics, 2012, 214(1) : 325-346.
[41] Parker M T. What is Ascape and why should you care. Journal of [66] Lazer D M J, Pentland A, Watts D J. Computational social science:
Artificial Societies and Social Simulation, 2001, 4(1): 5. Obstacles and opportunities. Science, 2020, 369(6507): 1060-1062.
[42] Tisue S, Wilensky U. Netlogo: A simple environment for modeling [67] Goodfellow I J, Pouget A J, Mirza M. Generative adversarial networks.
complexity. In: Proceedings of International Conference on Complex ArXiv Preprint ArXiv, 2014: 1406-2661.
Systems. Shanghai, China: 2004, 21: 16-21. [68] Nigel G, Rosaria C. Artificial Societies: The Computer Simulation of
[43] Carl H, Steven C B. Artificial Societies: A Concept for Basic Research Social Life. London, England: University College London Press,1995.
on the Societal Impacts of Information Technology. Santa [69] Tu X, Terzopoulos D. Artificial fished: physics, locomotion, perception,
Monica,CA,1991 behavior. In: Proceedings of ACM Computer Graphics, Annual
[44] Nigel G. Rosaria Conte: Artificial societies, The Computer Simulation Conference Series, Proceedings of Siggraph, 1994: 43-50.
of Social Life. London, England: University College London Press, [70] Börgers T, Sarin R. Learning through reinforcement and replicator
1995. dynamics. Journal of Economic Theory, 1997, 77(1): 1-14.
[45] Epstein, J M, Axtell R. Growing Artificial Societies: Social Science [71] Arthur W B. Designing economic agents that act like human agents: A
From the Bottom Up. Washington, D.C, USA: Brookings Institution behavioral approach to bounded rationality. The American Economic
Press,1996. Review, 1991, 81(2): 353-359.
[46] Arthur W B, Holland J, LeBaron B, Palmer R, Tayler P. Asset pricing [72] Byrne R W, Russon A E. Learning by imitation: A hierarchical approach.
under endogenous expectations in an artificial stock market. The Behavioral and Brain Sciences, 1998, 21(5): 667-684.
Economy as an Evolving Complex System II. 1997:15–44 [73] Fudenberg D, Levine D. Learning in games. European Economic
[47] Basu N,Pryor R, Quint T. ASPEN:A microsimulation model of the Review, 1998, 42(3-5): 631-639.
economy. Computational Economics, 1998,12:223-241. [74] Kaniovski Y M, Young H P. Learning dynamics in games with
[48] Grieves M W. Virtually Intelligent Product Systems: Digital and stochastic perturbations. Games and Economic Behavior, 1995, 11(2):
Physical Twins. Complex Systems Engineering: Theory and Practice, 330-363.
2019. [75] Jordan J S. Bayesian learning in repeated games. Games and Economic
[49] Tao F, Zhang H, Liu A, Nee A Y. Digital twin in industry: Behavior, 1995, 9(1): 8-20.
State-of-the-art. IEEE Transactions on Industrial Informatics, 2018, [76] Mniszewski S M, Del Valle S Y. EpiSimS: large-scale agent-based
15(4):2405-2415. modeling of the spread of disease. Los Alamos National Lab.(LANL),
[50] Farsi M, Daneshkhah A, Hosseinian F A, Jahankhani H. Digital twin USA, 2013.
technologies and smart cities. Springer, 2020. [77] Jain A K, Mao J, Mohiuddin K M. Artificial neural networks: A tutorial.
[51] Wang Fei-Yue. Parallel system approach and management and control Computer, 1996, 29(3): 31-44.
of complex systems. The Journal of Control and Decision, 2004, [78] Whitley D. A genetic algorithm tutorial. Statistics and Computing, 1994,
19(5):485-489. 4(2): 65-85.
[52] Wang F Y. An Investigation on Social Physics: Methods and [79] Alavi M, Henderson J C. An evolutionary strategy for implementing a
Significance. Complex Systems and Complexity Science, 2005, decision support system. Management Science, 1981, 27(11):
2(3):13-22. 1309-1323.
[53] Wang F Y. Toward a paradigm shift in social computing: the ACP [80] Kennedy J, Eberhart R. Particle swarm optimization. In: Proceedings of
approach. IEEE Intelligent Systems, 2007, 22(5):65-67. ICNN'95-International Conference on Neural Networks. Perth, Australia,
[54] Wang F Y. A framework for social signal processing and analysis: from 1995,4:1942-1948.
social sensing networks to computational dialectical analytics. Scientia [81] Karaboga D, Akay B. A comparative study of artificial bee colony
Sinica: Informationis, 2013, 43(12): 1598-1611. algorithm. Mathematics and Computation,2009, 214(1), 108-132.
[55] Wang F Y, Li X, Mao W. An ACP-based approach to intelligence and [82] Dodds P S, Watts D J. A generalized model of social and biological
security informatics. Studies in Computational Intelligence, 2015, contagion. Journal of Theoretical Biology, 2005, 232(4): 587-604.
563(1):69-86.
[83] Chen W, Yuan Y, Zhang L. Scalable influence maximization in social
[56] Mei H. Understanding “software-defined” from an OS perspective: networks under the linear threshold model. In: Proceedings of 2010
technical challenges and research issues. Science China Information IEEE international conference on data mining. IEEE, 2010: 88-97.
Sciences, 2017, 60(12): 126101.
[84] Hoppitt W, Laland K N. Social Learning: an Introduction to
[57] Busoniu L, Babuska R, Schutter B. A comprehensive survey of Mechanisms, Methods, and Models. Washington D.C,USA: Princeton
multiagent reinforcement learning. IEEE Transactions on Systems, Man, University Press, 2013.
and Cybernetics, 2008, 38(2): 156-172.
[85] Bachelor G, Brusa E, Ferretto D, Mitschke A.. Model-based design of
[58] Xue X, Wang S F, Zhang L J, Feng Z Y, Guo Y D. Social learning complex aeronautical systems through digital twin and thread concepts.
evolution (SLE): computational experiment-based modeling framework IEEE Systems Journal, 2020, 14(2):1568-1579.
of social manufacturing. IEEE Transactions on Industrial Informatics,
2019, 15(6): 3343-3355. [86] Wang F Y, Zheng N N, Cao D. Parallel driving in CPSS: A unified
approach for transport automation and vehicle intelligence. IEEE/CAA
[59] Wilensky U. NetLogo heatbugs model. [Online], available: Journal of Automatica Sinica, 2017, 4(4): 577-587.
http://ccl.northwestern.edu/netlogo/models/Heatbugs. November 16,
2020. [87] Li L, Lin Y L, Zheng N N. Artificial intelligence test: a case study of
intelligent vehicles. Artificial Intelligence Review, 2018, 50(3):
[60] Schelling T C. Dynamic models of segregation. Journal of Mathematical 441-465.
Sociology,1971,1(2):143-186.
[88] Montgomery D C. Design and Analysis of Experiments. Hoboken, USA,
[61] LeBaron B. Agent-based computational finance. Handbook of John Wiley & Sons, 2017.
Computational Economics, 2006, 2: 1187-1233.
[89] Marcos Lopez de Prado. Machine Learning for Asset Managers.
[62] Carley K M, Gasser L. Computational organization theory. Multiagent Cambridge, United, Kingdom: Cambridge University Press,2020.
Systems: A Modern Approach to Distributed Artificial Intelligence,
[90] Newell A, Simon H A. Human Problem Solving. New Jersey:
1999: 299-330.
Prentice-Hall, 1972.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 23
[91] Hamilton J. Time Series Analysis. 1st ed. New Jersey, USA: Princeton
University Press,1994. Xiao Xue, born in 1979. Professor at the School of Computer
[92] Kahn H, Wiener A J. The next thirty-three years: a framework for Software, College of Intelligence and Computing, Tianjin
speculation. Daedalus, 1967: 705-732. University. Also an adjunct professor at the School of
[93] Rao A S, Georgeff M P. BDI agents: From theory to practice. In: Computer Science and Technology, Henan Polytechnic
Proceedings of International Conference on Mechanical and Aerospace University. His main research interests include service
Systems, 1995,95:312-319. computing, computational experiment, the Internet of things,
[94] Pearl J. Causal inference in statistics: An overview. Statistics Surveys, etc.
2009, 3: 96-146.
Xiangning Yu, born in 2000. Undergraduate student at the
[95] YANG M, XIONG Z. Model validation-Methodological problems in
College of Intelligence and Computing, Tianjin University.
agent-based modeling, System Engineering –Theory & Practice, 2013,
33(6): 1458-1470. Her current research interests include service computing
and computational experiment.
[96] Gao J, Zhang Y C, Zhou T. Computational socioeconomics, Physics
Reports, 2019, 817:1-104.
[97] Salganik M J. Bit by bit: Social Research in the Digital Age. New Jersey,
USA: Princeton University Press, 2017. Deyu Zhou, born in 1995. PhD student in the School of
[98] Sundberg R. Multivariate calibration-direct and indirect regression
Software, Shandong University, Jinan, China. Her
methodology. Scandinavian Journal of Statistics, 1999, 26(2): 161-207. current research interests include service computing and
[99] Werker C, Brenner T. Empirical Calibration of Simulation Models
computational experiment.
Report. Papers on Economics and Evolution, 2004.
[100] Malerba F, Nelson R, Orsenigo L. 'History-friendly' models of industry
Xiao Wang, Associate Professor in the State Key
evolution: the computer industry. Industrial and Corporate Change, 1999,
8(1): 3-40.
Laboratory for Management and Control of Complex
Systems, Institute of Automation, Chinese Academy of
[101] Markisic S, Neumann M, Lotzmann U. Simulation Of Ethnic Conflicts Sciences, and executive president of Qingdao Academy of
In Former Jugoslavia. In: Proceedings of the International Conference
Intelligent Industries. Her research interests include social
on Energy, Chemical, Materials Science, 2012: 37-43.
network analysis, social transportation, cybermovement
[102] Ravi B, Miodownik D, Nart J. Rescape: An agent-based framework for organizations, and multi-agent modeling.
modeling resources, ethnicity, and conflict. The Journal of Artificial
Societies and Social Simulation, 2008, 11(27).
Zhangbing Zhou, born in 1975. Professor at China University
[103] Axelrod R, Bennett D S. A landscape theory of aggregation. British of Geosciences, Beijing, China, and Adjunct Professor at
Journal of Political Science, 1993: 211-233. TELECOM SudParis, Evry, France. He has authored over 100
[104] Paul E J. Agent-based modeling: what I learned from the artificial stock referred papers. His research interests include process-aware
market. Social Science Computer Review, 2002, 20(2): 174-186. information systems and sensor network middleware.
[105] Yeh C H, Yang C Y. Examining the effectiveness of price limits in an
artificial stock market. Journal of Economic Dynamics and Control,
Fei-Yue Wang, State Specially Appointed Expert and Director
2010,34(10):2089–2108.
of the State Key Laboratory for Management and Control of
[106] Laver M., Sergenti E. Party Competition: An Agent-Based Model. Complex Systems. His current research focuses on methods
Princetion, USA: Princeton University Press,2012. and applications for parallel systems, social computing, parallel
[107] Epstein J M. Modelling to contain pandemics. Nature, 2009, 460(7256): intelligence, and knowledge automation.
687-687.
[108] Schneider K A, Ngwa G A, Schwehm M. The COVID-19 pandemic
preparedness simulation tool: CovidSIM. BMC Infectious Diseases,
2020, 20(1): 1-11.
[109] Ghorbani A, Lorig F, Bruin B. The ASSOCC simulation model: A
response to the community call for the COVID-19 pandemic. [Online],
available: https://rofasss. org/2020/04/25/the-assocc-simulation-model,
2020, April 1,2021.
[110] Pescarmona G, Terna P, Acquadro A, Pescarmona P, Russo G, and
Terna S. How can ABM Models become part of the policy-making
process in times of emergencies .The S.I.S.A.R. Epidemic Model.
Review of Artificial Societies and Social Simulation, 20th Oct 2020.
[Online],available: https://rofasss.org/2020/10/20/sisar/.
[111] Zheng S, Trott A, Srinivasa S. The AI economist: improving equality
and productivity with ai-driven tax policies. ArXiv Preprint
ArXiv:2004.13332,2020.
[112] Peter D, Emmanuel S. The case for a progressive tax: from basic
research to policy recommendations. Journal of Economic Perspectives,
2011,25(4):165–190,
[113] Shi J C, Yu Y, Da Q. Virtual-taobao: Virtualizing real-world online
retail environment for reinforcement learning. In: Proceedings of the
AAAI Conference on Artificial Intelligence. Hawaii, USA, 2019, 33(01):
4902-4909.
[114] Robert E, Kelli L. Decision making in a virtual environment:
effectiveness of a semi-immersive "Decision Theater" in understanding
and assessing. Human-Environment Interactions. AutoCarto.2006,
8:1922
[115] Public Sector Digest Research Staff. More than slightly ahead of the
curve. Public Sector Digest. 2006, 10:3641