AI and Software Testing: Vikram Raghuwanshi Senior Consultant

Download as pdf or txt
Download as pdf or txt
You are on page 1of 23

AI and Software Testing

Vikram Raghuwanshi
Senior Consultant
Capgemini Technology Services

1
ABSTRACT

This whitepaper gives an introduction to


Artificial Intelligence; how AI touches our
lives on a day to day basis, & how AI
systems work.

It discusses aspects of the application of


AI in QA and explains what it takes to
develop an AI based solution.

The paper showcases a Capgemini AI


based tool and concludes by highlighting
the benefits of AI in QA.
What is AI
Jeff Bezos, Amazon.com Inc FORBES

“Artificial Intelligence (AI) is a self improving,


enabling horizontal layer that is solving problems Artificial Intelligence is the broader
that were in the realm of science fiction for the past concept of machines being able to
several decades” carry out tasks in a way that we
would consider “smart”.

SAS STANFORD

It is the science and engineering of making


Artificial intelligence (AI) makes it possible intelligent machines, especially intelligent
for machines to learn from experience, computer programs. It is related to the similar
adjust to new inputs and perform human- task of using computers to understand human
like tasks. intelligence, but AI does not have to confine
itself to methods that are biologically observable.
AI, ML and Analytics

Artificial Intelligence is the MACHINE LEARNING


broader concept of enabling PREDICTIVE ANALYTICS
machines to carry out tasks
and to think and reason just
as humans do

ML is a current application of
AI based on the idea that ARTIFICIAL
machines be given access to INTELLIGENCE (AI)
data and allowed to learn for
themselves

Data analytics, forms the


core of “learning process”
that machines undergo to
make sense of the data that
it digests
Fig Source: http://futurearchitectureplatform.org/news/28/ai-architecture-intelligence/
Application of AI

Fraud Meeting regulatory Boost customer


Detection requirements engagement
• AI has the ability to identify • AI is used to ensure that • Artificial intelligence will assist in
fraudulent behavior, as well as regulatory requirements are met the creation of customized and
identify what the next pattern of and that data is kept with intelligent products and services,
suspicious behavior will be monitoring done on a real-time with new features, more intuitive
• AI algorithms can preempt such basis. This allows issues to be interactions (e.g. speech) and
fraudulent transactions and can flagged a lot sooner advisory skills (e.g. personal
BANKING lead to huge costs savings for financial management)
banks and its customers

Computer High risk groups Epidemic


aided diagnosis identification outbreak prediction
• AI is being used • AI is being fed huge • ML and AI technologies are also being applied to
extensively to read volumes of data related to monitoring and predicting epidemic outbreaks
and interpret patients historical medical around the world, based on data collected from
complex radiology, records that helps in satellites, historical information on the web, real-
pathology reports to identifying whether a time social media updates, and other sources.
help doctors arrive patient is in a high risk • Support vector machines and artificial neural
at early diagnosis group for any particular networks have been used, for example, to predict
disease say stroke, malaria outbreaks, taking into account data such
HEALTHCARE cardiovascular diseases or as temperature, average monthly rainfall, total
cancer. number of positive cases, and other data points.
How AI based systems
work
AI based systems employ a host of advanced statistical
algorithms that processes huge volumes of input data in
order to come up with extremely useful insights and/or
predictions

An AI workflow typically involves 3 major parts:

Model: the system that makes predictions

Parameters: used by model to form its decisions

Learner: the system that adjusts the parameters, and in


turn the model, by looking at differences in predictions
versus actual outcome.
Architecture of AI system

AI-OPTIMIZED MODELS DATA


MASTER DATA TRAINING VISUALIZATION DATA INGESTION:

DATA INGESTION ENRICHED DATA


UNSTRUCTURED LEAN / SMART
DATA / AI-READY

Data Source 1
DATA ENRICHMENT:
TRAINING DATA
DASHBOARDS

MODEL

Data Source 2
NOTIFICATIONS STATISTICAL MODEL:

TESTING DATA APPLY MODEL

TRENDS / GRAPHS
Data Source 3

DATA VISUALIZATION:
What you need to build
AI systems (Languages)
Python
• Choice of Python for AI projects also stems from the fact that there are
• Python is a popular and widely used language and a rising plenty of useful libraries that can be used in AI for example, Numpy
star for machine learning offers scientific computation capability, Scypy for advanced computing
and Pybrain for machine learning in Python
• It can seamlessly be used with the data structures and other
frequently used AI algorithms • Easy to learn, program and debug
• Tons of reading materials and helpful resources available online.

R
• Encompasses enormous number of machine learning algorithms,
• R is a workhorse for statistical analysis and by extension
and advanced implementations too written by the developers of the
machine learning
algorithm
• Hugely popular among data scientists, statisticians, data
• Complex statistical algorithms can be written in a few lines of code
analysts and enthusiasts
• Plenty of extremely useful R packages such as dplyr, plyr to
• It is the default language to understand and explore data
manipulate data; ggvis, lattice and ggplot2 to visualize data; caret for
using statistical methods and graphs
machine learning
What you need to build AI
systems (Platforms &
Skillsets)
RapidMiner KNIME
• Licensed but offers a free version with
restriction on input data rows • Opensource data analysis platform

• Graphical user interface-based • Flexible, and low total cost of ownership


data science platform
• Graphical user interface-based
• Ease of use for all data scientists and skill data science platform
levels & has “large and vibrant” user community

IBM Watson Skillsets


• Advanced and a powerful analytics
platform with a breath of offerings
- Knowledge Studio • Data Scientists: Competent in using advanced Machine
- Natural Language Classifier Learning & Statistical algorithms
- Tone analyzer
- Speech to Text, Text to Speech • Big Data Architect: Well versed with handling huge datasets
running into terabytes
• Possess advanced NLP features
• High costs involved
Applications of AI in
Software Testing

Test Plan Support Chat Bots

Test Suite Analysis Defect analysis

RTM analysis
An futuristic AI Exhaustive Reporting
application that
Identification of right
Test Cases
will address Preserving Of test
Artifacts
various aspects
Continuous Integration Machine Learning
of testing
Prerequisites for using AI
based system in Testing
• Data Acquisition & Cleansing • Relevant access to data source

• Test plan • AI Engine needs access rights to


various systems. For example, for ALM
it needs query editor access, &
• Test Cases
developer access for TFS / SVN etc.,

• Defects
• Access to Data Science Platform

• Requirements Traceability Matrix


• RapidMiner, KNIME, RStudio

• Code Quality Reports


• Resources skilled in Data Science,
• Downtime Logs Machine Learning, AI
Insights from AI solution

AI solution gets various data feeds; it then gives critical predictive analysis to the
testing teams across project’s lifecycle assisting them to keep adjusting to various
moving parameters influencing outcomes of the project.

Asset analytics: Staff analytics:

• Captures key metrics and make • Assists in understanding the distribution of


recommendations on improving leverage from capabilities, specific skills among the complete
assets. team & to track real time progress,

• Analyses asset’s usage trend, execution • Predicts which resource is best suited for what
duration, attached criticality through tasks (& auto assigns such tasks to identified
requirements, defects mapping etc. resources based on bandwidth)
Insights from AI solution

CUSTOMER ANALYTICS: RIGHTSIZING SOLUTIONS:


• Sentiment analysis by analyzing customer feedback from
multiple social channels
• Identifies (from customer feedback) key issues in • Combination of various analytical utilities guiding testing
applications designs, behavior, usage patterns etc. organization to size their testing appropriately real time
allowing testing teams to plan better around digital testing
coverage.

DELIVERY ANALYTICS: PREDICTIVE ANALYTICS:

• Predicting likelihood of a Test Case failure (Defect


• When to start / stop Testing Prediction)

• Provides real time test data consumption and business • Predicting vulnerable (prone to defects) modules in the
requirement coverage Application Under Test
• Predicting the right Test Case pack for a new requirement
Benefits of Using AI in QA
• Shift left testing for faster time-to-market &
improved quality
• AI application will effectively help in
generating and optimizing test cases,
prioritizing testing and automation, enhancing IMPROVED QUALITY
UI testing and reducing tedious analysis & FASTER TIME-TO-
tasks. MARKET

• Unattended execution – In case of failures


during Test Execution, AI system shall DEFECT FORENSICS
execute recovery routines and achieve
unaided execution
MINIMIZE MANUAL
• Scientific approach for defect forensics:
EFFORT
(Application Stability, defect hotspots, Defect
pattern and failure predictions)
• Social media analytics: Mechanism to
analyze data from social media to
build trends by demographics; forming the
basis of test suites
Example of an actual AI
enabled System for QA

Capgemini’s Smart QA is an analytics-driven platform that helps banks, financial


institutions, and insurers build an end-to-end ecosystem, which includes testing assets,
a test environment, test data and performance data

It has An analytics engine, which includes cognitive


capabilities powered by smart techniques and insights
which test architects can use to prioritize what needs
to be tested, optimize testing efforts, and identify areas
of improvement.

Smart QA also knows how much testing is required


and which assets should be used.
SmartQA - Conceptual View
AI features in SmartQA
Resource Prediction

Resource Prediction
• Dashboard aggregates data of defects that have been
logged by the developers
• The aggregated historical defect data will be analyzed
based on severity and type of defect and prediction can
happen for future developer defects during releases
• The dashboard offers analysis, retrospection and
prescription of defects that have been logged Risk Based Test Case Selection

Risk Based Testing


• Exact test cases that need to be executed based on
priority.
• Prediction for next test execution
• Complexity of test case execution
What-If analysis
What-If analysis
• What if Analysis predicts defects at code & release level
• What if Analysis offers various levers which can help
predict percentage defects that may happen
• The levers can be adjusted as per desired test quality
and release cycle period.
AI features in SmartQA

TEST CASE RECOMMENDATION AUTO / SELF ROOT CAUSE ANALYSIS PREDICT THE TEST CASE FAILURE

List of Available Selected Test Case TC3 Actual TC status Predicted TC status
Test Cases for execution

Recommended Test Did the Test Case


Cases list for TC3 “Support” & “Conf”
Fail? Yes or No

The Decision Tree algorithm works on the concept of Entropy and


This Use case is on similar lines as the product The Logistic Regression’s output helps in assigning
looks to classify the input data based on gaining maximum
recommendation on popular e-commerce sites a quantitative value to each Test case
information and stability.
Conclusion

AI and its applications are increasingly gaining acceptance in


every vertical and/or service line that the organizations
operate in.

Industry gurus believe testing is certainly one area which


could immensely benefit from AI.

At this juncture it is worthwhile to note that while AI is


promising & emerging, it is fairly complex and has a steep
learning curve initially.

However with proper planning on resources, use cases,


software, hardware and on overall architecture, one shall
definitely be able to overcome the said challenges and
leverage the benefits for a successful AI based solution for QA
References

• Wikipedia - en.wikipedia.org • https://www.capgemini.com/service/capgemini


s-smart-qa-for-financial-services/
• https://en.wikipedia.org/wiki/Evolution
ary_computation • https://www.knime.com/knime-analytics-
platform
• https://www.ibm.com/watson/
• Fausett, L. Fundamentals of Neural Networks:
• https://www.openai.com Architecture,

• http://www.ipsoft.com • Algorithms and Applications.


Prentice Hall 1994.
• http://www.wipro.com/holmes/
Author Biography

Vikram is having 12+ years of rich IT experience.

He has worked on support, S/W Development, S/W


testing and Automated testing.

He has keen interest in current & emerging


technologies. His educational qualifications include
BSc. ( PCM ) & MCA
THANK YOU

You might also like