Agile Testing

Download as pdf or txt
Download as pdf or txt
You are on page 1of 43

AGILE TESTING PRACTICES

Building quality in for faster releases

Hosts: Joe Justice


Dave Sharrock

© 2014 Scrum Inc.


© 2011 Scrum Inc.
: Who We Are
Scrum Inc. is the Agile leadership company of Dr. Jeff Sutherland,
co-creator of Scrum. We are based in Cambridge, MA.

We maintain the Scrum methodology by:


• Capturing and codifying evolving best practices
• Conducting original research on organizational
behavior
• Adapting the methodology to an ever-expanding set
of industries, processes and business challenges
Dave  Sharrock   Joe  Jus2ce  

We also help companies achieve the full benefits of Scrum


through our comprehensive suite of support services:
• Training (Scrum Master, Product Owner, Agile Leadership, online courses, etc.)
• Consulting (linking Scrum and business strategy, customizing Scrum)
• Coaching (hands-on support to Scrum teams)
• Publishing and new content development

We run our services company using Scrum as the primary management


framework, making us a living laboratory on the cutting edge of “Enterprise

© 2014 Scrum Inc.


Scrum”

Find  out  more  at  www.scruminc.com.    


Agenda
• Agile Testing Costs & Benefits.
• Top Agile Testing Practices.
• Test Driven Development, Test First Development, DevTest, DevTestOps.
• Unit Testing, Integration Testing, Regression Testing, User Acceptance Testing.
• Tools to get testing done quickly.
• Modern test tools for your Weird/Backward/Deprecated/Unique environment.
• Cucumber and Ruby on the as400 with tools written by Cheezy, at Nationwide
Insurance.
• TDD in Hardware and Red Green Refactor.
• The Business case and numbers for exploratory testing, manual testing, and
building quality in.
• 24 times the cost to repair a bug that escaped the sprint.
• Eliminating Technical Debt with Elegant Testing.
• Outsourced QA?
• How to Crush your Competitors with Agile Testing.
• Part of the Solution? Gated Check-in and Rapid Roll Back in hardware and
Software.
• The mindset of managing a successful agile testing practice.
• Code Craftpersonship, shared ownership, elegance, building quality in.
• MIND BLOWING CONCLUSION: DevTestOps.
• You build your test, you build your code, you integration test your code, you
check in your automated tests, you build your deployment package and its tests,
you update your deployment environment, you deploy your tests, you deploy

© 2014 Scrum Inc.


your code. You do all of this in a pair. When you do this, competitors can’t keep
up.
© 2014 Scrum Inc.
David Rico. Business Value of Agile Methods. 2013.
Winning at Agile Testing: be2

• - move from 8-12 hours unplanned downtime per


release every 90 days to 1 hours planned
downtime (2 hours for DB updates) every 2
weeks
- eliminated hot fixes as a normal release
occurrence
- eliminated need for post-release testing (I
know, who thought there was ever a need for
this!) by PO group
- went from <1% test coverage to getting on for
30-50% (1000’s of tests) in about 18 months
using DoD and a concerted effort/understanding
from team

© 2014 Scrum Inc.


WSJ: Microsoft CEO bans test teams

• Dev after test is too slow! Banned “to cut costs


while building software faster” –WSJ.
• Triggered immediate stock climb, peaking at 10%
shareholder gain. -NYSE

Source:  Wall  Street  Journal:  

© 2014 Scrum Inc.


h;p://blogs.wsj.com/cio/2014/07/15/microsoG-­‐plots-­‐agile-­‐development-­‐course-­‐as-­‐talk-­‐on-­‐job-­‐cuts-­‐loom/  Source:  NYSE  via  
google  finance.  Source:  Bloomberg:  
h;p://www.bloomberg.com/news/2014-­‐07-­‐10/microsoG-­‐ceo-­‐nadella-­‐outlines-­‐vision-­‐with-­‐more-­‐overhauls.html  
Top Agile Testing Practices

1   Test inside the Sprint


I. Observed norm
for teams
releasing each
2   Automate tests sprint.

3   Gated Check-In, Rapid Roll Back II. Observed norm


in competitive
markets.
4   TDD

5   Red, Green, Refactor


III.
Observed in the
fastest companies.

© 2014 Scrum Inc.


6   DevTestOps
8

© 2014 Scrum Inc.


Testing – Ideal Case

End users

v1.0.0 v1.1.0

Scrum team Sprint 1 Sprint 2

Timeline

© 2014 Scrum Inc.


Source, Adapted From: Henrik Kniberg
Testing – The Sad, Common Alternative

End users

v1.0.1

Acceptance test team 1.0 acceptance test

Bug!

v1.0.0 v1.0.1 v1.1.0

Scrum team Sprint 1 Sprint 2

Timeline

© 2014 Scrum Inc.


Source, Adapted From: Henrik Kniberg
Agile Testing
• We know you aren’t doing it, most of the teams we work
with have this as their #1 reason for failing the sprint
(work carrying over), so let’s give you to the tools to
make it happen.

© 2014 Scrum Inc.


Tools to make this happen quickly

• Cucumber
• Plain language DSL tool.
• Product Owners can use directly.
• FitNesse
• Wiki driven ATD framework.
• Collaborative and information sharing.
•  The  
Power PowerTools  
Tools
• 2nd gen test framework.
• Possibly the most elegant, compact.
• Docker
• 2nd gen Virtual Machine service.
• Ultra compact VM files.

© 2014 Scrum Inc.


• Automate tests across thousands of test instances.
Central Desktop

• Venture company with no testers.


• Cucumber experts writing scripts with the product
owners to accompany new backlog stories.
• Developers get a User Story and a red test.
• The PO says “Dev team, turn these green and
you are done.”
• CTO is chief PO, and can hit the deploy button at
the end of any sprint.
• Enabled by policy, early on, Remediate Technical
Debt as soon as it is found.

© 2014 Scrum Inc.


Acceptance Test Driven Development (ATDD)
Tools: Fit and Cucumber

FIT (Framework for Integrated Test) Cucumber


and Fitnesse (Wiki front end) • Tool for natural language scenarios
• Test specified in table format

• Developers generates classes (“fixtures”) In order to ensure my account is correct


As a Registered User
to hook into application I want to check my recent activity

• Users/testers use Wiki or Excel to Scenario: Recent Account Activity


Given I am a registered user “Jsmith”
specify inputs and outputs And I am logged in with password “xyx123”
And I have had account activity in the last 45 days
And I am on the home page
numerator   denominator   quotient   When I click the “Recent Activity” button
Then I should see the “Account Activity” Page
And I should see a list of my activity over the last 45 days

10   2   5   Scenario: No Recent Account Activity


Given I am a registered user “Jsmith”
And I am logged in with password “xyx123”
12.6   3   4.2   And I have had no account activity in the last 45 days
And I am on the home page
When I click the “Recent Activity” button
Then I should see the “Select Account Activity Period” Page
100   4   25 expected And I should get a message: “You had no activity in the last

© 2014 Scrum Inc.


45 days, please select a time frame to review”
24 returned  
Modern test tools are available for your weird/
backward/deprecated/unique environment.
• Cucumber and Ruby on the as400: with tools
by Jeff “Cheezy” Morgan, at Nationwide Insurance.

• Arduino Test Fixture


for Scrum Hardware.
• Red Light, Green Light.
• Test Logic onboard.

© 2014 Scrum Inc.


• $24 USD.
• Ships same day.
Sources:  Na2onwide's  Applica2on  Development  Center,  Danielle  Roecker,  Team  WIKISPEED.  
Systematic
• Already 98% on time and on budget, CMMI level 5.
• How to improve from there? Half cost to deliver,
project management cost down 80%, and reduce
defects by 40%!
• How to do that? By monitoring the metric that is king
of them all:
• Time to fix a bug.
• Monitored by control charts.
• If time went over 2 hours, Scrum Masters immediately
performed root cause analysis and systemized the
resolution.
• Carsten Jakobson, their internal process champion, has
paired to published 5 papers documenting this practice
and it’s success, peer reviewed.
• Measured bug fix time 24x less if resolved in sprint.
• Couldn’t afford NOT to exploratory test, manual test,
automated test, and BUILD QUALITY IN.

© 2014 Scrum Inc.


• Repeated at Palm on web operating system code.

Sources:  IEEE  digital  Library,  “Scrum  and  CMMI  Going  From  Good  To  Great”.  Jeff  Sutherland  and  Carsten  Jakobson.  
Gartner - Technical Professional Advice
2012 Planning Guide: Application Delivery Strategies

• Business users are losing patience with old-school IT culture.


Relationships are tense and resentful. Legacy systems and practices
impede agility. Bottom line - GET AGILE
• Adopt a product perspective.
• Say goodbye to waterfall.
• Improve cross-competency collaboration.
• Launch a deep usability discipline.
• Start a technical debt management program.

© 2014 Scrum Inc.


Technical Debt & Release Planning
Remaining
story points

400

We’ll be done
by sprint 10!
300

Sorry, we’re late!


We should
200 definitely by done Um... we’re done
by sprint 12! when we’re
done!

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Sprint

© 2014 Scrum Inc.


Source: Adapted from Henrik Kniberg
Dealing with Technical Debt

Definition of Done Definition of Done


• .... bla bla .... • .... bla bla ....
No increased technical debt Technical debt decreased
Vmax

Vactual
velocity

!
ng pace
easi
Sustainable pace Incr

Second step
First step (optional)

Slow down Slow down even more


Stop accumulating debt Start repaying debt

© 2014 Scrum Inc.


time
Source: Adapted from Henrik Kniberg
The Mindset of Managing a Successful Agile
Testing Practice

A  • Elegance. UI tests that rely on click maps or control


placement are brittle, UI tests that call the button click
event or control programmatically (pinouts in hardware)
are robust!

Shared Ownership. Test doesn’t QA Dev’s solution, we


B  • don’t call our solution done until it meets our Definition of
Done.

• Build Quality In. Done solutions are not just tested, but
C   refactored in place to (re)establish elegant architecture in
that piece of the solution.

D  • Craftspersonship. Always inspired and inspiring to


improve the Definition of Done:
• User Acceptance Testing,

© 2014 Scrum Inc.


• Unit testing,
• integration testing,
• regression testing,
• User Acceptance testing.
OUTSOURCED QA?
Xebia Case study: Building a new railway information
system (ProRail)

• ProRail rescued a failed waterfall project to build a new scheduling system and automated
railway station signs at all Netherlands railway stations
• An 8 person half-Dutch and half-Indian Scrum team started the project and established local
velocity.
• After establishing local velocity at 5 times other waterfall vendors on the project, the Indian
half of the team went back to India.
• Scrum teams run all XP practices inside the Scrum including intensive pair programming.

© 2014 Scrum Inc.


• The customer completes acceptance testing on all features during each Sprint.
• Done at the end of the Sprint means customer has accepted the code as ready for production.
• Defect rates are less than 1 per 1000 lines of code and steadily getting lower.
Integrating analysis, testing and
implementation
• Analysts, testers and developers define together how a user story will be tested
using FitNesse
• Testers write functional tests while the user story is being implemented
• Developers write ‘fixtures’, which link tests with the production code
• Regression tests are run continuously
• 95% of Bugs are found and fixed within the Sprint

© 2014 Scrum Inc.


Crush Your Competitors with Agile Testing
• If you are the CEO, like Microsoft: BAN TEST TEAMS.
• If you are the Director of Engineering:
• Identify and encourage Test Leads as XP practices champions in each team.
• Configure build servers for each team yourself, or bring in a consultant to do it.
• Turn on Gated Check-in.
• Make Automated Test Coverage visible, measure each team’s increase in code
coverage.
• If you are the team Test Lead or XP practices champion:
• Read up on elegant architecture and radical refactoring.
• Practice your own ability as DevTestOps.
• Share with everyone interested. Host a book club.
• Make visible the number of people on your team who can dev, test, deploy.
• If you are the team Product Owner:
• Write backlog in a test tool as a failing test.
• Demand all estimates include no increased technical debt.
• Demand Definition of Done includes Refactor In Place.
• If you are the team Scrum Master:
• Measure time to fix a bug.

© 2014 Scrum Inc.


• Like Systematic, If more than 2 hours for a bug, immediately perform root cause
analysis and systematize the resolution.
Ques2ons?  

© 2014 Scrum Inc.


Stay Connected
Scruminc.com  
• For up coming events, new content releases, and more!  
ScrumLab  
• articles, online courses, tools, and papers on all things scrum  
• www.scruminc.com/scrumlab  
Blog  
• http://www.scruminc.com/category/blog/
Online Courses  
• advance your scrum with our online courses. Visit the Scrumlab
store to view upcoming topics.  
Twitter, Facebook, and G+  
• @ScrumInc, @JoeJustice0, scrum and scrum inc.

2014Scrum
©2012 Inc.
ScrumInc.
25
Acceptance Tests

26

© 2014 Scrum Inc.


Modern Agile Acceptance Model

• The move toward Acceptance Test Driven Development requires a


more complete model of progressing requirements acceptance
• Example model (sharper definition of done)
• Conditions of Satisfaction - broad terms
• Acceptance Criteria - further refined
• Examples - actual scenarios or data
• Executable Examples - Executable Spec

27

© 2014 Scrum Inc.


Simple Scenarios

• Suppose we are creating a registration function


• It consists of specifying email (as User ID) and Password

Enter your
Email and
Password to
Register  
Email Address:  
Password:  
Register  

© 2014 Scrum Inc.


Acceptance
Condition of Satisfaction
• Conditions of Satisfaction are high level criteria established with an
initial Epic/Feature/User Story
• In our previous example, the conditions of acceptance for the password would be:
• Ensure the password is not easy to crack.

• The PO would define the conditions of acceptance in concert with


business stakeholders

Enter your
Email and
Password to
Register  
Email Address:  
Password:  
Register  

© 2014 Scrum Inc.


Acceptance
Examples Make It Real
• Actual examples are the ultimate way to communicate
requirements:
Password   Expected   Expected Message   Comment  

abc123   Invalid   Your password must be at least 8 characters


and no more than 12 charcters long  
abcdefghi   Invalid   Your password must contain at least one
character and one number  
1aaaaaaaa   Valid   Why valid?  

ajx972dab   Valid  

• The PO works closely with a tester, developer, and business


stakeholder to define these criteria

30

© 2014 Scrum Inc.


Acceptance
Acceptance Criteria
• Acceptance Criteria specifies the details that the team can
then build against:
• Following the example for Password:
• Must be at least 8 characters and no more than 12
• Must contain only alpha-numberics and the period
• Must contain at least one digit
• Must contain at least one character
• Etc. (there may be more criteria)

• The PO works closely with a tester, developer, and business


stakeholder to define these criteria

Enter your Email


and Password to
Register  
Email Address:  
Password:  

© 2014 Scrum Inc.


Register  
MIND BLOWING NEXT STEP
• DevTestOps

© 2014 Scrum Inc.


Conclusions
• If you are the CEO, like Microsoft: BAN TEST TEAMS.
• If you are the Director of Engineering:
• Identify and encourage XP practices champion on each team.
• Configure build servers for each team yourself, or bring in a consultant to do it.
• Turn on Gated Check-in.
• Make Automated Test Coverage visible, measure each team’s increase in code
coverage.
• If you are the team XP practices champion:
• Read up on elegant architecture and radical refactoring.
• Practice your own ability as DevTestOps.
• Share with everyone interested. Host a book club.
• Make visible the number of people on your team who can dev, test, deploy.
• If you are the team Product Owner:
• Write backlog in a test tool as a failing test.
• Demand all estimates include no increased technical debt.
• Demand Definition of Done includes Refactor In Place.
• If you are the team Scrum Master:
• Measure time to fix a bug.

© 2014 Scrum Inc.


• Like Systematic, If more than 2 hours for a bug, immediately perform root cause
analysis and systematize the resolution.
Distributed Testing?

34

© 2014 Scrum Inc.


OUTSOURCED QA?
Xebia Case study: Building a new railway information
system (ProRail)

• ProRail rescued a failed waterfall project to build a new scheduling system and automated
railway station signs at all Netherlands railway stations
• An 8 person half-Dutch and half-Indian Scrum team started the project and established local
velocity.
• After establishing local velocity at 5 times other waterfall vendors on the project, the Indian
half of the team went back to India.
• Scrum teams run all XP practices inside the Scrum including intensive pair programming.

© 2014 Scrum Inc.


• The customer completes acceptance testing on all features during each Sprint.
• Done at the end of the Sprint means customer has accepted the code as ready for production.
• Defect rates are less than 1 per 1000 lines of code and steadily getting lower.
Testing Implementation

• New Product Development:


• WIKISPEED Red/Green card example.
• Start with epics phrased as tests
• Split until experiment can be run and tested in less
than a sprint.
• Known Product Development:
• How to rescue a failed Scrum.
• Add testers to the Scrum team to meet the real
definition of done, DEPLOYABLE.

© 2014 Scrum Inc.


Xebia/ProRail Defect Tracking
• Defect rate gets lower and lower as code base increases in size
• 95% of defects found inside iteration are eliminated before the end of the iteration

• TDD, pair programming, continuous integration. Same tools and


techniques onshore and offshore.
• Daily Skype video Scrum meeting of team across geographies.
• SmartBoards, wikis, and other tools used to enhance communication.

© 2014 Scrum Inc.


• Indians say it feels exactly the same in India as it does in Amsterdam.
They do the same thing in the same way.
Categories of Testing

• User Acceptance Testing,


• Unit testing,
• integration testing,
• regression testing,
• User Acceptance testing.

© 2014 Scrum Inc.


The mindset of managing a successful agile
testing practice.
• Code Craftpersonship, shared ownership,
elegance, building quality in.

© 2014 Scrum Inc.


Good Agility Starts with Good Testing

• Gated Check-in and Rapid Roll Back in hardware


and Software.

© 2014 Scrum Inc.


Agile Testing
How is testing approached differently in traditional and agile
contexts?

Requirements
Build
Testing?
Deploy

RBTD repeat – intermediate.


RTBTDTR repeat – skillful.

© 2014 Scrum Inc.


Velocity and Technical Debt

42

© 2014 Scrum Inc.


The numbers for exploratory testing, manual
testing, automated testing, and building quality in:
• It costs 24x as much to fix a bug that escapes
the current sprint.
• Corroborated studies in the U.S. and Europe.
• Found at Palm and Systematic.
• The solution?
• Testing in the sprint saves 24 times the bug fix
time, implemented by testers joining dev team and
only testers can move cards across backlog.

© 2014 Scrum Inc.


Sources:  IEEE  digital  Library,  “Scrum  and  CMMI  Going  From  Good  To  Great”.  Jeff  Sutherland  and  Carsten  Jakobson.  

You might also like