How to improve your

manual testing
…  without getting bored!

Alex Schladebeck

(exploratory testing and session-based test management)

23.10.2014 1
Short introduction
BREDEX This talk
• Individual software • Why exploratory testing?
development • How to use it
• Enterprise applications • How to plan and organise it
• Quality assurance
• Based on Jubula
• Tool-independent services

23.10.2014 Eclipse Con Europe 2014 2

 Wait, what?!
If you‘re such test automation champions, why
are we talking about manual testing?

23.10.2014 Eclipse Con Europe 2014 3

 Testing as detective work

Testing is gathering information with the intention of informing a decision – Jerry Weinberg

23.10.2014 Eclipse Con Europe 2014 4

Gathering information
It is impossible to plan tests in advance to cover every
possible condition
If you try to plan / write all possible tests, you will
only ever write them, never execute them
You can only plan based on what you know
• If the map and the territory differ, believe the territory

Elisabeth Hendrickson, Explore It!

23.10.2014 Eclipse Con Europe 2014 5

 Testing vs. Checking
Are there
Intended any
behaviour? other
risks?

Testing
Checking
(Exploring)

Scripted test Unscripted test

(manual or (manual only)
automated)

23.10.2014 Eclipse Con Europe 2014 6

A couple of videos

23.10.2014 Eclipse Con Europe 2014 7

1. Observational blindness
Focus on one aspect makes Errors avoid well-tested paths
us miss other things
• The narrower the view, the wider
the ignorance

23.10.2014 Eclipse Con Europe 2014 8

2. Orders of ignorance
because as we know, there are known knowns—there are things we
know  we  know…  

We also know there are known unknowns; that is to say we know

there  are  some  things  we  do  not  know…  

But there are also unknown unknowns—the ones we don't know

we don't know.

Donald Rumsfeld, US Secretary of Defense.

Department of Defense News Briefing. Feb 12 2002

23.10.2014 Eclipse Con Europe 2014 9

2. Orders of ignorance
0th order ignorance 2nd order ignorance
• We provably know something • We  don’t  even  know  what  we  
• a scripted test that shows don’t  know
us the same questions being • exploratory testing to find out
answered again and again what  we  don’t  know
• Repeat thinking, not steps
1st order ignorance • Find the holes in the net
• We  know  what  we  don’t  know
• scripted and exploratory
testing can help here: ask the
questions that will give us the
knowledge

23.10.2014 Eclipse Con Europe 2014 10

Not scripted? Clicking around?

23.10.2014 Eclipse Con Europe 2014 11

Exploratory testing
Exploratory testing is not ad hoc
Test design, execution and analysis all happen at the same time
• No scripting – but normal test design practices can be used (personas,
decision tables,  …)
• Pre-defined test results not necessary
• Thoughts and process not documented in advance
Situational and emergent practice – context is important
• The puzzle itself influences the strategies for solving it
• Steps and test cases are designed based on context and on observation

…  This sounds kind of difficult, but it‘s actually our normal human
practice of reasoning.
23.10.2014 Eclipse Con Europe 2014 12
Exploratory testing

23.10.2014 Eclipse Con Europe 2014 13

When should I use Exploratory Testing?
To find out unknown unknowns
To find out how the software works
• E.g. when the specifications aren‘t detailed
When you need to find important information quickly
To keep test script writing to a minimum
To find new information and new questions
To avoid observational blindness

23.10.2014 Eclipse Con Europe 2014 14

How do I use Exploratory Testing?
How do I know how the program should behave?
What should I test?
How do I know what is a problem?

23.10.2014 Eclipse Con Europe 2014 15

Sources and oracles
Sources Oracles
• Places you can get • How to know if there is a
information problem
• Specification • Mental models
• Requirements • Own gut feeling
• Documentation • Consistency
• Experience • Is there a problem here?
• Sales promises
• Discussions
• Customer Heuristics
• Users • Rules of thumb to guide
• Product menus, toolbars, testing and to identify
context menus problems

23.10.2014 Eclipse Con Europe 2014 16

Heuristics – what to look at / for
CRUD: Create, read, update, delete
Goldilocks: Too big, too small, just right
Users and scenarios: personas, extreme personalities,
soap operas, tours
Follow the data: Enter – Search – Report – Export –
Import – Update – View
HICCUPSF: History, Image, Comparable Products, Claims,
User expectations, Product, Purpose, Statutes, Familiar
Problems
Areas of potential instability: interoperation, multiple files,
network,  …

23.10.2014 Eclipse Con Europe 2014 17

Advantages and disadvantages
Advantages Disadvantages
• Less preparation than • Not reviewable in advance
scripting • Not reproducible
• Complete specification not • Not good in low-availability
required environments
• Adaptable to new
information • Hard to organise and
• Important errors found control
quickly • Where to start?
• More interesting • Session-based test
management

23.10.2014 Eclipse Con Europe 2014 18

Session-based test management
A way of organising and reporting on Exploratory Testing

Session-based test management recipe

Charters What areas to test
Sessions Time to do the testing
Notes Observations, questions, feelings, tests performed
Debriefing Information transfer

23.10.2014 Eclipse Con Europe 2014 19

 Define
charters

Prioritise
Debrief and
assign

Session –
Observe
down the
and note
rabbithole!
23.10.2014 Eclipse Con Europe 2014 20
Charters
Explore [area] with [resources] to discover [information]

Charters come from:

• Stakeholders, team, risks, quality attributes, places where we
require information, the nightmare headline game
Defined in advance but can be changed
(Re)prioritised before each new session

23.10.2014 Eclipse Con Europe 2014 21

Session – down the rabbit hole
Uninterrupted timebox
• 30-120 minutes
One charter
• Down the rabbithole
• If you find another hole, write it down new charter for next
session
Use heuristics to create and execute tests

23.10.2014 Eclipse Con Europe 2014 22

Notes
What did I do?
What did I observe?
• Videos, screenshots
How do I feel about the quality of this charter?
Ratio test:setup
Ratio charter:opportunity
• Opportunity = new areas that occur while testing

23.10.2014 Eclipse Con Europe 2014 23

Debrief
PROOF
• Past: what happened
• Results: what was achieved?
• Obstacles: what blocked the testing?
• Outlook: What needs to be done?
• Feeling: how do I feel about this?

23.10.2014 Eclipse Con Europe 2014 24

Test day
8.00 Define, prio, choose charters 1.15 Define, prio, choose charters
8.30 Session 1.30 Session
10.00 Debrief 3.00 Debrief
10.15 Break 3.15 Bug entry
10.30 Define, prio, choose charters
10.45 Session
12.00 Debrief
12.15 Lunch

23.10.2014 Eclipse Con Europe 2014 25

When to reproduce / enter bugs?

23.10.2014 Eclipse Con Europe 2014 26

A good mix
Daily / in each sprint Before a release
Automated tests Manual test checklists
• Regression tests • Semi-scripted
• As part of acceptance Exploratory testing days
testing for new features
tests that will become
with session based test
regression tests management
Exploratory testing
• Manual testing of new
features
• Can use SBTM

23.10.2014 Eclipse Con Europe 2014 27

Thanks!
alexandra.schladebeck@bredex.de
@alex_schl

www.bredexsw.com
testing.bredex.de

www.eclipse.org/jubula

23.10.2014 Eclipse Con Europe 2014 28

References
Jerry Weinberg, Perfect Software and other Illusions about Testing
Orders of ignorance http://www.corvusintl.com/CACM002-5OI.htm
Exploratory testing
• Explore it! Elisabeth Hendrickson (Pragmatic Bookshelf)
• http://www.developsense.com/blog/2012/04/all-oracles-are-heuristic/
• http://www.satisfice.com/articles/what_is_et.shtml
• http://www.satisfice.com/tools/procedure.pdf
• http://www.flickr.com/photos/softwaretestingclub/7173828227/sizes/o/in/photostream/

Cheat sheets and mnemonics

• http://www.qualityperspectives.ca/resources_mnemonics.html
• http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
• Session-based test management
• http://www.ministryoftesting.com/resources/session-based-test-management/
• http://www.satisfice.com/articles/sbtm.pdf

23.10.2014 Eclipse Con Europe 2014 29