Chapter seven

Evaluation Techniques
 Topics

 Roles of evaluation.
Goals of evaluation.
Evaluation Methods.
 Evaluation with user.
 Evaluation without user.
 Introduction

Evaluation
 Tests usability and functionality of system
 Occurs in laboratory, field and/or collaboration with users
 Evaluate both design and implementation
 Should be considered at all stages in the design life cycle.
 Cont..

Roles of evaluation
 Needs to assess designs.
 Needs to test systems.
 Ensure that systems behave as expected.
 Ensure that systems meet user’s requirements.
 Cont..
Goals of Evaluation
 assess extent of system functionality.
 Assess effect of interface on users.
 Identify specific problems.

In HCI we evaluate interfaces and systems to:

– Determine how usable they are for different user groups.
– Identify good and bad features to inform future design.
– Compare design choices to assist us in making decisions.
– Observe the effects of specific interfaces on users.
 Evaluation methods
Inspection methods (without users)
 Heuristic evaluations
 Cognitive Walkthroughs
User Tests (with users)
 Observational evaluation
 Experimental evaluation
 Query techniques [interview, questionnaires]
Inspection methods (without users)

 Heuristic evaluations
 Cognitive Walkthroughs
 Heuristic evaluation
 Heuristics – means rules or guidelines that are used to inform the
inspection.
 So, heuristics evaluation means an evaluation based on some
restricted guidelines.
 A heuristic evaluation is a usability inspection method for
computer software that helps to identify usability problems in the
user interface (UI) design.
 It specifically involves evaluators examining the interface and
judging its compliance with recognized usability principles (the

"heuristics"). 
 Cont..

 Heuristic method provides a simple list of design guidelines

which the evaluators uses to examine the interface screen by
screen.
 Reports violation of the guidelines as likely user problems.
 The 10 Nielsen's heuristics

1. Visibility of system status [Feedback]

The system should always keep users informed about current state
and actions through appropriate visual cues and feedback within
reasonable time. 
 Cont..

2. Match between system and the real world [Speak users’ language]
 The system should speak the users' language, with words, phrases and concepts
familiar to the user, rather than system-oriented terms.
 Follow real-world conventions, making information appear in a natural and
logical order. 
 Cont..

3. User control and freedom [Clearly marked exits]

 Users often choose system functions by mistake and will
need a clearly marked "emergency exit" to leave the
unwanted state without having to go through an extended
dialogue.
 Support undo and redo.
 Cont..

If you close the tab mistakenly from the Firefox browser

you can browse again from the history, you don’t have to remember.
 Cont..

4. Error prevention [Prevent errors]

 Even better than good error messages is a careful design which
prevents a problem from occurring in the first place.
 Either eliminate error-prone conditions or check for them and present
users with a confirmation option before they commit to the action. 
 Cont..

5. Consistency and standards

 Users should not have to wonder whether different words,
situations, or actions mean the same thing.
 Follow platform conventions.
 Cont..

6. Recognition rather than recall

 Minimize the user's memory load by making objects, actions, and
options visible.
 The user should not have to remember information from one part
of the dialogue to another.
 Instructions for use of the system should be visible or easily
retrievable whenever appropriate.
Cont..
 Cont..

7. Flexibility and efficiency of use [Shortcuts]

 Accelerators --unseen by the novice user --may often speed up
the interaction for the expert user such that the system can cater
to both inexperienced and experienced users.
 Allow users to do frequent actions easily and quicly. 
Cont..
 Cont..

8. Aesthetic and minimalist design [Simplicity]

 Dialogues should not contain information which is irrelevant or
rarely needed.
 Every extra unit of information in a dialogue competes with the
relevant units of information and diminishes their relative
visibility.
Cont..
 Cont..

9. Help users recognize, diagnose, and recover from errors

[Precise and constructive error messages]
Error messages should be expressed in plain language (no codes),
precisely indicate the problem, and constructively suggest a
solution. 
 Cont..

10. Help and Documentation

 Even though it is better if the system can be used without
documentation, it may be necessary to provide help and
documentation.
 Any such information should be easy to search, focused on the
user's task, list concrete steps to be carried out, and not be too
large.
Cont..
 Cognitive Walkthroughs
 The cognitive walkthrough method is a usability inspection method used
to identify usability issues in interactive systems, focusing on how easy
it is for new users to accomplish tasks with the system.
 It concentrate more on the difficulties users may experience in learning
to operate an applications to perform a given task.
 A cognitive walkthrough starts with a task analysis that specifies the
sequence of steps or actions required by a user to accomplish a task, and
the system responses to those actions.
 Cont..
 The designers and developers of the software then walk through
the steps as a group, asking themselves a set of questions at each
step.
 Data is gathered during the walkthrough, and afterwards a report
of potential issues is compiled.
 Finally the software is redesigned to address the issues identified.
 After the task analysis has been made the participants perform the
walkthrough by asking themselves a set of questions for each
subtask. Typically four questions are asked:
 Cont..
 Will the user try to achieve the effect that the subtask has? E.g. Does the
user understand that this subtask is needed to reach the user's goal?
 Will the user notice that the correct action is available?  E.g. is the button
visible?
 Will the user understand that the wanted subtask can be achieved by the
action? E.g. the right button is visible but the user does not understand the text
and will therefore not click on it.
 Does the user get appropriate feedback? Will the user know that they have
done the right thing after performing the action?
By answering the questions for each subtask usability problems will be noticed.
User Test methods [with users]

-- > Observational evaluation

-- > Experimental evaluation
-- > Query techniques
- interviews
- questionnaires
 Observational evaluation
 user observed performing task.
 user asked to describe what he is doing and why, what he thinks is
happening etc.
 Evaluators
-- > watches and records what users said and their action.
-- > take a note what the user said about the system.

Advantages
– simplicity - requires little expertise
– can provide useful insight
– can show how system is actually use
 Experimental evaluation

 controlled evaluation of specific aspects of interactive behavior

 evaluator chooses hypothesis to be tested
-- [a particular functionality of the system to be tested]
 a number of experimental conditions are considered which differ
only in the value of some controlled variable.
 changes in behavioral measure are attributed to different conditions
 Cont..
Evaluation factors
 Subjects
– who – representative, sufficient sample
 Variables
– things to modify and measure
 Hypothesis
– what you’d like to show
 Experimental design
– how you are going to do it
 Query techniques
Interviews
 analyst questions user on one-to -one basis usually based on prepared
questions
 informal, subjective and relatively cheap.
Advantages
– can be varied to suit context
– issues can be explored more fully
– can elicit user views and identify unanticipated problems.
Disadvantages
– very subjective
– time consuming
 Cont..
Advantages
Questionnaires
– quick and reaches large user group
 Set of fixed questions given to users.
– can be analyzed more strictly
 Need careful design
Disadvantages
– what information is required?
– less flexible
– how are answers to be analyzed? – less examining
 Styles of question:
general, open-ended, scalar, multi-choice,
ranked