Chapter+12

toc

=Evaluation=

12.1 Evaluating interactive system design
- Evaluation means reviewing, trying out or testing a design Steps in undertaking a simple evaluation project:



**Step**
> > > > > > > || > Obtaining feedback to inform early design concepts: - you may need to evaluate initial concepts, especially if the application is novel for your users. Paper prototypes can help, or even software if this can be produced rapidly.
 * 1) Establish the aims of the evaluation, the intended users and context of use for the software; obtain or construct scenarios illustrating how the applications will be used
 * 1) Select evaluation methods- should be a combination of expert review and end-user testing
 * 1) Carry out expert review
 * 1) Plan user testing; use the results of the expert review to help focus this
 * 1) Recruit users and organize testing venues and equipment
 * 1) Carry out testing
 * 1) Analyze results, write up report back to designer

Deciding between different design options: - During development, designers have to decide between options.

Checking for usability problems: - testing will identify potential problems once a stable version of the technology is available. **Formative evaluation:** results help to form, or shape the design.

Assessing the usability of a finished product: - this may be used to test against in-house guidelines, or formal usability standards such as ISO 9241, or to provide evidence of usability required by a customer for example the timed to complete a particular set of operations. This type of evaluation is sometimes termed **summative.**

As a means of involving people in the design process: - In a participatory design approach, user help designer set the goals for the evaluation work. Involving users has great benefits in terms of eventual uptake and use of the technology.

Assessing use in practice: - Researchers undertake long-term evaluations as a means of understanding the success of particular technologies. Such an approach is relatively rare in commercial practice, through indirect data are often collected and used to inform the next release of a product. These might include problems reported by customers or end-users, customer reactions to salespersons’ pitch or requests for modification.

12.2 Expert evaluation basics
The most widely used form of expert review, heurist evaluation, involves expert checking the application systematically against a list of principles, guidelines or ‘heuristic’ for good design. Usability heuristic draw on psychological theory and practical experience.

Examples of design principals—or heuristics:


 * 1) visibility
 * 2) consistency
 * 3) familiarity
 * 4) affordance
 * 5) navigation
 * 6) control
 * 7) feedback
 * 8) recovery
 * 9) constraints
 * 10) flexibility
 * 11) style
 * 12) conviviality

-The list can be summarized by the three over-arching usability principles of learn ability, effectiveness and accommodation.

How many evaluators: Ideally, several (5) with expertise in interactive systems should review. It has been noted that lone evaluators only find 35% of problems. It is also helpful if a severity rating is added. (ex. Scale of 1-3)

12.3 The IMPACT model for user evaluation: expert analysis will not find all problems, particularly those which result from a chain of wrong actions or are linked to fundamental misconceptions.

- The decisions to be made in planning can be encapsulated in the acronym ‘IMPACT’


 * Intention:** deciding the aims for evaluation help to determine the type of data required. It is useful to write down the main questions you need to answer.

The data we are interesting in at this stage is largely qualitative (non-numerical).

Metrics (and measures): Some metrics are helpful in evaluating many types of applications from small mobile communication devices to office systems. Three things to keep in mind when deciding metrics:
 * 1) just because something can be measured, it doesn’t mean it should be
 * 2) Always refer back to the overall purpose and context of use of the technology
 * 3) Consider the usefulness of the data you are likely to obtain against the resources it will take the test against the metrics.

People:
 * The most important people in evaluation are users. Analysis work should have identified the characteristics of intended users, but sometimes you will have to work with the designers to obtain this information.
 * Relevant data can include knowledge of the activities the technology is intended to support, skills relating to input and output devices, experience, education, training and physical and cognitive capabilities.
 * You need 3-5 persons to participate in a test

Summary

 * differing aims for evaluation require different questions to be answered.
 * expert review and end-user testing are both effective, but should be used together as complementary methods.
 * almost any degree of user testing can reveal useful insights, but care must be taken in generalizing from a small number of users.
 * factors to take into account in user testing can be summarized as Intention, Metrics, People, Activities, Context and Technologies (IMPACT).
 * data collection techniques for evaluation mirror those for analysis.