Usability requirements: how to specify, test and report usability
Specifying and testing usability requirements
How to specify usability requirements
A new Common Industry Format for requirements (CIF-R) is being developed. Copies can be obtained by joining the Industry Usability Reporting project (IUSR).
Defining usability requirements involves three related activities: analysing the context of use, defining task scenarios that can be tested, and specifying requirements for effectiveness, efficiency and satisfaction for each scenario.
Analyse context of use
The first stage is to collect and agree detailed information about:
- Who are the intended users and what are their tasks? (Why will they use the system? What is their experience and expertise?)
- What are the technical and environmental constraints? (What types of hardware will be used in what organisational, technical and physical environments?)
Arrange a half-day meeting. Invite stakeholders who have knowledge about the intended users and usage. This may include:
- project manager
- user representative(s)
The first two are key areas. You will also need a facilitator and a person to record the information provided during the meeting. To obtain information on the context of use, a detailed checklist will be needed.
Discuss and fill in each item on the context checklist. Try to obtain consensus where there is uncertainty or disagreement. If information is missing, agree how this can be obtained. Avoid prolonged discussion of minor issues.
Define scenarios of use
Next identify scenarios of use (use cases) to provide examples of usage as an input to design, and to provide a basis for subsequent usability testing. Scenarios specify how users carry out their tasks in a specified context. To maintain design flexibility, they should not specify what product features are used.
Try to generate scenarios to cover a wide range of situations, not just the most common ones or those of most interest to the design team. Try to include problem situations that will test the system concept, not just straightforward scenarios.
Specify requirements for effectiveness, efficiency and satisfaction
For each chosen task and user type, estimate:
- the acceptable task time and the optimum target
- how to score effectiveness by agreeing what errors the user might make
- the effectiveness target
- the satisfaction target.
If there is an existing or competitor system, it can be evaluated by the selected users and tasks with the objective that usability for the new system should be at least as good as for the old system.
For more information see the methods in the Requirements column in the table of methods.
How to test usability requirements
To obtain valid results, at least 8 representative users need to be tested with realistic tasks. Usability testing needs to be carefully planned and carried out. In addition to assessing whether usability requirements have been met, major usability problems can be identified, including problems related to the specific skills and expectations of the users.
- It is important that the users, tasks and environment used for the test are representative of the intended context of use.
- Select the most important tasks and user groups to be tested.
- Select users who are representative of each user group. 8 or more users of each type are required for reliable measures.
- Produce a task scenario and input data and write instructions for the user (tell the user what to achieve, not how to do it).
- Plan sessions allowing time for giving instructions, running the test, answering a questionnaire, and a post-test interview.
- Invite developers to observe the sessions. If developers cannot be present, videotape the sessions, and show developers edited clips.
- Two administrators are normally required: one to interact with the user, and one to note problems and to speak to any observers.
- If possible use one room for testing, linked by video to another room for observation.
- Observe the user without making any comments.
- Welcome the user, and give the task instructions.
- Do not give any hints or assistance unless the user is unable to complete the task.
- Observe the interaction and note any problems encountered.
- Time each task.
- At the end of the session, ask the user to complete a satisfaction questionnaire such as SUMI.
- Interview the user to confirm they are representative of the intended user group, to gain general opinions, and to ask about specific problems encountered.
- Assess the results of the task for accuracy and completeness.
- Summarise the results of the satisfaction questionnaire, task time and effectiveness (accuracy and completeness) measures.
- Document the results in the Common Industry Format.
- Produce a list of usability problems, categorised by importance (use post-it-notes to sort the problems), and an overview of the types of problems encountered. The summative measures will help place the problems in order of importance.
- Arrange a meeting with the project manager and developer to discuss whether and how each problem can be fixed.
There is more information on the UsabilityNet web site.
Copyright © 2002 Serco Ltd. Reproduction permitted provided the source is acknowledged.