User survey for design
User surveys are a means of finding out how the software or web site is likely to be used by a specific set of users, and who these users are likely to be. The answers user surveys provide must be relevant to the issues that are important to the design team. User surveys are traditionally carried out by post, but increasingly, the internet is used for this purpose.
User surveys only provide benefits if they provide answers to questions that design teams raise. Because it is possible to survey a large number of users, usage profiles from user surveys can be relied upon, if the correct methodology has been used. User surveys are often analysed statistically, and this gives moderately hard, objective data. However, there are many sources of bias and a poorly designed survey can do more harm than good.
The method of carrying out a survey is described differently in different sources. However, the following stages form a common core. It is easy to get lost in the technology of surveying and so the best advice is to learn from experience, guided by the sources recommended below.
Focusing the survey
Find out what are the major decision points, or areas of uncertainty in the thinking of the design team with regard to the usage of the product. Focus in on those areas and find out what needs to be discovered. Involve the decision makers in the development of the survey, find out when they need the information by, and what organisational contexts are likely to be affected by the presentation of the results. See Patton’s book for involving decision makers, and that by Akin et al. for analysis of decision points.
It is useful to employ the Context of Use analysis method as a way of discovering what aspects of the Context of Use are understood and what aspects pose problems or raise uncertainties.
Consider whether some (or perhaps all) of these issues may not be more easily resolved using methods other than surveys (e.g. by placing hit counters on a web site, or by looking at patterns of sales.)
Creating the survey instrument
There are many guidelines on how to formulate questions and how to lay out surveys to make them ‘respondent friendly.’ Remember that information about ‘how often’, or ‘how much’ is often coloured in respondents’ minds, and that if your questions start asking about respondent’s attitudes it will become difficult to interpret the results. Reliable questions focus on simple things that occur relatively infrequently, and on preferences from a fixed set of alternatives. An important element of designing a survey is to develop the concept of trust between yourself and the respondents. Dillman’s book is a good guide to this.
Use open-ended questions sparingly, but always include an ‘Other (please specify):’ option at the end of a list of choices.
Testing the survey
It is absolutely essential to test the survey before sending it out or releasing it. A survey test must be done in conditions as close to the real as possible, down to the issue of whether to include a stamp on the return envelope. Another useful technique is to do a ‘walk through’ the survey with a small number of typical respondents, asking them what they understand by each question as they go through the survey.
Conducting the survey
The sampling frame must be established: everyone from whom you require information must have an even chance of replying to the survey. Sampling theory is complex, and is best left to a statistician, but in essence you have to state how you define the total population from whom you want information, and then how you take an unbiased sample from that population.
A good survey method will target the sample, by sending a warning, then the survey itself, and following up on non-responders with reminders or second copies of the survey. As much as possible, send surveys to individuals rather than anonymously or to roles such as ‘managing director’ or ‘secretary.’ Some kind of reward may also be associated with filling out the survey (for instance, charity donations.) Surveys which follow this kind of methodology can usually bring in much more than the 20% of sample that is usually expected from single mailshot surveys. Reporting rates of up to 70% have been noted.
Analysing the results
How the results should be analysed should be clear from the outset, from the focusing activities. Coding and tabulating the data should be as automatic as possible, so as to rule out possibilities for random error or bias to creep in. A spreadsheet is a very useful tool for keeping raw survey results, which can then be exported to a statistics package such as SPSS. For large surveys (response rates of more than 1,000) a database may be more suitable.
The two most useful statistical procedures to use when analysing survey results are:
- counting of frequencies of response to options;
- cross-tabulating responses to one series of questions or options against another series of questions or options.
Presenting the results
If you have taken care to include the decision makers in the focusing and design of the survey, you will generally find that there is a pressure on you to deliver the results. When presenting results, always give the headline news first; then follow that up with a detailed analysis of how you got there; and finish with a conclusion based on the data. You will naturally form your own opinions or biases as you work with the data. These are important, and you should present these, carefully marking them as your extrapolations from the data so as not to confuse objective fact with your subjective opinion.
Alkin, MC, K Dalliak and P White: Using evaluations: does evaluation make a difference? Sage: Newbury Park CA, 1969.
Patton, QM: Utilization-focused evaluation. Sage: Newbury Park CA, 1986.
Dillman, DA: Mail and internet surveys: the tailored design method (2nd Ed) Wiley, 2000.
The ethnographic method is the only viable alternative to user surveys; this method however is time consuming, laborious, and does not easily yield quantitative information.
The user survey in design has the purpose of decreasing the amount of uncertainty in the design team’s minds. Most of the information from a design survey should find its way into the Context of Use analysis.
The book by Dillman contains many case studies and anecdotes relating to the practice of carrying out surveys. Patton also provides anecdotes of a rather more gritty nature.
Oppenheim, AN: Questionnaire design, interviewing, and attitude measurement (New Edition). Continuum press: London and New York, 1992.
See also the page on User Surveys on this site for another view of how to conduct post-release surveys: both accounts are complementary and support each other.©UsabilityNet 2006. Reproduction permitted provided the source is acknowledged.