Evaluating evaluation: an empirical examination of novel and conventional usability evaluation methods

Software designers and software users have different views of software. Designers see all the parts that make up a system, and which are usually hidden from the users, who only see the interface. Conventional usability evaluation methods strive for objectivity in their measurement of user/system interaction, yet it is often the subjective experience of using a system which is important. Whiteside et al (1988) propose contextual research as a way of getting hold of this type of data, but a number of problems present themselves. It is difficult to observe users in context without intruding and altering the nature of the interaction, especially as much of the interesting parts of an interaction are internal to the user, and not available for direct observation. Methods such as 'thinking aloud' verbal protocols (which may make such phenomena explicit) are both intrusive, and likely to alter the phenomena being observed. The PROTEUS method represents a potential solution, midway between formal, empirical methods (for example measuring behavioural performance on bench mark tests), and direct observation of users in their normal environments. In the PROTEUS method a number of direct observations of user behaviour are made by a shell program, which controls and monitors the presentation of a number of prototypes for a system. Questionnaire data is collected interactively. Users provide their own evaluation of the systems through a process of construct elicitation, using an automated repertory grid program (the Construct Elicitation System). In this way both objective, behavioural data is gathered, as well as subjective evaluative data elicited from users in an open ended manner. During recent co-operative working with an industrial partner, the PROTEUS method was evaluated with a number of other usability evaluation techniques. Keystroke models of the stimulus systems were prepared. A keystroke log of the interaction as recorded. Video records of the interactions were kept, and survey data was collected. This material has allowed a comparison of the different methods to be drawn up.