How many users do you need for a usability test? The standard answer for the past ten years has been "5." This is due largely to Jakob Nielsen's research on the matter. And, I suppose, many people like this because its much easier to grab 5 subjects than 20 subjects. But, as Larry Constantine reports from the CHI 2003 conference, some people are beginning to argue that the 5 user rule isn't quite right.
True to their aim, the panel reviewed and summarised relevant work already reported elsewhere and previously discussed. As an erstwhile firm believer in the small numbers approach, Jared Spool started the panel by recounting how his views had changed after an experience with one client who insisted on testing with at least 18 users. Expecting to uncover fewer and fewer problems as testing progressed, Spool and company were surprised that new problems were still showing up at about the same pace after 16 users. Their experience was supported by Rolf Molich, whose well-known CUE (Comparative Usability Evaluation) studies have the same software evaluated by a number of different usability testing labs. In the second study in that series, for example, the 7 teams returned almost completely different, mostly non-overlapping results. Of the total 310 usability problems uncovered, 75% were identified by but a single testing team and missed by the others, and only one problem showed up in the findings of every team.
Posted by Karl
May 1, 2003 12:59 PM