Jakob Nielsen has a very useful June 2006 on how to structure usability tests for your online creation. The article gives you rough estimates of how many users to recruit, how to structure feedback and testing sessions, adn what you can learn from them.
We can define usability in terms of quality metrics, such as learning time, efficiency of use, memorability, user errors, and subjective satisfaction. Sadly, few projects collect such metrics because doing so is expensive: it requires four times as many users as simple user testing.
Many users are required because of the substantial individual differences in user performance. When you measure people, you’ll always get some who are really fast and some who are really slow. Given this, you need to average these measures across a fairly large number of observations to smooth over the variability.
—–
Based on the above analysis, my recommendation is to test 20 users in quantitative studies. This is very expensive, because test users are hard to come by and require systematic recruiting to actually represent your target audience. Luckily, you don’t have to measure usability to improve it. Usually, it’s enough to test with a handful of users and revise the design in the direction indicated by a qualitative analysis of their behavior. When you see several people being stumped by the same design element, you don’t really need to know how much the users are being delayed. If it’s hurting users, change it or get rid of it. You can usually run a qualitative study with 5 users, so quantitative studies are about 4 times as expensive. Furthermore, it’s easy to get a quantitative study wrong and end up with misleading data. When you collect numbers instead of insights, everything must be exactly right, or you might as well not do the study. Because they’re expensive and difficult to get right, I usually warn against quantitative studies. The first several usability studies you perform should be qualitative. Only after your organization has progressed in maturity with respect to integrating usability into the design lifecycle and you’re routinely performing usability studies should you start including a few quant studies in the mix.
