As the grey hairs have collected, I have heard a range of arguments over the years about how many testers are enough for user testing, (or even alpha and beta testing). Some argue that you should have large samples for statistical certainty, while others argue much less. I am a much less kinda guy?
So how “much less” is enough for user testing? I’m of the mindset that 3-5 is fine as long as you are very organized in your testing depending on how much you need to get in the weeds with your test. Three users usually cover about 75% of the problems while 5 will increase that yield to 85%… a good grade especially when you are pressed for time. This post largely deals with those who are pressed for time.
Would I do the same number of users even for alpha and beta tests? Yup (although I would push toward 5 in most cases). From my point of view, if you are organized and make sure you guide the testing to focus on key workflows, you will be able to quickly identify most major problems. Continued review will only reveal less and less critical issues.
Think about it. If I wrote a sentence that said “this Is saf;ljknot a gud way to right a sentanz”… you could quickly tell me, as could most other people, what I did wrong. I don’t need an army to see the problems. The same is true for software. Consequently, “statistically valid samples” are not necessary because most problems are easily demonstrated from simple use and observation.
What are the benefits of a focused test with 3-5 testers?
- You can schedule your tests and render the results in the same day.
- You limit your feedback to a focused package of data that is easy to manage.
- A small number of users usually discover your worst issues very quickly.
- You can focus the testers on specific workflows starting with the most critical and working back.
- Quick tests with small groups can be very focused allowing you to clean up your code in a series of rounds rather than one giganta-round.
- In most cases, you’ll cover 75%-85% of all your problems.
- The list goes on, but you get the point…
Now for beta testing, you might say that this is not valid. I disagree… although I would ideally push for around 5. If I couldn’t get 5 and I was pressed for time, I would settle for no less than 3 and them make sure I worked closely with my beta testers handling the tests almost like extended user tests.
What are some things I would do in organizing a “controlled” beta? Here they are:
- Identify my most critical to least critical workflows organizing my unbiased work oriented questions in advance.
- ID and organize my beta testers explaining exactly how the testing is to be done, when it will be done and how long it will take. If there are a schedule of tests over time, then the users would know that in advance.
- Organize any observers well in advance setting their expectations too.
- Confirm the users before each test through email (and phone if they do not reply).
- Test and compile the data in one day so that it can be fed into the agile machine for processing and production.
- Reward and thank the users for the effort so they go away as positive ambassadors for my company.
Of course, real testing always depends on what you need to achieve. Therefore, if I needed to do more traditional beta testing, I would do it. But my point here is that often we don’t have the time, so doing it in a quick and highly focused manner should quickly get you most of the information you need to improve your product.
Finally, since many people like to see what others think, here is of list of articles with commentary that agrees with my thinking at least partly or in full:
- Why you only need to test with 5 users, Nielson, 2000: A very detailed study of diminishing returns as you increase the number of users.
- Why you only need to test with 5 users, Measuring Usability, 2010: Again, a good study looking at the raw numbers of usability testing.
- Steve Krug: You’re NOT doing usability testing? Are you nuts?, Learning LibTech, Univ of Toronto: Notes from a presentation by Steve Krug on the benefits of a 3 person sample size. The same information can be found in his book “Rocket Surgery Made Easy.” I quote this article here because I couldn’t post the book.
