How to analyze a usability study

Posted by: Josh Walsh in User Experience.

Most of the value you gain from a usability testing session comes from the analysis after the session is complete. I have been involved in a few sessions recently where no formal analysis has been conducted. I believe this is a mistake.

Traditionally, the analysis portion of a usability session takes quite a long time. The end results is usually what I call a “Dead Document.”

“Dead Documents” are reports that have value, but negatively impact productivity. This is usually because it takes too long to create the document, or there isn’t enough time to consume it.

I have developed a quick and dirty system for analyzing the data. It requires only a few hours to create and only a few minutes to consume.

Conduct a quality study

The quality of the results directly depends on the quality of your study. For the purposes of this post, I’m going to assume that your study was conducted properly. (Quality questions, good prototypes, people in your target demographic, etc…).

Use your brain

Even though you were the administrator of the study, you were also a participant. Your brain already has a good idea of the results. Take a few moments to write down your hunches. I like to mindmap, but you may prefer to doodle or freewrite. You should not worry about structuring or organizing your thoughts, just let them flow out of your head and onto paper.

Here are a few examples:

“A few people had a hard time understanding how the dropdown navigation works. I think it’s because they were distracted by the dancing Santa Claus on the opposite side of the screen.”

“It was obvious that people weren’t seeing the ‘logout’ link, even though I thought it was fairly obvious.”

In the next steps, we will try to prove or disprove your theories using the real data you collected.

Prepare to Review

Write down the questions or use cases that you asked your participants to discuss. We are going to grade each response with one of the following values.

• 1 – Indicates that the user was able to complete the test without assistance.
• 0 – Indicates that the user was able to complete the test after some help.
• -1 – Indicates a complete failure. Could not complete the test even with assistance.

In the end, we will sum up the results to come up with a pass/fail score. Before we can do that, we need a baseline score to compare against. Before you review the tapes and notes you need to decide what you think a “passing” score should be.

For example: You were testing a login form with 5 participants. Since this is a crucial feature to get correct, a score of 5 (all participants passed without assistance) is your baseline.

Another example: You were testing the ability to rename a bank account inside an online banking application with the same 5 participants. This is not critical, so a passing grade of 3 may be acceptable to you.

Another example: You asked the 5 participants to explain how they would contact support if they were having problems. A score of 4 would be acceptable here.

I setup a simple spreadsheet to keep myself organized. This easily translates into a deliverable document once completed.

Analyze the Results

Start reviewing your tapes and notes to extract the results of the tests, and plug them into your document. Write down any observations that you think are important. Keep track of things that were said that may be important later.

Sum up the total results across the participants and compare that with your baseline. I like to notate passing and failing tests in red and green.

Write a short paragraph for each test indicating the level of success, and a note about what was done right or what could be done better. Note anything interesting that your participants may have told you.

Write an overview paragraph at the top of the document explaining the results of the test overall. Which areas of the system need the most improvement, and which are working well.

The finished document is usually 3-5 pages of information which can be quickly scanned and understood.