January 5th, 2005
Moderating Remote Usability Sessions
B|P conducts moderated usability sessions. In addition to our revolutionary remote usability technology, our moderating technique is the most visible aspect of our practice to clients. Skillful moderating provides us with the user data we need to answer study research goals.
This document is both theoretical backgrounder and practical guide to B|P moderating techniques. We constantly evolve our moderating techniques; this document will reflect this innovation with periodic updates.
The Four Phases of Moderating
Moderating a usability session with one user consists of four phases:
Summarizing the study goals in your own words immediately before the session focuses your mind on the most important task at hand. Why are you conducting the study? Validate your summary with another member on the team. Your short summary of goals is your anchor during the session. It will keep you on track -- no matter where the session takes you.
- Summarize Goals of the Study
- Build Rapport with the User
- Find the Passionate Task
- Conduct the Session
Building rapport starts when you get on the phone with the user. You establish the trust and roles of the relationship that you need to maintain throughout the session. This can be challenging over the phone, since you do not have non-verbal tools such as body language or eye-contact at your disposal. Establish trust through your interest in them as humans and their product-in-use context. Be an apprentice to their world. Creating this type of relationship will make the user comfortable enough to eliminate their worries about what you are thinking or feeling. This is your goal as moderator.
Rapport Building Techniques
Finding the passionate task follows the building of rapport. Now that the user is aware that you have a genuine interest in how she works and uses the product you are testing, find out more. Make the user talk about how they work, how they use the product. Zero in on the task that seems to be the most important to them. Now make sure this task relates closely to the study goal.
Finding the Passionate Task Technique
Conducting the session should seem to the user as if they are driving, while responding to nonintrusive questions from the moderator. After asking the user to perform a task (that makes sense in the context of the user), the moderator observes and provides input only to:
- Keep the session on track (study goals)
- Have the user clarify or repeat
- Re-establish lost rapport
During the session the moderator’s focus is keeping the flow of the user natural while obtaining the best possible material for later analysis. This material has to provide answers to the study goals.
Conducting the Session Techniques
Rapport Building Techniques
During test set-up time:
- The first minutes with the user on the phone are critical; the moderator needs to communicate their positive regard and empathy while also coming across confident.
- Use the participant’s name and show interest in them.
- Focus on coming across calm and cool even though you are doing a number of different things
- While the Ethnio session is loading, engage the user and begin to understand user motivations
Assuring/Encouraging the User
- “This is exactly the kind of feedback we were hoping to get” – especially when quiet users start to ‘think aloud’
- “This is very helpful – you’re doing just what we need you to do”
- “Thanks for hanging in there…it appears this [process] is harder than it needs to be”
- Once you have assured users that they are doing fine, probe for more information about what’s causing them to be stuck. Keep in mind, when users get stuck they are doing us a favor by uncovering things that aren’t working well.
Keep the User Comfortable
- If you have to interrupt the user, be ready to apologize while remaining confident. You don’t want the user to become standoffish because you keep probing.
- Neutral affirmations When providing short responses use things like “uh huh,” ”okay,” ”yes,” but avoid responses like “correct,” “that’s good” or “Excellent” or “that was a success/successful”
- Be willing to take responsibility for tasks and questions that weren’t clear to the user. Say things like, “I don’t think I was clear about [task or question]”
Finding the Passionate Task Techniques
- Start an easy conversation about the user’s context of use with the tested product. Learn from the user.
- Take time establishing the passionate task.
- Create rapport by being interested in the user.
- Embrace the “master apprentice model” from Contextual Design. As a researcher you are there to learn from the users.
- Form an image in your mind of how the user is interacting with the product in their work context.
- Summarize and validate what you learn from the user.
- Drive the session with: “Does the site meet the user goal?” rather than “Is the user able to perform the tasks we ask her to do?”
- Only tasks that can be tied to realistic user goals make sense. Try to avoid asking users to do things they never would do in their native environment.
- Ask “why” a user wants to do their passionate task.
- Ask: “If you were to summarize why you come to the site…”
- In the wrap-up questions, ask users why they would use the system or site.
Conducting the Session Techniques
- Encourage questions but don’t answer them – if needed, say “That’s a really great question but forgive me if I don’t answer it yet – maybe you’ll discover the answer as you go”
- Use the user vocabulary – If they call an icon something specific, use the same word.
- Use open ended questions – similar to having users summarize instead of summarizing ourselves, get users to elaborate by asking an open ended question.
- What will that do?
- What are you trying to do right now?
- Hmm, interesting, tell me more about that.
- What does that mean to you?
- Listen for nonspecific utterances – when you here hmm, ah, oh, they usually represent “a cognitive iceberg” and it’s a good time to probe.
- Know when to be quiet – When people are thinking, let them think for 15-20 seconds and then probe, or rephrase the question.
- Let the user decide when they are done – unless they are using features unrelated to the study goals.
- End tasks if appropriate – keep the goals of the project in mind, and if we are only concerned about the beginning of a task, end the task early. For example, if we are interested in whether users can locate PDFs, we can end the task after users have located the PDF and before they start diving into the content.
- Be flexible – let the user lead & be ready to go off script
Be led by user exploration. Use the script more as a guide to see if bases are covered.
- Be unbiased about particularly “useful” user comments
Don’t get too excited about user comments that coincide with your own hypothesis. Just record it. This bias will show to client and erode research credibility.
- Ask user to repeat
This is good for obtaining better quality user comments, or for probing deeper. User might say something under their breath that’s negative and embarrassing, make them repeat it.
- Ask user to clarify
Same function as above.
- Keep user in use-mode (not critique mode)
Try to avoid asking users what color they would prefer, how they would design something. Keep users using, not critiquing.
- Allow the embarrassing silence before moving on
When the users stops or does not know what to do, allow the embarrassing silence to occur for a while, before helping user out. This makes for great video showing user behavior as evidence for usability problems.
- Moderate with “harvesting data for analysis” in mind
Think about how the data gathered can be used for analysis – during the session. Is the current user behavior/comment useful to answer study goals, or uncover user motivations?
||By Benjamin Lerch
|| Picture Coming Soon