This is Not a Dress Rehearsal
In this business we like to say that the person with the freshest eyes has a perspective that is the least tainted. So as the new kid at Bolt | Peters, I thought Iâ€™d talk about my take on this whole remote usability thing.
When I first talked to Cyd about how B|P recruits participants in real time, while theyâ€™re in the midst of the task weâ€™re actually looking to test, it was a total eureka moment for me. I realized that live recruiting was a brilliant answer to a pain point I hardly realized I had as a usability moderator.
The way it works is, we throw up a DHTML overlay inviting users to participate directly from the website weâ€™re testing. If weâ€™re testing just a single page or sub-section of a site, we recruit from there. Without moving away from what theyâ€™re doing, users answer a little questionnaire, which we use to pre-qualify them. We only call users who meet the recruiting requirements. If weâ€™re fast, we can have users on the phone for the study within minutes of filling out the questionnaire. Itâ€™s the ultimate in low maintenance recruiting!
Not only is there no pre-session phone interview, no phone tag, no scheduling users, no emailing out confirmations to remind them to show up where and when they said they would. But you actually get real users, who have experience with the site youâ€™re dealing with. They donâ€™t have to fake it.
I canâ€™t tell you how many marginally-qualified people Iâ€™ve moderated in the past and what a drag it can be. In the past, whenever Iâ€™ve recruited users to bring them into a lab, I would spend the first part of every session getting the user â€œin the zone.â€ â€œHave you ever posted pictures on the Internet? No? Well, could you imagine a situation where you might want to post pictures some day??â€ It was like being an acting coach, trying to warm up the user and get them into their role. â€œImagine youâ€™re an IT manager and you actually do care about purchasing enterprise software.â€
A lot of the users I tested were a little too good at getting into role. Since most of the recruiting was done using Craigslist ads, Iâ€™m afraid I had some professional participants on my hands. There seems to be an entire population thatâ€™s earning a livelihood from the ETC section of Craigslist. Their unique skill is in working the focus group and usability testing system; the higher the incentive offered, the more they would be able to elaborate on their â€œqualifications.â€
Of course there is something to be said for the adage that any average Joe should be able to navigate your site. And testing with a Craigslist professional is better than no testing at all. But itâ€™s a very different game when you test with real users. Especially when you catch them very close to the task youâ€™re most interested in learning about. People are relaxed and at ease, and their behaviors are an open book. They donâ€™t need to rehearse; they just need to be themselves.
When we call them up, users often seem pleasantly surprised at actually â€œwinningâ€ something off one of those pop-up boxes (We do offer 75 Amazon bucks for participating). Itâ€™s like theyâ€™re the subject of a reality TV spot for the duration of the interview. Since theyâ€™re in their own space, users sometimes try to multi-task while theyâ€™re on the line with us. We see this as a definite plus. Weâ€™re peeping into the messiness of their everyday lives and unpredictable online habits. This gives us insights into their actual experience that we just couldnâ€™t get any other way.
One question I did have at the beginning was about whether people would really drop everything, in the middle of their work day to do a 45-minute usability session? What I learned after just a few sessions is that the answer is almost always: â€œFor 75 bucks? You betcha.â€ That much hasnâ€™t changed. â€œLemme just close my office door here, and Iâ€™d be glad to talk with you…â€