This is Not a Dress Rehearsal

In this business we like to say that the person with the freshest eyes has a perspective that is the least tainted. So as the new kid at Bolt | Peters, I thought I’d talk about my take on this whole remote usability thing.

When I first talked to Cyd about how B|P recruits participants in real time, while they’re in the midst of the task we’re actually looking to test, it was a total eureka moment for me. I realized that live recruiting was a brilliant answer to a pain point I hardly realized I had as a usability moderator.

The way it works is, we throw up a DHTML overlay inviting users to participate directly from the website we’re testing. If we’re testing just a single page or sub-section of a site, we recruit from there. Without moving away from what they’re doing, users answer a little questionnaire, which we use to pre-qualify them. We only call users who meet the recruiting requirements. If we’re fast, we can have users on the phone for the study within minutes of filling out the questionnaire. It’s the ultimate in low maintenance recruiting!

Not only is there no pre-session phone interview, no phone tag, no scheduling users, no emailing out confirmations to remind them to show up where and when they said they would. But you actually get real users, who have experience with the site you’re dealing with. They don’t have to fake it.

I can’t tell you how many marginally-qualified people I’ve moderated in the past and what a drag it can be. In the past, whenever I’ve recruited users to bring them into a lab, I would spend the first part of every session getting the user “in the zone.” “Have you ever posted pictures on the Internet? No? Well, could you imagine a situation where you might want to post pictures some day??” It was like being an acting coach, trying to warm up the user and get them into their role. “Imagine you’re an IT manager and you actually do care about purchasing enterprise software.”

A lot of the users I tested were a little too good at getting into role. Since most of the recruiting was done using Craigslist ads, I’m afraid I had some professional participants on my hands. There seems to be an entire population that’s earning a livelihood from the ETC section of Craigslist. Their unique skill is in working the focus group and usability testing system; the higher the incentive offered, the more they would be able to elaborate on their “qualifications.”

Of course there is something to be said for the adage that any average Joe should be able to navigate your site. And testing with a Craigslist professional is better than no testing at all. But it’s a very different game when you test with real users. Especially when you catch them very close to the task you’re most interested in learning about. People are relaxed and at ease, and their behaviors are an open book. They don’t need to rehearse; they just need to be themselves.

When we call them up, users often seem pleasantly surprised at actually “winning” something off one of those pop-up boxes (We do offer 75 Amazon bucks for participating). It’s like they’re the subject of a reality TV spot for the duration of the interview. Since they’re in their own space, users sometimes try to multi-task while they’re on the line with us. We see this as a definite plus. We’re peeping into the messiness of their everyday lives and unpredictable online habits. This gives us insights into their actual experience that we just couldn’t get any other way.

One question I did have at the beginning was about whether people would really drop everything, in the middle of their work day to do a 45-minute usability session? What I learned after just a few sessions is that the answer is almost always: “For 75 bucks? You betcha.” That much hasn’t changed. “Lemme just close my office door here, and I’d be glad to talk with you…”