Categories
blog Live Recruiting Remote Research Uncategorized

Why Are You Still Scheduling Participants?

Most  user researchers feel they get the best results if they recruit and schedule study participants themselves.  We agree. There’s an art to it, which makes it hard to outsource, but it takes a lot of time.  An awful lot.  Take a minute and think about how many hours you or the people you work with on user research spent on recruiting and scheduling last year. And if you’re conducting guerilla-style research with friends and family or any kind of participants, then we love what you’re doing, but this is for the projects that have more specific criteria and typically require an agency or detailed recruiting effort.

If you’re doing that, please click through for more discussion and a poll.

Categories
blog Remote Hacks Remote Research

The Super Technical Guide to GoToMeeting for Remote Usability on a Mac

When Whitney Hess tweeted this week about needing help with a remote usability test, we jumped on the opportunity to help. “Simple,” we said! “That’s what we do everyday!”

Need usability testing help!

Well, not exactly simple but do-able without too much effort.

Here’s what Whitney told us she needed:

  • A way to call a user in her home
  • Have that user share her desktop (Mac or PC)
  • Ask the user a few questions, and observe what the user does
  • Record both the audio and video in a single file — on a Mac
  • Take notes all the while
  • Oh, and make it low cost while you’re at it

And here’s what we told Whitney:

diagram

First, you’re going to need the follow “equipment”:

  • A laptop and extra monitor
  • A USB headset
  • A set of headphones
  • GoToMeeting (GTM)
  • iSHowU HD
  • Optional: An Fiio amp to boost the audio in the headphones

Then, setup GoToMeeting to manage the screensharing and audio. We keep a persistent meeting ID in use, which we bookmark on a landing page. This landing page is where we go to launch the meeting (as the organizer), where our clients connect, and where we direct our users. GoToMeeting doesn’t use a browser plugin; instead they use Java or ActiveX to deliver an application that runs at the OS level. It works on both Macs and PCs, although Mac users can only share their primary desktop.

These are the settings you want to use in GTM:
diagram-GTM

Basically, you want to disable a bunch of features to make it feel like it’s just you and the user in the meeting. This includes disabling the Attendee List and turning off Entry/Exit beeps. Unfortunately, GTM always announces the number of callers on the line when the user connects either by phone or mic & speakers.

Audio beeps off

Pay attention to the Audio Output setting: you have to hijack the audio from GTM and send it to iShowU HD to record. In fact, you’ll need to install that driver (Soundflower 2ch) to allow you to record audio from native applications.

GTM - audio settings

“MUTE ALL” is also this wonderful trick we learned from the good folks at GTM. If you do it the very first thing, then everyone who joins the meeting afterwards will come in muted by default! This lets you rest assured that any clients who connect late won’t be heard by the participant.

Next, use iShowU HD to record just a portion of your screen. You can use pretty standard settings for video capture, but make sure you choose the correct Audio settings: turn on ‘Audio from Applications’ to capture that audio stream from GTM that we hijacked and the Mic Input from your Headset (DSP55).

diagram-ishowu

This is what the Advanced -> Audio panel should look like:

iShowU audio settings

Finally, make sure your Mac System Preferences are set properly. This is used to make sure you can hear and communicate during the call! Set the output to your headphones, and the input to your USB headset.
diagram-Sysprefs

Here’s what the settings look like:
sys-sound

Good luck getting things set up! This actually works brilliantly — but we’re always on the lookout for new and better solutions. Let us know what you’ve found that works for you!

Categories
blog HCI Remote Research Research innovation usability

Usertesting.com Review by Rolf Molich

[Guest author: Rolf Molich owns and manages DialogDesign, a small Danish usability consultancy that he founded in 1993. Rolf conceived and coordinated the Comparative Usability Evaluation studies CUE-1 through CUE-8 in which almost 100 professional usability teams tested or reviewed the same applications.]

Usertesting.com – A usable and useful recruiting service

Usertesting.com offers unattended, 15-minute usability test sessions at reasonable prices with participants from the US, Canada and the United Kingdom. I have some experience with their service. I am pleased with it, but you should be aware that they really excel at recruiting users from the most broad demographics to spend time looking at your site, not usability analysis.

Usertesting.com home page
Usertesting.com home page

“Use it and your site will get better”

On their home page, Usertesting.com quotes Evan Williams, Twitter Co-founder: “Use it and your site will get better”. I respectfully disagree. Usertesting.com is a great recruiting service, but a skilled usability professional is still required watch the videos and extract useful usability recommendations. The opinions from the participants are mostly worthless. Participants’ behavior, not their opinions is important.

I used Usertesting.com in a recent comparative usability measurement study of Budget.com. Usertesting.com ran about 30 sessions for me with about 20 users. Since my five car rental tasks on Budget.com took about 25 minutes to complete, I had to split the tasks into two sessions and ask their customer service to get the same test participants to do two sessions with different tasks. This worked like a breeze, but it took me some extra time to handle and write instructions for two sessions.

19$, 29$, 39$, and counting?

At the time of my study in April 2009, Usertesting.com was charging just 19$ per test participant. Shortly after, they raised the price to 29$ per test participant. Now they have announced yet another price increase to 39$ per test participant. At 19$ per participant their price/performance ratio was amazing. At 29$ it was good. At 39$ I am not so sure, considering that you only get 15 minutes test time. At 4 x 39$ = 156$ per hour their service is less competitive compared to traditional recruiting. Despite their price increases, they still pay their participants 10$ per session.

High quality

Amazing turnaround time. In late December 2009 I used Usertesting.com to run three test sesssions of the Enterprise.com car rental service. The test sessions were completed within one hour from the time I submitted the request. Because of the time difference between the US and Denmark where I live, I sometimes submit requests at 4 am East Coast time, but even these requests are honored almost instantly.

Their test participants are great. So Usertesting.com’s screening process seems to work. One or two participants out of 25 were rather talkative and offered 10 minutes of worthless personal opinions about the website in addition to their helpful task solutions. Fortunately, this happened rarely and I simply filtered out the opinions and looked at actual behavior.

Video and sound from their recordings is good. I never had a problem understanding what a participant was saying or doing. See the sample screen from a recording below. In contrast, I’ve had professional usability consultancies who without blushing sent me usability test recordings where the video was blurred or the sound was a loud humming.

Screen showing usertesting.com video replay
Screen showing usertesting.com video replay

Their customer service is excellent. I got a fast and sensible response each time I asked them a question or submitted a comment.

Their customer service told me that their agreements with participants allow the clients any use of the recordings they want – including public presentation of the videos. This seems ethically defensible because you can’t see participants’ faces and you mostly don’t get their names.

A few limitations and caveats

For unattended test sessions where no moderator is available to correct misunderstandings on the spot, you may want to test your instructions to test particpants carefully. You must provide your own pre-session interview questions and the all-important debriefing questions.

Usertesting.com only provides rudimentary demographic information about participants as shown in the screenshot below. If you want information about approximately where participants live, what their profession is and if they’ve used the site or similar sites before, you must use precious test time to ask in your test instructions.

Rudimentary participant demographics on usertesting.com
Rudimentary participant demographics on usertesting.com

Usertesting.com’s setup allows the participant to interrupt the recording at any time. This may be OK for qualitative testing, but it spoiled a few of my task time measurements. Also, in a few cases test participants seemingly left the session for about 5 minutes without explanation – no sound and nothing moved on the video. In one case it seemed to me that a test participant had rehearsed the test questions before they clicked the recording button and started the real session. I must add, though, that if I expressed just the slightest bit of dissatisfaction with a participant, their customer service immediately refunded me what I had paid. Of course moderated sessions aren’t perfect either – and most often they don’t have a money back guarantee.

Some of my colleagues have argued that Usertesting.com offers “professional” test participants. It is true that on 2-3 occasions test participants inadvertently showed me their test session dashboard which showed that they had been doing about 10 test sessions recently. Personally, I don’t mind this. I have yet to see a participant who was so experienced in user testing that it prevented me  from getting the data I wanted.

All in all: Recommended

Usertesting.com has a great concept and implements it with care. Before their recent price increase, I would even have said “Highly recommended”.

Resoures

www.usertesting.com

The comparative study of Budget.com that I used usertesting.com to test (CUE-8).

Categories
blog Remote Research

Using Remote Research to Inform Social Interaction Design (SxD)

[Guest author: Brynn Evans is a digital anthropologist, design researcher, and author who studies social interaction design and social search. She extends a thousand thanks and a bear hug to Tony Tulathimutte for help in editing this post!]

What is social interaction design?

Social interaction design (SxD) is the practice of designing for person-to-person interactions mediated by a computer interface, going beyond pure usability and human-computer interaction. Even fairly solitary experiences like editing a Wikipedia page occur in a social context in which other users’ past interactions influence what new editors contribute.

“It’s the interactions among users that informs design.” —Adrian Chan

sxd sketch
[Sketch and original photo by Kai Chan Vong]

What’s a good example of an SxD problem?

conversation threadVark.com is a question-answering service that routes users’ questions to people in their extended networks who may have relevant knowledge of the topic. The original service operates through IM, Twitter, and email; more recently an iPhone app has been developed.

Let’s consider the difference between the mobile and desktop experiences of Vark.com. Both asking and answering activities work rather well in desktop email and IM. In contrast, responding on-the-go is awkward—more often than not, we’re distracted, hurried, or unable to type a coherent answer without bumping into a fire hydrant.

There’s also an assumption that the answer resides solely in our heads, when in reality, providing an answer often requires sharing links or performing a quick search—that is, we may not have the answer immediately on hand, but we know where to look.

Furthermore, successful answers often manifest as conversations on the desktop (example above), in which messages are exchanged in a back and forth manner so that the questioner can clarify her question and the answerer can refine her response. This type of sustained interaction is much harder to establish with on-the-go users.
mobile vark
Finally, iPhone prompts (right) often lack enough information about the nature of the question or your relationship to the questioner. One reason for Vark’s success is that it seeks out answers from people within an extended, personal network, naturally building trust and accountability into the system. But without knowing how you know the questioner, the iPhone app experience feels instead intrusive and disruptive, and lacks any strong social motivator to respond.

Why is remote research useful for SxD?

Traditionally, user-centered designers conducted field studies or shadowed someone to learn more about their practices. The digital space complicates matters—not only is it difficult to shadow someone, but people’s actions are so fluid and varied that it’s hard to isolate specific behaviors in order to study them.

Remote research has emerged as a great way to do needs-finding for SxD, for three reasons:

First, it’s hard to recreate interactions between two or more people in a lab setting. Last year when I was studying user interactions during social search tasks, I realized that I needed to talk to multiple people: both the user who posed the question as well as the people who provided replies. I started by observing the questioning process: how the question was phrased, which communities or individuals were questioned, the historical relationships between the parties. Then I explored the answering process: answerers’ perception of the request, why they chose to reply, if they had a history of interacting like this.

What’s interesting is that answers provided over social networking sites (like Twitter and Facebook) were mostly jokes or “nudges” to attract the user’s attention (“Hey, remember me?”). But answerers in private channels (email, IM, phone) were more serious and thoughtful because people were contacted directly and had longstanding relationships with the user (“She asked me personally, and she’s helped me in the past”).

Second, social interactions unfold over time, and their repercussions aren’t always apparent in a hour-long lab study. I recall one user in my social search study who asked a question on ping.fm. He received a prompt reply which “seemed right”, so he reported it as his “final answer”. I followed up two days later to see if he had received any other replies. In fact, the conversation thread on ping.fm had progressed, and the community had collectively concluded that the earlier reply was incorrect. This observation was only made possible by the passage of time.

Third, social interactions are best understood within the context where they occured. Not just physical location, but also past history (between the people interacting) and reasons for having the interaction. For example, my sister tweets about her new startup, but I’m not familiar with her a field and don’t have a professional relationship with her, so I seldom reply to her tweets. However, when she emails, calls, or writes on my Facebook wall, I reply instantly—even on an unfamiliar topic. If you were only studying my Twitter use, you might wrongly conclude that I’m an ingrateful sister, but this interpretation would be taken out of the full context of my relationship with her.

Thus, whether you’re designing for healthcare, fitness, games, dating, or online privacy, it’s critical to gain insight into where, when, and why people to act the way they do. Community engagement through social media will differ substantially depending on people’s personalities, reputation, location, local culture and rules, nature of their relationships, and history of the community. Remote research methods—like experience sampling, remote observations, and critical incident surveys—are great tools for understanding the many facets of social behavior, and suggest productive avenues for pursuing SxD.

Additional resources:
Social interaction design salon (group blog)
Digital ethnography for social interaction design (slide show)


Categories
blog Remote Research

Give the Gift of Remote Research

82238906_011c194b49_b

In the past few weeks, remote user research has been touted as a solution to climate change and a swine flu containment strategy—but did you know that it also makes an amazing holiday gift?

We’re giving 15% off in December on the following services:

  • Live moderated remote research: We recruit your website’s visitors using Ethnio and call them right away to conduct a live, 40-minute behavioral research study. You get to watch and participate while it all happens. Great for formative design research, usability research, and design validation!
  • Automated research: capture the behavior of hundreds of users on specially-designed tasks to get answers to specific usability questions.
  • Expert design evaluation: After nearly 10 years and over 200 user research projects, we can evaluate your product for major and minor design and interaction issues.
  • Remote research training: Want to learn how to recruit users live from your website, use screen sharing tools, and record and edit remote research sessions? We’ll train you.
  • Weird, insane user research projects: Got a user research project that you have no idea how to approach? We’re experts at finding new ways to research stuff—check out our automotive and video game research projects for examples of the crazy methods we’ve come up with.

Imagine the look on your design team’s collective face as they get answers to their toughest interaction design problems. I mean, what else were you going to get them, a leopard-print Snuggie? A Chia Obama?

To get started, fill out the form below, and we’ll get back to you, usually in less than a day!

(Photo via H Dickens on Flickr)

Categories
blog Conferencery Live Recruiting Moderator Strategies Presentations and talks Remote Research

User Research Friday, Nov. 7th at Mighty!!!

USER RESEARCH FRIDAY is upon us!

MORE INFO!

REGISTER NOW!

The Lineup

Indi Young – Author, “Mental Models: Aligning User Strategy with Business”

Talk: “How Mental Models Helped Teams Do What They Dreamed”

Steve Portigal – Portigal Consulting

Talk: “Research and Design: Ships in the Night?”

Dan Saffer – Author, “Designing the User Experience”

Talk: “How to Lie with Design Research”

Aviva Rosenstein, PhD

Director of User Experience Analysis, Ask.com

Talk: “Fake Ethnography vs. Real Ethnography”

Dr. Kris Mihalic

Sr Research Manager, Yahoo! Mobile

Talk: “What Mobile Research Accomplishes in 15 Minutes”

Nate Bolt – El Presidente, Bolt | Peters

Intro and what-not

Categories
blog Moderator Strategies Remote Research

Remote is Better, pt. 2: We’re in ur computer

In our continuing “Remote is Better” series, we explain yet another benefit of remote research methodologies: since you get to talk to people who are using their own computers, you get to see all the fascinating stuff on their computer desktops and web browsers (with their permission, of course). Since we use UserVue to screenshare with the users we talk to, we can see everything on a users’ desktop while we talk to them–this is a great way to get your users talking about sites and programs they use the most, and it always gives you a sense of their computer experience and usage. (While we can’t post any of the desktops we’ve seen, I’ll be glad to offer my own: see below!)

Bookmarks. Forget personas–if you want a real in-depth look at how people use their computers, just take a look at their bookmarks. How many are there? How are they organized–are they organized? What’s especially fascinating is if you’re talking to one of your website’s power users: what other bookmarks do they have that are similar or related to your site? Get your users talking about the last time they visited those sites. You get a surprisingly rich idea of what kind of internet users they are, and what they really want out of your site.

Desktops. If bookmarks give you a good sense of the kind of things your users do online, desktops are a great way of seeing what they do offline: at a glance, you can see the shortcuts, program links, and files they’ve stowed at arm’s reach. You can even get personality hints: are you dealing with a busy professional type with a clutter problem? A compulsive neatnik? A power-using settings-tweaker? Even the desktop backgrounds can give you unexpected human touches: family photos, artwork, and vacation photos can all give you a feel for the user. Even the factory default wallpaper tells you something: Here’s a user who doesn’t want to spend all day fiddling with settings; or, here’s a user who doesn’t know how to change their background.

Usage. More than just the stuff that’s physically present on their computers, you have to pay attention to how they’re using it. Do they have a million other tabs open at the same time as yours? Do they have a computer that’s so slow, they can finish an entire Reuben sandwich while your page is loading? Do they always use their bookmarks to go straight to a page that’s four clicks deep into your navigation? Pay attention–it’s all good stuff.

Ethics. Your users are letting you into their lives; don’t abuse the access. There’s a fine line between harmlessly observing users’ technological ecosystems, and invading their privacy. Here’s one big Do and one big Don’t for gauging whether you’re on the side of Good nor Not Good, but as always, let your conscience be your guide:

DO be transparent. Users should know before the session begins exactly the kind of access you’ll have to their computer, whether or not you’ll record the session, and what you’ll do about the recording. If they’ve got questions or reservations, don’t try to strongarm them into consenting–after all, if you’re doing live recruiting, you can always just snag another user.

DON’T invade people’s privacy. Avoid bringing up any material that might make the user feel like they’re being intrusively scrutinized, or which has the potential for awkward situations. Overstepping your bounds can make your user feel anxious, untrusting, and defensive, where what you want is casual, warm, and relaxed. Examples of things you DON’T want to say: “Hey, I see there in your web history that you just went shopping for underpants! Want to tell me all about it?”, “Can you show me what’s in all those image files on your desktop?”

Keeping your eyes open for the right details on your users’ computers can help give you a more complete picture of the real people who are using your website or software product. We’re all for anything that puts
the emphasis back on treating users like people, rather than like talking pinatas that you whack opinions out of.

Categories
Best Practices blog Moderator Strategies Remote Research

Remote is Better, pt. 1: Getting Clients’ Hands Dirty

In this inaugural entry in our B|P continuing series “Remote is Better”, we discuss how separating the moderator and the user eliminates the need for “two-way mirrors”–now you can get your clients into the driver’s seat with you (metaphorically). We show you how!

Lots of people think of remote research as a trade-off or a compromise–a cheap, quick alternative for when you can’t get users in the lab face-to-face. What often gets overlooked are the many, many qualitative benefits of testing remotely: if done properly, remote research can give you all kinds of data and insight that would be impossible to get otherwise. Of course, doing it properly means you need to know what you’re doing. Wouldn’t it be nice if there were people around with years of remote research experience, who were nice (or dumb) enough to give away all their best practices on their official blog?

Categories
blog Remote Research

Remote Tasting, or: We Brought Mike to Lunch


We wanted a way to include our B|P homie Mike in our holiday lunch, but he works from London now, and flying him out just for lunch seemed a tad much. So instead, we decided to bring him with us remotely. All we needed was one laptop with a big-ass battery, one EVDO revA card, two skype accounts w/ the new MPEG4 video codec, one booster to place at his chair so the laptop would be at table level, and one friendly waitress at the Slow Club, in San Francisco.

Mike and his wife Natalie ate Indian food at the same time we ate our lunch, and we could barely hear what he said during lunch but it was overall awesome. Seriously. Full set of photos on flickr


Categories
Best Practices blog Remote Research

HOW-TO Use UserVue Internationally

At the clubs, people are always asking the same thing: “I love UserVue for remote user research, but I hate that I can’t use it internationally!” For those unhappy party people, here is a guide to dialing internationally with UserVue.

WHAT YOU NEED:

  • Desk phone
  • Computer w/ Skype and UserVue
  • Microphone headset for the computer
  • Moderator
  • Twix brand chocolate bar

LET’S DO THIS:

STEP 1:

Download and register for an account.

STEP 2: Login to Skype and start a UserVue session, like normal.




STEP 3:

Click “Call” in UserVue.


STEP 4:

Put your desk phone number in as Your Phone Number put the Skype call-in number as Participant’s Phone Number, and then click “Dial”.



STEP 5: Have the first Twix bar (there are two to a package). You’ll need the energy for the next few steps.


STEP 6:

Pick up your desk phone when it rings, and then dial “1” when the cold mechanical voice prompts you.

STEP 7:

Answer the call in Skype. Your desk phone will be connected to your Skype line.

STEP 8:

Mute and silence the desk phone. You won’t be speaking through it at all.

STEP 9:

In Skype, click Add Caller, and enter the participant’s number. If the built-in nternational line menu doesn’t work for any reason, then try setting the country to US and then typing in 011, the desired country code, then the participant’s number. Click Call.

STEP 10:

When the user picks up, you win. Direct him/her through the UserVue process like normal.

STEP 11:

Consume the second Twix bar–you’ve earned it. (There should be no more Twix bars. If you still have a Twix bar remaining by the end of this study, then you did it wrong, and the police will already be on their way to your house.)

(OR JAIL)

A WORD OF WARNING

Though it works and you can usually talk to your users just fine, there is a drop in the sound quality of the UserVue recordings. Also, the usual caveats apply when calling internationally with Skype, i.e. it drops out sometimes. You have been warned.