Tag: screensharing

Categories
blog Research innovation

How B|P Researched Spore

b|p gaming observation room

So we’ve been beating around the bush for the past year or so about this top secret game research project of ours, and now that the game has finally hit the streets, we’re bringing the news to you in all its full glory. How did we test the game? Who did we talk to? Do we have lots and lots of video to back it up? Let’s start from the beginning.

[vimeo] http://vimeo.com/1704123 [/vimeo]

Focus groups suck

Basically our highest priority for this study was to stay far, far away from the usual (and sadly dominant) market research-based focus group-style, ahem, bullshit. You know how it is: sterile lab environment, two-way mirrors, cubicles, soul-killing focus-groupthink, creepy guy with a clipboard standing behind you, nodding. We were not going to do that. No way, no how, no focus groups.

Revenge of the SNERD

So what did we do, then? Since we weren’t able to actually go out to users’ homes (though we definitely pushed that idea), our Dear Leader Mr. Nate Bolt formed SNERD (the Simulated Native Environment Research Division) and set out to create a lab environment that was as similar as possible to a typical gaming setup: a desk, a chair, a laptop, and no other participants, research moderators, or observers to distract them. Unlike a focus group study, where participants would play the game for an hour and then talk about it in a group for another hour (ref. aforementioned bullshit), we had our participants play for six hours over two nights, without any explicit directions or instructions, except for one, which was to think-aloud as they played. We used TeamSpeak to hear what they were saying, and only occasionally dropped in with questions.

EA gets in on it

We, the research moderators and the observers from EA, were in a separate room, where we broadcasted the game screens, player’s faces, and TeamSpeak audio live. Even the EA peeps got in on the act: if they had questions, they let me know, so that I could address the players myself and rephrase any potentially leading or distracting questions (“Are you having a lot of fun now?”) into neutral moderator-speak (“Tell me what you’re doing here.”). This way, they were able to figure out what mattered to them, without swaying the feedback one way or the other.

PENS is mightier than the pen

Since we mostly do qualitative research, we don’t do many surveys and questionnaires, so to get the quantifiable data EA wanted, we worked with a great company in NY called Immersyve, who designed questionnaires and analyzed them using a set of fun and engagement metrics called PENS (Player Experience of Need Satisfaction). To keep players from getting bored or distracted when filling out the surveys, we used unobtrustive touchscreens instead of pen and paper, kept the surveys brief (<2 min), and instead of interrupting them, we asked them to fill the surveys out whenever they wanted to take a break from playing whatever phase they were on.

Player station

Oh yeah and it was also kind of insane

Not gonna lie, stuff got pretty B-A-N-A-N-A-S. It took practice to wrap our heads around the setup: six stations in different parts of the building, broadcasting 18 live streams of video to the observation room plus six streams of audio (gameplay video over VGA, webcam video over VNC, live touchscreen view over VNC, and TeamSpeak Chat over IP) while recording the gameplay, voice, and webcam data at the same time. I had to keep an eye on everyone’s progress, listen equally to all six users, take time-stamped notes so I could analyze the videos later, and keep an eye on my computer’s performance, since it was running something like five million screensharing applications. (On some of our first run-throughs, we had so much equipment running that we blew out the power in the entire building. Sploops.) Of course, from the gamers’ perspective, they were just kicking back and playing Spore.

393 Hours of HD Gamer Video

And that, in a nutshell, was it! Over the course of a year, we tested 59 users, logging 393 hours of individual gameplay. And to top it all off, we recorded the whole damn thing: player footage, conversations, and gameplay. The whole point of the study was to get the gamers comfortable so they’d say and do the kind of things they might not do in a self-consciousness provoking focus group or lab environment. Don’t believe us? Check it out:
[vimeo] http://vimeo.com/1704058 [/vimeo]

Categories
blog Moderator Strategies Remote Research

Remote is Better, pt. 2: We’re in ur computer

In our continuing “Remote is Better” series, we explain yet another benefit of remote research methodologies: since you get to talk to people who are using their own computers, you get to see all the fascinating stuff on their computer desktops and web browsers (with their permission, of course). Since we use UserVue to screenshare with the users we talk to, we can see everything on a users’ desktop while we talk to them–this is a great way to get your users talking about sites and programs they use the most, and it always gives you a sense of their computer experience and usage. (While we can’t post any of the desktops we’ve seen, I’ll be glad to offer my own: see below!)

Bookmarks. Forget personas–if you want a real in-depth look at how people use their computers, just take a look at their bookmarks. How many are there? How are they organized–are they organized? What’s especially fascinating is if you’re talking to one of your website’s power users: what other bookmarks do they have that are similar or related to your site? Get your users talking about the last time they visited those sites. You get a surprisingly rich idea of what kind of internet users they are, and what they really want out of your site.

Desktops. If bookmarks give you a good sense of the kind of things your users do online, desktops are a great way of seeing what they do offline: at a glance, you can see the shortcuts, program links, and files they’ve stowed at arm’s reach. You can even get personality hints: are you dealing with a busy professional type with a clutter problem? A compulsive neatnik? A power-using settings-tweaker? Even the desktop backgrounds can give you unexpected human touches: family photos, artwork, and vacation photos can all give you a feel for the user. Even the factory default wallpaper tells you something: Here’s a user who doesn’t want to spend all day fiddling with settings; or, here’s a user who doesn’t know how to change their background.

Usage. More than just the stuff that’s physically present on their computers, you have to pay attention to how they’re using it. Do they have a million other tabs open at the same time as yours? Do they have a computer that’s so slow, they can finish an entire Reuben sandwich while your page is loading? Do they always use their bookmarks to go straight to a page that’s four clicks deep into your navigation? Pay attention–it’s all good stuff.

Ethics. Your users are letting you into their lives; don’t abuse the access. There’s a fine line between harmlessly observing users’ technological ecosystems, and invading their privacy. Here’s one big Do and one big Don’t for gauging whether you’re on the side of Good nor Not Good, but as always, let your conscience be your guide:

DO be transparent. Users should know before the session begins exactly the kind of access you’ll have to their computer, whether or not you’ll record the session, and what you’ll do about the recording. If they’ve got questions or reservations, don’t try to strongarm them into consenting–after all, if you’re doing live recruiting, you can always just snag another user.

DON’T invade people’s privacy. Avoid bringing up any material that might make the user feel like they’re being intrusively scrutinized, or which has the potential for awkward situations. Overstepping your bounds can make your user feel anxious, untrusting, and defensive, where what you want is casual, warm, and relaxed. Examples of things you DON’T want to say: “Hey, I see there in your web history that you just went shopping for underpants! Want to tell me all about it?”, “Can you show me what’s in all those image files on your desktop?”

Keeping your eyes open for the right details on your users’ computers can help give you a more complete picture of the real people who are using your website or software product. We’re all for anything that puts
the emphasis back on treating users like people, rather than like talking pinatas that you whack opinions out of.

Categories
Best Practices blog Moderator Strategies Remote Research

Remote is Better, pt. 1: Getting Clients’ Hands Dirty

In this inaugural entry in our B|P continuing series “Remote is Better”, we discuss how separating the moderator and the user eliminates the need for “two-way mirrors”–now you can get your clients into the driver’s seat with you (metaphorically). We show you how!

Lots of people think of remote research as a trade-off or a compromise–a cheap, quick alternative for when you can’t get users in the lab face-to-face. What often gets overlooked are the many, many qualitative benefits of testing remotely: if done properly, remote research can give you all kinds of data and insight that would be impossible to get otherwise. Of course, doing it properly means you need to know what you’re doing. Wouldn’t it be nice if there were people around with years of remote research experience, who were nice (or dumb) enough to give away all their best practices on their official blog?

Continue reading “Remote is Better, pt. 1: Getting Clients’ Hands Dirty”