So we’ve been beating around the bush for the past year or so about this top secret game research project of ours, and now that the game has finally hit the streets, we’re bringing the news to you in all its full glory. How did we test the game? Who did we talk to? Do we have lots and lots of video to back it up? Let’s start from the beginning.
[vimeo] http://vimeo.com/1704123 [/vimeo]
Focus groups suck
Basically our highest priority for this study was to stay far, far away from the usual (and sadly dominant) market research-based focus group-style, ahem, bullshit. You know how it is: sterile lab environment, two-way mirrors, cubicles, soul-killing focus-groupthink, creepy guy with a clipboard standing behind you, nodding. We were not going to do that. No way, no how, no focus groups.
Revenge of the SNERD
So what did we do, then? Since we weren’t able to actually go out to users’ homes (though we definitely pushed that idea), our Dear Leader Mr. Nate Bolt formed SNERD (the Simulated Native Environment Research Division) and set out to create a lab environment that was as similar as possible to a typical gaming setup: a desk, a chair, a laptop, and no other participants, research moderators, or observers to distract them. Unlike a focus group study, where participants would play the game for an hour and then talk about it in a group for another hour (ref. aforementioned bullshit), we had our participants play for six hours over two nights, without any explicit directions or instructions, except for one, which was to think-aloud as they played. We used TeamSpeak to hear what they were saying, and only occasionally dropped in with questions.
EA gets in on it
We, the research moderators and the observers from EA, were in a separate room, where we broadcasted the game screens, player’s faces, and TeamSpeak audio live. Even the EA peeps got in on the act: if they had questions, they let me know, so that I could address the players myself and rephrase any potentially leading or distracting questions (“Are you having a lot of fun now?”) into neutral moderator-speak (“Tell me what you’re doing here.”). This way, they were able to figure out what mattered to them, without swaying the feedback one way or the other.
PENS is mightier than the pen
Since we mostly do qualitative research, we don’t do many surveys and questionnaires, so to get the quantifiable data EA wanted, we worked with a great company in NY called Immersyve, who designed questionnaires and analyzed them using a set of fun and engagement metrics called PENS (Player Experience of Need Satisfaction). To keep players from getting bored or distracted when filling out the surveys, we used unobtrustive touchscreens instead of pen and paper, kept the surveys brief (<2 min), and instead of interrupting them, we asked them to fill the surveys out whenever they wanted to take a break from playing whatever phase they were on.
Oh yeah and it was also kind of insane
Not gonna lie, stuff got pretty B-A-N-A-N-A-S. It took practice to wrap our heads around the setup: six stations in different parts of the building, broadcasting 18 live streams of video to the observation room plus six streams of audio (gameplay video over VGA, webcam video over VNC, live touchscreen view over VNC, and TeamSpeak Chat over IP) while recording the gameplay, voice, and webcam data at the same time. I had to keep an eye on everyone’s progress, listen equally to all six users, take time-stamped notes so I could analyze the videos later, and keep an eye on my computer’s performance, since it was running something like five million screensharing applications. (On some of our first run-throughs, we had so much equipment running that we blew out the power in the entire building. Sploops.) Of course, from the gamers’ perspective, they were just kicking back and playing Spore.
393 Hours of HD Gamer Video
And that, in a nutshell, was it! Over the course of a year, we tested 59 users, logging 393 hours of individual gameplay. And to top it all off, we recorded the whole damn thing: player footage, conversations, and gameplay. The whole point of the study was to get the gamers comfortable so they’d say and do the kind of things they might not do in a self-consciousness provoking focus group or lab environment. Don’t believe us? Check it out:
[vimeo] http://vimeo.com/1704058 [/vimeo]
