Here's the thing - Bolt | Peters was acquired by Facebook. This site will live on.

Stop Bullshit Research in Five Easy Steps

Posted on May 22nd, 2008 by Tony Tulathimutte
2,370 comments

Anyone in the UX field who’s worked for a few companies will recognize a type of moderated research that gives off a reek of inauthenticity. Tell me if this sounds familiar: one moderator and six users sit around a table in a converted meeting room. The moderator tells the users, each of whom have been prescheduled and screened through a recruiting agency, to go to a prototype website and pretend they’re looking for a 20 GB googlydooter, or whatever. The users go into their cubicles, where the prototype is brought up on six identical, factory-default computers. Some of the users finish in five minutes, some don’t finish at all, but everyone gets exactly fifteen minutes to finish their task. (The early finishers drum their fingers in boredom, waiting for the moderator to call time.) Finally, the moderator brings up a projection of the prototype, and asks the users to voice their opinions, one-at-a-time, keeping their responses brief, to give everyone time to speak. The process lasts about 1-2 hours, making everyone kind of tired. The participants are paid their incentives, and the moderator drives home, wiping bitter tears from his eyes as he pulls into his driveway.

How could that possibly have been useful? he thinks to himself. What has my life come to?

Why Bullshit Happens

Naturally, one might ask: if it’s so terrible, so ineffective, and such a waste of the time and life force of everyone involved, why does it keep happening? Three reasons:

  1. Accountability. In order to justify spending on research to the men upstairs, companies need to be “sure” that the research they’re doing is “working”. The easiest way to ensure this is by coming up with a bullet-point list of Questions, then making sure that every participant gives an Answer. You hand the answers to your bosses, they nod approvingly: you’ve done your job.
  2. It’s easy. Making a list of Questions, asking people what they think point-blank, collecting a list of answers. It’s completely cut and dry, no error there. Of course, this process assumes that what the participants say is actually reliable–but isn’t that kind of absurd to assume that what someone says in a controlled environment to a person with a clipboard in front of six strangers actually reflects what they really think and believe?
  3. It works… kinda. We won’t go out and say that no valuable findings come out of this process, but we’ll also wager that it creates more problems than it solves. When all of your findings are based off of what people think they think, rather than how they act, it can set a misleading direction for the entire project that can cause more problems down the road. On top of that, this kind of research isn’t likely to uncover the sort of things you never even considered while putting your questions together–the kind of findings that inspire true innovation.

How to Stop That Shit

This, of course, is a topic that could fill many books (or blogs for that matter), but allow us to offer a few correctives to the worst offenses of user research as it’s commonly practiced:

  • Native environments. Let’s just put it out there: focus groups are soulless. For the average person, there’s absolutely nothing natural or even pleasant about sitting in a meeting room, no matter how inoffensively decorated or well-deodorized. On to[ of that, it’s frustrating to sit while other people prattle on about some website when all you want to do is say your piece and collect your check. So whenever possible, stick to native environments: make the effort to speak with your users on their own terms and in their own technological ecosystem. (Naturally, this is most easily achieved remotely.)
  • Get on your users’ time. Don’t force them to use your app or prototype when you want them to, but try to schedule your research around the time when they would naturally use it. If it doesn’t exist yet, you’re screwed. If it does, you can use Ethnio to recruit participants live from your web site.
  • Talk to real people. Of course, the only users who are going to have passionate tasks at all are going to be the ones who would actually use your product. When you do your recruiting, you’ve got to screen your users to make sure you’re not just talking to someone who wants an incentive check. Recruiting companies happen to be terrible at this–does anyone actually know where their recruits are coming from? By far, the most reliable source of recruits are the people who are actually use your product, and if possible, you should talk to them while they’re using your product. (And if you’re wondering how to contact and screen these users, live, with a minimum of muss and fuss, why don’t you check out our completely free online recruiting tool?
  • Ditch the script. A moderator script is useful as a checklist of things you know you need to find out about, but don’t be afraid of going off-script; again, you’ve got to let your participants talk about what they really care about, with regard to your product. So your moderator’s got to be flexible: if the user wants to go into a different part of the website than the script calls for, let them–you might not hit every script target with every user, but you’ll discover much more than you expected you would. If you really want to understand how people use your product, you have to let to show you how they really use your product, not just how you think they’ll be using it.

Of course, some of this applies more to research on finished, commercially available products than prototypes, and more to high-level user experience than usability; that said, this is just one tiny fragment of a micron of a speck of what needs to happen to reverse the flood of B.S. research that every day in labs and meeting rooms across the nation. Check back on the B|P blog often for more ranting about the state of UX, or, if you’re mad as hell and not going to take it anymore, hit us up on the website tip.

Tags: , , , , , , , , , , , , ,

2,370 Responses to “Stop Bullshit Research in Five Easy Steps”

  1. Maddison
    Dec 22, 2016

    Hey! Would you mind if I share your blog with my myspace group?
    There’s a lot of folks that I think would really enjoy your content.

    Please let me know. Many thanks


  2. more info
    Dec 22, 2016

    Wow! Thank you! I constantly wanted to write on my site something like that. Can I take a portion of your post to my website?


  3. Excellent goods from you, man. I’ve understand your stuff previous to and
    you’re just too wonderful. I really like what you’ve acquired here, certainly like what you’re saying and the way in which you
    say it. You make it enjoyable and you still care
    for to keep it smart. I can not wait to read much more
    from you. This is actually a terrific website.


  4. Website
    Dec 22, 2016

    You are a very clever person!


  5. Read This
    Dec 22, 2016

    You can definitely see your expertise within the work you write. The sector hopes for more passionate writers like you who are not afraid to mention how they believe. Always go after your heart.


  6. plumbing services
    Dec 22, 2016

    I keep listening to the news update speak about getting free online grant applications so I have been looking around for the best site to get one. Could you advise me please, where could i acquire some?


  7. learn more
    Dec 22, 2016

    You made some good points there. I did a search on the issue and found most people will consent with your site.


  8. learn more
    Dec 22, 2016

    Hiya, I am really glad I have found this info. Nowadays bloggers publish just about gossips and internet and this is actually irritating. A good site with interesting content, this is what I need. Thank you for keeping this web site, I’ll be visiting it. Do you do newsletters? Can not find it.


  9. visit here
    Dec 22, 2016

    hey there and thank you for your info – I have definitely picked up anything new from right here. I did however expertise a few technical issues using this website, as I experienced to reload the web site lots of times previous to I could get it to load correctly. I had been wondering if your web host is OK? Not that I am complaining, but slow loading instances times will very frequently affect your placement in google and could damage your high quality score if ads and marketing with Adwords. Anyway I’m adding this RSS to my email and could look out for a lot more of your respective interesting content. Ensure that you update this again soon..


  10. taxi vtc
    Dec 22, 2016

    What’s up to every body, it’s my first visit of this webpage; this blog includes awesome and in fact excellent stuff in support
    of readers.


  11. get more info
    Dec 22, 2016

    Thank you so much for giving everyone an extraordinarily special opportunity to read critical reviews from this site. It’s always very enjoyable and jam-packed with amusement for me and my office colleagues to search your web site a minimum of thrice in a week to see the newest tips you have. And definitely, we are at all times amazed with all the wonderful tactics you give. Selected 1 areas in this post are particularly the finest we have all had.


  12. Wonderful paintings! This is the type of info that are supposed to be shared across the
    web. Shame on Google for now not positioning this publish upper!
    Come on over and talk over with my site . Thanks =)


  13. s3.amazonaws.com
    Dec 22, 2016

    They are either extremely excited by the prospect of moving forward with things and they can sense that you are trying to
    move factors along but this also tends to make them extremely nervous.


  14. Visit Website
    Dec 22, 2016

    Excellent read, I just passed this onto a friend who was doing a little research on that. And he actually bought me lunch since I found it for him smile So let me rephrase that: Thanks for lunch!


  15. Hello There. I found your blog using msn. This is a really well written article. I will make sure to bookmark it and come back to read more of your useful information. Thanks for the post. I’ll certainly return.


  16. Good article and right to the point. I don’t know if this is really the best place to ask but do you folks have any thoughts on where to employ some professional writers? Thanks in advance :)


  17. Click Here
    Dec 22, 2016

    Great beat ! I wish to apprentice while you amend your site, how can i subscribe for a blog website? The account helped me a acceptable deal. I had been a little bit acquainted of this your broadcast provided bright clear concept


  18. Marketing
    Dec 22, 2016

    check below, are some entirely unrelated sites to ours, having said that, they’re most trustworthy sources that we use


  19. Discover More
    Dec 22, 2016

    Normally I do not learn post on blogs, however I wish to say that this write-up very forced me to take a look at and do so! Your writing style has been amazed me. Thanks, quite great article.


  20. Find Out More
    Dec 22, 2016

    Hello. remarkable job. I did not expect this. This is a splendid story. Thanks!