Here's the thing - Bolt | Peters was acquired by Facebook. This site will live on.

Stop Bullshit Research in Five Easy Steps

Posted on May 22nd, 2008 by Tony Tulathimutte

Anyone in the UX field who’s worked for a few companies will recognize a type of moderated research that gives off a reek of inauthenticity. Tell me if this sounds familiar: one moderator and six users sit around a table in a converted meeting room. The moderator tells the users, each of whom have been prescheduled and screened through a recruiting agency, to go to a prototype website and pretend they’re looking for a 20 GB googlydooter, or whatever. The users go into their cubicles, where the prototype is brought up on six identical, factory-default computers. Some of the users finish in five minutes, some don’t finish at all, but everyone gets exactly fifteen minutes to finish their task. (The early finishers drum their fingers in boredom, waiting for the moderator to call time.) Finally, the moderator brings up a projection of the prototype, and asks the users to voice their opinions, one-at-a-time, keeping their responses brief, to give everyone time to speak. The process lasts about 1-2 hours, making everyone kind of tired. The participants are paid their incentives, and the moderator drives home, wiping bitter tears from his eyes as he pulls into his driveway.

How could that possibly have been useful? he thinks to himself. What has my life come to?

Why Bullshit Happens

Naturally, one might ask: if it’s so terrible, so ineffective, and such a waste of the time and life force of everyone involved, why does it keep happening? Three reasons:

  1. Accountability. In order to justify spending on research to the men upstairs, companies need to be “sure” that the research they’re doing is “working”. The easiest way to ensure this is by coming up with a bullet-point list of Questions, then making sure that every participant gives an Answer. You hand the answers to your bosses, they nod approvingly: you’ve done your job.
  2. It’s easy. Making a list of Questions, asking people what they think point-blank, collecting a list of answers. It’s completely cut and dry, no error there. Of course, this process assumes that what the participants say is actually reliable–but isn’t that kind of absurd to assume that what someone says in a controlled environment to a person with a clipboard in front of six strangers actually reflects what they really think and believe?
  3. It works… kinda. We won’t go out and say that no valuable findings come out of this process, but we’ll also wager that it creates more problems than it solves. When all of your findings are based off of what people think they think, rather than how they act, it can set a misleading direction for the entire project that can cause more problems down the road. On top of that, this kind of research isn’t likely to uncover the sort of things you never even considered while putting your questions together–the kind of findings that inspire true innovation.

How to Stop That Shit

This, of course, is a topic that could fill many books (or blogs for that matter), but allow us to offer a few correctives to the worst offenses of user research as it’s commonly practiced:

  • Native environments. Let’s just put it out there: focus groups are soulless. For the average person, there’s absolutely nothing natural or even pleasant about sitting in a meeting room, no matter how inoffensively decorated or well-deodorized. On to[ of that, it’s frustrating to sit while other people prattle on about some website when all you want to do is say your piece and collect your check. So whenever possible, stick to native environments: make the effort to speak with your users on their own terms and in their own technological ecosystem. (Naturally, this is most easily achieved remotely.)
  • Get on your users’ time. Don’t force them to use your app or prototype when you want them to, but try to schedule your research around the time when they would naturally use it. If it doesn’t exist yet, you’re screwed. If it does, you can use Ethnio to recruit participants live from your web site.
  • Talk to real people. Of course, the only users who are going to have passionate tasks at all are going to be the ones who would actually use your product. When you do your recruiting, you’ve got to screen your users to make sure you’re not just talking to someone who wants an incentive check. Recruiting companies happen to be terrible at this–does anyone actually know where their recruits are coming from? By far, the most reliable source of recruits are the people who are actually use your product, and if possible, you should talk to them while they’re using your product. (And if you’re wondering how to contact and screen these users, live, with a minimum of muss and fuss, why don’t you check out our completely free online recruiting tool?
  • Ditch the script. A moderator script is useful as a checklist of things you know you need to find out about, but don’t be afraid of going off-script; again, you’ve got to let your participants talk about what they really care about, with regard to your product. So your moderator’s got to be flexible: if the user wants to go into a different part of the website than the script calls for, let them–you might not hit every script target with every user, but you’ll discover much more than you expected you would. If you really want to understand how people use your product, you have to let to show you how they really use your product, not just how you think they’ll be using it.

Of course, some of this applies more to research on finished, commercially available products than prototypes, and more to high-level user experience than usability; that said, this is just one tiny fragment of a micron of a speck of what needs to happen to reverse the flood of B.S. research that every day in labs and meeting rooms across the nation. Check back on the B|P blog often for more ranting about the state of UX, or, if you’re mad as hell and not going to take it anymore, hit us up on the website tip.

Tags: , , , , , , , , , , , , ,

1,687 Responses to “Stop Bullshit Research in Five Easy Steps”

  1. lloan
    Dec 01, 2016

    What i don’t realize is if truth be told how you’re not really a lot
    more smartly-liked than you might be right now. You’re very
    intelligent. You understand therefore significantly in the case of this matter, made me for my part believe
    it from a lot of various angles. Its like men and women don’t
    seem to be fascinated until it is one thing to accomplish with Girl gaga!
    Your personal stuffs excellent. All the time
    maintain it up!

  2. lloan
    Dec 01, 2016

    I could not resist commenting. Well written!

    Dec 01, 2016

    This is my first time pay a visit at here and i am truly happy to read all at one place.

  4. lloan
    Dec 01, 2016

    certainly like your web site but you need
    to check the spelling on several of your posts. Several of them are rife
    with spelling issues and I find it very bothersome to inform the reality nevertheless I will surely
    come again again.

  5. Just want to say your article is as surprising. The
    clarity on your submit is simply excellent and that i could
    suppose you are an expert in this subject. Fine together with your permission allow me to grasp your RSS feed
    to stay up to date with approaching post. Thank you a million and please carry
    on the gratifying work.

  6. Charlene Boatwright
    Dec 01, 2016

    Hello! Someone in my Facebook group shared this site with us so I came to
    take a look. I’m definitely enjoying the information. I’m
    book-marking and will be tweeting this to my followers!
    Outstanding blog and superb style and design.

  7. You are so awesome! I do not think I’ve read something like this before.
    So wonderful to find someone with genuine thoughts on this subject.
    Really.. thanks for starting this up. This web site is one thing that’s needed on the web, someone with a
    bit of originality!

  8. lloan
    Dec 01, 2016

    Hmm it appears like your site ate my first comment (it was extremely long) so I guess I’ll just sum it up what I wrote
    and say, I’m thoroughly enjoying your blog. I too am an aspiring blog blogger but I’m still new to the whole thing.
    Do you have any recommendations for beginner blog writers?

    I’d genuinely appreciate it.

  9. lloan
    Dec 01, 2016

    It is not my first time to visit this web site, i am browsing this web
    page dailly and obtain nice information from here every day.

    Dec 01, 2016

    Do you have a spam issue on this site; I also am a blogger, and I was
    wanting to know your situation; many of us have created some nice methods and we are looking to
    trade solutions with other folks, be sure to shoot me
    an email if interested.

  11. lloan
    Dec 01, 2016

    Great beat ! I would like to apprentice while you amend your web site, how could i subscribe for a blog site?
    The account aided me a acceptable deal. I had been tiny bit acquainted of this your broadcast provided bright clear idea

  12. Unquestionably imagine that that you stated. Your favourite reason seemed to be
    on the internet the easiest factor to remember of. I say
    to you, I certainly get irked at the same time as other folks think about issues
    that they just do not understand about. You controlled to hit the nail upon the top and outlined out the whole thing with no need side-effects , other folks can take a signal.
    Will probably be back to get more. Thank you

  13. lloan
    Dec 01, 2016

    Thanks for ones marvelous posting! I certainly enjoyed reading it, you are a great author.
    I will remember to bookmark your blog and will come back later on. I want
    to encourage yourself to continue your great writing, have a nice

  14. lloan
    Dec 01, 2016

    Why viewers still use to read news papers when in this technological world the whole thing is presented on web?

  15. lloan
    Dec 01, 2016

    My family every time say that I am wasting my time here at web, however I know I am getting knowledge every day by reading
    thes fastidious articles or reviews.

  16. lloan
    Dec 01, 2016

    Excellent blog right here! Also your website lots up very fast!
    What host are you using? Can I am getting your associate hyperlink on your host?
    I wish my website loaded up as fast as yours lol

  17. Boyd Kyle
    Dec 01, 2016

    This is the right blog for anybody who really wants to
    find out about this topic. You understand a whole lot its almost tough to argue with you (not that I personally would want to?HaHa).

    You certainly put a fresh spin on a topic that’s been discussed for
    a long time. Excellent stuff, just great!

  18. I simply could not depart your web site prior to suggesting that I extremely
    loved the usual information a person supply to your guests?
    Is going to be again frequently to check up on new posts

  19. just beneath, are several absolutely not associated web sites to ours, nevertheless, they may be certainly really worth going over

  20. Hi there! This is my first comment here so I just wanted
    to give a quick shout out and tell you I genuinely enjoy reading your blog posts.
    Can you recommend any other blogs/websites/forums that go over the same topics?
    Many thanks!

  21. Marylou Bastyan
    Dec 01, 2016

    I like the valuable information you supply in your articles.
    I will bookmark your blog and check again right here frequently.
    I’m fairly certain I will be told many new stuff right here!
    Best of luck for the following!

  22. lloan
    Dec 01, 2016

    I read this paragraph completely on the topic of the resemblance of latest and preceding
    technologies, it’s amazing article.

  23. I?m not that much of a online reader to be honest but your
    blogs really nice, keep it up! I’ll go ahead and bookmark your site to come back
    later. Many thanks

  24. lloan
    Dec 01, 2016

    Fantastic blog! Do you have any tips for aspiring writers?
    I’m planning to start my own site soon but I’m a little lost on everything.
    Would you propose starting with a free platform like WordPress or go for a paid option? There are so many options out there that I’m completely overwhelmed ..
    Any ideas? Thanks a lot!

  25. Mason Armer
    Dec 01, 2016

    Remarkable! Its truly remarkable post, I have got much clear idea about from this piece of writing.

  26. Excellent blog here! Also your web site loads up fast! What web host are you using? Can I get your affiliate link to your host? I wish my site loaded up as fast as yours lol

  27. Fine way of describing, and nice paragraph to take information concerning my presentation subject
    matter, which i am going to convey in school.

  28. free icon
    Dec 01, 2016

    one of our visitors lately encouraged the following website

  29. obat pembesar penis
    Dec 01, 2016

    My brother suggested I might like this website. He was totally right. This post actually made my day. You can not imagine just how much time I had spent for this info! Thanks!

  30. It is not my first time to visit this website, i am browsing this web site dailly and obtain fastidious facts from here daily.

  31. Deb Schaw
    Dec 01, 2016

    I think everything posted made a ton of sense. But, what about
    this? suppose you typed a catchier title? I am not suggesting your content is
    not solid, however suppose you added a post title that makes people desire more?
    I mean Stop Bullshit Research in Five Easy Steps | Bolt | Peters is a little plain. You ought to peek at Yahoo’s home page and watch how they create article
    headlines to grab viewers to open the links. You might add a video or a related picture or two
    to get people interested about what you’ve got to say.
    Just my opinion, it could bring your posts a little bit more interesting.

  32. Thank you for the good writeup. It in fact was a amusement account it.
    Look advanced to more added agreeable from you! By the way, how can we communicate?

  33. lloan
    Dec 01, 2016

    Thanks for any other magnificent article. The place else may anybody get that
    kind of information in such a perfect method of writing?
    I have a presentation subsequent week, and I’m on the look for such info.

  34. Every when inside a whilst we opt for blogs that we read. Listed below are the most recent internet sites that we pick

  35. Soeharto
    Dhini Aminarti
    Tommy Kurniawan
    berita ari ari terkini
    memuaskan suami
    Ramalan 2015
    fakta wanita
    berita sahabat rasulullah
    berita komunitas Islam
    berita makanan terbaru

  36. we came across a cool web page that you just could possibly take pleasure in. Take a look in case you want

  37. one of our visitors lately advised the following website

Leave a Reply