Here's the thing - Bolt | Peters was acquired by Facebook. This site will live on.

Usertesting.com Review by Rolf Molich

Posted on Apr 6th, 2010 by rolfmolich
387 comments

[Guest author: Rolf Molich owns and manages DialogDesign, a small Danish usability consultancy that he founded in 1993. Rolf conceived and coordinated the Comparative Usability Evaluation studies CUE-1 through CUE-8 in which almost 100 professional usability teams tested or reviewed the same applications.]

Usertesting.com – A usable and useful recruiting service

Usertesting.com offers unattended, 15-minute usability test sessions at reasonable prices with participants from the US, Canada and the United Kingdom. I have some experience with their service. I am pleased with it, but you should be aware that they really excel at recruiting users from the most broad demographics to spend time looking at your site, not usability analysis.

Usertesting.com home page

Usertesting.com home page

“Use it and your site will get better”

On their home page, Usertesting.com quotes Evan Williams, Twitter Co-founder: “Use it and your site will get better”. I respectfully disagree. Usertesting.com is a great recruiting service, but a skilled usability professional is still required watch the videos and extract useful usability recommendations. The opinions from the participants are mostly worthless. Participants’ behavior, not their opinions is important.

I used Usertesting.com in a recent comparative usability measurement study of Budget.com. Usertesting.com ran about 30 sessions for me with about 20 users. Since my five car rental tasks on Budget.com took about 25 minutes to complete, I had to split the tasks into two sessions and ask their customer service to get the same test participants to do two sessions with different tasks. This worked like a breeze, but it took me some extra time to handle and write instructions for two sessions.

19$, 29$, 39$, and counting?

At the time of my study in April 2009, Usertesting.com was charging just 19$ per test participant. Shortly after, they raised the price to 29$ per test participant. Now they have announced yet another price increase to 39$ per test participant. At 19$ per participant their price/performance ratio was amazing. At 29$ it was good. At 39$ I am not so sure, considering that you only get 15 minutes test time. At 4 x 39$ = 156$ per hour their service is less competitive compared to traditional recruiting. Despite their price increases, they still pay their participants 10$ per session.

High quality

Amazing turnaround time. In late December 2009 I used Usertesting.com to run three test sesssions of the Enterprise.com car rental service. The test sessions were completed within one hour from the time I submitted the request. Because of the time difference between the US and Denmark where I live, I sometimes submit requests at 4 am East Coast time, but even these requests are honored almost instantly.

Their test participants are great. So Usertesting.com’s screening process seems to work. One or two participants out of 25 were rather talkative and offered 10 minutes of worthless personal opinions about the website in addition to their helpful task solutions. Fortunately, this happened rarely and I simply filtered out the opinions and looked at actual behavior.

Video and sound from their recordings is good. I never had a problem understanding what a participant was saying or doing. See the sample screen from a recording below. In contrast, I’ve had professional usability consultancies who without blushing sent me usability test recordings where the video was blurred or the sound was a loud humming.

Screen showing usertesting.com video replay

Screen showing usertesting.com video replay

Their customer service is excellent. I got a fast and sensible response each time I asked them a question or submitted a comment.

Their customer service told me that their agreements with participants allow the clients any use of the recordings they want – including public presentation of the videos. This seems ethically defensible because you can’t see participants’ faces and you mostly don’t get their names.

A few limitations and caveats

For unattended test sessions where no moderator is available to correct misunderstandings on the spot, you may want to test your instructions to test particpants carefully. You must provide your own pre-session interview questions and the all-important debriefing questions.

Usertesting.com only provides rudimentary demographic information about participants as shown in the screenshot below. If you want information about approximately where participants live, what their profession is and if they’ve used the site or similar sites before, you must use precious test time to ask in your test instructions.

Rudimentary participant demographics on usertesting.com

Rudimentary participant demographics on usertesting.com

Usertesting.com’s setup allows the participant to interrupt the recording at any time. This may be OK for qualitative testing, but it spoiled a few of my task time measurements. Also, in a few cases test participants seemingly left the session for about 5 minutes without explanation – no sound and nothing moved on the video. In one case it seemed to me that a test participant had rehearsed the test questions before they clicked the recording button and started the real session. I must add, though, that if I expressed just the slightest bit of dissatisfaction with a participant, their customer service immediately refunded me what I had paid. Of course moderated sessions aren’t perfect either – and most often they don’t have a money back guarantee.

Some of my colleagues have argued that Usertesting.com offers “professional” test participants. It is true that on 2-3 occasions test participants inadvertently showed me their test session dashboard which showed that they had been doing about 10 test sessions recently. Personally, I don’t mind this. I have yet to see a participant who was so experienced in user testing that it prevented me  from getting the data I wanted.

All in all: Recommended

Usertesting.com has a great concept and implements it with care. Before their recent price increase, I would even have said “Highly recommended”.

Resoures

www.usertesting.com

The comparative study of Budget.com that I used usertesting.com to test (CUE-8).

Tags: , , , , , , ,

387 Responses to “Usertesting.com Review by Rolf Molich”

  1. we are assholes
    Dec 01, 2016

    Have you ever considered creating an e-book or guest authoring on other sites?
    I have a blog based on the same ideas you discuss and would love to have you
    share some stories/information. I know my visitors would value your work.
    If you are even remotely interested, feel free to send me an e mail.


  2. Homepage
    Dec 01, 2016

    As a Newbie, I am permanently searching online for articles that can be of assistance to me. Thank you


  3. Read More
    Dec 01, 2016

    I was recommended this website by my cousin. I’m not sure whether this post is written by him as nobody else know such detailed about my trouble. You are incredible! Thanks!


  4. Find Out More
    Dec 01, 2016

    I am not sure where you are getting your info, but great topic. I needs to spend some time learning more or understanding more. Thanks for excellent information I was looking for this information for my mission.


  5. visit here
    Dec 01, 2016

    Keep functioning ,remarkable job!


  6. i need a job now
    Dec 01, 2016

    Hello there! I know this is kinda off topic however I’d figured I’d ask. Would you be interested in exchanging links or maybe guest authoring a blog post or vice-versa? My blog addresses a lot of the same subjects as yours and I think we could greatly benefit from each other. If you’re interested feel free to send me an e-mail. I look forward to hearing from you! Terrific blog by the way!


  7. more info
    Dec 01, 2016

    Hello there, just became alert to your blog through Google, and found that it is truly informative. I am gonna watch out for brussels. I’ll be grateful if you continue this in future. Lots of people will be benefited from your writing. Cheers!


  8. andarine results
    Dec 01, 2016

    I really appreciate this post. I¡¦ve been looking all over for this! Thank goodness I found it on Bing. You have made my day! Thank you again


  9. hello!,I like your writing very much! percentage we be in contact extra about your post on AOL? I require a specialist on this area to solve my problem. Maybe that’s you! Having a look forward to peer you.


  10. 24 h betreuung
    Dec 01, 2016

    I have been absent for a while, but now I remember why I used to love this web site. Thank you, I will try and check back more often. How frequently you update your web site?


  11. Find Out More
    Dec 01, 2016

    Wow, marvelous weblog layout! How lengthy have you been blogging for? you made blogging glance easy. The overall look of your website is fantastic, as well as the content!


  12. It’s an awesome article in favor of all the online visitors; they will get advantage from it I
    am sure.


  13. more info
    Dec 01, 2016

    You are a very clever person!


  14. more info
    Dec 01, 2016

    I’m so happy to read this. This is the kind of manual that needs to be given and not the accidental misinformation that is at the other blogs. Appreciate your sharing this greatest doc.


  15. Go Here
    Dec 01, 2016

    I have been reading out many of your stories and i can state clever stuff. I will definitely bookmark your site.


  16. Home Page
    Dec 01, 2016

    I have read some just right stuff here. Certainly value bookmarking for revisiting. I surprise how much attempt you place to make this type of excellent informative website.


  17. I do not know whether it’s just me or if perhaps everyone else encountering problems with your website. It appears like some of the text on your posts are running off the screen. Can someone else please comment and let me know if this is happening to them as well? This might be a issue with my browser because I’ve had this happen previously. Many thanks


  18. website
    Dec 01, 2016

    My coder is trying to convince me to move to .net from PHP.
    I have always disliked the idea because of the expenses.
    But he’s tryiong none the less. I’ve been using Movable-type on numerous websites for about a year and
    am nervous about switching to another platform. I have heard fantastic things about blogengine.net.
    Is there a way I can import all my wordpress posts
    into it? Any help would be really appreciated!


  19. sex
    Dec 01, 2016

    Howdy! I know this is kinda off topic but I’d figured I’d ask.
    Would you be interested in exchanging links or maybe guest authoring
    a blog article or vice-versa? My site addresses a lot of the same topics as yours and I think we could greatly benefit from each other.

    If you might be interested feel free to shoot me an email.
    I look forward to hearing from you! Terrific blog by the way!


  20. baltimore md seo
    Dec 01, 2016

    I HATE getting to reduce poster board due to the factI am so OCD about the cuts becoming ideal and straight, So,I purchased precut half size poster board from Michael’s.


  21. Hi! Do you know if they make any plugins to
    protect against hackers? I’m kinda paranoid about
    losing everything I’ve worked hard on. Any suggestions?


  22. post568870
    Dec 01, 2016

    Excellent blog here! Also your web site loads up very fast!
    What web host are you using? Can I get your affiliate link to your host?
    I wish my site loaded up as quickly as yours lol


  23. Hello just wanted to give you a brief heads up and let you know a few of the pictures aren’t loading properly. I’m not sure why but I think its a linking issue. I’ve tried it in two different web browsers and both show the same outcome.


  24. ceramic flat iron
    Dec 01, 2016

    Howdy! I realize this is kind of off-topic but I had to ask. Does managing a well-established blog like yours require a lot of work? I’m completely new to writing a blog but I do write in my journal everyday. I’d like to start a blog so I can share my own experience and thoughts online. Please let me know if you have any ideas or tips for brand new aspiring bloggers. Thankyou!


  25. click site
    Dec 02, 2016

    This post gives clear idea for the new viewers of blogging, that actually
    how to do blogging.


  26. One Piece scan 829
    Dec 02, 2016

    Il a retrouvé son frère Sabo, a tenu tête à un amiral, et surtout, il s’est
    fait un sacré paquet de nouveaux copains !


  27. Hmm it appears like your blog ate my first comment (it was extremely long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog. I too am an aspiring blog blogger but I’m still new to everything. Do you have any helpful hints for first-time blog writers? I’d definitely appreciate it.


  28. Hello! This is my first visit to your blog! We are a collection of volunteers and starting a new initiative in a community in the same niche. Your blog provided us beneficial information to work on. You have done a extraordinary job!


  29. Saved as a favorite, I love your site!


  30. rigger boots
    Dec 02, 2016

    My programmer is trying to persuade me to move to .net from PHP. I have always disliked the idea because of the expenses. But he’s tryiong none the less. I’ve been using Movable-type on several websites for about a year and am anxious about switching to another platform. I have heard great things about blogengine.net. Is there a way I can transfer all my wordpress posts into it? Any kind of help would be really appreciated!


  31. Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates. I’ve been looking for a plug-in like this for quite some time and was hoping maybe you would have some experience with something like this. Please let me know if you run into anything. I truly enjoy reading your blog and I look forward to your new updates.


  32. The other day, while I was at work, my sister stole my iphone and tested to see if it can survive a 25 foot drop, just so she can be a youtube sensation. My apple ipad is now destroyed and she has 83 views. I know this is entirely off topic but I had to share it with someone!


  33. Hey there! I could have sworn I’ve been to this blog before but after reading through some of the post I realized it’s new to me. Anyways, I’m definitely happy I found it and I’ll be bookmarking and checking back often!


  34. I am really enjoying the theme/design of your website.
    Do you ever run into any web browser compatibility problems?

    A handful of my blog audience have complained about my website not
    operating correctly in Explorer but looks great in Opera.
    Do you have any ideas to help fix this problem?


  35. Hyundai
    Dec 02, 2016

    The info talked about inside the write-up are some of the best accessible


  36. Make money working from home, it’s easy!


  37. stalik hankishiev
    Dec 02, 2016

    Sites of interest we have a link to



Leave a Reply