Here's the thing - Bolt | Peters was acquired by Facebook. This site will live on.

Usertesting.com Review by Rolf Molich

Posted on Apr 6th, 2010 by rolfmolich
1,260 comments

[Guest author: Rolf Molich owns and manages DialogDesign, a small Danish usability consultancy that he founded in 1993. Rolf conceived and coordinated the Comparative Usability Evaluation studies CUE-1 through CUE-8 in which almost 100 professional usability teams tested or reviewed the same applications.]

Usertesting.com – A usable and useful recruiting service

Usertesting.com offers unattended, 15-minute usability test sessions at reasonable prices with participants from the US, Canada and the United Kingdom. I have some experience with their service. I am pleased with it, but you should be aware that they really excel at recruiting users from the most broad demographics to spend time looking at your site, not usability analysis.

Usertesting.com home page

Usertesting.com home page

“Use it and your site will get better”

On their home page, Usertesting.com quotes Evan Williams, Twitter Co-founder: “Use it and your site will get better”. I respectfully disagree. Usertesting.com is a great recruiting service, but a skilled usability professional is still required watch the videos and extract useful usability recommendations. The opinions from the participants are mostly worthless. Participants’ behavior, not their opinions is important.

I used Usertesting.com in a recent comparative usability measurement study of Budget.com. Usertesting.com ran about 30 sessions for me with about 20 users. Since my five car rental tasks on Budget.com took about 25 minutes to complete, I had to split the tasks into two sessions and ask their customer service to get the same test participants to do two sessions with different tasks. This worked like a breeze, but it took me some extra time to handle and write instructions for two sessions.

19$, 29$, 39$, and counting?

At the time of my study in April 2009, Usertesting.com was charging just 19$ per test participant. Shortly after, they raised the price to 29$ per test participant. Now they have announced yet another price increase to 39$ per test participant. At 19$ per participant their price/performance ratio was amazing. At 29$ it was good. At 39$ I am not so sure, considering that you only get 15 minutes test time. At 4 x 39$ = 156$ per hour their service is less competitive compared to traditional recruiting. Despite their price increases, they still pay their participants 10$ per session.

High quality

Amazing turnaround time. In late December 2009 I used Usertesting.com to run three test sesssions of the Enterprise.com car rental service. The test sessions were completed within one hour from the time I submitted the request. Because of the time difference between the US and Denmark where I live, I sometimes submit requests at 4 am East Coast time, but even these requests are honored almost instantly.

Their test participants are great. So Usertesting.com’s screening process seems to work. One or two participants out of 25 were rather talkative and offered 10 minutes of worthless personal opinions about the website in addition to their helpful task solutions. Fortunately, this happened rarely and I simply filtered out the opinions and looked at actual behavior.

Video and sound from their recordings is good. I never had a problem understanding what a participant was saying or doing. See the sample screen from a recording below. In contrast, I’ve had professional usability consultancies who without blushing sent me usability test recordings where the video was blurred or the sound was a loud humming.

Screen showing usertesting.com video replay

Screen showing usertesting.com video replay

Their customer service is excellent. I got a fast and sensible response each time I asked them a question or submitted a comment.

Their customer service told me that their agreements with participants allow the clients any use of the recordings they want – including public presentation of the videos. This seems ethically defensible because you can’t see participants’ faces and you mostly don’t get their names.

A few limitations and caveats

For unattended test sessions where no moderator is available to correct misunderstandings on the spot, you may want to test your instructions to test particpants carefully. You must provide your own pre-session interview questions and the all-important debriefing questions.

Usertesting.com only provides rudimentary demographic information about participants as shown in the screenshot below. If you want information about approximately where participants live, what their profession is and if they’ve used the site or similar sites before, you must use precious test time to ask in your test instructions.

Rudimentary participant demographics on usertesting.com

Rudimentary participant demographics on usertesting.com

Usertesting.com’s setup allows the participant to interrupt the recording at any time. This may be OK for qualitative testing, but it spoiled a few of my task time measurements. Also, in a few cases test participants seemingly left the session for about 5 minutes without explanation – no sound and nothing moved on the video. In one case it seemed to me that a test participant had rehearsed the test questions before they clicked the recording button and started the real session. I must add, though, that if I expressed just the slightest bit of dissatisfaction with a participant, their customer service immediately refunded me what I had paid. Of course moderated sessions aren’t perfect either – and most often they don’t have a money back guarantee.

Some of my colleagues have argued that Usertesting.com offers “professional” test participants. It is true that on 2-3 occasions test participants inadvertently showed me their test session dashboard which showed that they had been doing about 10 test sessions recently. Personally, I don’t mind this. I have yet to see a participant who was so experienced in user testing that it prevented me  from getting the data I wanted.

All in all: Recommended

Usertesting.com has a great concept and implements it with care. Before their recent price increase, I would even have said “Highly recommended”.

Resoures

www.usertesting.com

The comparative study of Budget.com that I used usertesting.com to test (CUE-8).

Tags: , , , , , , ,

1,260 Responses to “Usertesting.com Review by Rolf Molich”

  1. Pincode ahmedabad
    Dec 22, 2016

    Hello there! I simply want to give you a
    big thumbs up for the great information you have
    right here on this post. I will be coming back to your website for
    more soon.


  2. Visit Website
    Dec 22, 2016

    I am just writing to make you be aware of what a superb encounter my friend’s princess found reading your site. She picked up such a lot of details, most notably what it’s like to possess an incredible coaching nature to make other people just grasp chosen specialized matters. You really did more than my expected results. Thanks for delivering those good, safe, edifying and even easy guidance on the topic to Lizeth.


  3. Get More Info
    Dec 22, 2016

    It is in point of fact a nice and useful piece of info. I am happy that you just shared this helpful information with us. Please keep us informed like this. Thank you for sharing.


  4. Learn More Here
    Dec 22, 2016

    Wonderful paintings! This is the kind of info that are meant to be shared around the web. Shame on the search engines for no longer positioning this submit upper! Come on over and discuss with my website . Thank you =)


  5. Website
    Dec 22, 2016

    I’ve been surfing online more than 3 hours today, but I never discovered any attention-grabbing article like yours. It¡¦s pretty worth enough for me. In my view, if all web owners and bloggers made good content material as you did, the internet can be much more useful than ever before.


  6. Read More Here
    Dec 22, 2016

    Hiya, I am really glad I have found this info. Nowadays bloggers publish just about gossips and internet and this is actually irritating. A good site with interesting content, this is what I need. Thank you for keeping this web site, I’ll be visiting it. Do you do newsletters? Can not find it.


  7. Click This Link
    Dec 22, 2016

    I am constantly searching online for ideas that can facilitate me. Thanks!


  8. adidas nmd
    Dec 22, 2016

    Once you have acquired your teenager in the retailer, it is not enough to just try the sneakers on. They must be given a literal check run, or
    at least a check jog.


  9. Find Out More
    Dec 22, 2016

    I want to express my appreciation to the writer just for bailing me out of this type of setting. After looking through the world wide web and getting views that were not beneficial, I assumed my entire life was well over. Existing without the presence of solutions to the difficulties you have solved all through your entire write-up is a crucial case, and ones that might have negatively damaged my entire career if I hadn’t come across your blog. Your own personal mastery and kindness in dealing with all areas was tremendous. I don’t know what I would’ve done if I had not discovered such a step like this. I can now look forward to my future. Thanks for your time very much for this reliable and results-oriented help. I will not hesitate to refer your web site to anyone who requires assistance about this issue.


  10. visit here
    Dec 22, 2016

    I have been absent for a while, but now I remember why I used to love this web site. Thank you, I will try and check back more often. How frequently you update your web site?