Categories
blog Events

Escape the Lab – May 6th

Come hang with us in San Francisco or attend remotely on May 6th from 9am – 4:30pm PST. We’re holding our third Escape The Lab workshop on remote UX research methods and tools. You get hands-on training with the latest moderated and un-moderated remote UX research tools.
Space is limited as we’re only allowing 10 in-person and 10 remote attendees, so register soon. It’s about 50% sold-out right now. Also, we’re giving away two FREE spots for whoever tweets the best answer to why they love or hate remote research with the tag #ETLAB.
Date: Thursday, May 6th, 2010
Time: 9am – 4:30pm
Place: Bolt | Peters User Experience at 576 Natoma Street, San Francisco, CA
Cost: $399 In-person or $199 Remote. $299 for URF Alumni.
More info: http://escapethelab.com/
Hope to see you there!

Categories
blog HCI Remote Research Research innovation usability

Usertesting.com Review by Rolf Molich

[Guest author: Rolf Molich owns and manages DialogDesign, a small Danish usability consultancy that he founded in 1993. Rolf conceived and coordinated the Comparative Usability Evaluation studies CUE-1 through CUE-8 in which almost 100 professional usability teams tested or reviewed the same applications.]

Usertesting.com – A usable and useful recruiting service

Usertesting.com offers unattended, 15-minute usability test sessions at reasonable prices with participants from the US, Canada and the United Kingdom. I have some experience with their service. I am pleased with it, but you should be aware that they really excel at recruiting users from the most broad demographics to spend time looking at your site, not usability analysis.

Usertesting.com home page
Usertesting.com home page

“Use it and your site will get better”

On their home page, Usertesting.com quotes Evan Williams, Twitter Co-founder: “Use it and your site will get better”. I respectfully disagree. Usertesting.com is a great recruiting service, but a skilled usability professional is still required watch the videos and extract useful usability recommendations. The opinions from the participants are mostly worthless. Participants’ behavior, not their opinions is important.

I used Usertesting.com in a recent comparative usability measurement study of Budget.com. Usertesting.com ran about 30 sessions for me with about 20 users. Since my five car rental tasks on Budget.com took about 25 minutes to complete, I had to split the tasks into two sessions and ask their customer service to get the same test participants to do two sessions with different tasks. This worked like a breeze, but it took me some extra time to handle and write instructions for two sessions.

19$, 29$, 39$, and counting?

At the time of my study in April 2009, Usertesting.com was charging just 19$ per test participant. Shortly after, they raised the price to 29$ per test participant. Now they have announced yet another price increase to 39$ per test participant. At 19$ per participant their price/performance ratio was amazing. At 29$ it was good. At 39$ I am not so sure, considering that you only get 15 minutes test time. At 4 x 39$ = 156$ per hour their service is less competitive compared to traditional recruiting. Despite their price increases, they still pay their participants 10$ per session.

High quality

Amazing turnaround time. In late December 2009 I used Usertesting.com to run three test sesssions of the Enterprise.com car rental service. The test sessions were completed within one hour from the time I submitted the request. Because of the time difference between the US and Denmark where I live, I sometimes submit requests at 4 am East Coast time, but even these requests are honored almost instantly.

Their test participants are great. So Usertesting.com’s screening process seems to work. One or two participants out of 25 were rather talkative and offered 10 minutes of worthless personal opinions about the website in addition to their helpful task solutions. Fortunately, this happened rarely and I simply filtered out the opinions and looked at actual behavior.

Video and sound from their recordings is good. I never had a problem understanding what a participant was saying or doing. See the sample screen from a recording below. In contrast, I’ve had professional usability consultancies who without blushing sent me usability test recordings where the video was blurred or the sound was a loud humming.

Screen showing usertesting.com video replay
Screen showing usertesting.com video replay

Their customer service is excellent. I got a fast and sensible response each time I asked them a question or submitted a comment.

Their customer service told me that their agreements with participants allow the clients any use of the recordings they want – including public presentation of the videos. This seems ethically defensible because you can’t see participants’ faces and you mostly don’t get their names.

A few limitations and caveats

For unattended test sessions where no moderator is available to correct misunderstandings on the spot, you may want to test your instructions to test particpants carefully. You must provide your own pre-session interview questions and the all-important debriefing questions.

Usertesting.com only provides rudimentary demographic information about participants as shown in the screenshot below. If you want information about approximately where participants live, what their profession is and if they’ve used the site or similar sites before, you must use precious test time to ask in your test instructions.

Rudimentary participant demographics on usertesting.com
Rudimentary participant demographics on usertesting.com

Usertesting.com’s setup allows the participant to interrupt the recording at any time. This may be OK for qualitative testing, but it spoiled a few of my task time measurements. Also, in a few cases test participants seemingly left the session for about 5 minutes without explanation – no sound and nothing moved on the video. In one case it seemed to me that a test participant had rehearsed the test questions before they clicked the recording button and started the real session. I must add, though, that if I expressed just the slightest bit of dissatisfaction with a participant, their customer service immediately refunded me what I had paid. Of course moderated sessions aren’t perfect either – and most often they don’t have a money back guarantee.

Some of my colleagues have argued that Usertesting.com offers “professional” test participants. It is true that on 2-3 occasions test participants inadvertently showed me their test session dashboard which showed that they had been doing about 10 test sessions recently. Personally, I don’t mind this. I have yet to see a participant who was so experienced in user testing that it prevented me  from getting the data I wanted.

All in all: Recommended

Usertesting.com has a great concept and implements it with care. Before their recent price increase, I would even have said “Highly recommended”.

Resoures

www.usertesting.com

The comparative study of Budget.com that I used usertesting.com to test (CUE-8).

Categories
blog Events

Escape The Lab – Aug. 26

Want to learn remote research? Bolt | Peters is hosting a one-day workshop on August 26th, and you’re invited. Give us a day and we can teach you all the rocket surgery you need to conduct qualitative studies the real-time, native environment way.

Date: Wednesday, August 26th, 2009
Time: 9am – 4:30pm. Sign-in starts at 8:30am, drinks and schmoozing afterwards
Place: Bolt | Peters User Experience at 60 Rausch St., unit 102, San Francisco, CA
More Info: http://escapethelab.com
Cost: $399. Register now (space very limited). 1/2 off for students and underemployed.
By: Bolt | Peters User Experience, the makers of Ethnio

Bolt | Peters Instructors

Cyd Harrell, Director of Research
Frances James, Lead UX Researcher
Nate Bolt, CEO

Who Should Attend?

Researchers, designers, and product managers who want to watch real people use technology from the comfort of their own desks. (While saving travel costs and the planet!)

What We’ll Cover

  • Strengths and weaknesses of remote ux research
  • Study design & scripting
  • Participant recruiting options
  • Moderating in the remote environment
  • Tools for screen sharing, recording, and communication
  • What can go wrong and what to do about it

What You’ll Take Home

  • A Trapper Keeper full of script outlines, consent forms, and software comparisons
  • A starter account for Ethnio online recruiting
  • A coupon for 20% off our forthcoming book, Remote Research
  • 15% discount on all Rosenfeld Media books
  • A newfound confidence in conducting your own remote research!

Register now at:

http://escapethelab.com/register.html (Space is superduper limited.)
Hope to see you there!

Bolt | Peters User Experience

Categories
blog B|P Livin'

B|P Intern Tells All!

The raw, uncut dirt on Bolt|Peters from our intern-for-a-day, UCSD Cog Sci student Jason van Merle. Enjoy! -Tony

——–

So over spring break I decided to skip the margaritas and take my sliver of free time to visit with Bolt | Peters in San Francisco. Last spring, Nate Bolt and Mike Towber had given a presentation to my cognitive engineering class about B|P and the remote usability testing work they do. After spending so much time studying cognitive ethnography, which emphasizes getting your hands dirty and observing users in context, I got curious how one might observe user interaction without being able to actually view the user! They have an impressive client list, and I found their presentation very interesting, so I thought I’d fly up to San Francisco and check them out. They were kind enough to let me intern for the day.

Categories
blog Conferencery

UPA Remote Usability Sessions

Most folks we talked to at the UPA conference this year in Colorado had conducted remote moderated testing using some combination of web conferencing tools and Morae. There were a couple sessions on remote usability tools, and they focused primarily on web apps like WebEx, Breeze, GoToMeeting, etc. One speaker, Mike, did mention that tools like ethnio moving forward will be the way to go since they are made for usability profesionals, which is cool.