Most folks we talked to at the UPA conference this year in Colorado had conducted remote moderated testing using some combination of web conferencing tools and Morae. There were a couple sessions on remote usability tools, and they focused primarily on web apps like WebEx, Breeze, GoToMeeting, etc. One speaker, Mike, did mention that tools like ethnio moving forward will be the way to go since they are made for usability profesionals, which is cool.
I’m hitting up Adaptive Path’s UX Intensive workshop in Chicago this week, and helping them out as the event photographer and box mover. There has been some dope discussions of research methods today by Todd Wilkens.
Achewood, an internet comic where independently wealthy cats write advice columns and robots lie, comments astutely on the usability profession:
To support the growth of our research service business and the success of Ethnio, we have some new folks that have joined the team in the past couple of months. Cyd Harrell brings nine years of experience in UX to our team as our new Director of UX Research. Before joining the team, Cyd was a director at Charles Schwab for their representative facing financial service application. That’s RFFSA if you want to get technical.
Also joining the team is Bryan Tsao. Bryan is the 5th graduate of the UCSD HCI program in 5 years to join our team. In addition to helping B|P out with user research and the development of Ethnio, Bryan is getting his masters at from UC Berkeley in the new iSchool (Information Management and Systems). Welcome Cyd and Bryan! Now if you meet people from the iSchool at CAL, don’t say “so you go to iSchool?,” because it sounds like you are asking if they go to high school, and that’s just awkward.
We are proud to announce that B|P will be leveraging its experience in remote research to expand into Remote Robotic Dog Treat Dispensers. Working with Lego, or actually, working with the toy robot kit we bought from Lego, Nate and Mike built this little robotic arm that receives IM’s with the word “treat” and then dispenses a small treat to the office dog.
You’ll soon be able to IM the wild pooch a treat and watch it on our office webcam. The venerable Make zine picked it up, and you can check out the video below! We know what you’re thinking, and no, it’s not clear to anyone at the office why we built this, either.
It’s wild how many of our Ethnio customers have been asking for just a little bit of testing assistance for their remote research projects. So many of you are expert practitioners real physical world research, and just need a little assistance in conducting remote research. Whether it’s study design, training, moderating a couple users, helping combine moderated and automated remote methods, using MindCanvas or RelevantView, or just making sure your remote study goes smoothly, we’re down to help. B|P can ensure that the executive team watching your first remote study will denounce corporate social hierarchy and just give you a hug. Unless you don’t want a hug from your executive team, in which case we’ll make sure you get a nice email.
B|P is excited to announce that international dialing and call recording is now included free-of-charge with all Ethnio licenses. But what is Ethnio anyway? It’s the world’s first remote usability web application that allows you to watch and record anyone using a computer in the world with the express purpose of making their computer easier to use. Now you can expand your user research to international users! No need to fly to Japan and France to conduct usability studies unless you’re craving sushi or croissants on the corporate dime. Also, some of the latest organizations that have started using Ethnio are Adobe, CareerBuilder, and HP. They’re off and running with both assisted studies, where our practitioners help with set-up, moderating, or live recruiting, and doing all the remote research on their own.
We completed a rapid remote usability study for Genentech recently for their internal time-reporting tool, and got them up and running with Ethnio. Rather than run off at the mouth about how much we helped them, we’ll just pass along their words:
“The B|P project was a total success. Their research for Genentech actually elevated the importance of usability to the executive level here. It’s now a focus for our entire I.T. organization. The old time-reporting system flooded our Service Desk with support calls the day it launched. The new application launch barely made a ripple. ”
sarah murgel
manager, user experience & interface design, corporate information technology — genentech
Thanks to everyone that came out for drinks and UX research discussions at User Research Friday a few months ago here at B|P! Along with KDA Research and Uzanto, we put on this little conference and had a wild turn-out, with talks from folks including Ravit Lichetenberg from HP, Wendy Castleman from Intuit, Indi Young, Lane Becker / Todd Wilkens from Adaptive Path, and many others. We’ve got some videos and notes of the sessions up onuserresearchfriday.com which has a sign up to hear about the next one. We’ll probably make advanced UX research the focus of the next event, since many fine citizens let us know they were familiar with the broad concepts we discussed. So bring your smarty pants next time, because it’s gonna get complicated. And crunk.
A big thanks to those that used the Ethnio alpha and beta over the last year, and participated in our Ethnio usability testing! We have more Ethnio news coming soon, and B|P is proud to announce three products as part of the Ethnio remote usability family:
To support the development of Ethnio, and the continued demand for remote usability studies, we have some new folks that have joined the team since our last update. Mike Towber is the fourth member here at B|P to graduate from UC San Diego’s Distributed Cognition HCI program. Also joining the team is Lead Software Architect Julian Wixson. Julian brings 15 years of professional programming experience to the development of Ethnio. Kate Nartker now plays the role of Office Superstar, handles all our accounting, and keeps the office and all of us in line. You can see the whole team here.
We’re pleased to announce that we’re hiring for two positions, a User Experience Research Specialist, and a User Research Intern! Since so many people are hiring during these Boom #2 days, you are probably sick of seeing these requests, and we can only hope that your frustration with the frequency of large-scale technology industry job market fluctuations will not prevent you from sending us someone that might be a good fit!
Whew! From Vancouver to Montreal to Broomfield, CO, we enjoyed meeting lots of you on our recent conference tour. You can get flickr’d out with photos of folks hanging at ourbooth at CHI or UPA here. Thanks to all those who stopped by and kicked it. It was great to hear about your recent remote research and experience with Ethnio.
B|P has made our remote usability study pricing public, with an interactive spreadsheet. You just enter in how many users you might like in a study, choose which deliverables you’d like, and it will give you a price range. Consulting firms hate to make their pricing public, but we think it makes things just a little easier.To see pricing, visit our remote usability testing page, and click on B|P Cost Estimator on the right.
Our favorite usability comic site, OK-Cancel, is featuring a guide to remote usability testing written by Nate. It will definitely not make sense to anyone who is not involved with usability in some fashion. It will make even less sense if you have not had your soul sucked away by the computer industry and are leading a happy healthy life outside of the interwebs. More power to you, friend.
For all you multi-channel retailers, this month’s Multichannel Merchant has an article, ‘It’s The User, Stupid’ on how usability can “quickly gel differing opinions,” from your business stakeholders, designers, and developers. They quote Nate like crazy, which is always dangerous.
Come say hello to Bolt | Peters and the get fresh crew at Adaptive Path’s User Experience Week this year, August 14-17, in Washington DC, where Nate will be presenting ‘Playing Games and Cursing: The Truth About Remote User Research’ with Rashmi Sinha. Nate and Rashmi will be discussing Ethnio andMindCanvas, Rashmi’s saucy Game-like Elicitation Methods (GEMs) research web service. Together they’ll show how interface research that is typically done in a lab can be done remotely, and will help people curse at software and web sites less, by playing games and exhibiting more natural behavior.
Blog we created with explanations of the different remote usability methods and links to the latest tools and methods.
Dolby asked B|P to conduct usability research as part of an effort to completely re-design the Dolby website. Based on the findings from the first round of research that B|P conducted, Dolby was able to improve key areas of the site in the redesign, including site navigation, content structure and screen flow. The second phase was completed in November measuring usability improvements of the re-designed site compared to the previous one.
Techinsurance and B|P share the belief that improved user experience leads directly to the success of online products in the marketplace. This mutual understanding was so strong that both parties committed to an agreement structured around payment as a percentage of improved conversion rates for contractedge.com, a legal software web site. Techinsurance implemented the B|P usability recommendations for contractedge.com, and as a result saw a conversion rate increase of 103%!
Find out how you can structure a similar relationship with B|P.
To meet growing customer demand, we have made three new hires in 2004. Senior Information Architect Benjamin Lerch brings to B|P eight years of user interface research and design experience; Information Architect Brian Enright is the third graduate of UC San Diego’s Distributed Cognition HCI program to join B|P. Jocelyn Wine is our new office manager handling the ever-increasing load of administrative tasks. Thank you Jocelyn!
We’ve moved our satellite office! It’s now at 854 Folsom Street in San Francisco — right down the block from Yerba Buena Gardens and MOMA. We now have four permanent remote usability testing stations here, which can be used concurrently to test up to 20 users per day, as well as big leather couches and a full bar. Come visit! Of course, we still have our main office at One Market Street in San Francisco where we conduct our lab-based usability test sessions.
Bolt | Peters would like to wish you happy holidays and a wonderful new year. We are wrapping up the year here with a new office, three new team members, and the Alpha release of our new remote usability testing software! We would love to hear what you’ve been up to, so drop us a line and let us know
Want to conduct your own remote usability study? We walk you through every step of the research, from recruiting to observation.
This is a case-study comparing remote and lab-based usability testing. Bolt | Peters conducted two parallel Usability Studies on the corporate web site of a fortune 1000 software company in January 2002. Both studies used identical test plans but one was executed in a traditional usability laboratory and the other was conducted remotely, using an online screen-sharing tool to observe behavior.
You can also find more information on remote usability tools at the Remote Usability Wiki.
Our comparison of methods showed key differences in the areas of time, recruiting, and resource requirements as well as the ability to test geographically distributed user audiences. The table below provides a snapshot of the key differences we found comparing the two usability testing methods. There appeared to be no significant differences in the quality and quantity of usability findings between remote and in-lab approaches.
|
The table below breaks down the process for each of the recruiting, testing, and analysis phases. The left-hand column describes the lab study details, the right-hand column describes the remote study details.
Lab Recruiting
Recruiting for the lab-based study was outsourced to a professional recruiting agency. Ten users were recruited, screened and scheduled by G Focus Groups in San Francisco, including two extra recruits in case of no-shows. The total time required to recruit 8 users using the recruiting agency was 12 days. Agency assisted recruiting successfully provided seven test subjects for the lab study. The eighth recruit did not properly meet the testing criteria. |
Remote Recruiting
Recruiting for the remote usability study was conducted using a “live” online pop-up from the software company’s corporate website. The recruiting pop-up, hosted by B|P, offered the exact screening questions used by G Focus Groups to recruit users for the lab study. Users in both studies were selected based on detailed criteria such as job title and annual company revenues. Respondents to the online screener that met the study’s qualifications were contacted in real-time by B|P moderators. The online recruiting method took one day and recruited eight users total from California, Utah, New York, and Oregon. Normally the live screener requires 4 days of lead time to setup, but in this case it was completed for a previous project so setup was not necessary. |
Lab Environment
The lab study was also conducted from the software company’s in-house usability lab. The recruits for the lab study went to the lab in Pleasanton, CA to participate in the study and used a Windows PC to participate. In addition to users’ audio and screen movement capture, user’s facial expressions were also recorded. The video track of user facial expressions did not yield additional usability findings. |
Remote Environment
The remote usability study was conducted using B|P’s portable usability lab from software company’s headquarters in Pleasanton, California. The live recruits participated from their native environments and logged on to an online meeting allowing the moderators to view the participants’ screen movements. The users’ audio and screen movements were captured to be made into a highlights video. |
Lab Findings
The in-lab study uncovered similar issues of similar quality to the client when compared with the remote study results. The laboratory method uncovered 98 key findings, which is slightly lower than the in-lab results. The difference is too small to be statistically significant, but illustrates a trend with remote testing that more findings seem to surface than in lab testing. |
Remote Findings
The remote study uncovered usability issues of high value to the client. The number of key usability findings was slightly higher compared to the in-lab study. The difference of key findings is statistically negligible, however we believe there is a probability that remote testing could potentially yield a higher number of findings. |
Although both lab and remote methods delivered similar results, we found the following
key differences:
Remote usability testing delivers cost and time saving advantages with the benefit of testing globally distributed user audiences in their native environments. The remote testing approach is ideal for:
The in-lab usability testing method requires more resources and is limited to testing users on location. There are however, usability research projects that leverage the physical proximity of tester and participant.
1. Comp. Study of Synchronous Remote and Traditional In-Lab Usability EvaluationMethods Master’s Thesis Virginia Polytech, 3/2004
http://scholar.lib.vt.edu/theses/available/etd-05192004-122252/unrestricted/Thesis_Prakaash_Selvaraj.pdf
2. An Empirical Comparison of Lab and Remote Usability
Human Interface Design Dept, Fidelity Investmentshttp://home.comcast.net/~tomtullis/publications/RemoteVsLab.pdf
3. Two part article from IBM User Experience Team on remote usability testing
http://www-106.ibm.com/developerworks/web/library/wa-rmusts1/
http://www-106.ibm.com/developerworks/web/library/wa-rmusts2.html
