Purpose
This is a case-study comparing remote and lab-based usability testing. Bolt | Peters conducted two parallel Usability Studies on the corporate web site of a fortune 1000 software company in January 2002. Both studies used identical test plans but one was executed in a traditional usability laboratory and the other was conducted remotely, using an online screen-sharing tool to observe behavior.
You can also find more information on remote usability tools at the Remote Usability Wiki.
Summary
Our comparison of methods showed key differences in the areas of time, recruiting, and resource requirements as well as the ability to test geographically distributed user audiences. The table below provides a snapshot of the key differences we found comparing the two usability testing methods. There appeared to be no significant differences in the quality and quantity of usability findings between remote and in-lab approaches.
|
Lab |
Remote |
|
 |
 |
| Number of users |
8 |
8 |
| Recruiting Method |
Recruiting Agency |
Online Live Intercept |
| Recruiting Duration |
12 days |
1 day |
| Testing Duration |
2 days |
1 day |
| Location |
Pleasanton, CA |
CA, OR, NY, UT |
| Avg. Session Duration |
85.6 min |
51.5 min |
| Total Key Findings |
98 |
114 |
| Approximate Cost |
$26,000 |
$17,000 |
| Deliverables |
Report, Highlight Video |
Report, Highlight
Video, Survey
Responses |
|
Detailed Comparison of Methods
The table below breaks down the process for each of the recruiting, testing, and analysis phases. The left-hand column describes the lab study details, the right-hand column describes the remote study details.
Lab Recruiting
- 3 rd party recruiting agency schedules users
- Agency selects users based on recruiting criteria
- Only local users are selected to avoid travel expenses
- Duration: 12 days
Recruiting for the lab-based study was outsourced to a professional recruiting agency. Ten users were recruited, screened and scheduled by G Focus Groups in San Francisco, including two extra recruits in case of no-shows. The total time required to recruit 8 users using the recruiting agency was 12 days. Agency assisted recruiting successfully provided seven test subjects for the lab study. The eighth recruit did not properly meet the testing criteria. |
|
Remote Recruiting
- Pop-up screener on the website
- Practitioner selects users based on responses to screener questions
- Ability to cost-efficiently recruit globally distributed users
- Duration: 1 day
Recruiting for the remote usability study was conducted using a “live” online pop-up from the software company’s corporate website. The recruiting pop-up, hosted by B|P, offered the exact screening questions used by G Focus Groups to recruit users for the lab study. Users in both studies were selected based on detailed criteria such as job title and annual company revenues. Respondents to the online screener that met the study’s qualifications were contacted in real-time by B|P moderators. The online recruiting method took one day and recruited eight users total from California, Utah, New York, and Oregon. Normally the live screener requires 4 days of lead time to setup, but in this case it was completed for a previous project so setup was not necessary. |
Lab Environment
- User in controlled lab environment
- Test limited to users on location
- Practitioner sees user screen on her computer
- Practitioner sees user through a one-way mirror
- User and practitioner interact via microphone and speakers
- User audio, screen video and facial expressions are captured
The lab study was also conducted from the software company’s in-house usability lab. The recruits for the lab study went to the lab in Pleasanton, CA to participate in the study and used a Windows PC to participate. In addition to users’ audio and screen movement capture, user’s facial expressions were also recorded. The video track of user facial expressions did not yield additional usability findings. |
|
Remote Environment
- User in native environment
- Ability to test globally distributed users from one location
- Practitioner sees user screen on her computer
- User and practitioner interact via telephone
- User audio and screen video are captured
- Lab can be set up at client site, or client can observe remotely
The remote usability study was conducted using B|P’s portable usability lab from software company’s headquarters in Pleasanton, California. The live recruits participated from their native environments and logged on to an online meeting allowing the moderators to view the participants’ screen movements. The users’ audio and screen movements were captured to be made into a highlights video. |
|
Lab Findings
- High quality of usability findings
- 98 usability issues uncovered
- Identical test plan
- Highlights video with P.I.P.
The in-lab study uncovered similar issues of similar quality to the client when compared with the remote study results. The laboratory method uncovered 98 key findings, which is slightly lower than the in-lab results. The difference is too small to be statistically significant, but illustrates a trend with remote testing that more findings seem to surface than in lab testing. |
|
Remote Findings
- High quality of usability findings
- 116 usability issues uncovered
- Identical test plan
- Highlights video with audio
The remote study uncovered usability issues of high value to the client. The number of key usability findings was slightly higher compared to the in-lab study. The difference of key findings is statistically negligible, however we believe there is a probability that remote testing could potentially yield a higher number of findings. |
Conclusion: Choosing a Testing Method for Your Project
Although both lab and remote methods delivered similar results, we found the following
key differences:
- Geographic reach
- Time, cost and logistics
- User environment
- Perceived value of physical laboratories within your organization
When to use Remote Testing
Remote usability testing delivers cost and time saving advantages with the benefit of testing globally distributed user audiences in their native environments. The remote testing approach is ideal for:
- E-commerce web sites
- Large, informational web sites
- Web Applications
- Intranets
When to use Lab Testing
The in-lab usability testing method requires more resources and is limited to testing users on location. There are however, usability research projects that leverage the physical proximity of tester and participant.
- Highly secure client/server applications
- Handheld Apps or other products with a significant hardware component (requires observation of physical interactions)
- Test sessions lasting three hours or longer
- Projects where the client wants to physically observe the users
Further Reading
1. Comp. Study of Synchronous Remote and Traditional In-Lab Usability EvaluationMethods Master’s Thesis Virginia Polytech, 3/2004
http://scholar.lib.vt.edu/theses/available/etd-05192004-122252/unrestricted/Thesis_Prakaash_Selvaraj.pdf
2. An Empirical Comparison of Lab and Remote Usability
Human Interface Design Dept, Fidelity Investmentshttp://home.comcast.net/~tomtullis/publications/RemoteVsLab.pdf
3. Two part article from IBM User Experience Team on remote usability testing
http://www-106.ibm.com/developerworks/web/library/wa-rmusts1/
http://www-106.ibm.com/developerworks/web/library/wa-rmusts2.html
