Purpose

This is a case-study comparing remote and lab-based usability testing. Bolt | Peters conducted two parallel Usability Studies on the corporate web site of a fortune 1000 software company in January 2002. Both studies used identical test plans but one was executed in a traditional usability laboratory and the other was conducted remotely, using an online screen-sharing tool to observe behavior.

You can also find more information on remote usability tools at the Remote Usability Wiki.

Summary

Our comparison of methods showed key differences in the areas of time, recruiting, and resource requirements as well as the ability to test geographically distributed user audiences. The table below provides a snapshot of the key differences we found comparing the two usability testing methods. There appeared to be no significant differences in the quality and quantity of usability findings between remote and in-lab approaches.

	Lab	Remote

Number of users	8	8
Recruiting Method	Recruiting Agency	Online Live Intercept
Recruiting Duration	12 days	1 day
Testing Duration	2 days	1 day
Location	Pleasanton, CA	CA, OR, NY, UT
Avg. Session Duration	85.6 min	51.5 min
Total Key Findings	98	114
Approximate Cost	$26,000	$17,000
Deliverables	Report, Highlight Video	Report, Highlight Video, Survey Responses

Detailed Comparison of Methods

The table below breaks down the process for each of the recruiting, testing, and analysis phases. The left-hand column describes the lab study details, the right-hand column describes the remote study details.

Lab Recruiting

3 rd party recruiting agency schedules users
Agency selects users based on recruiting criteria
Only local users are selected to avoid travel expenses
Duration: 12 days

Recruiting for the lab-based study was outsourced to a professional recruiting agency. Ten users were recruited, screened and scheduled by G Focus Groups in San Francisco, including two extra recruits in case of no-shows. The total time required to recruit 8 users using the recruiting agency was 12 days. Agency assisted recruiting successfully provided seven test subjects for the lab study. The eighth recruit did not properly meet the testing criteria.

Remote Recruiting

Pop-up screener on the website
Practitioner selects users based on responses to screener questions
Ability to cost-efficiently recruit globally distributed users
Duration: 1 day

Recruiting for the remote usability study was conducted using a “live” online pop-up from the software company’s corporate website. The recruiting pop-up, hosted by B|P, offered the exact screening questions used by G Focus Groups to recruit users for the lab study. Users in both studies were selected based on detailed criteria such as job title and annual company revenues. Respondents to the online screener that met the study’s qualifications were contacted in real-time by B|P moderators. The online recruiting method took one day and recruited eight users total from California, Utah, New York, and Oregon. Normally the live screener requires 4 days of lead time to setup, but in this case it was completed for a previous project so setup was not necessary.

Lab Environment

User in controlled lab environment
Test limited to users on location
Practitioner sees user screen on her computer
Practitioner sees user through a one-way mirror
User and practitioner interact via microphone and speakers
User audio, screen video and facial expressions are captured

The lab study was also conducted from the software company’s in-house usability lab. The recruits for the lab study went to the lab in Pleasanton, CA to participate in the study and used a Windows PC to participate. In addition to users’ audio and screen movement capture, user’s facial expressions were also recorded. The video track of user facial expressions did not yield additional usability findings.

Remote Environment

User in native environment
Ability to test globally distributed users from one location
Practitioner sees user screen on her computer
User and practitioner interact via telephone
User audio and screen video are captured
Lab can be set up at client site, or client can observe remotely

The remote usability study was conducted using B|P’s portable usability lab from software company’s headquarters in Pleasanton, California. The live recruits participated from their native environments and logged on to an online meeting allowing the moderators to view the participants’ screen movements. The users’ audio and screen movements were captured to be made into a highlights video.

Lab Findings

High quality of usability findings
98 usability issues uncovered
Identical test plan
Highlights video with P.I.P.

The in-lab study uncovered similar issues of similar quality to the client when compared with the remote study results. The laboratory method uncovered 98 key findings, which is slightly lower than the in-lab results. The difference is too small to be statistically significant, but illustrates a trend with remote testing that more findings seem to surface than in lab testing.

Remote Findings

High quality of usability findings
116 usability issues uncovered
Identical test plan
Highlights video with audio

The remote study uncovered usability issues of high value to the client. The number of key usability findings was slightly higher compared to the in-lab study. The difference of key findings is statistically negligible, however we believe there is a probability that remote testing could potentially yield a higher number of findings.

Conclusion: Choosing a Testing Method for Your Project

Although both lab and remote methods delivered similar results, we found the following
key differences:

Geographic reach
Time, cost and logistics
User environment
Perceived value of physical laboratories within your organization

When to use Remote Testing

Remote usability testing delivers cost and time saving advantages with the benefit of testing globally distributed user audiences in their native environments. The remote testing approach is ideal for:

E-commerce web sites
Large, informational web sites
Web Applications
Intranets

When to use Lab Testing

The in-lab usability testing method requires more resources and is limited to testing users on location. There are however, usability research projects that leverage the physical proximity of tester and participant.

Highly secure client/server applications
Handheld Apps or other products with a significant hardware component (requires observation of physical interactions)
Test sessions lasting three hours or longer
Projects where the client wants to physically observe the users