Conducting a remote unmoderated usability testing

A quick-start guide

Mr. John was a UX guy at a food delivery restaurant which was in the process of building a mobile app for the ease of their users and more sales for the business. Mr. John was well aware of the rising expectations of the users with experience on app and so he wanted to test the app idea and its usability with a set of users.

But there were couple of challenges with his present project — tight budget and short timeline. Someone just suggested un-moderated usability testing. But John was worried…. because he had never conducted an un-moderated mobile usability testing before..

Mr. John is definitely a character in my imagination, but the situation is not an imaginary one.. This article will talk about a step wise approach to un-moderated usability testing and why one should go for it.

What is unmoderated usability testing?

It is a technique of testing that evaluates the usability of an interface or product; parameters such as

• the ease of use,
• efficiency and
• satisfaction customers have with the interface.

It is similar to in-person usability testing however participants complete tasks in their own environment without a facilitator present. The tasks are pre-determined and are presented to the participant via an online testing platform.

Steps for un-moderated usability testing :

The un-moderated usability testing is a three fold process-

1. Preparation for the test
2. Sharing the prototype link
3. Find out what’s happening quantitatively

Preparation for the test —

#1. To begin with, define the specific objectives for the test. It’s important to fully understand why the research is being conducted. This will help with recruitment, designing the questions and tasks, and provide a reference point for analysis and subsequent discussions later.

#2. Write the tasks very clearly. Because you won’t be present to help explain the tasks to the participants, you need to make sure the scenarios are clear and specific. Avoid adding extraneous information in a task, which may confuse the test participants. Having said all of this, the one thing I really want to emphasise on is — Keep it short.

Tip : Before you shoot the task to the users run a pilot run internally. This will eliminate any semantics or typo errors. Not just that, it will also help you understand if your users would understand your task.

#3. Recruit your users. Many of the tools let you choose participants from their pool of candidates, while some will let you add participants from your own list. Both have their upsides and downsides. What has to be kept in mind is that users should be as close to your user persona as possible.

Tip : Recruit more users…. No-show rate can be high. You also don’t know the quality of an un-moderated session until you’ve watched it. It’s better to add a few more users than you think you need in order to accommodate such problems.

Another way finding users — validate your prototypes with Magic URLs. These are social sharing URLs that can be shared on any channel — Facebook, Twitter, LinkedIn, Skype, Hangout, Slack etc.. Any user who tries out the prototype accounts for the user behaviour analysis.

#4. Select the right online tool for the test. There are a dozen tools on the internet that support un-moderated usability testing. But obviously one wants to choose the best one! Here are the factors that you should check for while selecting your tool.

• Records user sessions and shows time taken on each design screen. (A must have)
• Has an option of dividing the entire test as multiple tasks.
• Can test anything — paper prototypes, wireframes, mobile app prototype, website and everything else.
• Instant results
• Leave feedback in terms as comments and annotations.
• Shows user behaviour data in the form of heat maps and conversion funnels.
• Direct sync with majority of tools you are using.

Having said that, CanvasFlip is an obvious choice for any designers and UX experts across the globe. (Sign up and setup your un-moderated usability testing)

Sharing prototypes for testing —

Lots of planning done! It’s now time for execution. Share the prototype with the users via email or url. Ya this is a pretty simple step. There are a couple of things you should keep in mind during this time.

1. Be available!
   Have I lost it? It’s un-moderated testing! Accepted. But even for un-moderated studies, be available by email, if not by phone, to help with any potential user questions.
2. Schedule your email as per time zones
   The quarter marketer inside me speaks! Your users might belong to different time zones and sending the prototype link at one-time-fits-all will not work. Schedule the emails such that the user receives the email when it is 9.00 am — 11.00 am for him/her. This will ensure lower no-show rates.

Find out what’s happening quantitatively—

Quantitative research such as heat maps and conversion funnels do an excellent job at answering “how many” and “how much” type of questions. An initial analysis of numbers is required to set the conversation with the users, while trying to understand their mindset. Heatmaps, conversion ratio, time taken for a task, interactions performed etc are easy to collect with online tools. I use CanvasFlip for getting the quantitative data.

Why choose un-moderated over moderated usability testing?

1. More realistic feedback : With traditional usability testing in labs, you are putting users into an unfamiliar environment, making them use your unfamiliar devices rather than their own, and physically watching over their shoulders, directly interfering with their experience.
   But with un-moderated usability testing there are obvious chances that the user will slip into his “normal mode”. And the feedback you get will be much more relevant to how real people are using your app “in the wild.”
2. Accessing an audience that is geographically dispersed : When the audience is dispersed, it involves a lot of cost, effort and time to either get to them or call them for a testing. A remote usability testing makes a lot more sense in such situations.
3. To get a large sample of user behaviour data. A small sample of in person usability tests may be all that is required from a design perspective but if your stakeholders are used to seeing large samples then using big numbers may be simpler than trying to convince them of the value of the small sample.
   Further, where a new design is critical for an organisation or will have will have a substantial impact, the confidence gained from a large sample study can be valuable.
4. To compare two or more products or interfaces. Unmoderated usability testing is perfect for benchmarking studies — comparing either competitor products or different iterations of your own product. The ability to capture large sample sizes means that statistically significant differences between interfaces can be identified.

Over to you :

Un-moderated tests provide statistically significant usability results in a short period of time. And the beauty of the process is that one does not have to wait for any final interface. It can be done as early as on paper prototypes. So test often and test early!

Happy testing!