Methods and Types of User Testing

With all of the information floating around about user and usability testing, finding the information you need to get started with user testing can be tricky — that’s why I wrote this article.

We’ll go over two things here — methods of user tests and types of user tests.

Hopefully this can help you figure out the right path for testing your product with users.

Ways You Can Test Your Product With Users

In-person vs. Moderated vs. Automated Testing

The method that you test with users can yield different results. Each has appropriate situations and benefits. If you are having trouble choosing a method, this section should help.

Let’s break the methods of testing down into three buckets:

In-person User Testing

In person user testing is by most people’s opinion- the best method of testing with users. An in-person test is when the observer is in the same physical location as the tester. This provides several benefits to the research.

• It allows a more focused test by controlling, or helping control the environment
• Controlling the technology factors allow for less technical difficulties or distractions
• Being present with the person allows for better observation of human expressions
• It allows for the ability to questioning or directions from the observer to facilitate the test
• Having someone nearby will either put the tester at ease
• Testing in the tester’s own environment will also put them at ease

Some of the detractors of an in-person test are that:

• It’s expensive to execute if you have to travel to meet with a target user group
• It’s time intensive due to travel and travel prep — if that is something you have to do
• Having someone watch over them could make them uncomfortable

Moderated Remote User Testing

Moderated remote testing is a sweet spot in the middle. For the purposes of this description, when we refer to live testing we are talking about remote testing with users over a video or audio connection. Moderated remote testing allows observers to test with users quickly and effectively over a meeting connection.

With companies I have worked for in the past, I have implemented user tests over a typical online meeting application. You can use Skype, Join.me, WebEx and others. Most of these applications have a built in screen recording function that allows you to record all of the test on the screen.

When selecting an application, be sure that it can expose or highlight clicks that the tester makes. This will allow you to track clicks for your test metrics.

If you would like to use another video conferencing application to record your user test, then you could always use a screen recording application like Quicktime or SnagIt over you screen to record the test. The disadvantage to this is that you will miss out on being able to track clicks.

Automated Remote User Testing

Automated remote testing is when a test utilizes a software or service to implement the test remotely with users.

There are many companies that offer the service of hosting your directions or script and recording the user executing the test.

Services such as usertesting.com, trymyui.com, and userzoom provide some of the most scalable and affordable user testing platforms for automated user testing.

This method does come with with a caveat however. While this method is affordable, testers will often lose out on things such as context, the ability to ask the tester questions about their thoughts, and an often stringent limitation on the amount of time a test can take.

While In-person and Moderated testing are usually not formally limited to time constraints, other than the tester’s time, Automated Testing is typically fixed in time/scope — 5 minutes, 20 minutes, 10 questions, etc.

So which is the correct method for you? It all depends on what your time, budget and data needs are for a specific test. While one method might be good for certain tests, another method might be best for others.

Being able to record screen and faces is great, but don’t hog all the fun! It’s a great idea to have a colleague sit in with you and take notes. It’s not easy to administer the test itself and take notes. Possible, but not easy.

While there is always the ability to go back and watch the test footage later, there is something about hearing or seeing a test live that is fascinating and informative. That experience just doesn’t translate well into a video recording. I think part of it is that you are much more attentive to a live person using a product. You do have the opportunity to follow up with question in the moment, which is something not possibly with a recorded test.

Including other members of your project team in on the observation of a user test can help other people of the team make the same human connection and build empathy for your target users. In addition, it also helps build credibility for your user test and its results when other project team members that have sat in on the test. They are likely to help defend the user test and the target users after this has happened.

User Testing Etiquette

Be that any participating members follow user testing etiquette. User testing can raise emotions from people that you would not expect.

Sighing, speaking out and verbally disagreeing are common — and human reactions to what you might expect from an interaction testers make. Is this wrong? Not really, but it is in appropriate.

It’s a good idea to brief your colleagues who will be observing and/or taking notes during the user testing. Some common etiquette for a user test is below:

• Do not make audible or visible reactions to what users do during a test. Remain neutral.
• Do not guide the user outside of the script. If they ask for help like “what should I do?”, respond with “what do you think you should do”. If they decide to abandon the task, go ahead and help them. Sometimes testers will get frustrated and give up. Keep the test moving and proceed on to the next question after showing them how to proceed.
• Try to hold all of your questions to the end. As you go through the test you can take notes on specific tasks
• And in general, just make them feel welcomed and appreciated — they are giving you their time after all.

Variability of testing

Another important factor to take into consideration is variability. In layman’s terms, these are things that change, as opposed to constants — which are always the same.

Certain parts of a test can be variable, or things that change. Some of these things are location, time of day, the tester (obviously) and any observers are the most important things that come to mind.

However, when there is too much variability, or too many changes between tests, it can produce results that are not accurate.

Some things should be constant in order to have accurate results, or things that don’t change. Those things are the person administering the test, the test script, the device or prototype you are testing on, and the method in which you are testing (in-person vs. remote).

Types of User Tests

There are a number of data points you can collect from users that others would consider part of a user test. The three key user tests below are what I consider the most valuable for most businesses.

Usability testing

Usability tests are intended to test the usability of a product or service. An application, a web app, a website, etc. The primary goal for the type of test is to find usability issues with the design — ideally before it goes public to the larger user group.

A usability test involves people using a product so that you can get better insight into the ease and intuitiveness of a design. Observers watch and take notes for direct input on how real users use the system.

Usability can center around a number of things:

• Ease of use — Can the user do what they need to do easily? How many clicks does it take?
• Task Completion — Can the user complete the task?
• Legibility — Are the calls-to-action clear? Is the hierarchy of typography clear to the user?
• Time on task — Does it take a reasonable or expected amount of time to finish the task?

Example Usability Test — Data Entry

Let’s take an all too familiar (for me at least) example of a usability test — a test around data entry in a form.

Every day users enter data on forms — whether it is to sign up for a new subscription, schedule an appointment, or make an e-commerce purchase.

How usable entering data on a form can be measured. Below are just a few of the places you could do that.

• Are the input fields clearly marked?
• Are the sizes and locations of the input fields in a layout that is easy to fill out? Does the sequence of the input fields make sense to the user?
• Is there appropriate error handling for incorrect entries?

Benchmark Testing

A benchmark tests of a new design vs. an old design of your product, or tests your product against another product.

Let’s say you’re designing a new version of your application. You want to be sure that your new version is more valuable than your existing one. So you run a usability test on your existing one to get a baseline score. You then run the same test on your new design.

At this point, you have these two scores. Typically, most companies would only want to move forward with the new design if it met or exceeded the old design.

Example Benchmark Test — Redesign of Data Entry

Similar to the example above, let’s say you wanted to benchmark an existing design in order to compare a new design.

For each of the two designs, you could test entering three identical data points into input fields and submitting the data. If the data for the existing design was faster to complete or provided a better experience to the user when compared to the new design, you would know that you needed to improve your new design before releasing it.

Competitive benchmarking

Instead of benchmarking against your own design, the only difference with competitive benchmarking is against the design of a competitor’s product. This can be challenging at different levels. The tasks you base your test upon must be the same for both products, or else you risk invalidating the test.

Example Benchmark Test — Data Entry

Instead of benchmarking against your own existing design, you could benchmark against a competitor with your new design.

Some of the same questions could apply to this sort of test

• Are the input fields clearly marked?
• Are the sizes and locations of the input fields in a layout that is easy to fill out? Does the sequence of the input fields make sense to the user?
• Is there appropriate error handling for incorrect entries?
• Which one provided the better experience? Does the user have a preference?

A/B Testing

Testing one design against another to gauge preference and performance from users. This is similar to a benchmarking test, but tests both new designs instead of against existing ones.

Instead of trying to find a baseline to beat, A/B Testing (sometimes referred to as “split testing”) is a test for optimization.

Your goal is to choose the best here (both of which you have full control). The key difference between a benchmark and an A/B test is after a benchmark test, you may have to go back to the drawing board if your “variable” or new design score below your old one. That’s not the case with an A/B test.

There are many other things that some people bucket into a user test, things that may fall within user research, but are not actually considered a user test. Things like card sorting exercises, user interviews, task analysis or surveys, while similar, are not included in this article.

Example A/B Test — Data Entry

A/B tests are a comparison between a variable in two versions of a design. In order for results to be accurate from this sort of test, you can only change 1 thing.

An example of this might be a numerical input field for data entry. One version could be a simple “keyboard/number” text field and the other could be a multiple choice or even a graphical slider. You could text this input type between the two designs to find out which one was more effective, but that would be the ONLY thing you change between the two.

Multivariate Testing

This type of test is similar to an A/B test, except you are testing a number of variables (colors, content, design, etc.) at the same time. This is different than an A/B test in which you are just testing a single variable . The goal is to determine what combination of variables is the most successful.

Concept Testing

Concept testing is a way to validate a design concept with users or customers. The terminology around concept testing is a slippery slope, and some advocate that concept testing can be more of a marketing or business activity as opposed to a user research one. Additionally, the terms usability testing and concept testing are incorrectly misused together.

Concept testing often involves bringing entire concepts out to users for them to choose which one they like best.

Where as A/B testing might be testing a small portion of something like a call-to-action, a concept test might include something like entire workflows or entire app layouts.

Concept testing often involves using a low fidelity prototype to test the concept. The point of this type of test is to get feedback on the value of the prototype, and not necessarily the usability

Usability tests and concept tests are two separate things and should never be combined.