Thomas Saldanha.
//Training programme

User research & testing.

My experience of planning, recruiting, running and analysing user research, plus five core testing methodologies.

Jump to chapter
Use the arrow keys or click to navigate
Swipe or use the arrows
01 / 33
//Lunch & learn
Thomas Saldanha.

User research & testing

An introduction to planning, recruiting, running and analysing a study and some basic methodologies to consider.

Mentoring · Talk handoutSix chapters
//Contents

What this covers

01
Planning
Define the hypothesis, then choose the shape of the test.
02
Recruitment
How many, who, screening, compensation and consent.
03
Conduct the test
The pre-flight checklist and a testing script that travels.
04
Analysis
Turn what you watched into a documented, defensible decision.
05
General tips
The hard-won principles that keep a session on track.
06
Methodologies
Usability, guerrilla, A/B, card sort and tree test.
Who is this session for?

Built for anyone new to research and for the teams who commission studies. It gives you a foundation to work from and gives the business a clearer set of questions to bring to any study.

//Chapter 01

Planning

Frame the study before you build anything.

//01 · Planning

Define the purpose & type of test

Start with the hypothesis you want to challenge

It guides stakeholder conversations and keeps scope tight. Testing too much dilutes everything, so be focused. From there, filter the hundreds of methodologies with three decisions.

//01 · Planning

Moderated or unmoderated?

Moderated
A facilitator guides the participant throughout.
+Higher engagement; you can warm a quiet participant.
+Probe any in-session remarks live.
+Guide the study back on track if instructions are unclear or the tech fails.
+Complex tasks are likelier to succeed.
−Risk of bias, as you may inadvertently prompt.
−Hard to align both diaries efficiently.
−One participant at a time, per moderator.
−Risk of the observer effect.
Unmoderated
No one is present while they take the test.
+Runs at their convenience, no diaries to align.
+Test many at once and quicker end to end.
+Less risk of bias and observer effect.
−No one to unblock a stuck participant.
−No follow-up questions on in-session remarks.
−Hard to use for tests that need complex instructions.
//01 · Planning

In-person or remote?

In-person
Participant and facilitator share a room.
+Generally more engaged, as it is easier to build rapport.
+You catch subtle behaviour you would miss remotely.
+Control the setting and use hardware like eye-tracking.
+Replicate the real use environment.
−Hard to source a wide geographic pool, for travel cost and time.
−They generally cost more to conduct.
Remote
They are not in the same room.
+Cheaper to run, no travel or hotel cost.
+Easy to recruit a wide geographic pool.
−Harder to control the environment.
−No specialist hardware, no real-world setting.
//01 · Planning

Quantitative or qualitative?

Quantitative
Anything you can count or measure. “4 of 5 chose Pepsi over Coke.”
+Tells you what happened.
+Effective for testing a theory or hypothesis.
+Quicker to analyse.
−Will not tell you why.
Qualitative
Anything expressed in words. “It was much sweeter than I expected.”
+Tells you why.
+Good for a concept or experience.
−Often will not tell you what happened.
−Takes longer to analyse.
//01 · Planning

How will you document findings?

Think about documentation early. It shapes how you set the study up. Ask yourself:

Do you need a detailed document or will summary notes suffice?
Are you presenting to stakeholders or the internal product team?
How experienced is the audience in reading testing insights?
Will you have video or audio clips to support your insights?
Define success criteria
Pre-agree what passing looks like. E.g. This icon passes if 8 of 10 identify what it represents.
Agree it before launch, so you cannot move the goalposts later.
//Chapter 02

Recruitment

Find the right people and screen them well.

//02 · Recruitment

How many participants?

5 is the usual minimum for early concepts, but I personally prefer 10, especially unmoderated. If one session fails over tech etc you still have a clear majority rather than a potential 50/50 coin toss.
For optimisation studies like A/B tests you will recruit significantly more than five. The number varies per company, so use an online calculator to reach statistical relevance.
The number also varies with the severity of the consequences if something goes wrong. For example, you would not test something life-threatening only five times.
100 hrs ≈ 2½ wks
Do not over-recruit. 100 participants is 100 hours of footage, about two and a half weeks just to rewatch.
There is a well-known Nielsen Norman Group article on sample size: Why 5 users is enough
//02 · Recruitment

What characteristics do you need?

Should they know about your brand?
Should they be a new or existing customer?
If existing, how frequently should they have purchased?
Is any of the following important?
Age, Gender, Location, Salary or Employment
Do they have to use Android or iOS?
Be careful, there is a fine balance between letting everyone in and making it too difficult to find participants. The more specific you make the participant requirements, the more money it will cost to find them.
Where are you sourcing your participants?
Internal database
Cheaper, but blunt on targeting.
External recruiter
A fee, for a far larger, better-targeted pool.
//02 · Recruitment

Screening the pool

Defining participant characteristics filters the pool once. A second pass narrows it to the individuals who are actually eligible for your study, and that second pass is called a screener.

Eliminate early, biggest criteria first.
Do not reveal the study’s aim.
Ask open questions about real behaviour.
Avoid leading and yes/no questions.
Screening for a trainer brand, you would…
Avoid
“Do you regularly buy trainers?” (yes / no)
Better
“Which of these products have you purchased two or more times in the last six months?” Multi-select.
T-shirts, Jumpers, Jeans, Hoodies, Jackets, Shorts, Suits, Trousers, Trainers, Shirts
//02 · Recruitment

Compensating participants

Are you planning to compensate participants for their time?
In what form: cash, vouchers, discounts or products? Check with accounting before issuing cash, for tax reasons.
Compensation value should vary to reflect the length of time and complexity of each study.
Only compensate those who pass the screener and complete.
Pay promptly. Never make a participant chase you.
You pay for their time, not the quality of answers.
Unless they clearly ignored the instructions, release payment.
//02 · Recruitment

Capturing consent

Before launch, capture consent to take part and to have data processed. Provide the form well in advance of the session, so no one is pressured into signing without reading it.

Purpose of the study.
Tasks and duration.
Data storage and access.
Compensation.
The ability to withdraw consent at any time.
Consent for audio and video to be recorded.
If the study involves a minor, you will need approval from a parent or guardian.
Reference
Nielsen Norman Group publish an example consent form.
Example consent form
nngroup.com · PDF
//Chapter 03

Conduct the test

Run a calm, consistent session.

//03 · Conduct the test

Pre-test checklist

All participants have the URL, address and time of their test
Consent forms completed.
Compensation details on file.
Script written and dry-run.
Devices loaned and batteries full.
Third-party tools installed and working.
Screen, audio and video permissions granted.
Wi-Fi stable in the testing location.
Meeting-room booking confirmed.
Arrived well before the first participant
//03 · The testing script

Introduction

Warm the participant with small talk. The content does not matter, it eases them in and starts a natural flow.
Confirm contact details and that consent is signed.
Set the scene: purpose, who you represent, your role.
Reassure them there are no right or wrong answers. It is not a test of them, and nothing they say is taken personally.
Encourage them to vocalise their thoughts. Explain it’ll feel odd at first, but it shows you what they are seeing and the thinking behind their actions.
Give a high-level overview of the structure, e.g. “I will ask you to interact with a prototype, then follow up with some questions.”
Finish with “Any questions before we begin?”
A script is there to guide and prompt you, not to be read out word for word.
//03 · The testing script

During the test

Simple tasks first, to build confidence.
Keep questions open ended. Avoid anything likely to get a one word answer: it tells you half the story and does not encourage them to disclose opinions, thoughts and emotions.
Avoid leading questions. “How easy was checkout?” presumes it was easy, where “How was the checkout experience?” does not.
Do not presume answers on behalf of the participant. When an answer is ambiguous, ask them to expand and clarify.
Your primary role is to observe, so avoid talking where you can. It keeps you from introducing bias, and from interrupting a participant who is in flow.
Use a deliberate pause. Silence does more work than another question, and it gives them room to fill it.
Watch for someone answering to please you. Reassure them that they cannot get this wrong and that you value their honest thoughts.
If asked what something does, hand the question back with “what would you expect it to do?”
Prompts worth keeping to hand
“What do you mean by X?” “Could you give an example?” “What makes you think that?” “What would you expect?” “Tell me more about that.” “What were you expecting to happen?” “Is that what you expected?” “What would you do next?” “Why?”
//03 · The testing script

End of test

Before we end, is there anything else you would like to mention?

It lets them add context or raise something you never asked. Then thank them for their time and feedback and remind them to get in touch if compensation has not arrived within 48 hours.

//Chapter 04

Analysis

Turn what you observed into actionable recommendations.

//04 · Analysis

From insights to recommendations

01
Confirm compensation

Check every participant has been paid before anything else.

02
Review & document

Work the footage into the format you agreed up front.

03
Present & conclude

Present the findings to your stakeholder and agree the next steps.

Ends one of three ways Proceed as tested Investigate further Re-test
//04 · Analysis

Different methods require different analysis

The study you ran dictates how you analyse it. Below are a few examples of what you might look at.

Card sort
A similarity matrix, the groupings participants made and the language they used to name them.
Tree test
Success and directness rates per task, and the paths where people went wrong.
Usability test
Task success, a severity-rated list of issues and clips you can show.
Guerrilla test
Rough notes and an early signal, enough to spot a blocker.
A/B test
The difference in conversion between A and B, with a significance figure.
Agree the output format before you launch, not after.
It is the difference between writing findings up and hunting for them.
//04 · Analysis

Deciding what actually matters

Not everything you observe is worth acting on. Before you take anything to a stakeholder, weigh it up in four ways.

Frequency
How many participants ran into this issue. A single instance is a note to keep an eye on, a repeated one is a pattern.
Severity
How much it got in their way. Did it stop them completely, slow them down or just irritate them?
Cost to fix
What it would take to put right. Changing some copy and re-platforming are very different conversations to have.
Commercial impact
What it costs the business to leave alone. An issue on the checkout carries more weight than the same issue three levels into the help pages.
Weighing all four gives your stakeholder a list they can act on, rather than everything you noticed.
//04 · Analysis

Keep these three apart

Not everything can be quantified, but separating the three moves a finding from your opinion towards something closer to a fact.

Observation
“Three of ten opened the wrong tab first.”
Interpretation
“The tab labels do not describe what is inside them.”
Recommendation
“Rename the tabs using the words from the card sort.”

Collapsed into one sentence, a finding is easy to dismiss. Kept apart, a stakeholder can disagree with your recommendation without disputing what happened.

Expect them to disagree sometimes. They often hold commercial or technical context your study never saw, so do not take it personally.

//Chapter 05

General tips

What keeps a session on track, whatever the method.

//05 · General tips

Practice makes perfect

It is not about you

Participants do not know what you are looking for. If it doesn’t go to plan, learn from it.

Progress over perfection

You will never get the perfect setup. Just stay aware of the concessions you make.

Stay focused

Do not do too much. Chase the one thing that drives the most value.

//Chapter 06

Methodologies

Five methods worth knowing and where each one fits.

//06 · Methodologies

Usability testing

Behavioural

Observe real behaviour and reactions as people attempt a set of tasks with your product.

Process
You ask participants to complete a series of tasks while interacting with your product.
Whilst they conduct these tasks you may observe and record their reactions and behaviours using a range of tools such as eye-tracking, heat maps, video, audio and screen recording.
Where possible you want the testing environment to replicate the real-life conditions in which users would typically use your product.
Conducted with participants who are pre-recruited.
To avoid participant fatigue, we recommend tests should last no longer than 30-40 minutes.
Benefits
+Validates your product is intuitive and fulfils your end users’ expectations and needs.
+Receive unbiased feedback from real users, helping to validate or challenge stakeholder assumptions.
+Validate hypothesis and concepts prior to investing significant cost in large complex builds.
Use case
Usability testing is incredibly versatile and can be used in almost any use case where you want to test and validate a hypothesis.
//06 · Methodologies

Guerrilla testing

Behavioural

A quick, low-cost test in public spaces: ask passers-by to try a few simple tasks with your prototype.

Process
Unlike formal usability testing, participants are not pre-recruited so they won’t represent your real end user.
You are unlikely to be able to observe and record the participants’ reactions beyond a video and audio recording as you’re limited by the environment’s setup.
As per formal usability testing you still ask participants to complete a series of tasks while interacting with your product.
Due to the informal nature of guerrilla testing, we recommend limiting tests to no longer than 10 minutes to keep your participants engaged.
Benefits
+Relatively minimal planning and preparation required prior to launch.
+Validate simple hypothesis and concepts prior to committing to larger formal tests.
+Show the advantages of user testing to stakeholders, with the intention of convincing them to commission a larger study in the future.
Use case
Used primarily for small tests that need to be executed quickly with a minimal budget and it’s not essential you use a participant who accurately reflects your real user.
//06 · Methodologies

A/B testing

Optimisation

Change a single variable of an execution and compare whether version A or B performs better.

Process
You create two versions of the same execution, with the exception of a single variable.
You show half of your audience version A and the other half version B.
To avoid ambiguity you only change a single variable, so it’s clear what is the responsible factor for the performance change.
Benefits
+Utilises data-driven decision making rather than relying on gut feelings.
+A/B tests are often restricted to a smaller sub-section of your audience, so the impact of any major failures are often limited.
+Quick and easy way to optimise an existing experience.
+Cost-effective strategy as any negative dips in performance may be quickly identified and reverted.
Use case
A/B tests are incredibly versatile and can be used in almost any use case where you want to test two versions of a single execution.
//06 · Methodologies

Card sorting

Information architecture

Understand how users group and relate your content, by watching them organise it themselves.

Process
You provide the participants with a series of cards, each containing one piece of content. You ask them to sort the cards into groups that feel natural to them and provide a title for each group.
We analyse first party data alongside business priorities to identify which cards should be included.
To maintain participant focus, we recommend providing no more than 40 cards to sort.
Benefits
+Creates a logical and intuitive structure that aligns with your actual users’ expectations on where content should be located.
+Helps identify the natural language your users would use and expect to see for each section heading.
+By getting the structure right early on, it reduces the need for costly re-designs later in the project.
Use case
Used primarily to help inform a draft website information architecture and primary navigation.
//06 · Methodologies

Tree test

Information architecture

Evaluate how easily users navigate your information architecture, with no designed interface in the way.

Process
You ask participants to navigate through a non-designed text based navigation, to identify where they would expect to find a piece of content.
We analyse first party data alongside business priorities to identify which tasks should be included.
To maintain participant focus, we recommend asking participants to find no more than 10 tasks.
Benefits
+Validates if your information architecture is logical and easy to navigate.
+It separates the information architecture from the user interface, so there is no ambiguity if a test fails which part is responsible.
+Helps identify potential locations to display lateral links to re-direct a small portion of users who are unable to initially find the correct pathway.
Use case
Used primarily to validate your draft website information architecture and primary navigation.
//Thank you
Thomas Saldanha.

Now go and run that study

If your team could use a session like this or you just want to talk research, I am always happy to.

hello@thomas-saldanha.com LinkedIn uxtom.co.uk

Why I wrote it

Throughout my career, this has been the number one thing colleagues ask me to teach them, so I put it together properly once. It gives people new to testing a foundation to work from and gives the business a clearer set of questions to bring to any study.

Run this for your team

I run this with product, design, marketing and wider business teams, tailoring it around the studies they have coming up.

hello@thomas-saldanha.com