Guidelines on how to plan, recruit, moderate, and analyze a research.

Learn about preparing the test, User Recruitment, Scheduling interviews, Being a moderator, Test, Analyse, Document.

Preparing the test

1. Define the purpose

• Objective: The bigger question for which you’re trying to find answers
• Hypotheses: Assumed answers based on data/previous research is to be added as applicable
• Success: If the purpose is to test a design/hypothesis, then the tasks, scoring and evaluation criteria for the tasks have to be agreed upon.

2. Choose Research Type

• Exploratory
• Evaluative
• Validation

3. Choose Methodology

• Direct Face to Face
• Survey Form
• Audio/Video Call — Scheduled/Random
• Focus Group Discussions
• User Testing tools

4. Define Participants

• Behavior (eg. visited page > 5times)
• Demographic (eg Age, Location, Salary. etc)

5. Build Questionnaire

• Create questions and tasks based on the objective of the research
• Avoid leading questions

User Recruitment

• Send limited mailers to the users exactly matching the participant criteria for the test
• The quality of participants sourced for the interview is more important than the quantity
• Make sure the participants are aware of the basics of the session (the type of questions, time taken, etc)
• Why they are taking part in the interview/ their benefits & our benefits should be clearly explained to them at the beginning
• Mailers inviting the candidates should clearly explain the purpose to sign up
• In addition, a further level of filtration with a basic form related to the questionnaire is recommended to weed out participants that have signed up just to vent about the product

Scheduling Interviews

• Only the users meeting the specific behavior/demographic requirements of a test should be made eligible to participate
• In addition, check if the profile is active on our platform by checking the last login, or any other parameters based on the participant definition
• Share profiles with the Customer Support team, who would help to schedule the interview with the participants on a shared calendar — on the defined date and time slots

Being a moderator

• 2 Moderators are recommended per interview — one for observing emotions and taking notes in detail, and the other to probe the participant and engage them
• Should be comfortable with the mother tongue of the participant
• Should have good communication skills, and should observe and gather insights from things that are also not said in the interview
• Should have the time to prepare and actually conduct these interviews

If you’re chosen as a Notetaker :

Observe and write in shorthand every conversation. Don’t miss out on the emotions, and use the right words to describe them. Remain silent during the entire conversation, Don’t make the candidate nervous when writing notes.

If you’re chosen as the interviewer:

Get a brief from the UX Designer about the purpose of the test, the questionnaire to be asked, probing questions, tasks if any, evaluation criteria, etc. Prepare the questions to be asked in advance, and learn about the basic user profile before the call. (this will help you filter the relevant questions to be asked and to empathize better on the call)

• Level Down. Don’t introduce yourself as a PM, rather as a mere Test Moderator, who has been assigned a task to collect answers for few questions prepared by the product team
• Talk slowly, loudly, and clearly, they might be having a bad connection or other disturbances
• Have the user’s profile ready and ask only questions relevant to them.
• Don’t ask questions that have obvious answers, the user could get offended.
• Ask them questions about product features so that they wouldn’t doubt the authenticity of the call.
• Ask Personna questions in parallel to some other question being answered about the product. (Find the right time)
• Prepare probing questions, and probe them to talk to you more about a problem.
• Don’t give them the answers. The purpose of the research is to understand what the customer knows and not to teach them how to use the product
• Act Dumb. If you act too knowledgeable — participants might tend to ask unnecessary doubts and not attempt to solve problems by themselves.
• Pilot and practice

Test/Interview

Before the test:

Introduce yourself as a moderator and brief both your and their role for this research clearly to them.

• Moderator’s Role: I am here to understand the expectations you have, your goals and motivations, and potential problems you are trying to solve using our product(I have a colleague helping me take notes and observe)
• Participant’s Role: During this session, I would like you to share your point of view frankly as this will help us improve our product. There is nothing right or wrong and any opinion is welcomed. If you have any questions, comments, or areas of confusion while you are interviewing please let me know.

Probing Questions during the test

• Why is that?
• When did you notice that?
• How do you do that?
• How come you feel that way?
• What’s an example of that?
• What’s the reason for that?
• That’s interesting. Tell me more
• What is the next step that you plan to do?
• What is it that you are waiting for next?

After the test:

• Compile all answers into the assigned sheet under the respective question
• Explain answers in detail as possible, immediately after the interview
• Make sure you capture all the emotions and explain that using the right words

Analyze

• Organize Ideas and data using an affinity diagram
• Create hypothesis based on similar patterns
• Discuss with team Conduct another round of research based on this hypothesis

Document

• The status of the research and upcoming researches planned should be visible to all
• The findings from the research, hypothesis created, or results that are evaluated needs to be documented and shared
• Next steps for the research should also be defined if any, or marked as closed