User Testing Plan

Research Objectives

To find out if the design system used is user-friendly, and whether it allows users to read and engage with the content quickly.

Hypotheses

1. Users will be able to search for what they are interested in via hashtags, which are intuitive.
2. FAQ / Rules section is visible and easy enough to understand.
3. Write-ups are short and concise enough to hold user’s attention.
4. Picture summary is engaging enough to hold user’s attention.
5. Article pictures are engaging enough in terms of write-up, copy and design to hold user’s attention.
6. Summary video is of adequate length to hold user’s attention.
7. Content of summary video is engaging in terms of design, voice over and content to hold user’s attention.
8. Design of content (for both video and picture) is easy to view, read and is easy on the eye.

User profiles

1. Muhammad Syafiq
2. Shaina Kang

Both users are casual fans of the Karjenner family. Both are students or fresh graduates of university who enjoy hearing about Kardashian news from their friends or scrolling through news about the Kardashians after a long day. They both prefer having the flexibility and option of reading about news, or watching news about the Kardashians. They are also existing users of the Telegram app.

How am I conducting the tests?

Tests will be conducted remotely so that we will be able to accurately observe the users in their own environments when they would be using the Telegram channel. I will ask them to set up two cameras–one directly above their phones, and one capturing their expression. I will ask them a series of questions as they go through the Telegram Channel prototype and engage with the sample content.

I will take on the roles of moderator, note taker and observer through a video call.

Test setup

Participants were tested in 2 different scenarios and were given tasks to complete.

Scenario 1: A busy day where users do not have the time to read through all the news and have specific family members they would like to check up on.

1. Enter the Telegram channel
2. View FAQ
3. Search for #summary to look for the summary messages with the top 5 news for the day.
4. Watch summary video.
5. Read summary write-up / picture slideshow.

Scenario 2: A relaxed day where users want and have the time to engage in all news about the Karjenner family.

1. Enter the Telegram channel
2. View FAQ
3. Continue scrolling from “unread messages”, where you last left off.
4. Read news article and its picture slideshow.
5. Click into discussion channel to discuss about that particular news piece.
6. Search for any specific Kardashian member with hashtag of their names.
7. Search for genre of news you are interested in via hashtag.

Test measures

Post-task

Ease: Single Ease Question (Overall, how difficult or easy did you find this task (from a scale of 1–7)?)

Effectiveness: Task Completion Rate (Success: Completes without prompts, Partial Success: Completes with a few prompts, Failure: Unable to complete at all)

Efficiency: Efficiency Question (Overall, I’m satisfied with the amount of time it took to complete this task (from a scale of 1–7).)

Post-test

Usability: System Usability Scale Survey (From a scale of 1–5)

1. I think that I would like to use this system frequently.

2. I found the system unnecessarily complex.

3. I thought the system was easy to use.

4. I think that I would need the support of a technical person to be able to use this system.

5. I found the various functions in this system were well integrated.

6. I thought there was too much inconsistency in this system.

7. I would imagine that most people would learn to use this system very quickly.

8. I found the system very cumbersome to use.

9. I felt very confident using the system.

10. I needed to learn a lot of things before I could get going with this system.

Satisfaction: Satisfaction Question (On a scale of 1–7, how satisfied were you with this Telegram channel?)

Sample Questions (Scenario 1)

1. What is your first impression of the summary video and/or picture slideshow?
2. What do you like about the summary video and/or picture slideshow?
3. Which one do you prefer and why?
4. What could be improved or is missing?
5. How would you see yourself use the summary video and/or picture slideshow?
6. Were the hashtags intuitive or allow you to find what you were looking for easily? What did you like? What could be improved?
7. What do you think about the FAQ/Rules section? What did you like? What can be improved?

Questions (Scenario 2)

1. What is your first impression of the news picture slideshow?
2. What do you like about the news picture slideshow?
3. What is your first impression of the news picture slideshow write-up?
4. What do you like about the news picture slideshow write-up?
5. What could be improved or is missing?
6. How would you see yourself use the news picture slideshow?
7. Were the hashtags intuitive or allow you to find what you were looking for easily? What did you like? What could be improved?
8. What do you think about the FAQ/Rules section? What did you like? What can be improved?