User Testing Report

Resource Booking

Scenario 1

Task 1
Ease: Average SEQ = (7+7) / 2 = 7 out of 7

Effectiveness: Success = 2; Task completion rate = 100%

Efficiency = (7+7) / 2 = 7 out of 7

Task 2

Ease: Average SEQ = (6+6) / 2 = 6 out of 7

Effectiveness: Success = 1 ; Partial Success = 1 ; Task completion rate = (1 + 0.5) / 2 = 75%

Efficiency = (6+7) / 2 = 6 out of 7

Task 3

Ease: Average SEQ = (6+7) / 2 = 6.5 out of 7

Effectiveness: Success = 1 ; Partial Success = 1 ; Task completion rate = (1 + 0.5) / 2 = 75%

Efficiency = (6+7) / 2 = 6 out of 7

Task 4

Ease: Average SEQ = (7+7) / 2 = 7 out of 7

Effectiveness: Success = 1 ; Partial Success = 1 ; Task completion rate = (1 + 0.5) / 2 = 75%

Efficiency = (6+6) / 2 = 6 out of 7

Task 5

Ease: Average SEQ = (6+7) / 2 = 6.5 out of 7

Effectiveness: Success = 2; Task completion rate = 100%

Efficiency = (6+5) / 2 = 5.5 out of 7

Post-test

Usability: System Usability Scale Survey

1. I think that I would like to use this system frequently.

2. I found the system unnecessarily complex.

3. I thought the system was easy to use.

4. I think that I would need the support of a technical person to be able to use this system.

5. I found the various functions in this system were well integrated.

6. I thought there was too much inconsistency in this system.

7. I would imagine that most people would learn to use this system very quickly.

8. I found the system very cumbersome to use.

9. I felt very confident using the system.

10. I needed to learn a lot of things before I could get going with this system.

Total score: 92

Scenario 2

Task 1
Ease: Average SEQ = (7+7) / 2 = 7 out of 7

Effectiveness: Success = 2; Task completion rate = 100%

Efficiency = (7+7) / 2 = 7 out of 7

Task 2

Ease: Average SEQ = (6+6) / 2 = 6 out of 7

Effectiveness: Success = 1 ; Partial Success = 1 ; Task completion rate = (1 + 0.5) / 2 = 75%

Efficiency = (6+7) / 2 = 6 out of 7

Task 3

Ease: Average SEQ = (7+7) / 2 = 7 out of 7

Effectiveness: Success = 2; Task completion rate = 100%

Efficiency = (7+7) / 2 = 7 out of 7

Task 4

Ease: Average SEQ = (7+7) / 2 = 7 out of 7

Effectiveness: Success = 2; Task completion rate = (1 + 1) / 2 = 100 %

Efficiency = (6+6) / 2 = 6 out of 7

Task 5

Ease: Average SEQ = (6+7) / 2 = 6.5 out of 7

Effectiveness: Success = 1; Partial Success = 0.5; Task completion rate = (1+0.5) / 2 = 75%

Efficiency = (6+6) / 2 = 6 out of 7

Task 6

Ease: Average SEQ = (6+7) / 2 = 6.5 out of 7

Effectiveness: Success = 2; Task completion rate = 100%

Efficiency = (6+6) / 2 = 6 out of 7

Task 7

Ease: Average SEQ = (6+7) / 2 = 6.5 out of 7

Effectiveness: Success = 2; Task completion rate = 100%

Efficiency = (6+6) / 2 = 6 out of 7

Post-test

Usability: System Usability Scale Survey

1. I think that I would like to use this system frequently.

2. I found the system unnecessarily complex.

3. I thought the system was easy to use.

4. I think that I would need the support of a technical person to be able to use this system.

5. I found the various functions in this system were well integrated.

6. I thought there was too much inconsistency in this system.

7. I would imagine that most people would learn to use this system very quickly.

8. I found the system very cumbersome to use.

9. I felt very confident using the system.

10. I needed to learn a lot of things before I could get going with this system.

Total score: 87

Overall Reaction

• Liked that it was easily accessible on a platform they already used for leisure and to catch-up with friends
• As they were already familiar with the UI of Telegram, it was not hard to navigate and interact with the functions
• Enjoyed Telegram channel as an avenue to chat with fellow fans and keep up with news

Positives

• Hashtags allowed them to search for specific genres and family members specifically when they were in a rush
• Video, picture and written content gave them ample choice depending on their mood to engage with the content in different ways.
• FAQ Section gave them an inkling of how to navigate the channel. It was intuitive enough for first-time users
• Enjoyed the option of a discussion channel so that they could discuss with like-minded fans if their friends weren’t interested

Negatives

• Design: White background of pictures makes things too serious and boring, to try a pale yellow (tertiary colour scheme) to have contrast and to keep things light.
• Hashtags section was too cluttered; had to click around and explore to know what each one meant (especially the ones for genre).
• Voiceover for video was too fast and robotic as it was generated using AI. Static images made it boring as well. Made them disinterested to watch and would rather read the write-ups.

Suggestions and Areas of Improvement

• Daily summary posts might be too much for a casual fan who does not have much-vested interest in the family. It can be a weekly summary instead.
• 3 minutes for the summary video is a little too long for 2 news articles. Should be bite-sized pieces or to fit in more articles into the 3-minute video.
• 1 user found it hard to click on/find the FAQ as it is on the top of the Telegram screen. Could look into the implementation of bots who will send out direct messages to users who first join the channel.
• Photos with plain text are boring — use more pictures to keep interest of users.
• The voiceover was too fast. To be slower-paced, otherwise users can’t keep up.
• To explore having a more streamlined hashtag system as current one is too complicated/confusing.