Timeline Jan 2019 - March 2019 (8 weeks)
Target user Adults (Age 18-29)
My role Usability Study Researcher
Study Methodology Survey+Screener, Pre-Test Interview, Usability Testing
Team
Josh Holland (Researcher)
Sean Horita (Researcher)
Sean Vesce (Client)
Bobo Xu (Researcher)
Project Description
It was a usability study we collaborated with our client Very Very Spaceship (VVS), a video game R&D company focused on developing the next generation of location-based Augmented Reality (AR) games, to test two distinct navigation methods: Map and Photo with specific tasks to determine which were most useful, satisfying, and easiest to learn. The study results will be used to focus on future digital development efforts as part of a larger interactive game experience.
Final Presentation In The Class
Purpose and goal
Background:
Location-based AR games influence millions of players to participate in the real world while exploring in adventures; however, most players have been observed to sustain their full focus and attention on their device instead of the world around them.
Goal:
Explore the world through AR games, without the constant need to focus on their mobile devices. We were interested in gleaning insights about general human behavior, thing, and reactions while navigating in unfamiliar environments.
Research Questions
Which is most fun, challenging, and frustrating?
Which provides the best sense of adventure?
Which is preferred by participants?
Participant Characteristics
We conducted usability testing with 6 participants that had the following characteristics:
Adults, ages 18-29
Have used mobile navigation tools of on-foot wayfinding
Have an interest in independent on-foot exploration in urban environments
Were unfamiliar with the physical location of the tests (the Fremont neighborhood of Seattle)
Fell into one of three experience levels with playing location-based AR games:
– "Hard Core": Plays one or more of these games several times per week
– "Casual": Plays at least one of these games for more than 3 hours
– "No Experience": Has never played a location-based AR game.
I am wearing the Live-Link to do a pilot test
Methodology
Logistics
Test Dates:
Tuesday, Feb 26th, Weds, Feb 27th
Tuesday, March 5th, Weds, March 6th
Technical Requirements:
Proprietary Live-Link streaming device, a wearable camera designed to communicate the first-person perspective of each participant's test session back to test administrators in a remote office.
Live-Link streams and records the participant's POV (point of view), voice, GPS position, and allows two-way communication between the participant and the Moderator.
Test Location:
Fremont Neighborhood Exploration
Post Test Interview at Client's Office
Study Test Team
Moderator: Served as the main coordinator and main communication channel with the participant.
Scribe: Used a laptop to record notes and verbatim participant quotations.
Minder: Physically accompanied each participant to guarantee player safety throughout the test and as communication backup between the participant and the Moderator.
Pre-Test
Online Survey
I designed a survey screener by Google Form to carefully selected our participants based on the characteristics and were scheduled for an on-site test at the client's office.
Survey screener by Google Form
Prior to arrival, participants were required to review and sign a Non-Disclosure Agreement and Liability Release Agreement. The team verified that all test equipment was functioning normally.
On arrival, participants were greeted, we were introduced and the participant was "suited up" with a Live Link backpack, got their permission to take a photograph for internal test reporting purposes, and the team conducted a second check of the system's capabilities and functions. Please refer here for our full study script.
Pre-test Process draw by Bobo
Test
The participants were escorted by the Minder (Bobo) to the starting location outside the Client’s offices. I instructed the participants about the first challenge, reminded them to “think-aloud” as they embarked, and reminded them that if at any time they need support they should feel free to ask.
After the participant completed each of the two challenges, they were instructed to return to the starting location, during which they were interviewed about their experiences.
Test Process draw by Bobo
Participants were tasked to navigate from Client’s offices to one of two randomly selected destinations, using one of two randomly selected methods:
Destination 1: Bench under the George Washington Memorial Bridge
Destination 2: Dinosaur Shrub
All participants were tasked with two navigation challenges; we split the order of the challenges evenly across our six participants.
Task A: Landmark Photos
Participants were given a series of photographs of nearby landmarks to navigate.
When the player is within a few feet of the landmark, he/she is presented with the next photo in the sequence.
To reach the final destination, the player will need to travel to several sequential landmarks using contextual clues, like the color and details on each photo.
Series of Landmark Photos
Task B: Symbolic Map
The participant was given a static, stylized hand-drawn map that is representational, but not spatially accurate - similar to a “pirate map”.
Using a combination of memory, problem-solving, and intuition, the player is challenged to navigate to the target POI.
Stylized Map
Post-Test
Upon return to the starting location after the second test, the participants were escorted back to the Client’s office. The Test Moderator and I assisted Participants to remove the Live-Link system and Participants were escorted to a private conference room for a post-test debrief.
During the debrief, the Test Moderator conducted a short final survey to gain additional insights into the participant’s experience.
After the survey and discussion were complete, the participant was thanked for their participation, given a small gift ($20 Amazon Gift card) for their time.
Finally, the participant was reminded of the confidentiality requirements of the NDA.
Post-Test Process draw by Bobo
Data Collection and Analysis
Qualitative data:
“Think aloud” quotes from the participant, coupled with general observations about the participant behavior that was verbalized by the Moderator and Observers (Bobo). Also, the feedback of the task was collected after the conclusion of each one.
Quantitative data:
Time to reach each POI for each of the two tasks; time spent going “off-track” or significantly out of the straightest path to task completion; whether the participant gave up or was not able to complete a task.
One example of collecting the data
Once back in the office, we asked a series of questions concerning general impressions of participant’s opinions and attitudes around each task and the test overall, using a web-based survey with responses captured by the Scribe.
Post-test debrief Sample
Key trends and recommendations
Most participants preferred the Map to the Photos.
The Map was the “most fun” by a small margin, and the majority said the Map provided the greater sense of adventure.
“I liked the way the hand-drawn map - more personal - made me feel like I was going on an adventure - like a treasure map”
Data Analyzed Using Microsoft Excel By Bobo
Participants experience the challenge of navigating differently when they have an idea of what their destination is and generally where it is relative to their starting location, rather than being led to a destination through steps where they don’t exactly know where they will finally end up.
The Photos test ranked more highly for the Challenging and Frustrating ratings, though the overall scores for Frustrating were much lower than that of Fun, suggesting that both tests provided an interesting and stimulating experience. Participants overwhelmingly enjoyed discovering new points of interest and unfamiliar environments through a game-like experience.
There are four apparent trends and patterns inferred by our test data are listed below.
Trend 1:
Safety hazards and concerns are real, even for screenless location-based games.
Recommendations:
Consider ways to limit requirements for player attention and decrease requirements on mental load especially when the game detects a player is on the move. Consider providing ample warnings and reminders to players to be conscious of safety hazards.
Trend 2:
Getting a little lost is fun, getting a lot lost is not so fun.
Recommendations:
Keep a consistent level of challenge level without solving it for players. Pay attention to how the effects of elapsed time related to distance traveled and task progress.
Find a balance between detailed clues, task ease, and amount of game feedback as each affects the fun factor. Provide feedback when the participant is too far off track, but don’t immediately provide a solution.
Trend 3:
Participants varied in their strategies for getting back on track.
Recommendations:
Provide opportunities for multiple solution pathways as participants like having the creative freedom to solve challenges.
Also, consider graduated levels of specificity of clues
Trend 4:
Interpretation of landmarks from clues took different forms.
Recommendations:
Consider including context about surrounding areas when offering photo clues.
Reflection
The actual user studies in practice did not vary significantly from the original plan, but there were several alterations that occurred during the first tests and as results of a high-resolution pilot test.
Before
Cycled through the various roles.
Completed the post-test interview back in the office after both tasks were completed.
After
we largely gravitated to keeping the same roles, mainly due to the relatively high amount of coordination involved in administering the study.
In order to save time, we began conducting post-test interviews after each method, while participants were making their way back to the start point. This resulted in more fresh insights and dramatically reduced the overall time requirements for each test.
Future versions of this study could include more comprehensive tests to investigate wider age ranges and that could provide successively increasing levels of challenge.
Looking further into the future, digital versions of these methods could be tested that incorporate the lessons learned from these analog tests and bring the play content closer to that of a real AR game.
Thank you for reading > <
