Redefining the Highlight Experience
BACKGROUND
In the spring of 2020, the Hudl Media Business Unit decided to invest in a product team as part of the strategy to hit the team’s revenue goals for the coming years. The business unit hadn’t had the help of a product team for about 3 years, so unfortunately our users were enduring a variety of outdated experiences and the infrastructure in which they were built upon was also shaky.
The Media business at Hudl generates revenue from brands and media partners, not our users directly. Brands and media partners want to communicate and engage with their customers, primarily athletes, as effectively as possible and they choose Hudl as a means to do that. Successful relationships with both our users and brand/media partners relies on our ability to develop Hudl as an engaging content destination for our largest user cohort–athletes. Our team set out to do that through two high-level actions: creating content and consuming content. We articulated them through the following strategic initiatives:
Promote world-class content creation so athletes feel empowered to get noticed.
Harness the power of data so athletes, coaches, fans and brands can discover and engage with the moments that matter.
ROLE | DURATION | TEAM
Product Designer + Product Manager | Hudl
JUL 2020 - MAY 2021
My team was made up of an engineering lead, 3 engineers, and a QA. I also worked really closely with our Research and Data Science teams.
THE CHALLENGE
One of the most important products we provide for athletes is the Highlight–the ability to highlight their best moments in their games and share them with recruiters or friends and family as a means of getting noticed. The highlights product hadn’t been touched for over three years, which meant there were plenty of opportunities to improve the workflows from which an athlete watched their game video to eventually sharing their highlight reel, and all the steps in between. As a result, the number of interactions our Support team had related to highlights went up, and the number of highlights athletes created was on the steady decline. Consequently our advertising inventory also declined, inhibiting our ability to grow revenue from brand/media partners.
We needed to reinvest in the highlight creation experience to meet our users needs, as well as increase the amount of highlights created to meet the needs of our brand/media partners. The outcomes we hoped to achieve as a result of this work included the following:
Athletes will create more highlights as a result of the new experience.
Athletes will want to opt in to a new experience because it’s easier, faster, and smarter.
Coaches and athletes will create more highlights.
Athletes will be noticed more frequently.
Less support interactions related to athletes making highlights.
GETTING INTO ATHLETES’ SHOES
Our internal research team was able to help us better understand the athletes using Hudl so that we could ensure we were designing the experience that was going to meet their needs. Through a qualitative survey completed by about 1,000 users, a few of the following themes emerged from what Hudl athletes were saying:
I want to be a better athlete.
I have dreams for my future.
I care about my teammates and I want to be a good teammate.
I want to see content that is relevant to me.
I find a lot of value in Hudl by I also want it to be better.
The second phase of research consisted of user interviews to better understand the athlete experience using Hudl, specifically creating, sharing and watching highlights. We learned the following key takeaways about Hudl athletes:
Athlete motivations predict their behavior.
Athletes are figuring it (Hudl platform) out on their own.
Athlete watch highlights to get better, they create highlights to get noticed.
Athletes love what Hudl can do for them, but it could be easier.
PROCESS
When looking at the process in which our athletes create highlights, four main phases emerged: Finding Moments, Collecting Moments, Viewing Moments, and Sharing Moments. Within each of these phases I documented hypotheses and assumptions my team and I had discussed so that we could run experiments for each. I also found it really helpful to organize our Figma space with each phase as it’s own file, and within that file each page had the list of experiments and the accompanying designs. This seemingly simple organization made it really easy for the team to follow progress, collaborate, and provide feedback for each phase. It also eventually made it a smooth process to onboard a new designer we welcomed to the team months later.
We started our experiments within the first phase, Finding Moments. At the time my team was working on stabilizing our infrastructure so I focused on running experiments that I could test without the need of additional investment from the team. The first couple of experiments were run using interactive prototypes in Figma and shared with an athlete through a usability test over a Zoom call. I also manipulated an experience that was in the process of being built in production, Hudl Beta, using Chrome’s dev tools to show data that was specific to the athlete I was speaking with. Eventually the team was ready to begin focusing on helping run the experiments at a larger scale. Given the fact that we had about 1 million active athletes on Hudl, this was integral because we needed to focus on having quantitative over qualitative data to feel confident the results were statistically significant. Using the results from the first couple of experiments I ran, we were able to iterate on further experiments at scale.
RESULTS
We ran a total of 5 experiments under the Finding and Collecting Moments phases of the highlight creation process. Since we are still in the process of running some of the experiments and haven’t collected results, the following documentation are the results from two experiments run under the Finding Moments phase.
Experiment D | Finding Moments
Hypothesis
By filtering to a user's tagged Assist* game moments, users will be able to more easily create clips and highlight clips, resulting in more highlight clips being created compared to the current version of Hudl.
*Assist is a video breakdown service Hudl provides in which professional analysts break down game film so that coaches can focus on coaching and then analyze the game and athlete data 24 hours after the game is played.
Experiment
Send a push notification to an athlete who has been tagged in a game and filter tags by that athlete.
Design
The experiment was only conducted on athletes who had been tagged in basketball games submitted to Assist. We only sent push notifications to athletes tagged in “positive moments” so we weren’t asking them to highlight their turnovers, missed shots, etc. Users were randomly selected to go into the test or control groups. We measured the experiment success based on the concept of “potential highlights". We considered a potential highlight to be any time an athlete was tagged in a game submitted to Assist. At the time our baseline conversation rate was ~3.48% of potential highlights that get created into highlight clips.
Results + Takeaways
The result of the experiment was inconclusive. As of March 24, 2021, the conversion rates for the control and test groups are 3.4% and 2.87%, respectively. Having said that, the test group suffered from a lower number of potential highlights to make clips and reels from due to the fact they must receive the push notification after the game sent to Assist is tagged. Of the 10312 users in the test group, only 1770 had received the push notification. This could have been because they didn’t have push notifications turned on in the Hudl app, or it could have been because they had them turned off at the system level. Approximately 70% of these users had push notifications turned on in the Hudl app, which meant to a large degree those users were either not getting the push notification because they're turned off at the system level, or for some other unidentified reason. Due to the fact that we had other similar experiments to run that didn’t rely on opening a push notification, we decided not to invest in trying to get more users to have notifications on and just take the following takeaways into consideration moving forward:
Push notification data may not be sufficiently reliable
Relying on the push notification to do the heavy lifting of filtering the moments may not have been enough assistance to the end user
Experiment E | Finding Moments
Hypothesis
By filtering and creating clips and playlists from a user's tagged Assist game moments, users will be able to more easily create highlight clips, resulting in more highlight clips being created compared to the current version of Hudl.
Experiment
Send a push notification to an athlete who has been tagged in a game AND automatically create clips from their tags in a consolidated playlist.
Design
Similar to Experiment D, this experiment was only conducted on athletes who had been tagged in basketball games submitted to Assist and we only sent push notifications to athletes tagged in "positive moments”. Users were randomly selected to go into the test or control groups. The test group consisted of users who were not selected for the test group in Experiment D. Like Experiment D, we measured experiment success based on the concept of “potential highlights”. Our baseline conversation rate at the time was ~2.35% of potential highlights that get created into highlight clips.
Results + Takeaways
This test was a success on all metrics we tracked. The overall conversion metric, potential highlight percentage, ended at 2.84%. This was .69% higher than control. In addition, the average number of clips created per user was higher (1.66 for test vs 1.39 for control). The percent of users creating clips and reels was higher. The end result was 1600 more clips and 200 more reels created in the test group compared to the control. These represent increases of 20% and 9%, respectively. All the above results were statistically significant at the > 99% confidence level.
If we put these results into context of the 2020-21 basketball season there were 134,525 highlights made from 3,274,710 possible highlights which is a conversion rate of 4.11%. This was higher than our baseline conversion rate of 2.35%. We ran our experiment toward the end of the basketball season so if we assumed the conversion rate in season was 4.8%, we could have seen 157k highlights created which would represent a ~16% increase in inventory on the platform.
A consolidated playlist for moments resonates with users much more than relying on them to collect moments themselves.