Shopify Help - Increasing engagement while in-queue

THE CHALLENGE

When merchants were waiting in the queue to be connected to an Advisor, the Help Center Assistant (HCA) would indicate to the merchant that it could help while they waiting in the case they had any additional questions. The issue however, was that it didn’t guide the merchant in how it could help or what additional context the merchant could provide to speed up their resolution once they were connected with an Advisor. As a result, most merchants would sit idle in the queue.

OUR USERS

All merchants who use Shopify (or any users visiting Shopify Help Center) have the ability to interact with the Help Center Assistant. You do not need a subscription with Shopify to use the HCA, although if you are authenticated, your experience is somewhat personalized to your shop.

Shopify’s merchants range from small businesses run out of the owner’s homes, to multi-million dollar operations that employ hundreds of people. Due the vast differences in merchant’s needs, the Help Center and the HCA needs to be as generic as possible, while also understanding the nuances of variables of plan types Shopify offers.

Iterating on how to engage merchants

With the goal of engaging merchants while they were in queue, I began to brainstorm some ideas to prevent them from leaving the HCA and stay focused on meeting with an Advisor. I created some lo-fidelity concepts so that I could review with members of the engineering team to understand the effort needed to build them out. After getting a better understanding of scope, I laid out the high level concepts on a scale ranging from low effort (less scope) to higher effort (more scope) to help facilitate the conversations I was having with leaders of our product team. Below is a snippet of those concepts and the scope associated.

Issue details card concept

After presenting to members of our product management team, we decided to move forward with a more complex approach of gathering as much information from a merchant as possible while they were waiting in queue. We felt confident that this would do the following:

• Allow advisors to have as much context of the issue merchant’s were experiencing once they connected with them to be able to resolve their issue more efficiently

• Decrease the ‘dead on arrival’ rate which is when a merchant leaves the tab without exiting the queue and once an Advisor is ready to chat, the merchant is no longer there

We called this concept the ‘Issue details card’ because the merchant would be presented a series of cards within a message component that they could choose whether or not they wanted to answer the questions associated. A video of the concept can be seen below.

The three steps (cards) within the message component consisted of the following:

1. Confirm issue summary

   • The HCA would use AI to auto generate an issue summary based on the merchant <> bot conversation. This summary would be presented to the merchant and they would be able to edit it if it didn’t align with their issue.

2. Add screenshots

   • Merchants already had the ability to send attachments to advisors to provide more context of the issue they were experiencing but the feature wasn’t available until after the Advisor had started to chat with them. This step would allow the merchant to provide screenshots while they were in queue so that by the time the Advisor started chatting with the merchant, they already had the context and didn’t have to waste time asking the merchant for screenshots.

3. Add relevant links or error codes

   • Links/error codes are another layer of context that Advisors typically ask merchants to provide them so they can understand the issue at hand. Giving merchants the ability to add these in queue aligned with the step prior, intending to save both the merchant and Advisor time.

Refining the UI

Screenshots from Figma files of in progress work.

A change in scope

As a result of a change in the organization structure of our team at the time, the priorities for our team shifted. This meant that we needed to significantly pare down the scope of the project because the development team had new priorities to work on. We still were tasked with releasing an experiment to learn about how to engage merchants while they were waiting for an Advisor. Although some stakeholders felt strongly about prioritizing nudging merchants to send a first message to Advisors while they waited, I felt that merchants wouldn’t be open to doing that because of what the data we had was telling us. After working with our data science team, we knew that merchants rarely (close to 0%) sent any messages to the HCA when they entered the queue (although they had the ability to and the HCA would reply by trying to help them). Another interesting data point, was that ~25% of merchants sent screenshots to Advisors once they were chatting with them. This increased my confidence that merchants found providing screenshots to Advisors valuable in describing their issue to them.

My suspicion on why the engagement rate was so low was that because merchants knew they were going to be talking to a human they didn’t feel like they needed to converse with the chat bot they just finished trying. Instead of focusing on nudging merchants to send a message once again to the bot, I thought that nudging them to add a screenshot of the issue they were experiencing would help both the merchant and the Advisor while they were waiting, since they would have more confidence that the Advisor could help them resolve their query more quickly if they had the most context.

After presenting my hypothesis and pared down concept to the product leadership team, they approved the build of the experiment. Below is a video of the final experience. The two metrics we were concerned about were the following:

1. Decreasing average handle time

   • Average handle time is the time that an Advisor spends with a merchant on a chat

2. Increasing merchant engagement rate while in-queue

   • Messages or files that the merchant sends the HCA while they are waiting to meet with an Advisor

Experiment results : Increased merchant engagement by 35%

The experiment was rolled out to 10% of users and after 1 week we were able to collect results. Although the average handle time did not increase or decrease, the merchant engagement rate while they were in-queue increased from 0% to 35%.

This was a significant increase in engagement while merchants were in-queue. We decided to roll the experience out to 100% of users so that they could provide as much context to Advisors as possible while they waited to be connected, ideally providing them a better experience when they start chatting with them to resolve their issue.