What are the best practices to improve your IVA?

Updated on October 26th, 2021

+ More

What are the core objectives?How can you achieve these objectives?1. Automation %How to improve Automation %?2. Completion Rate of Funnels How to increase the completion rate and decrease drop-offs?3. User Count and Conversation Load How to improve User Count and Conversation Load?4. Feedback How to improve Feedback collection?

Intelligent Virtual Assistant (IVA) helps businesses automate their support. User queries are answered by these virtual assistants powered by NLU and Machine Learning. By being always available and providing a faster resolution to the users, these virtual assistants drive value for our business.

But just like an agent, virtual assistants also need to keep learning and improving over time for being effective. But how do we do this? This could be achieved only when we analyze data regularly and pick up feedback points for these virtual assistants.

After years of reading & analyzing data, Haptik has come up with few best practices that could be followed to make our virtual assistants grow.

What are the core objectives?

Virtual assistants are used for both Support and Lead Generation use cases. There are 2 basic objectives for Support and Lead Generation virtual assistants respectively -

Answer maximum queries correctly and instantly to divert call center volume
Increase leads for a business

How can you achieve these objectives?

These objectives can be met by tracking a few important metrics related to the virtual assistant. Haptik's Intelligent Analytics will help us do that! We can either track these metrics in real-time on the tool or run detailed analysis on the downloaded reports.

Intelligent Analytics has all types of metrics available related to the virtual assistant's efficiency and performance. We will go over the ones which will help us improve our virtual assistant over time.

1. Automation %

If a user's message is understood by the virtual assistant, then it gives the designed response and if a user's message is not understood by the virtual assistant then it gives a Bot Break Message or gets an agent involved. Bot Automation % tells us about what percentage of conversations were handled correctly by the virtual assistant and did not have either a bot break or an agent involved.

You should keep monitoring the Automation % regularly and make sure it does not drop below the threshold set in the beginning of the project planning.

An efficient virtual assistant will have a HIGH Bot Automation %.

How to improve Automation %?

Few points to keep in mind while you are building the IVA in order to keep the automation % high -

1. Ensure IVAs have a clear messaging upfront with the users on what the IVA can and can not do. This will aim to reduce out-of-scope queries. Keep updating this messaging every phase as more and more scope gets added to the IVA.

2. If you’re going with a phase-wise approach, you should add bot responses even for use cases to be built in future phases. These bot responses can either be escalation steps that transfer the chat to an agent or can be static responses informing the user that the bot will be able to address this query at a later stage. This will reduce the bot breaks expected for queries that are out of scope for that phase but are highly likely to be asked by users.

Now once the IVA has been taken live, monitor the IVA 1-week post go live. The ultimate goal is to increase virtual assistant's efficiency by making it answer all questions users are coming and asking. For this, we would need to find out messages where the virtual assistant is not giving desired responses. AI Recommendations, Word Analysis, and Message Analysis are different screens that can be used to achieve this.

AI Recommendations can be used to identify issues with the virtual assistant by finding out missing user intents and training deficits on existing intents. This information is provided by the algorithm by going through thousands of conversations. It suggests both utterances that existing use cases should be trained on as well as out-of-scope categories that are being asked.

Word Analysis can be used to check Bot Break Topics for prioritizing over broken messages based on the volume. We can fetch exact user messages which are resulting in these bot breaks to add them under the User Messages section of the virtual assistant.

Screenshot 2021-06-30 at 4 59 55 PM

To know the conversations in more detail once prioritization has been done, you can view these chats in the Message Analysis section. You can achieve this by selecting the Broken Messages filter.

You can see the actual messages sent by the user in the Body column or you can view the entire conversation.

If a similar intent is present on the virtual assistant, you can add this message in the User Messages section of the virtual assistant. If the intent does not exist, then you can create a new one on the virtual assistant and train the virtual assistant to understand such user queries. You should train the virtual assistant by adding maximum user variants to increase its efficiency. Once the virtual assistant is trained on the broken intents, the automation % will increase.

You can also download the Message Analysis report to get the entire dump at the Message or Conversation level. You can go through every single message received on the virtual assistant.

You can leverage this report to -

Analyze the actual user messages.
You can pivot the report to find major and common bot breaks.

The following is a snippet of the Message Analysis report -

If after a point, further analysis shows that the automation % reduction is majorly due to out-of-scope queries, then this could mean either or both:

1. Marketing efforts have been ineffective in defining the capabilities of the virtual assistant to the end-users.

2. Wrong use cases have been picked for the virtual assistant.

2. Completion Rate of Funnels

Smart Funnels help us track the effectiveness of the virtual assistant by tracking the completion rates and identifying the drop-off points of conversations started on the virtual assistant. This feature plays an important role to enhance user journeys and IVA content.

Various metrics like completion rate, drop-off rate, drop-off steps, and funnel hits help us to make changes in the virtual assistant to ensure users are completing the conversation flow and getting a resolution.

An efficient virtual assistant will have a HIGH Completion Rate and LOW Drop-Off Rate.

How to increase the completion rate and decrease drop-offs?

Even before coming to an increasing the completion rate of funnels, let us start with how to create these funnels? You can know more about it here.

Smart Funnels give the number of people initiating a user journey, the number of people completing the user journey, and places of drop-offs in the user journey. Once we have the funnels ready, we can monitor these metrics regularly to find the problem areas in the journey and fix them.

To decrease the funnel drop off rate, we can perform the below actions on the drop off points -

Go to Message Analysis to see the bot breaks happening once the user enters the flow
Modify the user journey by removing the unnecessary steps or questions asked in the journey
Modify the virtual assistant's responses
Add Follow-up messages on the virtual assistant so that users are prompted to continue their conversations and complete their user journey
Make sure that user journeys are not complicated and lengthy so that more and more users are likely to complete it

You can download the Smart Funnel report which will help you highlight the touchpoints in the user journey which are causing the maximum drop-offs along with the reasons for the drops. You can check the corresponding chat links to understand the pretext of the drop-off. There are various reasons for these drop-offs and you can read more about them here. On the basis of this analysis, you can find where virtual assistance needs more improvement in terms of training data or design considerations.

The following is a snippet of the Smart Funnel report -

3. User Count and Conversation Load

The Unique User section shows the number of unique users who have visited the virtual assistant in the given duration period. The conversations graph represents the conversations count. Unique users and conversations graph can reveal the discoverability and usage of the virtual assistant.

A highly used virtual assistant will have a HIGH Unique User and Conversation count.

How to improve User Count and Conversation Load?

To improve User & Conversation Count, you can take the below-suggested stages-

If the number of users who are visiting the virtual assistant is low, you can add Bot Prompt Triggers to grab users' attention
You can deploy the virtual assistant on additional channels such as WhatsApp, Facebook, Google Business Messages, and others
You can have a release plan to make users aware of the presence of virtual assistants and that they can seek help on it. Have few marketing initiatives started to spread awareness about the virtual assistant
You can use Deep links for the virtual assistant used in email signatures or email footers
Launch Message or Custom icons can also turn to be useful for catching user's attention
Whatsapp Notifications(HSMs) can be sent too to keep users engaged with the brand

Conversation count can be increased by adding new queries on the virtual assistant which users can come and ask. It may be related to the new products or services your brand has released.

Story Analysis will give you a deeper understanding of which use-cases are having more traffic than others. This will help you prioritize the use-case that should be improved first on the virtual assistant. You can compare these numbers week by week to understand the trend as well.

The following is a snippet of the Story Analysis report -

4. Feedback

Direct feedback given by the user is always a great indicator of the performance of the virtual assistant.

Make sure feedback collection is enabled on your virtual assistant. If not, you can enable it by following the stages, here.

User Rating & CSAT sections help us in finding this out as they give the average of all feedback provided by the users on a scale of 1-5 at the end of a conversation.

The feedback collected from the users can aid in making the user journeys easier and improving the virtual assistants' performance.

An efficient virtual assistant will have a HIGH User Rating.

How to improve Feedback collection?

There are few DOs and DONTs related to Feedback. Let us have a quick check at that -

Feedback collection should be done only when users have completed their journey and not in the middle of the conversation
Negative feedback received should be analyzed in detail to derive insights on bad user experience
User feedback should be distinguished into Bot vs Brand and Relevant vs Irrelevant and Bot vs Agent
CSAT section will help you analyze the reviews or comments left by the users. It might reveal why users are leaving negative feedback on the virtual assistant
Viewing the respective chats under Customer Reviews can tell you what was the user's query and why the virtual assistant was not able to resolve it. If the virtual assistant could not understand the query, then it can be added as a new intent to handle it in the future

Using the User Rating report, you can check all the ratings and reviews which users have given to the virtual assistant or the agent. You can also check the chat links in detail to further understand the reason for a low rating or a bad comment.

The following is a snippet of the User rating report -

Additional points to improve your virtual assistant

Building a flow if the data suggests that users are not happy with an FAQ approach.
Highlight all API errors immediately and take corrective measures.
Set usage, automation percentage, and API failure trigger to get notified in case of sudden drops in usage or bot performance. You can contact your CSM to enable the same on your virtual assistant.
Review chats where users showed dissatisfaction and create action items to improve user experience.
Arrange for UX retro with the Conversation Design team if the bot has too many breaking flows or user journeys.

If you have doubts regarding any of these best practices or you want to know about any of these in more detail, feel free to reach out to your Customer Success Manager.

Contact Us