Word Analysis is used for understanding the words that have caused Bot Break. In this article, we will go through how Word Analysis actually works.

Following are the steps that explain the working of Word Analysis. 

1. Let us take an input user query that caused a Bot Break
2. If Data Masking is enabled, then we run Data Masking on that message
3. On the encrypted message, we will remove common words (like the, is, on, ok, hi, etc.)
4. Next, we try to detect the phrases of 1 word/2 word/3 word length with the remaining part of the User Message and store it in the datalake

When we open the Word Analysis page, we will generate the counts of phrases that are created in step 4 and will show them on the UI.

Example:

User Message: Since morning I m only doing chat with you.

The detected phrases could be:

• 1 Word: since, morning, chat.
• 2 Word: since morning, morning chat
• 3 Word: since morning chat

Notice how the common/smaller words like I, m, only, you, etc. have been ignored.

What are the edge cases:

1. Because of Data masking
   • User Message: There is no update on my query QY5689A98
   • After Data Masking: There is no update on my query [bot_query_id|akwjdakwjdnkawjdna]
     When generating the phrases for this message, we might get something like:
     • 1 Word: update, query, bot_query_id.
     • 2 Word: query bot_query_id.
     • 3 Word: update query bot_query_id.
       
       
2. Because of words being removed
   • For this, you need to understand how the Message Analysis search works.
   • If you search for since morning chat, the results will contain all the messages with any of the 3 words (more priority is given to ones where more words match).
   • If you try to search for the phrase "since morning chat", then it will look for exactly those messages where the phrase comes as it is.
   • So, in the above case, if you search for since morning chat - you might see more results, and if you search for "since morning chat", then you might not see the results because there is no such User Message (the original one had some extra words in between)

This is how Word Analysis works.