Why did your chatbot fail miserably ?

Finding the Elusive ‘U’ in NLU

Meet Neo ! Neo is a talented developer who loves building stuff. One fine morning , Neo decides to take up a road, less travelled, decides to build a chatbot ! After a couple of keyword searches and skimming through dozens of articles with titles “build a chatbot in 5 mins” , “chatbot from scratch” etc, Neo figures out the basic components to be intent detection, Named Entity Recognition ,Text Matching for QnA . Another 30 mins of Google search and Neo has collected his arsenal, the state of the art implementations for these 3 components. His arsenal has the almighty Bert for NER, Ulmfit for Text classification, RoBERTa for text matching.

Excited to have decoded the path to greatness, Neo arranges annotated data and sets up the pipeline and calls it ‘The Matrix’ . Little did he know, the name would turn out to be the nemesis. Neo tests out some happy-flows and deploys the system gleefully awaiting well-behaved users.

To Neo’s Horror, The AI system proved to be poles apart from the expected promise of Natural language understanding. ‘the Matrix’ was as awkward as a cow on roller skates.

Neo’s users queried abut everything ‘the matrix’ was not trained for , fooling it more often than not. Happy-flows turned out to be a myth. we, humans as always beat the AI bot down to death, as if we were being challenged in intelligence supremacy . Battered and defeated , Neo decides to fix the ‘matrix’ by hook or crook . Not getting into the debate of enhancing NLP (Natural Language Processing) v/s plugging the gaps using CI (Conversational Interfaces) .

Neo decides to put together a checklist for his developer friends, the core issues in the NLU system nobody had told him before he walked through that mirage , He had thought was remarkably easy to put together.

1. Query Absoluteness Detection/ Fragment detection

Objective : To identify Whether a query is a fragment or a complete sentence . Query can be either syntactically incomplete , or semantically . As long as the intent is clear and the query does not lead to multiple answer possibilities , the query may be considered as ‘complete’

User query : “Virat Kohli”
Query absoluteness index: Fragment

User query : “Virat Kohli test average”
 Query absoluteness index: sentence

In the first case , there is no ambiguity as such , but the query seems incomplete to be answered correctly. Thus user has to be either asked about “ what do you want to know about Virat Kohli “ or trigger a fallback which can be a simple definition when the term in the query is an object or brief intro when its a person . Or do the latter and the ask the question.

Now consider the query : User : tell me about cricket

This query has to be disambiguated because it doesn’t clarify if the cricket is the insect or the sport.

Fragment detection can either trigger query elucidation or probing based on the intent type detected ( factoid or non-factoid ) , to disambiguate the query or complete the query in the scenario of incompleteness.

2. Adversarial Query detection

Objective : To identify Whether a query is valid / appropriate within the defined premises of domain, project, bot type, set standards or not.

User query : “How Many goals has Sachin Tendulkar scored in his international cricket career”
Query adversity index: adversarial

User query : “How Many runs has Sachin Tendulkar scored in his international cricket career”
Query adversity index: Valid

Answering invalid queries can be embarrassing , and as we add domains the system would be vulnerable to get fooled by subtle adversarial perturbation in the query which resemble very closely to the valid queries in the domain

Adversarial attacks are not uncommon

3. Query compression (Noise cancellation)

Objective : To cancel out noise in long multi-line queries to be able to detect the intent with higher precision. Also be helpful in making the query concise for improved accuracy in open-ended question answering.

User query : “can you please help me to get some information regarding the symptoms that may surface up when a person may have been infected with flu causing virus ”
Query compression index: Very noisy

Compressed query : “symptoms , flu”

User query : “i have lost my wallet I want to block my card what do i do” Query adversity index: noisy
Compressed query: “how do i block my card”

User queries can be noisy , its important for the conversational AI system to separate salient information pieces from the noise. Noise can be linguistic in nature e.g opening phrases like “can you please” or because of linguistic phenomenon like pied-piping or prepositional stranding .

In some cases it can just be an overload of information like the 3rd example, where the actual intent may get lost in the scramble of words

4a. Explicit compound-ness in User queries

Objective : User queries can have compound statements . Single intent, multiple statements . Or multiple intents.

This segment only caters to scenarios where the compound behaviour is explicit (for example , presence of conjunction/s)

User query : “i want to buy a pen and pencil”
Query compound ness index: explicit conjunctive compound statement 
Split type : split and inherit
Split queries : I want to buy a pen, I want to buy a pencil

User query : “add a burger and remove fries”
Query adversity index: explicit conjunctive compound statement 
Split type : split at conjunction
Split queries : “add burger, add fries”

User query : “i have an offer from Bain & company ”
Query adversity index: explicit conjunctive compound statement 
Split type : no split
Compressed query: “i have an offer from Bain & company”

User queries can have compound statements , which need to be resolved for handling them properly .

4b. Implicit compound-ness handling in User queries : Sentence boundary segmentation, punctuation restoration

Objective : User queries can have compound statements . Single intent, multiple statements . Or multiple intents.

This segment only caters to scenarios where the compound behaviour is implicit (for example , absence of conjunction/s still compound)

User query : “I want to buy a pen I want to buy a pencil ”
Query compound ness index: implicit compound statement Punctuation restored 
processed query : I want to buy a pen. I want to buy a pencil

User query : “add a burger remove fries”
Query adversity index: implicit compound statement Punctuation restored processed query: “add burger. add fries”

User query : “hi how are you can you book a flight to delhi”
Query adversity index: implicit compound statement
 processed query: “hi. how are you ? can you book a flight to delhi .”

User queries can have compound statements , which need to be resolved for handling them properly . Implicit compound statements can be handled by segmenting the sentences correctly by restoring the punctuations.

5. Query form Identification

Objective : To identify Whether a query is informative or contains enquiry or has instructions . Does the instructions mean read-write or the query has instruction to fetch info only .

E.G

“I like Indian cricket team “ can’t trigger a faq/open-ended QA flow . Its informative and doesn’t contain any instruction .

“I want to know about Indian cricket team” : has a declarative speech act but still requires some info retrieval . Note that previous query was also declarative , still the communicative action has to change.

“tell me about Indian cricket team” has a totally different speech act i.e imperative but still can have the same response as the previous one .

“show me this year’s stats of Indian cricket team “

Speech act : imperative
Query form : instruction
Still the response changes , because the response type has changed .

6. Context change detection in conversations

Objective : During the conversation , users tend to digress temporarily or transition completely to new intent or domain. It is important to detect such changes for smooth conversations .

Bot utterance : “please tell me the destination city to book your flight” User utterance : “can you book a movie ticket instead”
Context : Intent transition

Bot utterance : “please tell me the destination city to book your flight” User utterance : “can you first check the status of a flight ”
Context : Digression within domain

Bot utterance : “please tell me the destination city to book your flight” User utterance : “i do not want to book ticket now”
Context : cancellation

Bot utterance : “please tell me the destination city to book your flight” User utterance : “i do not remember the city name, can I give you the code”
Context : Non-cancellation negation, digression