Text Classification using Neural Networks

This article was written by gk_.

Understanding how chatbots work is important. A fundamental piece of machinery inside a chat-bot is the text classifier. Let’s look at the inner workings of an artificial neural network (ANN) for text classification.

We’ll use 2 layers of neurons (1 hidden layer) and a “bag of words” approach to organizing our training data. Text classification comes in 3 flavors: pattern matching, algorithms, neural nets. While the algorithmic approach using Multinomial Naive Bayes is surprisingly effective, it suffers from 3 fundamental flaws:

• the algorithm produces a score rather than a probability. We want a probability to ignore predictions below some threshold. This is akin to a ‘squelch’ dial on a VHF radio.
• the algorithm ‘learns’ from examples of what is in a class, but not what isn’t. This learning of patterns of what does not belong to a class is often very important.
• classes with disproportionately large training sets can create distorted classification scores, forcing the algorithm to adjust scores relative to class size. This is not ideal.

Join 30,000+ people who read the weekly Machine Learnings newsletter to understand how AI will impact the way they work and live.

As with its ‘Naive’ counterpart, this classifier isn’t attempting to understand the meaning of a sentence, it’s trying to classify it. In fact so called “AI chat-bots” do not understand language, but that’s another story.

If you are new to artificial neural networks, here is how they work.

To understand an algorithm approach to classification, see here.

Let’s examine our text classifier one section at a time. We will take the following steps:

• refer to libraries we need
• provide training data
• organize our data
• iterate: code + test the results + tune the model
• abstract

The code is here, we’re using iPython notebook which is a super productive way of working on data science projects. The code syntax is Python.

We begin by importing our natural language toolkit. We need a way to reliably tokenize sentences into words and a way to stem words. 

To read the whole article, with demonstration, click here.