Natural Language Processing (NLP) Definition

NLP allows computers to segment, assign meaning, and analyze human

communication in its natural forms. By doing so, it can be used to create insights
from texts, to associate meanings, to separate topics, to recognize speech and
entities from abbreviations to brands, and even to evaluate consumers’ feelings
using various statistical techniques.

The cornerstone of NLP involves statistical likelihoods that examine the

relationship between word and object pairs within a sentence. For example, a
consumer might accidentally say: “Boston is a civvy,” however NLP can
automatically correct a mispronunciation or typo by understanding that Boston
and city are highly related.

To examine user sentiment, an NLP model examines the relationship between

words and phrases and the frequency of their appearance across various user star
ratings.

What are terminology of Natural Language Processing (NLP)?

Artificial Intelligence (a standard definition): The term “artificial intelligence”
refers to the field of study and creation in computer science of an atmosphere
capable of replicating human thought.
Automatic Summarization: Applications like Summarizer use trained algorithms
to extract key information and ignore what is less relevant.
Chatbot: A computer program that conducts conversations with human users
mimicking human-like behavior, interpreting and sometimes learning from user
input. These programs are increasingly used to automate customer service
functions.
Deep Learning: This is a subset of machine learning (which is a subset of AI)
where the neural networks are layered to leverage enormous computational power
to process data more efficiently and to assist with mining discoveries from
unstructured data.
Machine Learning: This is a subset of AI where programs are created that
actually “learn” how to complete tasks with increasing efficiency over time.
Machines then take the very data they output to find the more effective way to go
about each process.
Relationship Extraction: The linking of entities in text such as connecting
“Richard Branson” with information that may be outside the text data set but
found elsewhere, like “founder of Virgin group” and other known data such as
“British Citizen.”
Natural Language Parsing: The process of working out the grammatical
structure of a sentence and classifying parts as subjects and objects, as well as
identify phrases within them.
Sentiment Analysis: The process of identifying human reception and opinions
from speech patterns within a dataset and assigning it a value from neutral to
very positive or negative without relying upon star ratings.
Speech Recognition: Conversion of human speech into text, often coupled with
recognition and action upon commands imbedded within the language.
Supervised Machine Learning: This is machine learning where human
supervision (and regular data input) are leveraged to promote the machine’s
improved processing cognition.
Text Mining: Mining of text data for patterns or frequency of word uses, but
unlike NLP does not consider semantics.

What is Natural Language Processing (NLP)?

NLP merges computational linguistics and data science, attempting to make
computers process language just like a human would. This allows an AI to
understand commands, sentiment and even slang, and respond in a natural way.
An NLP AI can be asked to perform tasks like the secretary of old, but with some
marked advantages. For example, instead of taking hours to highlight and tag
relationships within a few documents, Natural Language Processing harnesses the
power of AI to query enormous datasets and rapidly extract relationships in a way
that no human would be capable of parsing.

NLP breaks human language into meaningful bits using the following three step
process:

1. In the case of natural spoken language, speech is converted into text. It

breaks each part of the speech into phonemes, the smallest bit of sound, or
10-20mm clips and then re-affirms it heard correctly by using statistical
prediction of the surrounding clips, typically using the Hidden Markov Model.
You can see this in action as your smartphone updates its word by word
interpretation as you finish a sentence.

2. Sentences are semantically classified. Next, each bit of language is broken

down and classified grammatically into verbs, tenses, subject, object, etc.
 And once it’s both statistically affirmed that those words were the correctly
words and that the words have been grammatically classified, the computer
can understand the query and begin to respond.

3. A response is formulated. Queries such as “What’s the weather,” for

example, will likely have Google search the internet for the desired
information and then respond with the local weather conditions.

A truly impressive display of the capabilities of and NLP powered AI is shown in

this Google Assistant video. Google Assistant actually calls to make appointments
with a hair dresser, correctly navigating difficulties and handling non-standard
responses without outing itself as a machine.
As useful as this is, there will come a point in the near future where disclosures of
the use of NLP AI are required in service and concierge services.

What are the categories of Natural Language Processing (NLP)?

There are a wide variety of statistical techniques that can be applied to build a
language processing model. These techniques are typically be divided into the
categories of supervised versus unsupervised, each having their advantages—
whether that be the human training time required to generate inputs, the problem
type, or the scope of relevant datasets.
Unsupervised methods: Some methods of statistical analysis can be used to
extract meaningful data without training a model. Clustering is one such method,
and simply groups similar documents into sets. Latent Semantic Indexing is
another method that can identify key words and phrases that are associated to
each other within a data set.
In plan English? For example: the analysis might reveal that the words “boat” and
“ship” are highly related throughout a series of documents. Understanding that
relationship, the model will include data for “ship” when a search is performed for
“boat.”

Matrix Factorization is another tool used to create relational tables between

extremely large sets of inputs. In such a matrix, the model can understand that
the word “fish” is more likely to appear in a sentence with the word “bait” than
“bike.” This can help gain accuracy when there are other uncertainties in spelling
or pronunciation with auto-corrections.
Supervised Elements of specific features of language are tagged in a data-set and
then fed to the algorithm to its statistical model. The model learns to recognize
patterns and then can identify and automatically apply tags to new texts.
This model can be further improved by adding more tagged data, or reviewing
previous tags and correcting any mistakes once more information is available.

Some more specific model types associated with supervised learning and uses
include:

 Bayesian Networks (for spam filtering)

 Conditional Random Field (used to make predictions based on
neighboring data)
 Deep Learning – Neural Networks (for example, with self-driving cars)
 Maximum Entropy (in parts-of-speech tagging)

How Natural Language Processing (NLP) works?

In general, NLP tries to break naturally language data down into its most basic of
parts and then examine the relationships between those parts to extract meaning.
It can be used in a variety of ways, and the NLP model you construct should
reflect the type of problem you are trying to solve.

If, for example, you are trying to understand how customers feel about your
product or service based on their social media comments, a relatively simple text
classification model can be built by comparing text, emoticons and the ratings
they correspond to even if they never directly use obvious watch words such as
“like,” “dislike,” “hate,” etc.

To do this, we would program our NLP AI to match words, phrases and other bytes
of text with their corresponding star rating on a large review of the dataset
(including millions of ratings). From this, our NLP can infer things like a “:/”
typically meaning someone is less-than-satisfied, since its reviews it appears in
ratings with an average of 2.2 stars.

Other potentially less-obvious things can also arise from the data, such as slang,
character swearing (for example, “#$@@”), usage frequencies of exclamation
points and question marks.

Statistics in hand, the NLP can now automatically assign “sentiment” to a purely
text input. Actions then can be developed to respond, such as alerting a customer
service agent to directly respond to a negative comment or simply measuring
feedback from consumers about a new policy, product, or service on social media.

If you use Gmail, you’ve been seeing this in action for quite some time now. It
filters out spam, and auto-sorts emails into: Primary, Social and Promotions and
Updates based on language patterns it’s identified with each category.
Speech Queries
If you, however, want to build a system capable of recognizing and responding to
speech, you’ve got a few more steps ahead of you.
Remember breaking sentences down into subject, object, verb, indirect object,
etc. in elementary school? At least a little? Then you’ve done this type of work
before.
We’ll cover a quick example with the following sentence. “Andrew is on a flight to
Bali, an island in Indonesia”.
Tokenization:
Each word is first separated and broken down into tokens. “Andrew,” “is,” “on,” “a,”
“flight,” “to,” “Bali,” “,” “an,” “island,” “in,” “Indonesia,” “.”
Punctuation also becomes part of our token set because it affects meaning.
Parts of Speech Prediction:
Then we look at each word or token and try to determine whether it is a pronoun,
adjective, verb, etc. This is done by running the sentence through a pre-trained
“parts-of-speech” classification model which has already statistically examined the
relationships within of millions of English sentences.
Lemmatization:
This examines words to find the base form of each, understanding that “person” is
often the singular form of “people,” and that “is, was, were, am” are all forms of
“to be.”
Stop Word Removal:
Articles like “the, an, a” are often removed because of their high frequency. This
can cause relational confusion. However, each NLP’s list of stop words has to be
carefully crafted as there is no standard set to remove from all model applications.
Dependency Parsing:
In this phase, syntax structure is devised by to allow the AI to understand
sentence attributes such as subject, object, etc.
This allows the AI to understand that, while John and ball are both nouns, in the
phrase “John hit the ball out of the park,” John has done the action of hitting.
Open source parsers like spaCy can be used to define the properties of each word
and build syntax trees.
Name Entity Recognition:
The goal of this phase is to extract nouns from the next, including people, brand
names, acronyms, places, dates, etc.
A good NLP model can differentiate between noun types such as June the person
and June the month based on statistical inference of its surrounding words like the
presence of the preposition “in.”
Coreference Resolution:
Coreferencing tracks pronouns across sentences in relation to their entity. It is,
many argue, one of the most difficult steps in NLP programming.
At this stage, we have our parts of speech mapped as subjects, objects, verbs and
more. But our sentence model thus far only examines one sentence at a time, and
parsing is needed to match. For example, in:
“Andrew is on a flight to Bali, an island in Indonesia. He is planning on living
there. It has a warm climate”.
After coreferencing, our model would understand “he” and “it” were the subjects
of sentences two and three, but would not yet be able to connect those pronouns
to “Andrew” and “Bali,” respectively.
Due to this complicated nature, a more detailed explanation of the coreferencing is
beyond the scope of this article but can be read about in some recent research
from Sanford University.

What are components of Natural Language Processing (NLP)?

Source Data: Every natural language processing model needs training. Some
open source models come “pre-trained,” meaning those models will have to been
fed large data sets using supervised or unsupervised methods to correctly identify
patterns in new data and act accordingly.
Statistical Models: Without the ability to make predictions-based context, we
would still be in the days of rigid structure queries and commands. A host of
statistical methods and algorithms have been developing to train machines to
respond correctly to natural language.
Reference Libraries and Tools: Libraries and tools are continually being worked
on by major players in the field, and many of these resources are made available
to the public as an open source project. This means you can quickly perform
aspects NLP without having to write all the code and tag all the grammatical
datasets yourself.

What are applications of Natural Language Processing (NLP)?

Coming back to a 30,000-foot view of NLP in AI, where, exactly, is NLP used in
today’s technology? Where are you probably interacting with it even if you haven’t
noticed?
 Auto-correct uses NLP to provide grammatically compatible spelling
suggestions based on the other words in the sentence even when we vary
outside the usual misspellings.
 Chatbots using NLP can increasing automate support functions, as well as
intelligently direct queries to the correct department in cases that require
human interactions.
 Sentiment Analysis: NLP models are able to provide insights into users’
favorability opinions or tone left in their text by comparing input with
massive data sets that already contain ratings.
 Named Entity Recognition: Automatically identifies and classify nouns
within data sets for further analysis.
 Part-of-Speech Tagging also uses NLP.
 Relationship Extraction: The process of linking entities to each other, i.e.,
“Virgin Group” and “Richard Branson.”
 Topic Extraction: Tools like summarizer use LDA to automatically extract
key themes out of content.
 Machine Translation such as Google Translate combines both rules-based
learning of grammar and trained NLP algorithms of native texts, and user
submitted corrections.
 Virtual Assistants like Google Assistant can now convincingly make phone
calls to schedule, cancel or update appointments.
 Voice to Text: Siri and Alexa use statistical models like the Hidden Markov
Modelssystem (HMM) break sentence sounds into phonemes then
compare with them each other crosscheck statistical likelihood, resulting
in highly accurate transcriptions.
What are Natural Language Processing (NLP) tools?
AutoTag
This program generates topic tags from content automatically using LDA (Latent
Dirichlet Allocation) to discover the most important words in a document.

spaCy
An open source dependency parser using Python known for its accuracy in
mapping syntax trees, spaCy is becoming increasingly relevant in NLP
development.

textacy
Textacy is a Python library built on the spaCy parser that implements a number of
common data extraction methods.

PorterStemmer
PorterStemmer returns words to their root stems, like is, am, and are equate to
“to be.”

Summarizer
And Summarizer uses trained algorithms to extract key information and ignore
what is less relevant. Tokenizer reduces words and punctuation into individual
tokens.