Natural Language

Processing
NLP
What is Natural Language
processing?
• Process the information contained in natural language or human
language in the form of text or voice command.
• NLP stands for Natural Language Processing, which is a part of
Computer Science, Human language, and Artificial Intelligence. It is the
technology that is used by machines to understand, analyse,
manipulate, and interpret human's languages. It helps developers to
organize knowledge for performing tasks such as translation, automatic
summarization, Named Entity Recognition (NER), speech recognition,
relationship extraction, and topic segmentation.
• Also known as computational linguistic (CL),Human Language
technology (HLT),Natural LANGUAGE ENGINEERING (NLE)
Components of NLP

• There are the following two components of NLP -

1. Natural Language Understanding (NLU)
• Natural Language Understanding (NLU) helps the machine to
understand and analyze human language by extracting the metadata
from content such as concepts, entities, keywords, emotion, relations,
and semantic roles.

Auto-speech
I/P NLU NLG O/P
recognition
Natural Language Understanding
(NLU)
• NLU mainly used in Business applications to understand the
customer's problem in both spoken and written language.
NLU involves the following tasks -
• It is used to map the given input into useful representation.
• It is used to analyze different aspects of the language.
Natural Language Generation (NLG)

Natural Language Generation (NLG) acts as a translator that converts

the computerized data into natural language representation. It mainly
involves Text planning, Sentence planning, and Text Realization.
How to build an NLP pipeline

• There are the following steps to build an NLP pipeline –

• Step1: Sentence Segmentation
• Sentence Segment is the first step for building the NLP pipeline. It breaks the paragraph into
separate sentences.
• Example: Consider the following paragraph -
• Independence Day is one of the important festivals for every Indian citizen. It is celebrated on
the 15th of August each year ever since India got independence from the British rule. The day
celebrates independence in the true sense.
• Sentence Segment produces the following result:
• "Independence Day is one of the important festivals for every Indian citizen."
• "It is celebrated on the 15th of August each year ever since India got independence from the
British rule."
• "This day celebrates independence in the true sense."
• Step 2:
• Word Tokenizer is used to break the sentence into separate words or tokens.
• Step3: Stemming
• Stemming is used to normalize words into its base form or root form. For example, celebrates,
celebrated and celebrating, all these words are originated with a single root word "celebrate." The big
problem with stemming is that sometimes it produces the root word which may not have any meaning.
• For Example, intelligence, intelligent, and intelligently, all these words are originated with a single root
word "intelligen." In English, the word "intelligen" do not have any meaning.
• Step 4: Lemmatization
• Lemmatization is quite similar to the Stamming. It is used to group different inflected forms of the
word, called Lemma. The main difference between Stemming and lemmatization is that it produces the
root word, which has a meaning.
• For example: In lemmatization, the words intelligence, intelligent, and intelligently has a root word
intelligent, which has a meaning.
• Step 5: Identifying Stop Words
• In English, there are a lot of words that appear very frequently like
"is", "and", "the", and "a". NLP pipelines will flag these words as stop
words. Stop words might be filtered out before doing any statistical
analysis.
• Example: He is a good boy.
• Step 6: Dependency Parsing
• Dependency Parsing is used to find that how all the words in the sentence
are related to each other.
• Step 7: POS tags
• POS stands for parts of speech, which includes Noun, verb, adverb, and
Adjective. It indicates that how a word functions with its meaning as well as
grammatically within the sentences. A word has one or more parts of speech
based on the context in which it is used.
• Example: "Google" something on the Internet.
• In the above example, Google is used as a verb, although it is a proper noun.
• Step 8: Named Entity Recognition (NER)
• Named Entity Recognition (NER) is the process of detecting the
named entity such as person name, movie name, organization name,
or location.
• Example: Steve Jobs introduced iPhone at the Macworld Conference
in San Francisco, California.
• Step 9: Chunking
• Chunking is used to collect the individual piece of information and
grouping them into bigger pieces of sentences.
Stages in NLP
• Lexical
• Syntatical
• Semantical
• Discourse
• Pragmatic
Lexical/Morphological Analysis
• The lexicon of a language is its vocabulary that includes its words and
expression
• Morphological depicts analyzing ,identifying and description of
structure of words
• Lexical analysis involves dividing a text into paragraphs ,words and
sentences i.e. Tokens
• Eg.” Shidharth likes bikes” will be divided into tokens [Shidharth]
[likes] [bikes]
Syntactic Analysis
• Syntax concerns the proper ordering of words and its affect on
meaning
• This involves analysis of the words in a sentence to depict the
grammatical structure of the sentence
• Analyze whether the words in a sentence are ordered grammatically
right or not
• The words are transformed into structure that shows how the words
are related to each other
• Eg. “the girl the go to the school”, or “the school go to girl” this would
definitely be rejected by the English syntactic analyzer
Semantic Analysis
• Semantic concerns with literal meaning of words ,phrases, and
sentences
• This abstract the dictionary meaning or the exact meaning from
context
• The structures which are created by the syntactic analyser are
assigned meaning
• Example: colourless blue idea” .this would be rejected by the analyzer
as colourless blue do not makes any sense together
Discourse Integration
• Sense of the Context
• The meaning of any single sentence depends on the sentences that
precedes it and also invokes the meaning of the sentences that follow
it.
• Example: 1)the word “it” in the sentence “She wanted it ” depends on
prior discourse context.
• 2)”she goes to college “ she? As in who I am talking about?? Depends
on prior sentences
Pragmatic Analysis
• Pragmatics concerns the overall communicative and social context and its effect
on interpretation.
• It means abstracting or deriving the purposeful use of the language in
situations.
• Importantly those aspects of language which require world knowledge.
• The main focus is on what was said is reinterpreted on what it actually means.
• It is the study of intended and literal meaning of the sentence
• Example :1) ”Close the window ?” should have been interpreted as a request
rather than an order.
• 2)same sentence where “window” word is may referred as doors of window
also window operating system
Applications of Natural Language Processing (NLP):

• Spam Filters: One of the most irritating things about email is spam. Gmail uses natural language
processing (NLP) to discern which emails are legitimate and which are spam. These spam filters
look at the text in all the emails you receive and try to figure out what it means to see if it’s spam
or not.
• Algorithmic Trading: Algorithmic trading is used for predicting stock market conditions. Using
NLP, this technology examines news headlines about companies and stocks and attempts to
comprehend their meaning in order to determine if you should buy, sell, or hold certain stocks.
• Questions Answering: NLP can be seen in action by using Google Search or Siri Services. A major
use of NLP is to make search engines understand the meaning of what we are asking and
generate natural language in return to give us the answers.
• Summarizing Information: On the internet, there is a lot of information, and a lot of it comes in
the form of long documents or articles. NLP is used to decipher the meaning of the data and
then provides shorter summaries of the data so that humans can comprehend it more quickly.
• Sentiment Analysis
Sentiment Analysis is also known as opinion mining. It is used on the
web to analyse the attitude, behaviour, and emotional state of the
sender. This application is implemented through a combination of NLP
(Natural Language Processing) and statistics by assigning the values to
the text (positive, negative, or natural), identify the mood of the
context (happy, sad, angry, etc.)
• Machine Translation
Machine translation is used to translate text or speech from one natural
language to another natural language.
Example: Google Translator

• Spelling correction
Microsoft Corporation provides word processor software like MS-word,
PowerPoint for the spelling correction.
• Speech Recognition
Speech recognition is used for converting spoken words into text. It is
used in applications, such as mobile, home automation, video recovery,
dictating to Microsoft Word, voice biometrics, voice user interface, and
so on.

• Chatbot
Implementing the Chatbot is one of the important applications of NLP.
It is used by many companies to provide the customer's chat services.
• . Information extraction
Information extraction is one of the most important applications of NLP.
It is used for extracting structured information from unstructured or
semi-structured machine-readable documents.

• Natural Language Understanding (NLU)

It converts a large set of text into more formal representations such as
first-order logic structures that are easier for the computer programs to
manipulate notations of the natural language processing.
Why NLP is difficult?

• NLP is difficult because Ambiguity and Uncertainty exist in the language.

• Ambiguity
• There are the following three ambiguity –

• Lexical Ambiguity
• Lexical Ambiguity exists in the presence of two or more possible meanings of the sentence within a single
word.
• Example:
• Manya is looking for a match.
• In the above example, the word match refers to that either Manya is looking for a partner or Manya is
looking for a match. (Cricket or other match)

• .
Syntactic Ambiguity

• Syntactic Ambiguity exists in the presence of two or more possible

meanings within the sentence.
• Example:
• I saw the girl with the binocular.
• In the above example, did I have the binoculars? Or did the girl have
the binoculars?
Referential Ambiguity

• Referential Ambiguity exists when you are referring to something

using the pronoun.

• Example: Kiran went to Sunita. She said, "I am hungry."

• In the above sentence, you do not know that who is hungry, either
Kiran or Sunita