Download our e-book of Introduction To Python
Related Blog
Matplotlib - Subplot2grid() FunctionDiscuss Microsoft Cognitive ToolkitMatplotlib - Working with ImagesMatplotlib - PyLab moduleMatplotlib - Working With TextMatplotlib - Setting Ticks and Tick LabelsCNTK - Creating First Neural NetworkMatplotlib - MultiplotsMatplotlib - Quiver PlotPython - Chunks and Chinks View More
Top Discussion
How can I write Python code to change a date string from "mm/dd/yy hh: mm" format to "YYYY-MM-DD HH: mm" format? Which sorting technique is used by sort() and sorted() functions of python? How to use Enum in python? Can you please help me with this error? I was just selecting some random columns from the diabetes dataset of sklearn. Decision tree is a classification algo...How can it be applied to load diabetes dataset which has DV continuous Objects in Python are mutable or immutable? How can unclassified data in a dataset be effectively managed when utilizing a decision tree-based classification model in Python? How to leave/exit/deactivate a Python virtualenvironment Join Discussion
Top Courses
Webinars
AI with Python – Natural Language Processing
Jon Wood
2 years ago
Table of Contents
- Introduction
- Components of NLP
1. Natural Language Understanding (NLU)
2. Natural Language Generation (NLG)
- Difficulties in NLU
1. Lexical ambiguity
2. Syntax level ambiguity
3. Referential ambiguity
- NLP Terminology
- Steps in NLP
1. Lexical Analysis
2. Syntactic Analysis (Parsing)
3. Semantic Analysis
4. Discourse Integration
5. Pragmatic Analysis
Introduction
Natural Language Processing (NLP) refers to the artificial intelligence (AI) method of communicating with intelligent systems using a natural language such as English.
Processing of Natural Language is required when you want an intelligent system like a robot to perform as per your instructions, when you want to hear a decision from a dialogue based clinical expert system, etc.
The field of NLP involves making computers perform useful tasks with the natural languages humans use.
The input and output of an NLP system can be −
- Speech
- Written Text
Components of NLP
In this section, we will learn about the different components of NLP. There are two components of NLP. The components are described below −
1. Natural Language Understanding (NLU)
It involves the following tasks −
- Mapping the given input in natural language into useful representations.
- Analyzing different aspects of the language.
2. Natural Language Generation (NLG)
It is the process of producing meaningful phrases and sentences in the form of natural language from some internal representation. It involves −
- Text planning − This includes retrieving the relevant content from the knowledge base.
- Sentence planning − This includes choosing the required words, forming meaningful phrases, setting the tone of the sentence.
- Text Realization − This is mapping sentence plan into sentence structure.
Difficulties in NLU
The NLU is very rich in form and structure; however, it is ambiguous. There can be different levels of ambiguity −
1. Lexical ambiguity
It is at a very primitive level such as the word-level. For example, treating the word “board” as a noun or a verb?
2. Syntax level ambiguity
A sentence can be parsed in different ways. For example, “He lifted the beetle with a red cap.” − Did he use a cap to lift the beetle or he lifted a beetle that had a red cap?
3. Referential ambiguity
Referring to something using pronouns. For example, Rima went to Gauri. She said, “I am tired.” − Exactly who is tired?
NLP Terminology
Let us now see a few important terms in the NLP terminology.
- Phonology − It is the study of organizing sound systematically.
- Morphology − It is a study of the construction of words from primitive meaningful units.
- Morpheme − It is a primitive unit of meaning in a language.
- Syntax − It refers to arranging words to make a sentence. It also involves determining the structural role of words in the sentence and in phrases.
- Semantics − It is concerned with the meaning of words and how to combine words into meaningful phrases and sentences.
- Pragmatics − It deals with using and understanding sentences in different situations and how the interpretation of the sentence is affected.
- Discourse − It deals with how the immediately preceding sentence can affect the interpretation of the next sentence.
- World Knowledge − It includes general knowledge about the world.
Steps in NLP
This section shows the different steps in NLP.
1. Lexical Analysis
It involves identifying and analyzing the structure of words. Lexicon of a language means the collection of words and phrases in a language. Lexical analysis is dividing the whole chunk of text into paragraphs, sentences, and words.
2. Syntactic Analysis (Parsing)
It involves the analysis of words in the sentence for grammar and arranging words in a manner that shows the relationship among the words. The sentence such as “The school goes to the boy” is rejected by an English syntactic analyzer.
3. Semantic Analysis
It draws the exact meaning or the dictionary meaning from the text. The text is checked for meaningfulness. It is done by mapping syntactic structures and objects in the task domain. The semantic analyzer disregards sentences such as “hot ice-cream”.
4. Discourse Integration
The meaning of any sentence depends upon the meaning of the sentence just before it. In addition, it also brings about the meaning of immediately succeeding sentence.
5. Pragmatic Analysis
During this, what was said is re-interpreted on what it actually meant. It involves deriving those aspects of language which require real-world knowledge.
Liked what you read? Then don’t break the spree. Visit our insideAIML blog page to read more awesome articles.
Or if you are into videos, then we have an amazing Youtube channel as well. Visit our InsideAIML Youtube Page to learn all about Artificial Intelligence, Deep Learning, Data Science and Machine Learning.
Keep Learning. Keep Growing.