Page 1
* Images shown here are the property of individual organisations and are used here for reference purpose only.
Natural Language Processing
Introduction
Till now, we have explored two domains of AI: Data Science and Computer Vision. Both these domains
differ from each other in terms of the data on which they work. Data Science works around numbers
and tabular data while Computer Vision is all about visual data like images and videos. The third
domain, Natural Language Processing (commonly called NLP) takes in the data of Natural Languages
which humans use in their daily lives and operates on this.
Natural Language Processing, or NLP, is the sub-field of AI that is focused on enabling computers to
understand and process human languages. AI is a subfield of Linguistics, Computer Science,
Information Engineering, and Artificial Intelligence concerned with the interactions between
computers and human (natural) languages, in particular how to program computers to process and
analyse large amounts of natural language data.
But how do computers do that? How do they understand what we say in our language? This chapter
is all about demystifying the Natural Language Processing domain and understanding how it works.
Before we get deeper into NLP, let us experience it with the help of this AI Game:
Identify the mystery animal: http://bit.ly/iai4yma
Go to this link on Google Chrome, launch the experiment and try to identify the Mystery Animal by
asking the machine 20 Yes or No questions.
Were you able to guess the animal?
__________________________________________________________________________________
__________________________________________________________________________________
If yes, in how many questions were you able to guess it?
__________________________________________________________________________________
__________________________________________________________________________________
If no, how many times did you try playing this game?
__________________________________________________________________________________
__________________________________________________________________________________
What according to you was the task of the machine?
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
Page 2
* Images shown here are the property of individual organisations and are used here for reference purpose only.
Natural Language Processing
Introduction
Till now, we have explored two domains of AI: Data Science and Computer Vision. Both these domains
differ from each other in terms of the data on which they work. Data Science works around numbers
and tabular data while Computer Vision is all about visual data like images and videos. The third
domain, Natural Language Processing (commonly called NLP) takes in the data of Natural Languages
which humans use in their daily lives and operates on this.
Natural Language Processing, or NLP, is the sub-field of AI that is focused on enabling computers to
understand and process human languages. AI is a subfield of Linguistics, Computer Science,
Information Engineering, and Artificial Intelligence concerned with the interactions between
computers and human (natural) languages, in particular how to program computers to process and
analyse large amounts of natural language data.
But how do computers do that? How do they understand what we say in our language? This chapter
is all about demystifying the Natural Language Processing domain and understanding how it works.
Before we get deeper into NLP, let us experience it with the help of this AI Game:
Identify the mystery animal: http://bit.ly/iai4yma
Go to this link on Google Chrome, launch the experiment and try to identify the Mystery Animal by
asking the machine 20 Yes or No questions.
Were you able to guess the animal?
__________________________________________________________________________________
__________________________________________________________________________________
If yes, in how many questions were you able to guess it?
__________________________________________________________________________________
__________________________________________________________________________________
If no, how many times did you try playing this game?
__________________________________________________________________________________
__________________________________________________________________________________
What according to you was the task of the machine?
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
* Images shown here are the property of individual organisations and are used here for reference purpose only.
Were there any challenges that you faced while playing this game? If yes, list them down.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
What approach must one follow to win this game?
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
Applications of Natural Language Processing
Since Artificial Intelligence nowadays is becoming an integral part of our lives, its applications are very
commonly used by the majority of people in their daily lives. Here are some of the applications of
Natural Language Processing which are used in the real-life scenario:
Automatic Summarization: Information overload is a real
problem when we need to access a specific, important piece
of information from a huge knowledge base. Automatic
summarization is relevant not only for summarizing the
meaning of documents and information, but also to
understand the emotional meanings within the information,
such as in collecting data from social media. Automatic
summarization is especially relevant when used to provide an
overview of a news item or blog post, while avoiding
redundancy from multiple sources and maximizing the
diversity of content obtained.
Sentiment Analysis: The goal of sentiment
analysis is to identify sentiment among several
posts or even in the same post where emotion is
not always explicitly expressed. Companies use
Natural Language Processing applications, such as
sentiment analysis, to identify opinions and
sentiment online to help them understand what
customers think about their products and services
(i.e., “I love the new iPhone” and, a few lines later
“But sometimes it doesn’t work well” where the
person is still talking about the iPhone) and overall
Page 3
* Images shown here are the property of individual organisations and are used here for reference purpose only.
Natural Language Processing
Introduction
Till now, we have explored two domains of AI: Data Science and Computer Vision. Both these domains
differ from each other in terms of the data on which they work. Data Science works around numbers
and tabular data while Computer Vision is all about visual data like images and videos. The third
domain, Natural Language Processing (commonly called NLP) takes in the data of Natural Languages
which humans use in their daily lives and operates on this.
Natural Language Processing, or NLP, is the sub-field of AI that is focused on enabling computers to
understand and process human languages. AI is a subfield of Linguistics, Computer Science,
Information Engineering, and Artificial Intelligence concerned with the interactions between
computers and human (natural) languages, in particular how to program computers to process and
analyse large amounts of natural language data.
But how do computers do that? How do they understand what we say in our language? This chapter
is all about demystifying the Natural Language Processing domain and understanding how it works.
Before we get deeper into NLP, let us experience it with the help of this AI Game:
Identify the mystery animal: http://bit.ly/iai4yma
Go to this link on Google Chrome, launch the experiment and try to identify the Mystery Animal by
asking the machine 20 Yes or No questions.
Were you able to guess the animal?
__________________________________________________________________________________
__________________________________________________________________________________
If yes, in how many questions were you able to guess it?
__________________________________________________________________________________
__________________________________________________________________________________
If no, how many times did you try playing this game?
__________________________________________________________________________________
__________________________________________________________________________________
What according to you was the task of the machine?
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
* Images shown here are the property of individual organisations and are used here for reference purpose only.
Were there any challenges that you faced while playing this game? If yes, list them down.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
What approach must one follow to win this game?
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
Applications of Natural Language Processing
Since Artificial Intelligence nowadays is becoming an integral part of our lives, its applications are very
commonly used by the majority of people in their daily lives. Here are some of the applications of
Natural Language Processing which are used in the real-life scenario:
Automatic Summarization: Information overload is a real
problem when we need to access a specific, important piece
of information from a huge knowledge base. Automatic
summarization is relevant not only for summarizing the
meaning of documents and information, but also to
understand the emotional meanings within the information,
such as in collecting data from social media. Automatic
summarization is especially relevant when used to provide an
overview of a news item or blog post, while avoiding
redundancy from multiple sources and maximizing the
diversity of content obtained.
Sentiment Analysis: The goal of sentiment
analysis is to identify sentiment among several
posts or even in the same post where emotion is
not always explicitly expressed. Companies use
Natural Language Processing applications, such as
sentiment analysis, to identify opinions and
sentiment online to help them understand what
customers think about their products and services
(i.e., “I love the new iPhone” and, a few lines later
“But sometimes it doesn’t work well” where the
person is still talking about the iPhone) and overall
* Images shown here are the property of individual organisations and are used here for reference purpose only.
indicators of their reputation. Beyond determining simple polarity, sentiment analysis understands
sentiment in context to help better understand what’s behind an expressed opinion, which can be
extremely relevant in understanding and driving purchasing decisions.
Text classification: Text classification makes it possible to assign
predefined categories to a document and organize it to help you
find the information you need or simplify some activities. For
example, an application of text categorization is spam filtering in
email.
Virtual Assistants: Nowadays Google Assistant, Cortana,
Siri, Alexa, etc have become an integral part of our lives. Not
only can we talk to them but they also have the abilities to
make our lives easier. By accessing our data, they can help
us in keeping notes of our tasks, make calls for us, send
messages and a lot more. With the help of speech
recognition, these assistants can not only detect our speech
but can also make sense out of it. According to recent
researches, a lot more advancements are expected in this
field in the near future.
Natural Language Processing: Getting Started
Natural Language Processing is all about how machines try to understand and interpret human
language and operate accordingly. But how can Natural Language Processing be used to solve the
problems around us? Let us take a look.
Revisiting the AI Project Cycle
Let us try to understand how we can develop a project in Natural Language processing with the help
of an example.
The Scenario
The world is competitive nowadays. People face
competition in even the tiniest tasks and are expected to
give their best at every point in time. When people are
unable to meet these expectations, they get stressed and
could even go into depression. We get to hear a lot of cases
where people are depressed due to reasons like peer
pressure, studies, family issues, relationships, etc. and they
eventually get into something that is bad for them as well
as for others. So, to overcome this, cognitive behavioural
therapy (CBT) is considered to be one of the best methods
to address stress as it is easy to implement on people and
also gives good results. This therapy includes
Page 4
* Images shown here are the property of individual organisations and are used here for reference purpose only.
Natural Language Processing
Introduction
Till now, we have explored two domains of AI: Data Science and Computer Vision. Both these domains
differ from each other in terms of the data on which they work. Data Science works around numbers
and tabular data while Computer Vision is all about visual data like images and videos. The third
domain, Natural Language Processing (commonly called NLP) takes in the data of Natural Languages
which humans use in their daily lives and operates on this.
Natural Language Processing, or NLP, is the sub-field of AI that is focused on enabling computers to
understand and process human languages. AI is a subfield of Linguistics, Computer Science,
Information Engineering, and Artificial Intelligence concerned with the interactions between
computers and human (natural) languages, in particular how to program computers to process and
analyse large amounts of natural language data.
But how do computers do that? How do they understand what we say in our language? This chapter
is all about demystifying the Natural Language Processing domain and understanding how it works.
Before we get deeper into NLP, let us experience it with the help of this AI Game:
Identify the mystery animal: http://bit.ly/iai4yma
Go to this link on Google Chrome, launch the experiment and try to identify the Mystery Animal by
asking the machine 20 Yes or No questions.
Were you able to guess the animal?
__________________________________________________________________________________
__________________________________________________________________________________
If yes, in how many questions were you able to guess it?
__________________________________________________________________________________
__________________________________________________________________________________
If no, how many times did you try playing this game?
__________________________________________________________________________________
__________________________________________________________________________________
What according to you was the task of the machine?
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
* Images shown here are the property of individual organisations and are used here for reference purpose only.
Were there any challenges that you faced while playing this game? If yes, list them down.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
What approach must one follow to win this game?
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
Applications of Natural Language Processing
Since Artificial Intelligence nowadays is becoming an integral part of our lives, its applications are very
commonly used by the majority of people in their daily lives. Here are some of the applications of
Natural Language Processing which are used in the real-life scenario:
Automatic Summarization: Information overload is a real
problem when we need to access a specific, important piece
of information from a huge knowledge base. Automatic
summarization is relevant not only for summarizing the
meaning of documents and information, but also to
understand the emotional meanings within the information,
such as in collecting data from social media. Automatic
summarization is especially relevant when used to provide an
overview of a news item or blog post, while avoiding
redundancy from multiple sources and maximizing the
diversity of content obtained.
Sentiment Analysis: The goal of sentiment
analysis is to identify sentiment among several
posts or even in the same post where emotion is
not always explicitly expressed. Companies use
Natural Language Processing applications, such as
sentiment analysis, to identify opinions and
sentiment online to help them understand what
customers think about their products and services
(i.e., “I love the new iPhone” and, a few lines later
“But sometimes it doesn’t work well” where the
person is still talking about the iPhone) and overall
* Images shown here are the property of individual organisations and are used here for reference purpose only.
indicators of their reputation. Beyond determining simple polarity, sentiment analysis understands
sentiment in context to help better understand what’s behind an expressed opinion, which can be
extremely relevant in understanding and driving purchasing decisions.
Text classification: Text classification makes it possible to assign
predefined categories to a document and organize it to help you
find the information you need or simplify some activities. For
example, an application of text categorization is spam filtering in
email.
Virtual Assistants: Nowadays Google Assistant, Cortana,
Siri, Alexa, etc have become an integral part of our lives. Not
only can we talk to them but they also have the abilities to
make our lives easier. By accessing our data, they can help
us in keeping notes of our tasks, make calls for us, send
messages and a lot more. With the help of speech
recognition, these assistants can not only detect our speech
but can also make sense out of it. According to recent
researches, a lot more advancements are expected in this
field in the near future.
Natural Language Processing: Getting Started
Natural Language Processing is all about how machines try to understand and interpret human
language and operate accordingly. But how can Natural Language Processing be used to solve the
problems around us? Let us take a look.
Revisiting the AI Project Cycle
Let us try to understand how we can develop a project in Natural Language processing with the help
of an example.
The Scenario
The world is competitive nowadays. People face
competition in even the tiniest tasks and are expected to
give their best at every point in time. When people are
unable to meet these expectations, they get stressed and
could even go into depression. We get to hear a lot of cases
where people are depressed due to reasons like peer
pressure, studies, family issues, relationships, etc. and they
eventually get into something that is bad for them as well
as for others. So, to overcome this, cognitive behavioural
therapy (CBT) is considered to be one of the best methods
to address stress as it is easy to implement on people and
also gives good results. This therapy includes
understanding the behaviour and mindset of a person in their normal life. With the help of CBT,
therapists help people overcome their stress and live a happy life.
To understand more about the concept of this therapy, visit this link:
https://en.wikipedia.org/wiki/Cognitive_behavioral_therapy
Problem Scoping
CBT is a technique used by most therapists to cure patients out of stress and depression. But it has
been observed that people do not wish to seek the help of a psychiatrist willingly. They try to avoid
such interactions as much as possible. Thus, there is a need to bridge the gap between a person who
needs help and the psychiatrist. Let us look at various factors around this problem through the 4Ws
problem canvas.
Who Canvas – Who has the problem?
Who are the
stakeholders?
o People who suffer from stress and are at the onset of depression.
What do we know
about them?
o People who are going through stress are reluctant to consult a psychiatrist.
What Canvas – What is the nature of the problem?
What is the
problem?
o People who need help are reluctant to consult a psychiatrist and hence live
miserably.
How do you know
it is a problem?
o Studies around mental stress and depression available on various authentic
sources.
Where Canvas – Where does the problem arise?
What is the context/situation
in which the stakeholders
experience this problem?
o When they are going through a stressful period of time
o Due to some unpleasant experiences
Why Canvas – Why do you think it is a problem worth solving?
What would be of key
value to the stakeholders?
o People get a platform where they can talk and vent out their
feelings anonymously
o People get a medium that can interact with them and applies
primitive CBT on them and can suggest help whenever needed
How would it improve their
situation?
o People would be able to vent out their stress
o They would consider going to a psychiatrist whenever required
Page 5
* Images shown here are the property of individual organisations and are used here for reference purpose only.
Natural Language Processing
Introduction
Till now, we have explored two domains of AI: Data Science and Computer Vision. Both these domains
differ from each other in terms of the data on which they work. Data Science works around numbers
and tabular data while Computer Vision is all about visual data like images and videos. The third
domain, Natural Language Processing (commonly called NLP) takes in the data of Natural Languages
which humans use in their daily lives and operates on this.
Natural Language Processing, or NLP, is the sub-field of AI that is focused on enabling computers to
understand and process human languages. AI is a subfield of Linguistics, Computer Science,
Information Engineering, and Artificial Intelligence concerned with the interactions between
computers and human (natural) languages, in particular how to program computers to process and
analyse large amounts of natural language data.
But how do computers do that? How do they understand what we say in our language? This chapter
is all about demystifying the Natural Language Processing domain and understanding how it works.
Before we get deeper into NLP, let us experience it with the help of this AI Game:
Identify the mystery animal: http://bit.ly/iai4yma
Go to this link on Google Chrome, launch the experiment and try to identify the Mystery Animal by
asking the machine 20 Yes or No questions.
Were you able to guess the animal?
__________________________________________________________________________________
__________________________________________________________________________________
If yes, in how many questions were you able to guess it?
__________________________________________________________________________________
__________________________________________________________________________________
If no, how many times did you try playing this game?
__________________________________________________________________________________
__________________________________________________________________________________
What according to you was the task of the machine?
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
* Images shown here are the property of individual organisations and are used here for reference purpose only.
Were there any challenges that you faced while playing this game? If yes, list them down.
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
What approach must one follow to win this game?
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
__________________________________________________________________________________
Applications of Natural Language Processing
Since Artificial Intelligence nowadays is becoming an integral part of our lives, its applications are very
commonly used by the majority of people in their daily lives. Here are some of the applications of
Natural Language Processing which are used in the real-life scenario:
Automatic Summarization: Information overload is a real
problem when we need to access a specific, important piece
of information from a huge knowledge base. Automatic
summarization is relevant not only for summarizing the
meaning of documents and information, but also to
understand the emotional meanings within the information,
such as in collecting data from social media. Automatic
summarization is especially relevant when used to provide an
overview of a news item or blog post, while avoiding
redundancy from multiple sources and maximizing the
diversity of content obtained.
Sentiment Analysis: The goal of sentiment
analysis is to identify sentiment among several
posts or even in the same post where emotion is
not always explicitly expressed. Companies use
Natural Language Processing applications, such as
sentiment analysis, to identify opinions and
sentiment online to help them understand what
customers think about their products and services
(i.e., “I love the new iPhone” and, a few lines later
“But sometimes it doesn’t work well” where the
person is still talking about the iPhone) and overall
* Images shown here are the property of individual organisations and are used here for reference purpose only.
indicators of their reputation. Beyond determining simple polarity, sentiment analysis understands
sentiment in context to help better understand what’s behind an expressed opinion, which can be
extremely relevant in understanding and driving purchasing decisions.
Text classification: Text classification makes it possible to assign
predefined categories to a document and organize it to help you
find the information you need or simplify some activities. For
example, an application of text categorization is spam filtering in
email.
Virtual Assistants: Nowadays Google Assistant, Cortana,
Siri, Alexa, etc have become an integral part of our lives. Not
only can we talk to them but they also have the abilities to
make our lives easier. By accessing our data, they can help
us in keeping notes of our tasks, make calls for us, send
messages and a lot more. With the help of speech
recognition, these assistants can not only detect our speech
but can also make sense out of it. According to recent
researches, a lot more advancements are expected in this
field in the near future.
Natural Language Processing: Getting Started
Natural Language Processing is all about how machines try to understand and interpret human
language and operate accordingly. But how can Natural Language Processing be used to solve the
problems around us? Let us take a look.
Revisiting the AI Project Cycle
Let us try to understand how we can develop a project in Natural Language processing with the help
of an example.
The Scenario
The world is competitive nowadays. People face
competition in even the tiniest tasks and are expected to
give their best at every point in time. When people are
unable to meet these expectations, they get stressed and
could even go into depression. We get to hear a lot of cases
where people are depressed due to reasons like peer
pressure, studies, family issues, relationships, etc. and they
eventually get into something that is bad for them as well
as for others. So, to overcome this, cognitive behavioural
therapy (CBT) is considered to be one of the best methods
to address stress as it is easy to implement on people and
also gives good results. This therapy includes
understanding the behaviour and mindset of a person in their normal life. With the help of CBT,
therapists help people overcome their stress and live a happy life.
To understand more about the concept of this therapy, visit this link:
https://en.wikipedia.org/wiki/Cognitive_behavioral_therapy
Problem Scoping
CBT is a technique used by most therapists to cure patients out of stress and depression. But it has
been observed that people do not wish to seek the help of a psychiatrist willingly. They try to avoid
such interactions as much as possible. Thus, there is a need to bridge the gap between a person who
needs help and the psychiatrist. Let us look at various factors around this problem through the 4Ws
problem canvas.
Who Canvas – Who has the problem?
Who are the
stakeholders?
o People who suffer from stress and are at the onset of depression.
What do we know
about them?
o People who are going through stress are reluctant to consult a psychiatrist.
What Canvas – What is the nature of the problem?
What is the
problem?
o People who need help are reluctant to consult a psychiatrist and hence live
miserably.
How do you know
it is a problem?
o Studies around mental stress and depression available on various authentic
sources.
Where Canvas – Where does the problem arise?
What is the context/situation
in which the stakeholders
experience this problem?
o When they are going through a stressful period of time
o Due to some unpleasant experiences
Why Canvas – Why do you think it is a problem worth solving?
What would be of key
value to the stakeholders?
o People get a platform where they can talk and vent out their
feelings anonymously
o People get a medium that can interact with them and applies
primitive CBT on them and can suggest help whenever needed
How would it improve their
situation?
o People would be able to vent out their stress
o They would consider going to a psychiatrist whenever required
Now that we have gone through all the factors around the problem, the problem statement templates
go as follows:
Our People undergoing stress Who?
Have a problem of Not being able to share their feelings What?
While They need help in venting out their emotions Where?
An ideal solution would
Provide them a platform to share their thoughts
anonymously and suggest help whenever required
Why
This leads us to the goal of our project which is:
“To create a chatbot which can interact with people, help them
to vent out their feelings and take them through primitive CBT.”
Data Acquisition
To understand the sentiments of people, we need to collect their conversational data so the machine
can interpret the words that they use and understand their meaning. Such data can be collected from
various means:
1. Surveys 2. Observing the therapist’s sessions
3. Databases available on the internet 4. Interviews, etc.
Data Exploration
Once the textual data has been collected, it needs to be processed and cleaned so that an easier
version can be sent to the machine. Thus, the text is normalised through various steps and is lowered
to minimum vocabulary since the machine does not require grammatically correct statements but the
essence of it.
Modelling
Once the text has been normalised, it is then fed to an NLP based AI model. Note that in NLP, modelling
requires data pre-processing only after which the data is fed to the machine. Depending upon the type
of chatbot we try to make, there are a lot of AI models available which help us build the foundation of
our project.
Evaluation
The model trained is then evaluated and the accuracy for the same is generated on the basis of the
relevance of the answers which the machine gives to the user’s responses. To understand the
efficiency of the model, the suggested answers by the chatbot are compared to the actual answers.
Read More