CBSE Class 10 Artificial Intelligence Unit 6 - Natural Language Processing (NLP) Complete Notes

Unit 6: Natural Language Processing (NLP)

Natural Language Processing (NLP) is a sub-field of Artificial Intelligence (AI) that enables computers to analyse, understand and process human languages so that meaningful information can be derived from spoken or written communication.

Topics Covered in This Unit

• 6.1 Introduction to NLP
• 6.2 Applications of Natural Language Processing
• 6.3 Stages of Natural Language Processing
• 6.4 Chatbots
• 6.5 Text Processing
• 6.6 Natural Language Processing: Use Case Walkthrough

6.1 Introduction to NLP

Natural Language refers to languages used naturally by humans for communication, such as English, Hindi, Spanish and Japanese. NLP acts as a bridge between human language and computers by enabling machines to process and understand human communication.

Features of Natural Language

Human languages are complex and contain several characteristics that make them difficult for computers to understand.

Feature	Meaning
Rules-Based	Natural languages follow rules related to syntax, vocabulary and meaning.
Redundant	The same information can be expressed using different words or sentence structures.
Ever-Changing	Natural languages continuously evolve with new words, expressions and usages.
Context-Dependent	The meaning of a word or sentence may depend upon the surrounding context.

The Power of Context

The meaning of a word can change according to the context in which it is used.

• "His face turned red with anger." – Red refers to an emotional expression.
• "The stop sign is red." – Red refers to a colour.
• "I have a red rash on my arm." – Red describes a medical condition.

Humans can normally understand the intended meaning from context, whereas a computer needs NLP techniques to process such variations.

Computer Language

Computers communicate through computer languages such as Python, Java and C++.

• Humans communicate naturally using spoken or written languages.
• Computers require specific instructions in the form of programs.
• At the lowest level, computers process information using binary language (0s and 1s).

Importance of NLP

NLP helps convert human speech and text into information that can be processed by computers.

Aspect	Role of NLP
Human-Computer Communication	Enables people to communicate with machines using natural language.
Intent	Helps computers understand what a user wants to do.
Context	Helps interpret meaning according to the situation.
Information Processing	Converts human language into machine-processable information.

Real-Life School Example

Suppose a student says, "I'm going to study now." Here, study is used as a verb. In another sentence, "That's a great study on climate change.", the same word is used as a noun. NLP helps a computer understand such differences based on context.

Case Study: The Language Barrier App

Remya moved from India to Sweden for higher education. She struggled to find a part-time job because she did not know Swedish. A computer science student developed a real-time voice translator that converts Swedish speech into English audio.

1. Barrier faced by Remya: Linguistic barrier.
2. Why is NLP required? The application must process natural Swedish speech, understand its meaning and generate corresponding English speech.

Competency-Based Questions

Q1. Which feature of natural language allows the same message to be conveyed using different sets of words?

Answer: Redundancy.

Q2. Why can computers not understand human language intuitively like humans?

Answer: Computers require specific instructions and computational processes to interpret human language.

Q3. Explain why perfect syntax does not always mean perfect meaning.

Answer: A sentence can follow grammatical rules but still be logically meaningless. For example, "Chickens feed extravagantly while the moon drinks tea." has grammatical structure but does not convey a logical meaning.

Q4. A teacher uses an AI tool to summarize long research papers into five bullet points. Which AI domain is being used?

Answer: Natural Language Processing (NLP).

6.2 Applications of Natural Language Processing

NLP enables computers to process human language and is used in many applications that we encounter in everyday life.

1. Voice Assistants

Voice assistants process natural speech, identify the user's intent and perform the requested task.

Examples:

• "Hey Google, set an alarm at 3:30 pm."
• "Hey Alexa, play some music."
• "Hey Siri, what's the weather today?"

School Example: A student could use a voice-based "Study Buddy" application to search for notes by saying, "Find my Physics notes from last Tuesday."

2. Autogenerated Captions

Autogenerated captions convert spoken words into text, usually in real time. They improve accessibility for people who are hard of hearing and are also useful when audio cannot be heard clearly.

Examples:

• Auto-generated captions on YouTube.
• Live captions during online meetings.

3. Language Translation

Language translation converts text or speech from one language into another language.

Example: Google Translate can translate text and speech between different languages.

Student Example: An Indian student visiting Japan can use a translation application to communicate with local people.

4. Sentiment Analysis

Sentiment Analysis identifies the opinion or emotional tone expressed in textual data.

Sentiment	Meaning	Example
Positive	Expresses satisfaction or favourable opinion.	"The service was excellent."
Negative	Expresses dissatisfaction or unfavourable opinion.	"The service was very poor."
Neutral	Does not express a clearly positive or negative opinion.	"The service is available today."

School Example: A digital suggestion box can use sentiment analysis to identify negative feedback about school facilities or services.

5. Text Classification

Text Classification automatically places text or documents into predefined categories based on their content.

Examples:

• Classifying news articles as Sports, Politics or Technology.
• Email spam filtering by classifying messages as legitimate or spam.

6. Keyword Extraction

Keyword Extraction identifies the most important words or phrases from a document. It helps understand the main topics of a long text.

Example: From a research paper about climate change, keywords may include Climate Change, Glaciers and CO2 Levels.

Application Summary

Application	Purpose
Voice Assistants	Understanding spoken commands and user intent.
Autogenerated Captions	Converting speech into text.
Language Translation	Converting text or speech from one language to another.
Sentiment Analysis	Identifying positive, negative or neutral opinions.
Text Classification	Assigning text to predefined categories.
Keyword Extraction	Identifying important words and phrases.

Case Study: Feedback Bot

A smartphone company analyses 10,000 social media comments. The system identifies that many customers are happy with the camera but unhappy with the battery life.

Question: Which NLP application is being used?

Answer: Sentiment Analysis.

Case Study: Smart News App

Rohan develops an application that automatically tags news articles as Technology or Environment and highlights important words.

Answer:

1. Text Classification – categorises the article.
2. Keyword Extraction – identifies important words.

Competency-Based Questions

Q1. Which NLP application identifies emotions in customer feedback?

Answer: Sentiment Analysis.

Q2. A video displays everything the actors are saying in text form. Which NLP application is being used?

Answer: Autogenerated Captions.

Q3. Why are autogenerated captions useful?

Answer: They improve accessibility and allow users to understand spoken content through text.

6.3 Stages of Natural Language Processing

NLP systems process language through several stages. The five major stages are:

1. Lexical Analysis
2. Syntactic Analysis
3. Semantic Analysis
4. Discourse Integration
5. Pragmatic Analysis

1. Lexical Analysis

Lexical Analysis is the first stage of NLP. It breaks a large piece of text into smaller units such as paragraphs, sentences and words.

A lexicon is the collection of words and phrases available in a language.

Example: "The quick brown fox jumped over the lazy dog" can be divided into individual words or tokens.

2. Syntactic Analysis

Syntactic Analysis, also called Parsing, checks the grammatical structure and arrangement of words in a sentence.

Sentence	Result
The quick brown fox jumped over the lazy dog.	Grammatically correct.
The quick brown jumped fox over the lazy dog.	Grammatically incorrect.

3. Semantic Analysis

Semantic Analysis checks the meaning of words and sentences.

A sentence can be grammatically correct but logically meaningless.

Example: "Chickens feed extravagantly while the moon drinks tea."

The sentence has grammatical structure, but its meaning is not logically meaningful.

4. Discourse Integration

Discourse Integration examines the relationship between multiple sentences and uses previous context to understand references.

Example: "The quick brown fox jumped over the lazy dog. Then it went into the thick bushes."

The system needs to understand that it refers to the fox.

5. Pragmatic Analysis

Pragmatic Analysis is the final stage. It attempts to understand the intended meaning, relevance, context, sarcasm and expressions rather than relying only on the literal meaning of words.

Example: "Relax! I'm just pulling your leg."

The literal meaning suggests physically pulling someone's leg, whereas the intended meaning is that the person is joking.

School Example: If a teacher says, "What a quiet room this is!" to a noisy class, pragmatic analysis helps understand that the teacher is being sarcastic and actually wants the students to become quiet.

Stages of NLP at a Glance

Stage	Main Purpose	Key Idea
Lexical Analysis	Breaks text into smaller units.	Words / Tokens
Syntactic Analysis	Checks grammatical structure.	Grammar / Parsing
Semantic Analysis	Checks meaning.	Meaning
Discourse Integration	Connects information across sentences.	Context
Pragmatic Analysis	Understands intended meaning.	Intent / Relevance

Case Study: The Jumbled Message

An NLP system receives the sentence: "To the school going are we today." It identifies the sentence as incorrect. When the sentence is changed to "We are going to the school today.", it accepts it.

Answer: Syntactic Analysis is responsible because it checks the grammatical arrangement of words.

Case Study: The Sarcastic Chatbot

Anish writes, "Wow, this website is incredibly fast! I love waiting for ten minutes." The chatbot understands that Anish is actually unhappy.

Answer: Pragmatic Analysis helps understand the intended meaning and sarcasm.

Competency-Based Questions

Q1. Which is the first stage of NLP?

Answer: Lexical Analysis.

Q2. Which stage checks grammar?

Answer: Syntactic Analysis.

Q3. Which stage checks meaning?

Answer: Semantic Analysis.

Q4. Which stage identifies that "He" refers to a person mentioned earlier?

Answer: Discourse Integration.

Q5. Which stage helps understand sarcasm?

Answer: Pragmatic Analysis.

6.4 Chatbots

A Chatbot is a computer program designed to simulate human conversation through text, voice commands or both.

Chatbots can answer questions, troubleshoot problems and provide customer support. Their main goal is to understand the user's intent and provide a relevant response.

Examples of Chatbots

• Elizabot
• Mitsuku (Kuki)
• Cleverbot
• Singtel Support

Script Bot vs Smart Bot

Feature	Script Bot (Rule-Based)	Smart Bot (AI-Powered)
Ease of Creation	Very easy to create.	Requires advanced coding and data science skills.
Working	Follows a predefined script or set of rules.	Uses larger databases and can learn from new data.
Functionality	Limited to programmed questions and responses.	Can handle complex and varied queries.
Language Skills	Little or no advanced language processing.	Can understand context and intent.
Cost and Integration	Generally easier and less expensive to integrate.	More complex and expensive to build and maintain.

Real-Life Examples

Script Bot – School Canteen Menu Bot

A bot asks users to enter 1 for the lunch menu and 2 for prices. If a student asks, "Which snack is the healthiest?", the bot may not be able to answer because the question was not included in its script.

Smart Bot – Study Buddy

An AI tutor can explain a Physics concept in different ways according to the student's requirement and can handle a variety of questions.

Script Bot – School Admission Bot

A school website chatbot can answer repetitive questions related to admission forms, fee structure and school timings.

Case Study: Aditi's Economics Chatbot

Aditi develops a chatbot to answer Economics questions. She trains the software using large datasets covering different difficulty levels. The chatbot analyses questions and provides intelligent replies.

Answer: It is a Smart Bot and uses Natural Language Processing.

Case Study: Clothing Website Tracker

A clothing website uses a bot that looks for keywords such as "track order". If the user asks an unrelated question, the bot redirects the user to its FAQs.

Answer: It is a Script Bot because it follows predefined rules and cannot improvise beyond its programmed responses.

Competency-Based Questions

Q1. Which type of bot learns from its environment and experience?

Answer: Smart Bot.

Q2. Why do Script Bots have limited functionality?

Answer: They work according to predefined scripts and rules and cannot independently learn from new user inputs.

Q3. What is a major drawback of a rule-based chatbot?

Answer: Its learning is static. If a user's input is not covered by the predefined rules, the bot may fail to provide an appropriate response.

6.5 Text Processing

Human language is complex, while computers ultimately process information in numerical form. Text Processing converts natural language into a form that can be processed mathematically by computers.

1. Text Normalisation

Text Normalisation is the process of cleaning and transforming textual data into a consistent format.

The collection of textual data from documents is called a Corpus.

Steps of Text Normalisation

Step	Description
Sentence Segmentation	Dividing a corpus or paragraph into individual sentences.
Tokenization	Dividing sentences into tokens such as words, numbers and special characters.
Removing Stop Words	Removing common words such as "a", "an", "the" and "is" when they do not add significant information.
Removing Special Characters and Numbers	Removing unnecessary characters and numbers where appropriate.
Converting to Common Case	Converting text to a common case, usually lowercase.
Stemming	Reducing words to a root form by removing affixes.
Lemmatization	Converting words to meaningful dictionary-based root forms.

Important: Stop Words and Special Characters

Special characters should not always be removed. For example, in the email address support@school.com, the symbols @ and . are essential to the information.

Stemming vs Lemmatization

Stemming	Lemmatization
Removes affixes to obtain a root form.	Finds the meaningful dictionary form.
Usually faster.	Usually slower and more sophisticated.
May produce a non-meaningful word.	Produces a meaningful lemma.
Example: studies → studi	Example: studies → study

2. Bag of Words (BoW)

Bag of Words (BoW) is a technique used to extract features from text by counting the frequency of words.

In BoW, the order of words is not considered important. The focus is on the presence and frequency of words.

Steps in Bag of Words

1. Perform text preprocessing and normalisation.
2. Create a dictionary containing unique words.
3. Create a document vector by counting how many times each dictionary word appears in the document.
4. Assign 0 when a dictionary word does not occur in a document.

Example

Consider two documents:

• Document 1: Aman and Avni are stressed
• Document 2: Aman went to a therapist

A dictionary is created containing unique words from both documents. Each document is then represented by a numerical vector showing the frequency of each dictionary word.

3. TFIDF

TFIDF stands for Term Frequency – Inverse Document Frequency. It helps identify how important a word is within a document and across a collection of documents.

Term Frequency (TF)

Term Frequency represents how frequently a particular word occurs in a specific document.

Inverse Document Frequency (IDF)

Inverse Document Frequency reduces the importance of words that occur in many documents and gives greater importance to words that are relatively rare across documents.

Word Pattern	TFIDF Interpretation
Frequent in one document but rare across other documents	High importance / High TFIDF value
Occurs in almost every document	Low importance / Low TFIDF value
Occurs in every document	IDF becomes zero, resulting in zero TFIDF

Applications of TFIDF

1. Document Classification – classifying documents into categories.
2. Topic Modelling – identifying the main subjects of a collection of documents.
3. Information Retrieval – finding important information in documents.
4. Stop Word Filtering – reducing the importance of commonly occurring words.

Real-Life Examples

• Library Sorter: Bag of Words can identify frequently occurring keywords and help suggest a suitable category for a book.
• Essay Checker: Text Normalisation can clean student essays and reduce unnecessary variations in words.
• Research Paper Finder: TFIDF can help identify documents containing important and relatively rare terms.

Case Study: Document Vector Challenge

Consider the following documents:

• Doc 1: Students love studying AI
• Doc 2: AI is transforming education
• Doc 3: Teachers and students explore AI tools

A dictionary can be created using the unique words:

students, love, studying, ai, is, transforming, education, teachers, and, explore, tools

The vector for each document is constructed by counting the occurrence of each dictionary word. If a word does not occur in a document, its value is 0.

Why is word order not important in BoW? BoW focuses on the occurrence and frequency of words rather than their grammatical order.

Case Study: Email ID Dilemma

A student performs stop-word and special-character removal on: Contact us at support@school.com.

Answer: The symbols @ and . should not be removed because they are essential components of the email address.

Competency-Based Questions

Q1. What are the stemmed and lemmatized forms of "flies"?

Answer: Stemmed form: fli; Lemmatized form: fly.

Q2. Why is text converted to lowercase during preprocessing?

Answer: It ensures that words such as Apple and apple are not treated as separate words merely because of case differences.

Q3. In a corpus of 100 documents, the word "the" occurs in all 100 documents. What will its TFIDF value be?

Answer: Zero, because its IDF becomes log(100/100) = log(1) = 0.

6.6 Natural Language Processing: Use Case Walkthrough

This section connects NLP concepts with practical tools and applications, particularly Sentiment Analysis.

Examples of Code and No-Code NLP Tools

Feature	Code-Based NLP Tools	No-Code NLP Tools
Meaning	Require programming, commonly using Python.	Use visual and drag-and-drop interfaces.
Example 1	NLTK – Natural Language Toolkit for text-processing tasks.	Orange Data Mining – visual data analysis and data mining platform.
Example 2	SpaCy – open-source library for advanced NLP tasks.	MonkeyLearn – platform for text and sentiment analysis.

Code-Based NLP Tools

• NLTK: Natural Language Toolkit, a Python package used for text processing.
• SpaCy: An open-source NLP library used for advanced language-processing tasks such as Named Entity Recognition.

No-Code NLP Tools

• Orange Data Mining: Provides visual programming and drag-and-drop workflows.
• MonkeyLearn: Can be used for text analysis, sentiment analysis and social media monitoring.

Applications of Sentiment Analysis

Sentiment Analysis determines whether textual information expresses a positive, negative or neutral sentiment.

1. Customer Service

Businesses can analyse customer conversations and feedback to identify dissatisfaction, satisfaction or other emotional trends.

2. Voice of the Customer (VoC)

Voice of the Customer (VoC) helps organisations understand the gap between what customers expect and what they actually experience.

The information can provide actionable insights for improving products and services.

School Example

A school can analyse student or parent feedback to identify areas that receive positive or negative responses and use the findings for improvement.

Sentiment Analysis using Orange Data Mining

Orange Data Mining can be used as a no-code tool for performing sentiment analysis without writing complex Python programs.

Typical Workflow

1. Data Acquisition: Import a dataset containing textual information.
2. Pre-processing: Use text-processing tools to clean and prepare the data.
3. Sentiment Analysis: Analyse the text and assign sentiment information.
4. Visualisation: Use suitable visualisation or data-table tools to examine the results.

Case Study: Eco-Friendly School Campaign

A group of Class X students conducts an "Eco-Friendly School" campaign. They collect 200 student responses about plastic usage in the school canteen and analyse the responses using Orange Data Mining.

1. Which NLP technique should be used to understand the mood of the school?
   Answer: Sentiment Analysis.
2. Name one code-based NLP tool that could be used instead of Orange.
   Answer: NLTK or SpaCy.
3. What is the primary benefit of Orange?
   Answer: It is a No-Code tool that uses visual, drag-and-drop workflows, allowing users to perform data-mining and NLP tasks without advanced programming skills.

Competency-Based Questions

Q1. Which is an open-source NLP library designed for Python?

Answer: SpaCy.

Q2. What is Voice of the Customer?

Answer: It is an approach that helps understand the gap between what customers expect and what they actually experience.

Q3. Are NLTK and SpaCy No-Code tools?

Answer: No. NLTK and SpaCy are code-based NLP tools. Orange Data Mining and MonkeyLearn are examples of no-code/visual tools.

Q4. Sentiment Analysis classifies text into which three categories?

Answer: Positive, Negative and Neutral.

Quick Revision: Unit 6

Concept	Remember
NLP	AI field that processes human language.
Redundancy	Same information can be expressed in different ways.
Voice Assistant	Processes spoken commands and identifies intent.
Sentiment Analysis	Positive, Negative or Neutral.
Text Classification	Assigns text to predefined categories.
Keyword Extraction	Identifies important words and phrases.
Lexical Analysis	Breaks text into smaller units or tokens.
Syntactic Analysis	Checks grammar and sentence structure.
Semantic Analysis	Checks meaning.
Discourse Integration	Uses context across multiple sentences.
Pragmatic Analysis	Understands intent, relevance and implied meaning.
Script Bot	Rule-based and follows predefined instructions.
Smart Bot	AI-powered and capable of handling varied queries.
Text Normalisation	Cleans and standardises textual data.
Bag of Words	Represents text using word frequency.
TFIDF	Measures the importance of words.
NLTK	Code-based Python NLP toolkit.
SpaCy	Code-based open-source NLP library.
Orange Data Mining	No-Code / visual data-mining tool.
MonkeyLearn	Tool for text and sentiment analysis.

Exam-Oriented Key Points

• NLP is a sub-field of Artificial Intelligence.
• Lexical Analysis is the first stage of NLP.
• Syntactic Analysis deals with grammar and sentence structure.
• Semantic Analysis deals with meaning.
• Discourse Integration uses information from previous and subsequent sentences.
• Pragmatic Analysis focuses on intended meaning, context and relevance.
• Sentiment Analysis identifies positive, negative and neutral opinions.
• Bag of Words focuses on word occurrence and frequency; word order is not considered important.
• TFIDF identifies the importance of words within documents.
• NLTK and SpaCy are code-based NLP tools.
• Orange Data Mining is a no-code/visual tool.
• A Script Bot follows predefined rules.
• A Smart Bot can handle more complex queries using AI and data.