Working on Big Data Analytics
Class 12 · Artificial Intelligence
5.6 Working on Big Data Analytics
Big Data Analytics involves a systematic process of collecting, processing, cleaning and analyzing large volumes of data to discover meaningful patterns, trends and insights. These insights can help organizations improve operations and make better decisions.
The basic process of working on Big Data Analytics can be understood through four major steps: Gather Data → Process Data → Clean Data → Analyze Data.
Four Major Steps of Big Data Analytics
| Step | Purpose | Examples |
|---|---|---|
| 1. Gather Data | Collect data from different sources in structured and unstructured forms. | Cloud storage, mobile applications and IoT sensors. |
| 2. Process Data | Organise and process the collected data so that it can be used for analysis. | Batch Processing and Stream Processing. |
| 3. Clean Data | Remove errors, duplicate records, irrelevant information and formatting inconsistencies. | Removing duplicate student records or correcting inconsistent data formats. |
| 4. Analyze Data | Examine the processed and cleaned data to identify patterns, trends and useful insights. | Tableau, Apache Hadoop, Cassandra and MongoDB. |
Step 1: Gather Data
The first step in Big Data Analytics is to gather data from different sources. Organizations may collect both structured and unstructured data depending on their requirements.
Common Sources of Big Data
- Cloud storage
- Mobile applications
- IoT (Internet of Things) sensors
- Transactional systems
- Web applications
- Social media platforms
A school may gather student attendance records, examination marks, fee information, library transactions and learning activity data from different digital systems.
These datasets can later be processed and analyzed to identify attendance patterns, academic trends and other useful insights.
Step 2: Process Data
After collecting data, it needs to be processed and organised so that it can be analysed effectively.
Two important approaches to processing Big Data are Batch Processing and Stream Processing.
Batch Processing
Batch Processing involves collecting and processing large blocks of data together over a specific period of time.
A school processes the complete examination result data of all students at the end of a term to prepare reports.
Stream Processing
Stream Processing processes continuous streams of data immediately as they are produced. It is useful when information needs to be analysed with very little delay.
A website continuously monitors user activity and processes clicks and searches as they occur to understand current user behaviour.
Batch Processing vs Stream Processing
| Feature | Batch Processing | Stream Processing |
|---|---|---|
| Processing | Processes large blocks of data together. | Processes continuous streams of data. |
| Timing | Data is processed after it has been collected for a particular period. | Data is processed immediately as it is produced. |
| Speed | Generally takes more time to produce results. | Provides results with very little delay. |
| Suitable For | Large historical datasets and periodic reports. | Real-time monitoring and immediate decision-making. |
Step 3: Clean Data
The collected data may contain errors, duplicate records, missing information or irrelevant entries. Therefore, data cleaning is an essential part of Big Data Analytics.
Cleaning data improves its quality, accuracy and reliability before analysis.
Common Data Cleaning Activities
- Removing duplicate records.
- Correcting incorrect or inconsistent formats.
- Removing irrelevant information.
- Handling missing or incomplete data.
- Correcting errors in the dataset.
Poor-quality data can lead to incorrect analysis and misleading conclusions. Therefore, data should be cleaned before meaningful insights are generated.
Suppose a school database contains the same student's examination record twice. If both records are included during analysis, the calculated results may be inaccurate.
Removing the duplicate record is an example of data cleaning.
Step 4: Analyze Data
Once data has been gathered, processed and cleaned, it can be analyzed to discover meaningful patterns, trends and insights.
Big Data Analytics uses specialised software and technologies to convert large amounts of usable data into big insights.
Examples of Tools Used in Big Data Analytics
| Tool | Purpose / Use |
|---|---|
| Tableau | Used for data visualization and analytical dashboards. |
| Apache Hadoop | Used for storing and processing large datasets in a distributed environment. |
| Cassandra | A database technology designed to handle large-scale distributed data. |
| MongoDB | A database system commonly used for storing flexible, document-oriented data. |
Complete Big Data Analytics Workflow
Gather Data
│
▼
Process Data
│
▼
Clean Data
│
▼
Analyze Data
│
▼
Meaningful Insights
The complete workflow transforms raw data into useful information that can support better decisions and problem-solving.
Using Orange Data Mining for Big Data Analytics
Orange Data Mining can also be used as a visual environment for performing several data analysis tasks. Different widgets can be connected to create a workflow.
| Activity | Orange Widget | Purpose |
|---|---|---|
| Gather Data | File | Loads a dataset into the workflow. |
| Process Data | Preprocess | Performs data preprocessing and transformation. |
| Clean Data | Impute | Helps handle missing values in a dataset. |
| Analyze Data | K-Means, Logistic Regression, Scatter Plot | Supports clustering, prediction and visualization. |
Activity
Consider a school dataset containing student attendance, marks and participation information.
- Identify possible sources from which the data can be gathered.
- Decide whether Batch Processing or Stream Processing would be more suitable for preparing a term-end report.
- Identify two possible data-cleaning activities.
- Suggest two useful insights that could be obtained by analyzing the data.
Competency-Based Question
A school collects attendance data from its ERP system, examination marks from the examination module and participation data from various digital learning platforms. The school wants to identify students who may require additional academic support.
Explain how the four stages of Big Data Analytics can help the school achieve this objective.
Click to View Answer
First, the school gathers attendance, examination and participation data from different sources. The data is then processed and organised for analysis. During the cleaning stage, duplicate, incomplete or incorrect records are identified and corrected. Finally, the cleaned data is analyzed to identify patterns such as low attendance, declining marks or reduced participation. These insights can help the school identify students who may need additional academic support.
Think Like an AI Engineer
An online learning platform receives thousands of student activities every minute. It wants to identify unusual changes in student activity as soon as they occur.
Which type of data processing would be more appropriate: Batch Processing or Stream Processing? Explain your answer.
Click to View Answer
Stream Processing would be more appropriate because the platform needs to process student activity immediately as it is generated. This allows the system to identify changes or unusual patterns with very little delay.
Common Beginner Mistakes
- Confusing Batch Processing with Stream Processing.
- Assuming that data analysis can be performed reliably without cleaning the data first.
- Thinking that gathering data and analyzing data are the same step.
- Forgetting that data may come from multiple sources and formats.
- Assuming that every Big Data problem requires real-time processing.
- Confusing data processing with data analysis.
- Treating data cleaning as an optional activity.
Quick Revision
- Big Data Analytics converts large datasets into useful insights.
- The four major steps are Gather → Process → Clean → Analyze.
- Gather Data means collecting data from different sources.
- Batch Processing handles large blocks of data together.
- Stream Processing handles continuous data immediately as it is produced.
- Clean Data means removing errors, duplicates and irrelevant information.
- Analyze Data means discovering patterns, trends and useful insights.
- Tools used in Big Data Analytics include Tableau, Apache Hadoop, Cassandra and MongoDB.
Memory Trick
G → P → C → A
Remember:
- G → Gather Data
- P → Process Data
- C → Clean Data
- A → Analyze Data
Easy sentence: Gather, Process, Clean, Analyze.
Exam Tips
- Memorize the four major steps of Big Data Analytics in the correct order.
- Clearly differentiate between Batch Processing and Stream Processing.
- Remember that data cleaning improves the accuracy and reliability of analysis.
- Be prepared to list the four major steps: Gather, Process, Clean and Analyze.
- For competency-based questions, connect each stage with a practical real-world example.
- Remember examples of Big Data tools such as Tableau, Apache Hadoop, Cassandra and MongoDB.
Frequently Asked Questions (FAQs)
1. What are the four major steps involved in Big Data Analytics?
The four major steps are Gather Data, Process Data, Clean Data and Analyze Data.
2. What is Batch Processing?
Batch Processing processes large blocks of data together over a specific period of time.
3. What is Stream Processing?
Stream Processing processes continuous streams of data immediately as they are produced, making it suitable for real-time analysis.
4. Why is data cleaning important?
Data cleaning removes errors, duplicates, irrelevant information and inconsistencies so that the resulting analysis is more accurate and reliable.
5. What is the difference between data processing and data analysis?
Data Processing organises and prepares data for analysis, whereas Data Analysis examines the prepared data to discover patterns, trends and useful insights.
6. Name some tools used in Big Data Analytics.
Examples include Tableau, Apache Hadoop, Cassandra and MongoDB.
7. Which processing method is suitable for real-time data?
Stream Processing is suitable for real-time data because it processes information as it is generated.
8. Which processing method is suitable for periodic reports?
Batch Processing is suitable when large volumes of accumulated data need to be processed together, such as preparing periodic reports.
Summary
- Big Data Analytics is a systematic process of converting large datasets into meaningful insights.
- The process begins by gathering data from different sources.
- The gathered data is then processed using approaches such as Batch Processing or Stream Processing.
- Data is cleaned to remove errors, duplicates, irrelevant information and inconsistencies.
- Finally, the cleaned data is analyzed to discover patterns, trends and useful insights.
- The four-step workflow can be remembered as Gather → Process → Clean → Analyze.
- Big Data Analytics can support better decision-making in education, business, healthcare, industry and many other domains.
Next Topic: Mining Data Streams