UPSC Exam  >  UPSC Notes  >  Science & Technology for UPSC CSE  >  Big Data

Big Data | Science & Technology for UPSC CSE PDF Download

Introduction

Big Data simply refers to a large amount of data which is of structured, semi-structured or unstructured nature. The data pool is so voluminous that it becomes difficult for an organization to manage and process it using traditional databases and software techniques. Therefore, big data not only implies the enormous amount of available data but it also refers to the entire process of gathering, storing, and analyzing that data.

Big Data is characterized by 6Vs as described in the following figure

6 V’s of Big Data6 V’s of Big Data

Evolution of Big Data

  • In today’s world where data is the new oil, big data can hypothetically be assumed as a system that acquires crude oil and makes fuel out of it.
  • With the advent of the Internet of Things (IoT), data is now readily available. According to DOMO’s report, "Over 2.5 quintillion bytes of data are created every single day, and it’s only going to grow from there. By 2020, it is estimated that 1.7 MB of data will be created every second for every person on earth."

Significance of Big Data in India

  • With a population of around 1.3 billion, Big Data holds a significant position in the Indian context.
  • As per the study conducted by NASSCOM, “the Indian analytics industry is predicted to reach $16 billion mark by 2025.”

Applications of Big Data

  1. Governance: Big data is used in governance for the following purposes:
    •  Power Discoms use data collected from last-mile to cut down their Aggregate Technical & Commercial losses.
    • Security agencies and Police forces use big data to,
      (i) Prevent cyber-attacks
      (ii) Enhance security systems
      (iii) Detect card-related fraud cases
      (iv) Predict criminal activities, e.g – Crime and Criminal Tracking Networks and Systems (CCTNS)
    • In improving the quality of education.
    • Disaster Management: In understanding and mitigating the risks of disasters.
  2. Economy: Big data can provide huge benefits to various sectors of the economy like,
    • In Insurance Sector to improve customer experience & ensure their right to claim
    • In Banking Sector to manage financial data
    • To capture the production, price statistics, & calculate the resultant GDP
    • Evade risks & minimize losses for financial firms
    • Tax officials catching hold the tax evaders using Project Insight
    • Deregistration of lakhs of shell companies
    • In preventing money laundering & curbing terrorism financing by identifying the safe havens
  3. Health Care: Big Data in health care caters the following benefits:
    • Predicting diseases,
    • Prescribing medicines,
    • Optimizing treatment,
    • Using clinical data to improve patient care,
    • In critical Diagnostic tests,
    • Finding new cures (R&D)
  4. Agriculture and Food
    • Seed Selection
    • Geo-Tagging to keep the track record of agricultural assets in the country
    • Weather Forecasting
    • Irrigation & effective water management
    • Food Processing
    • Identification of Crop Diseases
  5. Digital Space
    • In the telecom sector- connecting the hinterland areas and bringing them to the mainstream,
    • On Social Media for targeting platform users,
    • Artificial Intelligence - Controlling home appliances,
    • Analyzing & Improving individual performance (at work, sports, or home) using wearable devices.

Government Initiatives & Interventions

  • NITI Aayog is currently working on a plan in collaboration with the private players to develop the ‘National Data & Analytics Platform’, which will act as a single source of sectoral data for citizens, policymakers, and researchers.
  • ‘Big Data Management Policy’ drafted by CAG for auditing large chunks of data generated by the public sector in the states and the union territories, is a great start.
  • Ministry of Statistics and Programme 
    Implementation has proposed to establish a ‘National Data Warehouse on Official Statistics’ which will leverage technology and use big data analytical tools to improve the quality of macro-economic aggregates.
  • Use of Direct Benefit Transfer in MGNREGA & Aadhaar for authentication & availing welfare scheme benefits helps in the filtering of ghost beneficiaries.
  • The Ministry of Agriculture has signed a deal with the ISRO to use satellites for mapping of agricultural assets.
  • Smart City Mission, Digital India, digital economy proposals like BHIM app, inter alia are important government initiatives that are using Big Data for achieving good governance in the country.

Challenges

  • Invasion of Privacy: Big Data Analytics leads to the twin problems of digitization, which are, Data privacy and Net neutrality.
  • Data Security: Several incidents of leakage of Aadhaar data underlined the need for the government to increase the security and safety of virtual data it collects from its citizens.
  • Technical Challenges: As the name suggests, Big data involves several inherent limitations like,
    (i) Inefficient infrastructure for data collection & management of such humongous data,
    (ii) Storage & computational problems,
    (iii) Scalability & streaming issues.
  • Challenges related to Governance: For effective policymaking using Big Data, the government needs to adopt a consistent dynamic approach. Constant evaluation of the generated feedback and flexibly changing the related policy structures will percolate the benefits to the lowest level.

Way Forward

  • Ethical issues related to data privacy need to get addressed.
    (i) Data protection law must incorporate some of the best practices followed in the World.
    (ii) It must strengthen cybersecurity in order to safely utilize the large pool of virtually available data.
  • Economically, the alliance of big data with blockchain technology will potentially change the way we see big data for now.
  • Infrastructural Requirement: To effectively analyze the large chunks of available data, well-equipped data centers are needed. It is essential to segregate the relevant data from the irrelevant data pool.
  • In a world where most of the governments & businesses rely more on new-age data sources such as satellite imagery, parking images, and night lights, Big Data governance could be a game-changer for India, where policies have for long been under-implemented.

Big Data is like a double-edged sword. It will be wise to handle it with utmost care and conscience.

The document Big Data | Science & Technology for UPSC CSE is a part of the UPSC Course Science & Technology for UPSC CSE.
All you need of UPSC at this link: UPSC
146 videos|358 docs|249 tests

Top Courses for UPSC

FAQs on Big Data - Science & Technology for UPSC CSE

1. What is Big Data?
Ans. Big Data refers to the large and complex sets of data that cannot be easily managed, processed, or analyzed using traditional data processing methods. It encompasses large volumes of data that include structured, semi-structured, and unstructured data from various sources.
2. How is Big Data different from traditional data processing methods?
Ans. Big Data is different from traditional data processing methods in terms of volume, velocity, and variety. Traditional methods are not equipped to handle the massive volumes of data generated in today's digital world. Additionally, Big Data is generated at a high velocity, requiring real-time or near-real-time analysis. Furthermore, Big Data includes a variety of data types, such as text, images, videos, and sensor data, which traditional methods struggle to handle.
3. What are the main challenges in processing Big Data?
Ans. Processing Big Data poses several challenges, including data storage, data integration, data quality, and data analysis. Storing large volumes of data requires scalable and cost-effective storage solutions. Integrating data from various sources with different formats and structures can be complex. Ensuring data quality and reliability is crucial for accurate analysis. Finally, analyzing Big Data requires advanced algorithms, tools, and technologies to extract valuable insights.
4. What are the benefits of analyzing Big Data?
Ans. Analyzing Big Data can provide several benefits, including improved decision-making, enhanced operational efficiency, better customer insights, and the ability to identify new business opportunities. By analyzing large datasets, organizations can uncover patterns, trends, and correlations that can help them make data-driven decisions. It also allows them to optimize their processes, identify bottlenecks, and improve overall efficiency. Understanding customer behavior and preferences can lead to personalized experiences and targeted marketing strategies. Finally, analyzing Big Data can help organizations identify emerging trends, market gaps, and potential areas for innovation.
5. What are some popular tools and technologies used for Big Data processing?
Ans. There are several popular tools and technologies used for Big Data processing, such as Apache Hadoop, Apache Spark, and NoSQL databases. Apache Hadoop is a widely used open-source framework that allows distributed processing of large datasets across clusters of computers. Apache Spark is another open-source framework that provides fast and flexible data processing capabilities. NoSQL databases, such as MongoDB and Cassandra, are designed to handle large volumes of unstructured and semi-structured data. These tools and technologies enable efficient storage, processing, and analysis of Big Data.
146 videos|358 docs|249 tests
Download as PDF
Explore Courses for UPSC exam

Top Courses for UPSC

Signup for Free!
Signup to see your scores go up within 7 days! Learn & Practice with 1000+ FREE Notes, Videos & Tests.
10M+ students study on EduRev
Related Searches

Summary

,

MCQs

,

Exam

,

mock tests for examination

,

pdf

,

video lectures

,

shortcuts and tricks

,

ppt

,

Big Data | Science & Technology for UPSC CSE

,

Semester Notes

,

practice quizzes

,

Sample Paper

,

Important questions

,

past year papers

,

Free

,

Big Data | Science & Technology for UPSC CSE

,

study material

,

Viva Questions

,

Previous Year Questions with Solutions

,

Big Data | Science & Technology for UPSC CSE

,

Objective type Questions

,

Extra Questions

;