Skip to main content

Big data

What is big data​

Big data is a term for a group of datasets so massive and sophisticated that it becomes troublesome to process using on-hand database-management tools or contemporary processing applications. Within the recent market, massive data trends to refer to the employment of user-behavior analytics, predictive analytics, or certain different advanced data-analysis methods that extract value from this new data echo system analytics.

Characteristics​

The concept of the 3Vs in big data is foundational in understanding the characteristics and challenges associated with big data. The 3Vs are:

  • Volume(size): Big data implies massive amounts of data. The size of data gets a very relevant role in determining the value out of the data, and it is also a key factor that determines whether we can judge the chunk of data as big. Hence, volume justifies one of the important attributes of big data.
  • Variety(Complexity): Big data encompasses diverse types of structured, semi-structured, and unstructured data from various sources. This diversity adds complexity to data storage, processing, and integration.
  • Velocity(speed): This refers to the increasing speed at which large amounts of data are generated, stored, and analyzed. Real-time processing is a key goal, allowing data to be processed as it is produced.
  • Other
    • Veracity(Quality): Veracity relates to the quality and accuracy of data. Big data can be noisy and uncertain, posing challenges in ensuring data accuracy and validity for meaningful analysis.
    • Valence(Connectedness): Valence represents the connectedness of data, with higher valence indicating denser connections. This affects the efficiency of data analysis.

Data Sources​

  • Social Media: Tweets, posts, comments, likes, shares, and other interactions on platforms like Facebook, Twitter, and Instagram.
  • IoT Devices: Sensor data from smart devices, wearables, and industrial equipment.
  • Transactional Data: Records from online purchases, banking transactions, and retail sales.
  • Log Data: Logs generated by servers, applications, and network devices.
  • Multimedia: Images, videos, and audio files from various sources, including surveillance systems, entertainment, and online streaming services.

Technologies​

  • Storage Solutions: Data lakes, Hadoop Distributed File System (HDFS), Amazon S3, Google Cloud Storage.
  • Processing Frameworks: Apache Hadoop, Apache Spark, Flink, Storm.
  • Databases: NoSQL databases like MongoDB, Cassandra, HBase; and NewSQL databases designed for high-performance queries on big data.
  • Data Warehouses: Amazon Redshift, Google BigQuery, Snowflake.
  • Analytics Tools: Apache Hive, Apache Pig, Elasticsearch, Kibana.

Applications​

  • Business Intelligence: Analyzing sales, customer behavior, and market trends to drive business strategy.
  • Healthcare: Personalizing patient care, predicting outbreaks, and optimizing operations.
  • Finance: Fraud detection, risk management, and personalized banking.
  • Manufacturing: Predictive maintenance, quality control, and supply chain optimization.
  • Smart Cities: Traffic management, energy optimization, and public safety.

NoSQL = Big Data?​

Using a NoSQL database in your application does not necessarily mean that your application is a big data application. While NoSQL databases are commonly used in big data applications due to their scalability and flexibility, the classification of an application as a big data application depends on several key factors beyond just the type of database used.

Why Non-Big Data applications use NoSQL?

Let's see

  • Volume:
    • Big Data: Applications handling vast amounts of data (terabytes, petabytes, or more).
    • Non-Big Data: Applications using NoSQL for reasons like flexibility or specific use cases but with smaller, manageable data volumes.
  • Variety:
    • Big Data: Applications dealing with a wide range of data types (structured, semi-structured, unstructured).
    • Non-Big Data: Applications using NoSQL may deal with specific data types but not necessarily a broad variety.
  • Velocity:
    • Big Data: Applications that require high-speed data ingestion, processing, and real-time analytics.
    • Non-Big Data: Applications using NoSQL may handle data efficiently but do not necessarily require real-time processing.

Criteria for Big Data Applications​

For an application to be truly considered a big data application, it typically exhibits most of the following characteristics:

  • Massive Data Volumes: Handling large datasets that exceed the capacity of traditional databases.
  • Diverse Data Types: Managing structured, semi-structured, and unstructured data.
  • High-Velocity Data Processing: Ingesting and processing data at high speeds, often in real-time.
  • Complex Analytics: Performing advanced analytics, machine learning, and complex queries.
  • Distributed Computing: Using frameworks like Hadoop or Spark to process data across multiple nodes.

Comparison of Data Modeling in Big Data and traditional Data​

While the core concepts of data modeling apply to both big Data and traditional data, the implementation and approach differ significantly due to the scale, complexity, and nature of the data.

  • Big Data modeling requires flexibility, scalability, and specialized tools to handle vast amounts of diverse and rapidly changing data.
  • In contrast, traditional data modeling focuses on structured data with established methods and technologies for managing smaller data volumes.

Similarities​

  1. Fundamental Principles: Both Big Data and normal data modeling rely on the same core principles, such as defining entities, relationships, and constraints.
  2. Goals: The objectives of data modeling—organizing data, ensuring data quality, facilitating data retrieval, and supporting data integration—are consistent across both environments.
  3. Processes: The basic steps of data modeling, including requirement analysis, conceptual modeling, logical modeling, physical modeling, and data governance, are applicable in both contexts.

Differences​

Scale and Complexity​

  • Big Data: Deals with massive volumes of data that can include diverse and unstructured data types. Models must handle high velocity and variety, often requiring distributed storage and processing.
  • Normal Data: Typically involves structured data with relatively smaller volumes, making it easier to manage with traditional relational databases.

Data Models and Storage​

  • Big Data:
    • Schema-on-Read: Schemas are applied to the data only when it is read, not when it is written.
    • Data Lakes: Large storage repositories that hold vast amounts of raw data in its native format.
    • NoSQL Databases: Non-relational databases (e.g., MongoDB, Cassandra) designed for scalability and flexibility.
    • Distributed Computing: Frameworks like Hadoop and Spark for processing large datasets across distributed systems.
  • Normal Data:
    • ER Diagrams: Entity-Relationship diagrams to design database schemas.
    • Normalization: Ensures data integrity and reduces redundancy.
    • SQL: Structured Query Language for querying and managing data.

Performance and Scalability​

  • Big Data: Models must be designed for horizontal scalability, allowing the system to handle increasing data volumes by adding more nodes to the cluster. Performance considerations are critical due to the sheer size and speed of data ingestion and processing.
  • Normal Data: Typically optimized for vertical scalability, where performance is enhanced by adding more resources (e.g., CPU, memory) to a single server. Performance tuning is more straightforward.

Tools and Technologies​

  • Big Data: Involves specialized tools like Hadoop (for distributed storage and processing), Spark (for fast, in-memory processing), Cassandra and HBase (for scalable NoSQL storage), and Neo4j (for graph data). Data modeling may also leverage data lake architectures.
  • Normal Data: Primarily uses relational database management systems (RDBMS) like MySQL, PostgreSQL, and SQL Server, along with traditional data modeling tools like ERwin, Microsoft Visio, and IBM InfoSphere Data Architect.

Key Considerations​

  • Schema Design
    • Big Data: Flexible schema design to accommodate evolving data structures. Schema-on-read approach allows for dynamic data modeling.
    • Normal Data: Fixed schema design requiring predefined structure. Schema-on-write approach ensures data consistency at the time of storage.
  • Data Governance
    • Big Data: More complex due to diverse data sources and formats. Requires robust policies for data quality, privacy, and security across distributed systems.
    • Normal Data: More straightforward governance with well-defined data sources and structures. Traditional data governance practices are often sufficient.
Last updated on