Foundations of Big Data and Python for Analytics

TL;DR

You'll understand what makes data "big" and why traditional tools break when dealing with it. You'll know which Python libraries handle different types of big data problems. You'll be able to choose the right approach for datasets that don't fit in memory.

1. The Mental Model

Big data isn't just "lots of data" — it's data that breaks your normal tools. When your laptop crashes trying to load a CSV, when your database queries time out, when your pandas DataFrame throws a memory error — that's big data. The solution isn't always "get a bigger computer" — it's often "use different tools that work smarter, not harder."

2. The Core Material

What Actually Makes Data "Big"

The classic definition uses three V's, but let's be practical. Your data is "big" when:
- Volume: It doesn't fit in your computer's RAM (usually >8-16GB for most laptops)
- Velocity: It's streaming in faster than you can process it
- Variety: It's messy, unstructured, or comes from many different sources

A fourth V matters too: Veracity — the data might be dirty, incomplete, or unreliable.

The Python Big Data Ecosystem

For datasets larger than RAM:
- Pandas + chunking: Still pandas, but reading files piece by piece
- Dask: Like pandas, but splits work across cores and machines
- Polars: Faster than pandas, uses lazy evaluation
- Vaex: Built for billion-row datasets, visualizes data without loading it all

For distributed computing:
- PySpark: Python wrapper for Apache Spark, handles massive datasets across clusters
- Ray: Modern alternative to Spark, better for machine learning workloads

For streaming data:
- Kafka-Python: Connects to Apache Kafka streams
- Apache Beam: Handles both batch and streaming data

For specific use cases:
- Modin: Drop-in pandas replacement that uses all your CPU cores
- CuDF: GPU-accelerated pandas (requires NVIDIA GPU)

Memory vs. Speed Trade-offs

Traditional pandas loads everything into RAM for speed. Big data tools make different trade-offs:

• Lazy evaluation: Tools like Dask and Polars build a plan of what to do, then execute only when you ask for results
• Columnar storage: Formats like Parquet store data by column, making analytics queries much faster
• Chunking: Process data in small pieces, combining results at the end
• Distributed processing: Split work across multiple machines

```me

Distributed Data Processing with PySpark

TL;DR

You'll learn how PySpark splits big datasets across multiple machines to process them in parallel. You'll understand RDDs and DataFrames as the two main data structures for distributed computing. You'll write real PySpark code that can scale from your laptop to a 100-node cluster.

1. The Mental Model

When your data gets too big for one machine, PySpark breaks it into chunks and sends those chunks to different computers. Each computer processes its chunk independently, then PySpark combines the results back together. That's the whole idea.

2. The Core Material

What Makes PySpark Different

PySpark is Python's interface to Apache Spark, a distributed computing framework. Unlike pandas which runs on one machine, PySpark automatically spreads your data and computations across multiple machines.

Here's the key difference:

import pandas as pd
# This runs on ONE machine
df_pandas = pd.read_csv("huge_file.csv")  # Might crash if file is too big

from pyspark.sql import SparkSession
# This can run across MANY machines
spark = SparkSession.builder.appName("MyApp").getOrCreate()
df_spark = spark.read.csv("huge_file.csv", header=True)  # Scales automatically

Core Data Structures

PySpark gives you two main ways to work with distributed data:

RDDs (Resilient Distributed Datasets) - The lower-level building blocks:

from pyspark import SparkContext
sc = SparkContext()

# Create an RDD from a list
numbers = sc.parallelize([1, 2, 3, 4, 5, 6, 7, 8])

# Operations are lazy - nothing happens yet
squared = numbers.map(lambda x: x ** 2)
filtered = squared.filter(lambda x: x > 10)

# Action triggers computation across cluster
result = filtered.collect()  # [16, 25, 36, 49, 64]

DataFrames - The high-level, SQL-like interface:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, avg, count

spark = SparkSession.builder.appName("Analysis").getOrCreate()

# Read data (automatically distributed)
df = spark.read.option("header", True).csv("sales_data.csv")

# Operations look like pandas but run distributed
result = df.groupBy("region") \
          .agg(avg("sales").alias("avg_sales"), 
               count("*").alias("total_orders")) \
          .orderBy("avg_sales", ascending=False)

# Show results
result.show()

Lazy Evaluation and Actions

PySpark uses lazy evaluation - it builds up a plan of what to do but doesn't actually do it unt

Big Data Machine Learning with MLlib and scikit-learn on Distributed Data

TL;DR

You'll learn how to train machine learning models on datasets too big for one machine using Spark MLlib and distributed scikit-learn. You'll understand when to use each framework and how to handle the unique challenges of distributed ML. By the end, you'll be able to build and deploy scalable ML pipelines that work across clusters.

1. The Mental Model

When your dataset doesn't fit in memory on one machine, you need distributed machine learning. MLlib runs natively on Spark clusters and handles partitioning automatically, while scikit-learn can be distributed using libraries like Dask or joblib. The key insight: some algorithms distribute naturally (like linear regression), others don't (like k-means clustering). That's the whole idea.

2. The Core Material

MLlib: Spark's Native ML Library

MLlib is built for distributed computing from the ground up. It operates on Spark DataFrames and RDDs, automatically distributing computation across your cluster.

from pyspark.sql import SparkSession
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.regression import LinearRegression
from pyspark.ml.evaluation import RegressionEvaluator

# Initialize Spark
spark = SparkSession.builder.appName("MLlibExample").getOrCreate()

# Load distributed data
df = spark.read.parquet("hdfs://large_dataset.parquet")

# Feature engineering on distributed data
assembler = VectorAssembler(inputCols=["feature1", "feature2", "feature3"], 
                           outputCol="features")
feature_df = assembler.transform(df)

# Train model - computation happens across cluster
lr = LinearRegression(featuresCol="features", labelCol="target")
model = lr.fit(feature_df)

MLlib's strength is seamless integration with Spark's ecosystem. You can read from HDFS, process with Spark SQL, train with MLlib, and write results back—all in one pipeline.

Scikit-learn with Distributed Computing

Scikit-learn wasn't designed for distributed computing, but you can distribute it using Dask or joblib backends.

```python
import dask.dataframe as dd
from dask_ml.model_selection import train_test_split
from dask_ml.linear_model import LinearRegression
from sklearn.linear_model import SGDRegressor
from joblib import parallel_backend

Option 1: Dask-ML (scikit-learn compatible)

df = dd.read_parquet("large_dataset.parquet")
X = df[["feature1", "feature2", "feature3"]]
y = df["target"]

X

Real-time Data Processing and Stream Analytics

TL;DR

You'll learn how to process continuous data streams as they arrive, rather than waiting for batches. You'll build real-time analytics pipelines that can handle millions of events per second. You'll understand when to use streaming vs batch processing and how to implement both approaches in Python.

1. The Mental Model

Traditional data processing is like doing laundry - you collect a full load, then wash it all at once. Stream processing is like a dishwasher that cleans plates as soon as you put them in. Instead of waiting for data to pile up, you process each piece the moment it arrives. That's the whole idea.

2. The Core Material

What Makes Streaming Different

Streaming systems process unbounded data - there's no "end" to the dataset. Each record has a timestamp, and you're always working with windows of recent data. The key challenge isn't just speed; it's handling late arrivals, out-of-order events, and maintaining state across millions of records.

Core Streaming Concepts

Windows: You can't process infinite data at once, so you create time-based or count-based windows. A 5-minute tumbling window processes all events that arrived in each 5-minute period. A sliding window continuously updates as new data arrives.

Watermarks: These handle late-arriving data. If you set a watermark of 30 seconds, you'll wait that long after a window should close before finalizing results, catching stragglers.

State Management: Unlike batch jobs that start fresh, streaming apps maintain state between records. You might track running averages, user sessions, or fraud detection scores that update with each new event.

Python Streaming Libraries

Apache Kafka with kafka-python: The most common setup for high-throughput streaming.

from kafka import KafkaConsumer, KafkaProducer
import json
from datetime import datetime

# Producer - sends events to stream
producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda x: json.dumps(x).encode('utf-8')
)

# Send events
for i in range(100):
    event = {
        'user_id': f'user_{i}',
        'timestamp': datetime.now().isoformat(),
        'action': 'click',
        'value': i * 10
    }
    producer.send('user_events', value=event)

producer.flush()

Stream Processing with Faust: A Python-native stream processing library.

```python
import faust
from datetime import timedelta

app =

Big Data Storage, Graph Processing, and Advanced Topics

TL;DR

You'll learn how to store massive datasets efficiently using columnar formats like Parquet, process graph data with NetworkX and distributed systems, and handle advanced big data challenges like streaming and machine learning at scale. These tools let you work with datasets that don't fit in memory and solve complex network problems. By the end, you'll know when to use each approach and how to avoid common performance traps.

1. The Mental Model

Big data isn't just "lots of data" — it's data that breaks your normal tools. Storage becomes about compression and column-oriented access patterns instead of rows. Graph processing means thinking in nodes and edges, not tables. That's the whole idea.

2. The Core Material

2.1 Columnar Storage with Parquet

Traditional CSV files store data row by row, but Parquet stores it column by column. This matters because most analytics queries only need a few columns from wide tables.

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
import numpy as np

# Create sample data
data = {
    'user_id': np.arange(1000000),
    'timestamp': pd.date_range('2023-01-01', periods=1000000, freq='1min'),
    'revenue': np.random.exponential(10, 1000000),
    'category': np.random.choice(['A', 'B', 'C'], 1000000)
}
df = pd.DataFrame(data)

# Write to Parquet with compression
df.to_parquet('sales_data.parquet', compression='snappy', index=False)

# Read only specific columns (this is where Parquet shines)
revenue_data = pd.read_parquet('sales_data.parquet', columns=['revenue', 'category'])

Parquet files are typically 10-20x smaller than equivalent CSVs and read much faster when you only need some columns.

2.2 Graph Data Structures and NetworkX

Graphs represent relationships: social networks, web links, transportation routes. NetworkX makes this intuitive in Python.

```python
import networkx as nx
import matplotlib.pyplot as plt

Create a directed graph

G = nx.DiGraph()

Add nodes and edges

G.add_nodes_from([1, 2, 3, 4, 5])
G.add_edges_from([(1, 2), (2, 3), (3, 4), (4, 5), (5, 1), (2, 4)])

Basic graph metrics

print(f"Nodes: {G.number_of_nodes()}")
print(f"Edges: {G.number_of_edges()}")
print(f"Is connected: {nx.is_weakly_connected(G)}")

Find shortest path

shortest = nx.shortest_path(G, source=1, target=4)
print(f"Shortest path from 1 to 4: {shortest}")

Calculate centrality measures

betweenness = nx.betweenness