Foundations of Big Data and Python for Analytics

TL;DR

You'll understand what makes data "big" and why traditional tools break when dealing with it. You'll know which Python libraries handle different types of big data problems. You'll be able to choose the right approach for datasets that don't fit in memory.

1. The Mental Model

Big data isn't just "lots of data" — it's data that breaks your normal tools. When your laptop crashes trying to load a CSV, when your database queries time out, when your pandas DataFrame throws a memory error — that's big data. The solution isn't always "get a bigger computer" — it's often "use different tools that work smarter, not harder."

2. The Core Material

What Actually Makes Data "Big"

The classic definition uses three V's, but let's be practical. Your data is "big" when:
- Volume: It doesn't fit in your computer's RAM (usually >8-16GB for most laptops)
- Velocity: It's streaming in faster than you can process it
- Variety: It's messy, unstructured, or comes from many different sources

A fourth V matters too: Veracity — the data might be dirty, incomplete, or unreliable.

The Python Big Data Ecosystem

For datasets larger than RAM:
- Pandas + chunking: Still pandas, but reading files piece by piece
- Dask: Like pandas, but splits work across cores and machines
- Polars: Faster than pandas, uses lazy evaluation
- Vaex: Built for billion-row datasets, visualizes data without loading it all

For distributed computing:
- PySpark: Python wrapper for Apache Spark, handles massive datasets across clusters
- Ray: Modern alternative to Spark, better for machine learning workloads

For streaming data:
- Kafka-Python: Connects to Apache Kafka streams
- Apache Beam: Handles both batch and streaming data

For specific use cases:
- Modin: Drop-in pandas replacement that uses all your CPU cores
- CuDF: GPU-accelerated pandas (requires NVIDIA GPU)

Memory vs. Speed Trade-offs

Traditional pandas loads everything into RAM for speed. Big data tools make different trade-offs:

• Lazy evaluation: Tools like Dask and Polars build a plan of what to do, then execute only when you ask for results
• Columnar storage: Formats like Parquet store data by column, making analytics queries much faster
• Chunking: Process data in small pieces, combining results at the end
• Distributed processing: Split work across multiple machines

flowchart TD
    A["Data Source"] --> B["Too big for pandas?"]
    B -->|No| C["Use pandas"]
    B -->|Yes| D["Streaming or batch?"]
    D -->|Batch| E["Fits on one machine?"]
    D -->|Streaming| F["Kafka/Beam"]
    E -->|Yes| G["Dask/Polars"]
    E -->|No| H["PySpark/Ray"]

3. Worked Example

Let's say you have a 50GB CSV file of customer transactions that crashes pandas. Here's how different tools handle it:

Pandas with chunking:

import pandas as pd

# Read in 100,000 row chunks
total_revenue = 0
chunk_size = 100000

for chunk in pd.read_csv('huge_transactions.csv', chunksize=chunk_size):
    total_revenue += chunk['amount'].sum()

print(f"Total revenue: ${total_revenue:,.2f}")

Dask (looks like pandas but works distributed):

import dask.dataframe as dd

# Lazy loading - doesn't read the file yet
df = dd.read_csv('huge_transactions.csv')

# This triggers the actual computation
total_revenue = df['amount'].sum().compute()
print(f"Total revenue: ${total_revenue:,.2f}")

Polars (faster, uses lazy evaluation):

import polars as pl

# Lazy query - builds execution plan
result = (
    pl.scan_csv('huge_transactions.csv')
    .select(pl.col('amount').sum())
    .collect()  # This executes the plan
)
print(f"Total revenue: ${result[0,0]:,.2f}")

Each approach handles the same 50GB file differently:
- Pandas chunking: Processes in small pieces, uses minimal memory
- Dask: May use multiple CPU cores, can scale to multiple machines
- Polars: Optimizes the query plan, often fastest for single-machine work

4. Production Pitfalls & Best Practices

4.1 Real-World Best Practices

• Start with file formats that matter: Convert CSVs to Parquet files — they're 5-10x smaller and much faster to read
• Use lazy evaluation when possible: Build your entire analysis pipeline before triggering computation with .compute() or .collect()
• Partition your data logically: Split files by date, region, or category so you only process what you need
• Monitor memory usage: Use htop or Activity Monitor to watch RAM usage during development
• Profile before optimizing: Use %%time in Jupyter notebooks to measure what's actually slow
• Cache intermediate results: If you'll reuse a computation, save it to disk as a Parquet file
• Set chunk sizes based on your RAM: For chunking, use sizes that leave plenty of free memory (aim for 50% of available RAM)

4.2 Common Bugs & Anti-Patterns

BAD: Loading everything then filtering

# Loads entire 50GB file into memory first
df = pd.read_csv('huge_file.csv')
recent_data = df[df['date'] > '2023-01-01']

GOOD: Filter while reading

# Only loads rows that match the condition
df = dd.read_csv('huge_file.csv')
recent_data = df[df['date'] > '2023-01-01'].compute()

BAD: Using loops instead of vectorized operations

# Processes one row at a time - extremely slow
total = 0
for chunk in pd.read_csv('file.csv', chunksize=10000):
    for _, row in chunk.iterrows():
        total += row['amount'] * row['tax_rate']

GOOD: Vectorized operations on chunks

# Processes entire chunks at once
total = 0
for chunk in pd.read_csv('file.csv', chunksize=10000):
    total += (chunk['amount'] * chunk['tax_rate']).sum()

BAD: Not handling memory cleanup

results = []
for file in huge_file_list:
    df = pd.read_csv(file)
    results.append(df.sum())  # Old DataFrames stay in memory

GOOD: Explicit cleanup

results = []
for file in huge_file_list:
    df = pd.read_csv(file)
    results.append(df.sum())
    del df  # Free memory immediately

BAD: Using the wrong data types

# Uses 64-bit integers for everything
df = pd.read_csv('file.csv')  # Default dtypes waste memory

GOOD: Optimize data types

# Specify efficient data types
dtypes = {'user_id': 'int32', 'category': 'category', 'amount': 'float32'}
df = pd.read_csv('file.csv', dtype=dtypes)

4.3 When to Use What (Decision Matrix)

5. Now Try It

Download a large dataset (try the NYC Taxi data from kaggle.com — look for files >1GB). Your task is to compute the average trip distance using three different methods:

1. Pandas chunking: Read the file in 50,000-row chunks and compute a running average
2. Dask: Load the file with dd.read_csv() and compute the mean
3. File conversion: Convert the CSV to Parquet format first, then compute the average

Time each approach using %%time in a Jupyter notebook.

Success looks like: You get the same answer from all three methods, you can see the performance differences, and you understand why Parquet is faster. The Parquet conversion should make subsequent analyses much quicker.