big data models python

A fourth V matters too: Veracity — the data might be dirty, incomplete, or unreliable.

For datasets larger than RAM:
- Pandas + chunking: Still pandas, but reading files piece by piece
- Dask: Like pandas, but splits work across cores and machines
- Polars: Faster than pandas, uses lazy evaluation
- Vaex: Built for billion-row datasets, visualizes data without loading it all

PySpark is Python's interface to Apache Spark, a distributed computing framework. Unlike pandas which runs on one machine, PySpark automatically spreads your data and computations across multiple machines.

Here's the key difference:

PySpark gives you two main ways to work with distributed data:

RDDs (Resilient Distributed Datasets) - The lower-level building blocks:

DataFrames - The high-level, SQL-like interface:

MLlib is built for distributed computing from the ground up. It operates on Spark DataFrames and RDDs, automatically distributing computation across your cluster.

MLlib's strength is seamless integration with Spark's ecosystem. You can read from HDFS, process with Spark SQL, train with MLlib, and write results back—all in one pipeline.

Scikit-learn wasn't designed for distributed computing, but you can distribute it using Dask or joblib backends.

Windows: You can't process infinite data at once, so you create time-based or count-based windows. A 5-minute tumbling window processes all events that arrived in each 5-minute period. A sliding window continuously updates as new data arrives.

Watermarks: These handle late-arriving data. If you set a watermark of 30 seconds, you'll wait that long after a window should close before finalizing results, catching stragglers.

Parquet files are typically 10-20x smaller than equivalent CSVs and read much faster when you only need some columns.

Bad: Loading entire Parquet file to get metadata

Good: Use parquet metadata

Bad: Converting large graphs to pandas then NetworkX

Good: Stream construction or use igraph

Bad: Synchronous message processing

Good: Async processing with batching