Big Data

A durable, high-throughput pipe for streaming data between systems in real time.

Like MapReduce, but it keeps data in memory instead of writing to disk between every step.

In a distributed system, you can't have perfect consistency, availability, and network reliability all at once.

Grow by buying one bigger machine, or grow by buying more ordinary ones.

Split your data across machines so no single one becomes the bottleneck.

Machines fail constantly at scale — the system has to expect that and survive it anyway.

When one computer isn't enough, split the work across many that talk to each other.

The original toolkit that made storing and processing web-scale data on cheap hardware possible.

A file system built assuming your hardware will fail — and designed not to care when it does.

Break the job into 'map' and 'reduce' steps, and let the cluster run them in parallel.

Not every dataset fits neatly into rows and columns — or needs to scale the same way.

The end-to-end path data takes from the moment it's generated to the moment it's acted on.

RDDs with a schema, SQL-like syntax, and a smart optimizer doing the heavy lifting for you.

The original, low-level building block Spark is built on — resilient, distributed, and immutable.

Spark doesn't actually do anything until you ask for a result — and that's the whole point.

Process data as it arrives, one event at a time — or wait and process it all together, later.

Not just 'a lot of data' — data too big, fast, or messy for one machine to handle alone.