intermediatebig-data

Spark RDDs (Resilient Distributed Datasets)

The original, low-level building block Spark is built on — resilient, distributed, and immutable.

Key Points

Immutable — an RDD can't be changed once created; transformations produce a new RDD instead of modifying the old one.
Distributed — its data is split into partitions spread across the cluster's machines.
Resilient — if a partition is lost due to node failure, Spark can recompute it using its recorded lineage of transformations, without needing full replication.
Lower-level than DataFrames — offers more control but less automatic optimization.
Mostly superseded in day-to-day use by the higher-level, more optimized DataFrame API.

What it is

An RDD (Resilient Distributed Dataset) is Spark's original core data structure: an immutable, distributed collection of objects spread across a cluster that can be processed in parallel and automatically recovers from node failures using its lineage of transformations.

Why it exists

Early distributed systems needed fault tolerance, which usually meant replicating data across machines — expensive in storage. RDDs exist to achieve fault tolerance differently: by remembering the sequence of operations (lineage) that built the data, so any lost piece can simply be recomputed rather than restored from a redundant copy.

Intuition

Instead of keeping three physical backup copies of a document (expensive), imagine keeping a detailed, precise recipe of every edit that was ever made to produce it. If the document is destroyed, you can perfectly reconstruct it just by replaying the recipe from the original source. That recipe is the RDD's 'lineage.'

When to use

Use RDDs when you need fine-grained, low-level control over data partitioning or custom transformations that the DataFrame API doesn't support.
Understand RDDs conceptually even if you mostly use DataFrames in practice, since DataFrames are built on top of them.

When not to use

Do not default to RDDs for typical structured data processing — DataFrames offer the same fault tolerance with far better automatic performance optimization.
Do not use RDDs when SQL-like queries would express the logic more simply and efficiently via DataFrames.

Common mistakes

Using the low-level RDD API out of habit for tasks that DataFrames would handle faster with less code.
Not understanding lazy evaluation — transformations on an RDD don't execute until an action is called.

In practice

Rarely used directly in modern data science pipelines, but understanding RDDs clarifies why Spark can recover gracefully from failures and why the DataFrame API — built on the same core idea — is both fault-tolerant and fast.

Was this concept helpful?

Spot an error or gap? Suggest an edit →

Related concepts

spark-dataframesspark-transformations-actionsapache-spark-intro