Introduction to Apache Spark
Like MapReduce, but it keeps data in memory instead of writing to disk between every step.
Key Points
- Processes data in memory across a cluster, avoiding the slow disk writes between steps that MapReduce required.
- Supports batch processing, streaming, SQL queries, and machine learning, all within one unified engine.
- Works with data stored in HDFS, S3, databases, or many other sources.
- Has APIs in Python (PySpark), Scala, Java, and R, making it accessible to data scientists and engineers alike.
- Built around two core abstractions: RDDs (low-level) and DataFrames (higher-level, more optimized).
What it is
Apache Spark is a distributed data processing engine that processes large datasets across a cluster, keeping intermediate data in memory rather than writing it to disk between steps like MapReduce did — making it dramatically faster, especially for iterative and interactive workloads.
Why it exists
MapReduce's habit of writing results to disk between every step made it painfully slow for jobs that needed multiple passes over the same data — like most iterative machine learning algorithms. Spark exists to keep data in memory across steps whenever possible, delivering major speed gains, especially for workloads that reuse the same data repeatedly.
Intuition
When to use
- Use Spark as the default choice for large-scale batch data processing today, over raw MapReduce.
- Use it for iterative workloads like training machine learning models on distributed data.
- Use it when you need one unified tool for batch processing, SQL queries, and streaming.
When not to use
- Do not use Spark for small datasets that fit comfortably in memory on one machine — the cluster overhead isn't worth it.
- Do not use it for workloads needing millisecond-level real-time responses on individual events — dedicated streaming tools like Kafka Streams may fit better.
Common mistakes
- Using Spark for a small dataset out of habit, when pandas would run the same job faster with none of the cluster overhead.
- Not understanding lazy evaluation, leading to confusion about when computation actually happens.
- Running out of memory because too much data is cached without considering cluster memory limits.
In practice
The industry-standard engine for large-scale data processing and feature engineering pipelines, often used to prepare data at scale before feeding it into machine learning models, either within Spark's own MLlib or exported for use elsewhere.
Was this concept helpful?
Related concepts