How-To Guides on Advanced Beginner

Troubleshooting OutOfMemoryError

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Estimated Time: About 15 minutes

TL;DR

Driver OOM: Reduce collect() result size, increase spark.driver.memory

Executor OOM: Increase partition count (repartition), increase spark.executor.memory

Diagnose First: Check Spark UI to identify where OOM occurs

Problem Definition#

The following error occurs during Spark application execution:

java.lang.OutOfMemoryError: Java heap space

Or:

Container killed by YARN for exceeding memory limits

This guide explains step-by-step how to diagnose and resolve OOM errors.

Prerequisites#

Item	Requirement	How to Verify
Spark Version	2.4 or higher (3.x recommended)	`spark-submit --version`
Java Version	8, 11, or 17	`java -version`
Spark UI	Accessible	Open `http://localhost:4040` in browser
Permissions	Can modify Spark settings	Verify spark-submit execution permissions

Supported Environments: Linux, macOS, Windows (WSL2 recommended)

Resolving Data Skew

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Estimated Time: About 20 minutes

TL;DR

Diagnosis: Compare Task Duration Min/Max in Spark UI Stages tab (10x+ difference = skew)

Enable AQE: spark.sql.adaptive.skewJoin.enabled=true (Spark 3.0+)

Manual Fix: Distribute hot keys using Salting technique

Problem Definition#

A Spark job mostly completes quickly but some Tasks take much longer:

Stage 3: 199/200 tasks completed... (last 1 running for tens of minutes)

This is Data Skew - data is concentrated on specific keys, causing those partitions to be overloaded.

Optimizing Shuffle

d8lzz1gpw@mozmail.com (kimbenji) — Fri, 16 Jan 2026 00:00:00 +0000

Estimated Time: About 20 minutes

TL;DR

Check Shuffle: Exchange node in df.explain() = shuffle occurs

Eliminate Unnecessary Shuffles: Perform multiple aggregations at once in the same group

Broadcast Join: Use broadcast() for small tables (tens of MB)

Shuffle Partition Count: Adjust spark.sql.shuffle.partitions (default 200)

Problem Definition#

Shuffle optimization is needed when you see these symptoms:

Symptom	Where to Check
Stage transitions take 10+ seconds	Spark UI → Jobs tab
Shuffle Read/Write is tens of GB or more	Spark UI → Stages tab
Timeout due to network I/O	Application logs
Many `Exchange` nodes in `explain()`	Execution plan output

Shuffle is the most expensive operation in Spark. The following operations cause shuffle: