Concepts on Advanced Beginner

Architecture

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

Spark consists of Driver (coordinator) + Executor (workers) + Cluster Manager (resource management)

All Transformations are represented as DAG, split into Job → Stage → Task on Action call

Memory uses Unified Memory model where Execution (computation) and Storage (cache) dynamically share

For Java/Spring developers, SparkSession is like Spring Container, Executor is like Thread Pool Worker

Target Audience: Developers with Java/Spring backend development experience

Prerequisites:

Java basic syntax and JVM memory structure understanding
Basic multithreading concepts (Thread, ExecutorService)
Basic distributed systems concepts (optional)

Understand how Spark applications run in a distributed environment. We’ll explain using concepts familiar to Java/Spring developers.

RDD Basics

d8lzz1gpw@mozmail.com (kimbenji) — Sat, 10 Jan 2026 00:00:00 +0000

TL;DR

RDD (Resilient Distributed Dataset) is a distributed immutable data collection, Spark’s basic data abstraction

Data processing through Transformation (lazy evaluation) and Action (immediate execution)

Automatic recovery on failure through Lineage

DataFrame/Dataset recommended now, but RDD useful when low-level control needed

Target Audience: Java/Spring developers, beginners learning Spark basics

Prerequisites:

Java Collections API (List, Map, etc.)
Lambda expressions and functional programming basics
Understanding of Architecture document

RDD is Spark’s most fundamental data abstraction. As the low-level API underlying DataFrame and Dataset, it’s essential for understanding how Spark works.

DataFrame and Dataset

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

DataFrame and Dataset#

DataFrame and Dataset are Spark’s modern high-level APIs. They are easier to use than RDDs and provide automatic optimization through the Catalyst Optimizer.

Concept Overview#

DataFrame#

A DataFrame is a distributed data collection organized into named columns. It’s similar to a relational database table or a DataFrame in Python/R.

// DataFrame is an alias for Dataset<Row>
Dataset<Row> df = spark.read().json("employees.json");

Dataset#

A Dataset is a distributed data collection with a specific type. It provides compile-time type safety.

Spark SQL

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Spark SQL#

Spark SQL is a Spark module for structured data processing. It supports both SQL queries and the DataFrame API, using the same execution engine (Catalyst Optimizer).

Benefits of Spark SQL#

Familiar SQL syntax: Leverage existing SQL knowledge directly
Optimization: Catalyst Optimizer automatically optimizes queries
Various data sources: Integration with JDBC, Parquet, JSON, Hive, etc.
DataFrame interoperability: SQL results returned as DataFrames

Deep Dive into Catalyst Optimizer#

Catalyst is Spark SQL’s query optimization engine. It analyzes user queries and optimizes them to generate efficient execution plans.

Transformations and Actions

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Transformations and Actions#

All Spark operations are classified into two categories: Transformations and Actions. Understanding this distinction is fundamental to Spark programming.

Core Concepts#

Transformation#

Transformations are operations that create new RDDs/DataFrames from existing ones.

Lazy Evaluation: Not executed immediately when called
Immutability: Creates new data without modifying original
DAG Construction: Adds operations to the execution plan (DAG)

// These lines don't execute - they just add to the DAG
Dataset<Row> df = spark.read().csv("data.csv");
Dataset<Row> filtered = df.filter(col("age").gt(30)); // Transformation
Dataset<Row> selected = filtered.select("name", "age"); // Transformation

// Nothing has executed yet!

Action#

Actions are operations that trigger actual computation and return results.

Partitioning and Shuffle

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Partitioning and Shuffle#

Partitioning is key to Spark performance. Understanding how data is distributed and optimizing it is crucial for large-scale data processing.

What is a Partition?#

A Partition is a logical division unit of RDD/DataFrame data. Each partition is processed on one node in the cluster.

DataFrame (100 million rows)
├── Partition 0 (10 million rows) → Executor 1, Task 0
├── Partition 1 (10 million rows) → Executor 2, Task 1
├── Partition 2 (10 million rows) → Executor 1, Task 2
├── ...
└── Partition 9 (10 million rows) → Executor 4, Task 9

Why Partitions Matter#

Partition Count	Impact
Too few	Low parallelism, possible memory issues, underutilized nodes
Too many	Scheduling overhead, small task inefficiency, increased shuffle cost
Optimal	Balanced load distribution, efficient resource usage

Checking and Adjusting Partition Count#

Checking Current Partition Count#

Dataset<Row> df = spark.read().parquet("data.parquet");
int partitions = df.rdd().getNumPartitions();
System.out.println("Partition count: " + partitions);

// Count data per partition
df.mapPartitions(
 iter -> {
 int count = 0;
 while (iter.hasNext()) { iter.next(); count++; }
 return Collections.singletonList(count).iterator();
 },
 Encoders.INT()
).collectAsList().forEach(System.out::println);

Guidelines for Determining Partition Count#

Recommended partition size: 100MB ~ 200MB
Minimum partition count: Total cluster cores × 2-4
Maximum partition count: Data size (GB) × 2-4

Example:

Caching and Persistence

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Caching and Persistence#

Learn how to leverage Spark’s in-memory computing capabilities to cache intermediate results for reuse.

What is Caching?#

Caching is storing RDD/DataFrame in memory or disk for reuse in subsequent operations.

Why Caching is Needed#

// Without caching: Inefficient
Dataset<Row> processed = df
 .filter(col("status").equalTo("ACTIVE"))
 .groupBy("category")
 .agg(sum("amount").alias("total"));

// Same data processed 3 times (full recomputation each time)
long count = processed.count(); // Job 1: Full computation
processed.show(); // Job 2: Full recomputation
processed.write().parquet("output"); // Job 3: Full recomputation

// With caching: Efficient
processed.cache(); // Register cache
long count = processed.count(); // Job 1: Compute + cache
processed.show(); // Read from cache (fast)
processed.write().parquet("output"); // Read from cache (fast)
processed.unpersist(); // Release cache

Basic Usage#

cache()#

Dataset<Row> df = spark.read().parquet("large-data.parquet");

// cache() call - Cache with default storage level
df.cache();

// Actual caching happens on first Action
df.count(); // Data processing + cache to memory

// Subsequent Actions read from cache
df.filter(col("x").gt(10)).count();
df.groupBy("category").count().show();

// Release cache
df.unpersist();

persist()#

Allows specifying storage level directly.

Structured Streaming

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Structured Streaming#

Structured Streaming is Spark’s stream processing engine. It processes real-time data using the same DataFrame/Dataset API as batch processing.

What is Structured Streaming?#

Structured Streaming treats stream data as an infinitely appending table. When new data arrives, it performs incremental processing.

Core Concepts#

Input Stream (Infinite Table)
 ↓
[Data Arrival] → [Incremental Processing] → [Result Update]
 ↓
Output (Continuously Updated Result Table)

Batch vs Streaming Code Comparison#

// Batch processing (static data)
Dataset<Row> batchDf = spark.read()
 .json("input/data.json");

Dataset<Row> result = batchDf
 .groupBy("category")
 .count();

result.write().parquet("output");

// Stream processing (dynamic data) - almost identical!
Dataset<Row> streamDf = spark.readStream() // Changed to readStream
 .json("input/");

Dataset<Row> result = streamDf // Same processing logic
 .groupBy("category")
 .count();

result.writeStream() // Changed to writeStream
 .outputMode("complete")
 .format("console")
 .start()
 .awaitTermination();

Basic Usage#

Reading Streams#

SparkSession spark = SparkSession.builder()
 .appName("Structured Streaming")
 .master("local[*]")
 .getOrCreate();

// File source (directory monitoring)
Dataset<Row> fileStream = spark.readStream()
 .schema(schema) // Schema required for streams
 .json("input/");

// Kafka source
Dataset<Row> kafkaStream = spark.readStream()
 .format("kafka")
 .option("kafka.bootstrap.servers", "localhost:9092")
 .option("subscribe", "topic-name")
 .load();

// Socket source (for testing)
Dataset<Row> socketStream = spark.readStream()
 .format("socket")
 .option("host", "localhost")
 .option("port", 9999)
 .load();

// Rate source (for testing - generates specified rows per second)
Dataset<Row> rateStream = spark.readStream()
 .format("rate")
 .option("rowsPerSecond", 10)
 .load();

Processing Streams#

Apply the same operations as regular DataFrames to stream DataFrames:

MLlib

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

MLlib (Machine Learning Library)#

MLlib is Spark’s distributed machine learning library. It enables training and prediction of machine learning models on large-scale datasets.

MLlib Overview#

Two APIs#

API	Package	Data Type	Status
spark.ml	`org.apache.spark.ml`	DataFrame	Currently recommended
spark.mllib	`org.apache.spark.mllib`	RDD	Maintenance mode

This guide covers the spark.ml (DataFrame-based) API.

Key Components#

Transformer: Data transformation (transform without fit)
Estimator: Trainable model (fit creates Transformer)
Pipeline: Connect multiple stages
Evaluator: Evaluate model performance
CrossValidator/TrainValidationSplit: Hyperparameter tuning

Basic Workflow#

import org.apache.spark.ml.classification.LogisticRegression;
import org.apache.spark.ml.classification.LogisticRegressionModel;
import org.apache.spark.ml.feature.VectorAssembler;

SparkSession spark = SparkSession.builder()
 .appName("MLlib Example")
 .master("local[*]")
 .getOrCreate();

// 1. Load data
Dataset<Row> data = spark.read()
 .option("header", "true")
 .option("inferSchema", "true")
 .csv("data.csv");

// 2. Create feature vector
VectorAssembler assembler = new VectorAssembler()
 .setInputCols(new String[]{"feature1", "feature2", "feature3"})
 .setOutputCol("features");

Dataset<Row> assembled = assembler.transform(data);

// 3. Train/test split
Dataset<Row>[] splits = assembled.randomSplit(new double[]{0.8, 0.2}, 42);
Dataset<Row> training = splits[0];
Dataset<Row> test = splits[1];

// 4. Train model
LogisticRegression lr = new LogisticRegression()
 .setFeaturesCol("features")
 .setLabelCol("label")
 .setMaxIter(100)
 .setRegParam(0.1);

LogisticRegressionModel model = lr.fit(training);

// 5. Predict
Dataset<Row> predictions = model.transform(test);
predictions.select("label", "prediction", "probability").show(10);

// 6. Save model
model.write().overwrite().save("models/logistic-regression");

// 7. Load model
LogisticRegressionModel loadedModel = LogisticRegressionModel.load("models/logistic-regression");

Feature Transformation#

VectorAssembler#

Combine multiple columns into a single feature vector:

Performance Tuning

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Performance Tuning#

Learn strategies and specific configuration methods for optimizing Spark application performance.

Tuning Principles#

Tuning Order#

Code optimization — Improve algorithms and logic
Data structure optimization — Partitioning, schema optimization
Resource configuration — Memory, CPU, partition count
Detailed settings — Spark configuration parameters

Measurement Tools#

Spark UI: Job, Stage, Task analysis
Event Log: Detailed execution history
Metrics: Executor metric monitoring

Code-Level Optimization#

1. Minimize Shuffle#

// Bad: Unnecessary shuffles
Dataset<Row> result = df
 .repartition(100) // Shuffle
 .groupBy("key") // Shuffle
 .agg(sum("value"))
 .orderBy("key"); // Shuffle

// Good: Only necessary shuffles
Dataset<Row> result = df
 .groupBy("key")
 .agg(sum("value"));
// Only add orderBy when truly needed

2. Early Filtering#

// Bad: Filter after join
Dataset<Row> joined = large.join(small, "key")
 .filter(col("status").equalTo("ACTIVE"));

// Good: Filter before join
Dataset<Row> filtered = large.filter(col("status").equalTo("ACTIVE"));
Dataset<Row> joined = filtered.join(small, "key");

3. Column Selection#

// Bad: Keep all columns
Dataset<Row> result = df.join(other, "id")
 .groupBy("category")
 .agg(sum("value"));

// Good: Select only needed columns
Dataset<Row> result = df.select("id", "category", "value")
 .join(other.select("id"), "id")
 .groupBy("category")
 .agg(sum("value"));

4. Broadcast Join#

import static org.apache.spark.sql.functions.broadcast;

// Broadcast small tables (tens of MB)
Dataset<Row> result = largeTable.join(
 broadcast(smallTable),
 "key"
);

// Set auto broadcast threshold
spark.conf().set("spark.sql.autoBroadcastJoinThreshold", "100MB");

5. Leverage Caching#

// Cache repeatedly used data
Dataset<Row> processed = df.filter(...).groupBy(...).agg(...);
processed.cache();

// Use multiple times
process1(processed);
process2(processed);
process3(processed);

processed.unpersist();

Data Structure Optimization#

1. File Formats#

Format	Compression	Column Pruning	Use Case
Parquet	High	Supported	Analytical queries (recommended)
ORC	High	Supported	Hive compatibility
Avro	Medium	Not supported	Streaming, schema evolution
JSON	None	Not supported	When compatibility needed
CSV	None	Not supported	Simple data exchange

// Save as Parquet (recommended)
df.write()
 .mode("overwrite")
 .option("compression", "snappy") // snappy, gzip, zstd
 .parquet("output/data");

2. Partitioning Strategy#

// Partition matching query patterns
df.write()
 .partitionBy("year", "month")
 .parquet("output/data");

// Partition pruning on query
Dataset<Row> jan2024 = spark.read()
 .parquet("output/data")
 .filter(col("year").equalTo(2024).and(col("month").equalTo(1)));
// Only reads year=2024/month=1 partition

3. Bucketing#

// Bucket by join key
df.write()
 .bucketBy(100, "user_id")
 .sortBy("timestamp")
 .saveAsTable("events");

// Join tables with same bucketing has no shuffle

4. Small File Problem Resolution#

// Adjust partitions before writing
df.coalesce(10) // Merge into 10 files
 .write()
 .parquet("output");

// Or specify max records per file
df.write()
 .option("maxRecordsPerFile", 1000000)
 .parquet("output");

Resource Configuration#

Memory Structure#

Executor Memory (spark.executor.memory)
├── Execution Memory: Shuffle, join, sort
├── Storage Memory: Cache, broadcast
└── User Memory: UDFs, user objects

Default ratio (spark.memory.fraction = 0.6):
├── Execution + Storage: 60%
└── User: 40%

Recommended Settings#

# Executor settings
spark.executor.instances=10 # Number of Executors
spark.executor.memory=8g # Executor memory
spark.executor.cores=4 # Cores per Executor

# Driver settings
spark.driver.memory=4g # Driver memory
spark.driver.cores=2 # Driver cores

# Parallelism
spark.default.parallelism=200 # Default RDD partition count
spark.sql.shuffle.partitions=200 # SQL shuffle partition count

Cluster Sizing Guide#

Total cores = spark.executor.instances × spark.executor.cores
Recommended partitions = Total cores × 2-4

Example: 50 Executors × 4 cores = 200 cores
→ Partition count: 400-800

Executor Memory Calculation#

spark.executor.memory = Container memory × 0.9 - Overhead

Example: 10GB container
- Overhead (10%): 1GB
- spark.executor.memory: ~8GB

Shuffle Optimization#

// Shuffle partition count (most important)
spark.conf().set("spark.sql.shuffle.partitions", "400");

// Shuffle buffer size
spark.conf().set("spark.shuffle.file.buffer", "64k");

// Shuffle compression
spark.conf().set("spark.shuffle.compress", "true");

// Shuffle spill threshold
spark.conf().set("spark.shuffle.spill.compress", "true");

AQE (Adaptive Query Execution)#

Automatic optimization in Spark 3.0+:

Deployment and Cluster Management

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Deployment and Cluster Management#

Learn how to deploy and manage Spark applications in various cluster environments.

Cluster Manager Types#

Type	Characteristics	Suitable Environment
Local	Single JVM	Development/Testing
Standalone	Spark built-in	Small dedicated clusters
YARN	Hadoop ecosystem integration	Existing Hadoop environments
Kubernetes	Container-based	Cloud native
Mesos	General-purpose scheduler	Mixed workloads

Local Mode#

Used for development and testing.

SparkSession spark = SparkSession.builder()
 .appName("Local App")
 .master("local") // Single thread
 .master("local[4]") // 4 threads
 .master("local[*]") // All cores
 .getOrCreate();

Standalone Cluster#

A simple cluster manager built into Spark.

Spark Connect

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Spark Connect (Spark 3.4+)#

Spark Connect is a new client-server architecture introduced in Spark 3.4. It enables connecting to remote Spark clusters with a thin client.

Traditional Approach vs Spark Connect#

flowchart TB
 subgraph Traditional["Traditional Approach"]
 App1[Application]
 Driver1[Driver + SparkContext]
 Exec1[Executors]
 App1 --> Driver1
 Driver1 --> Exec1
 end

 subgraph Connect["Spark Connect"]
 App2[Thin Client]
 Server[Spark Connect Server]
 Driver2[Driver]
 Exec2[Executors]
 App2 -->|gRPC| Server
 Server --> Driver2
 Driver2 --> Exec2
 end

Key Differences#

Aspect	Traditional Approach	Spark Connect
Architecture	Monolithic (Driver embedded)	Client-server separation
Client size	Hundreds of MB (full Spark)	Few MB (gRPC client)
Language support	JVM required	Language independent (gRPC)
Resource isolation	Driver resources needed on client	Resources used only on server
Upgrades	Client redeployment required	Only server upgrade needed
Stability	Job stops on client crash	Server runs independently stable

Spark Connect Advantages#

1. Lightweight Client#

<!-- Traditional approach: spark-core + spark-sql (~200MB) -->
<dependency>
 <groupId>org.apache.spark</groupId>
 <artifactId>spark-sql_2.13</artifactId>
 <version>3.5.1</version>
</dependency>

<!-- Spark Connect: lightweight client (~10MB) -->
<dependency>
 <groupId>org.apache.spark</groupId>
 <artifactId>spark-connect-client-jvm_2.13</artifactId>
 <version>3.5.1</version>
</dependency>

2. Isolated Execution Environment#

Client memory exhaustion doesn’t affect Spark jobs
Multiple clients can share the same cluster
Server continues work on client failure

3. Language Independence#

gRPC-based connectivity from various languages
Supports Python, Scala, Java, Go, etc.
More languages planned for future support

Server Setup#

Starting Spark Connect Server#

# Start standalone server
./sbin/start-connect-server.sh \
 --packages org.apache.spark:spark-connect_2.13:3.5.1 \
 --conf spark.connect.grpc.binding.port=15002

# Or start with spark-submit
spark-submit \
 --class org.apache.spark.sql.connect.service.SparkConnectServer \
 --packages org.apache.spark:spark-connect_2.13:3.5.1 \
 --conf spark.connect.grpc.binding.port=15002 \
 local:///dev/null

Running Server with Docker#

# docker-compose.yml
version: '3.8'
services:
 spark-connect:
 image: apache/spark:3.5.1
 command: >
 /opt/spark/sbin/start-connect-server.sh
 --packages org.apache.spark:spark-connect_2.13:3.5.1
 ports:
 - "15002:15002" # gRPC port
 - "4040:4040" # Spark UI
 environment:
 - SPARK_CONNECT_GRPC_BINDING_PORT=15002
 volumes:
 - ./data:/opt/spark/data

Server Configuration Options#

# spark-defaults.conf
spark.connect.grpc.binding.port=15002
spark.connect.grpc.arrow.maxBatchSize=4194304
spark.connect.grpc.maxInboundMessageSize=134217728
spark.connect.extensions.relation.classes=
spark.connect.extensions.expression.classes=
spark.connect.extensions.command.classes=

Java Client Usage#

Dependency Setup#

// build.gradle.kts
dependencies {
 // Spark Connect client (lightweight)
 implementation("org.apache.spark:spark-connect-client-jvm_2.13:3.5.1")
}

Connection and Usage#

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import static org.apache.spark.sql.functions.*;

public class SparkConnectExample {
 public static void main(String[] args) {
 // Connect to Spark Connect server
 SparkSession spark = SparkSession.builder()
 .remote("sc://localhost:15002") // Spark Connect URL
 .build();

 try {
 // Use existing DataFrame API identically
 Dataset<Row> df = spark.read()
 .option("header", "true")
 .option("inferSchema", "true")
 .csv("data/sales.csv");

 Dataset<Row> result = df
 .filter(col("amount").gt(100))
 .groupBy("category")
 .agg(
 sum("amount").alias("total"),
 count("*").alias("count")
 )
 .orderBy(col("total").desc());

 result.show();

 // Collect results
 List<Row> rows = result.collectAsList();
 for (Row row : rows) {
 System.out.println(row.getString(0) + ": " + row.getDouble(1));
 }

 } finally {
 spark.stop();
 }
 }
}

Connection URL Format#

// Basic connection
SparkSession.builder().remote("sc://hostname:port")

// Token authentication
SparkSession.builder().remote("sc://hostname:port;token=<auth_token>")

// User ID specification
SparkSession.builder().remote("sc://hostname:port;user_id=my_user")

// SSL/TLS
SparkSession.builder().remote("sc://hostname:port;use_ssl=true")

Supported Features#

Fully Supported#

Feature	Status	Notes
DataFrame API	✅ Supported	select, filter, groupBy, etc.
SQL queries	✅ Supported	spark.sql()
UDF (User-defined functions)	✅ Supported	Must be registered on server
File read/write	✅ Supported	CSV, JSON, Parquet
Aggregate functions	✅ Supported	sum, avg, count, etc.
Window functions	✅ Supported	rank, row_number, etc.
Joins	✅ Supported	inner, left, right, etc.

Limitations#

Feature	Status	Alternative
RDD API	❌ Not supported	Use DataFrame API
Streaming	⚠️ Partial support	Execute on server side
MLlib	⚠️ Partial support	Basic features only
GraphX	❌ Not supported	Use GraphFrame
Local file access	❌ Not supported	Server path or cloud storage

Spring Boot Integration#

package com.example.config;

import org.apache.spark.sql.SparkSession;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import jakarta.annotation.PreDestroy;

@Configuration
public class SparkConnectConfig {

 @Value("${spark.connect.url:sc://localhost:15002}")
 private String sparkConnectUrl;

 private SparkSession spark;

 @Bean
 @Profile("!local") // Use in non-local environments
 public SparkSession sparkSession() {
 this.spark = SparkSession.builder()
 .remote(sparkConnectUrl)
 .build();
 return spark;
 }

 @Bean
 @Profile("local") // For local development
 public SparkSession localSparkSession() {
 this.spark = SparkSession.builder()
 .appName("local-dev")
 .master("local[*]")
 .getOrCreate();
 return spark;
 }

 @PreDestroy
 public void cleanup() {
 if (spark != null) {
 spark.stop();
 }
 }
}

# application.yml
spring:
 profiles:
 active: local

---
spring:
 config:
 activate:
 on-profile: production

spark:
 connect:
 url: sc://spark-connect.internal:15002;token=${SPARK_TOKEN}

Best Practices#

1. Connection Pooling#

@Component
public class SparkConnectionPool {
 private final SparkSession spark;

 public SparkConnectionPool(SparkSession spark) {
 this.spark = spark;
 }

 // SparkSession is thread-safe so it can be shared
 public SparkSession getSession() {
 return spark;
 }
}

2. Large Result Handling#

// ❌ Collect all - memory risk
List<Row> allRows = df.collectAsList();

// ✅ Collect limited results only
List<Row> topRows = df.limit(1000).collectAsList();

// ✅ Save to file on server then download
df.write().parquet("s3://bucket/results/");

3. Error Handling#

import io.grpc.StatusRuntimeException;

try {
 Dataset<Row> result = spark.sql("SELECT * FROM table");
 result.show();
} catch (StatusRuntimeException e) {
 if (e.getStatus().getCode() == Status.Code.UNAVAILABLE) {
 logger.error("Spark Connect server connection failed");
 // Reconnection logic
 } else {
 throw e;
 }
} catch (AnalysisException e) {
 logger.error("SQL analysis error: {}", e.getMessage());
}

Migration Guide#

Migrating from Existing Code to Spark Connect#

// Before: Traditional approach
SparkSession spark = SparkSession.builder()
 .appName("MyApp")
 .master("spark://master:7077")
 .config("spark.executor.memory", "4g")
 .getOrCreate();

// After: Spark Connect
SparkSession spark = SparkSession.builder()
 .remote("sc://spark-connect:15002")
 .build();

// DataFrame API is used identically
Dataset<Row> df = spark.read().parquet("data.parquet");

Considerations#

No local file access: Use cloud storage instead of client local files
RDD API not supported: Convert to DataFrame/Dataset API
UDF server registration: User-defined functions must be pre-registered on server
Network latency: Some overhead from remote connection

Architecture - Spark cluster structure
Spring Boot Integration - Spring integration
Deployment - Cluster deployment methods

Concepts on Advanced Beginner

Architecture

RDD Basics

DataFrame and Dataset

DataFrame and Dataset#

Concept Overview#

DataFrame#

Dataset#

Spark SQL

Spark SQL#

Benefits of Spark SQL#

Deep Dive into Catalyst Optimizer#

Transformations and Actions

Transformations and Actions#

Core Concepts#

Transformation#

Action#

Partitioning and Shuffle

Partitioning and Shuffle#

What is a Partition?#

Why Partitions Matter#

Checking and Adjusting Partition Count#

Checking Current Partition Count#

Guidelines for Determining Partition Count#

Caching and Persistence

Caching and Persistence#

What is Caching?#

Why Caching is Needed#

Basic Usage#

cache()#

persist()#

Structured Streaming

Structured Streaming#

What is Structured Streaming?#

Core Concepts#

Batch vs Streaming Code Comparison#

Basic Usage#

Reading Streams#

Processing Streams#

MLlib

MLlib (Machine Learning Library)#

MLlib Overview#

Two APIs#

Key Components#

Basic Workflow#

Feature Transformation#

VectorAssembler#

Performance Tuning

Performance Tuning#

Tuning Principles#

Tuning Order#

Measurement Tools#

Code-Level Optimization#

1. Minimize Shuffle#

2. Early Filtering#

3. Column Selection#

4. Broadcast Join#

5. Leverage Caching#

Data Structure Optimization#

1. File Formats#

2. Partitioning Strategy#

3. Bucketing#

4. Small File Problem Resolution#

Resource Configuration#

Memory Structure#

Recommended Settings#

Cluster Sizing Guide#

Executor Memory Calculation#

Shuffle Optimization#

Shuffle-Related Settings#

AQE (Adaptive Query Execution)#

Deployment and Cluster Management

Deployment and Cluster Management#

Cluster Manager Types#

Local Mode#

Standalone Cluster#

Spark Connect

Spark Connect (Spark 3.4+)#

Traditional Approach vs Spark Connect#

Key Differences#