Appendix on Advanced Beginner

Glossary

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

Glossary#

Key terminology and concepts used in Spark. Each term links to related documentation.

Core Concepts#

Action#

Operations that trigger actual computation on RDD/DataFrame and return results. Examples include count(), collect(), show(), write(). → Transformations and Actions

Application#

A Spark program submitted by the user. Consists of a Driver and Executors. → Architecture

Broadcast Variable#

A shared variable distributed to all nodes as read-only. Used for efficiently sharing small datasets. → Performance Tuning

Catalyst Optimizer#

Spark SQL’s query optimization engine. Transforms logical plans into optimized physical plans. → Spark SQL

FAQ

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

FAQ#

Frequently asked questions and solutions to common problems.

General Questions#

Which Java versions does Spark support?#

Spark 3.5 supports Java 8, 11, and 17. Java 21 is not officially supported yet.

java -version # Check version

Can I use Spark with Java only, without Scala?#

Yes, you can. Spark fully supports the Java API. However, since the Spark runtime is written in Scala, Scala libraries are included in the dependencies.

What’s the difference between DataFrame and Dataset?#

DataFrame (Dataset<Row>): Has schema but no compile-time type checking
Dataset (Dataset<T>): Uses POJO types to provide compile-time type safety

In Java, DataFrame is an alias for Dataset<Row>.

References

d8lzz1gpw@mozmail.com (kimbenji) — Wed, 07 Jan 2026 00:00:00 +0000

References#

Official documentation and additional resources for learning Apache Spark.

Official Documentation#

Apache Spark Official Site#

Spark Official Site — Downloads, news, release information
Spark 3.5 Documentation — Current stable version documentation
Spark Latest Documentation — Latest version documentation

Programming Guides#

RDD Programming Guide — Detailed RDD API explanation
Spark SQL, DataFrames and Datasets Guide — SQL and DataFrame API
Structured Streaming Programming Guide — Real-time stream processing
MLlib Guide — Machine learning library
GraphX Programming Guide — Graph processing

Operations Guides#

Cluster Overview — Cluster architecture
Tuning Guide — Performance tuning
Monitoring Guide — Monitoring
Configuration — Configuration options
Security — Security settings

Cluster Manager Guides#

API Documentation#

Java API#

Spark Java API (Javadoc) — Java API reference
Dataset — DataFrame class
SparkSession — Entry point class
functions — Built-in functions

Scala API#

Spark Scala API (Scaladoc)

Additional Learning Resources#

Online Courses#

Databricks Academy — Official training from Spark’s co-creator company
Coursera: Big Data Analysis with Scala and Spark — EPFL’s Scala/Spark course
edX: Big Data Analytics Using Spark — UC San Diego course

Blogs and Documentation#

Databricks Blog — Latest Spark technology and use cases
Spark By Examples — Java, Scala, Python examples
Baeldung Spark Tutorials — Spark tutorials for Java developers

Community#

Stack Overflow - apache-spark — Q&A
Spark Mailing Lists — Developer mailing lists
GitHub - apache/spark — Source code and issue tracker

Data Sources#

Kafka — Streaming data source
HDFS — Distributed file system
Parquet — Columnar format
Delta Lake — Storage with ACID transaction support

Cluster Environments#

Hadoop YARN — Resource management
Kubernetes — Container orchestration

Cloud Services#

AWS EMR — AWS managed Spark
Google Dataproc — GCP managed Spark
Azure HDInsight — Azure managed Hadoop/Spark
Databricks — Unified Data Analytics Platform

Version Release Notes#

Performance Benchmarks#

TPC-DS Benchmark — Decision support system benchmark
Spark SQL Performance Tests — Databricks performance testing tools

Recommended Books#

Beginner#

Learning Spark, 2nd Edition (O’Reilly) — Jules S. Damji et al.
Spark: The Definitive Guide (O’Reilly) — Bill Chambers, Matei Zaharia

Advanced#

High Performance Spark (O’Reilly) — Holden Karau, Rachel Warren
Spark in Action, 2nd Edition (Manning) — Jean-Georges Perrin

Appendix on Advanced Beginner

Glossary

Glossary#

Core Concepts#

Action#

Application#

Broadcast Variable#

Catalyst Optimizer#

FAQ

FAQ#

General Questions#

Which Java versions does Spark support?#

Can I use Spark with Java only, without Scala?#

What’s the difference between DataFrame and Dataset?#

References

References#

Official Documentation#

Apache Spark Official Site#

Programming Guides#

Operations Guides#

Cluster Manager Guides#

API Documentation#

Java API#

Scala API#

Additional Learning Resources#

Online Courses#

Blogs and Documentation#

Community#

Related Technology Documentation#

Data Sources#

Cluster Environments#

Cloud Services#

Version Release Notes#

Performance Benchmarks#

Recommended Books#

Beginner#

Advanced#