You’ve read the docs. You can write queries using SparkSQL or PySpark. You’re pretty sure you understand how Spark works under the hood. Then an interviewer asks you to actually reason through a shuffle, and that confidence gets shaky fast.

Been there.

What actually got these concepts to stick for me was working through math problems tied to Spark’s internals. You have to understand the mechanics or the numbers don’t add up. So I put together a set of exercises built around exactly that.

Full disclosure: I generated the first draft with AI, then went through the whole list myself to make sure every question actually holds up and reflects what gets asked in real interviews.

Stuck on a question? Check the solution. Still not clicking? Feed it to an AI and ask it to explain it differently. That’s what I did. It forced me to think about the gaps I didn’t even know I had.

Give it a try.

Exercise List 1 Link to heading

Section 1 — Partitioning Link to heading

Exercise 1.1 — Basic partition count Link to heading

You have a 500 GB dataset in S3 stored as uncompressed Parquet. Spark’s default spark.sql.files.maxPartitionBytes is 128 MB. How many partitions will Spark create when reading this data?

Exercise 1.2 — Compression skews the math Link to heading

Same 500 GB dataset, but now it’s Snappy-compressed Parquet with a compression ratio of ~4:1 (i.e., 500 GB on disk decompresses to ~2 TB in memory). Spark decides partition count based on the file size on disk, not decompressed size. Explain why this can cause OOM errors on executors, and calculate the actual in-memory size per partition if the file-based partitioning still targets 128 MB per partition.

Exercise 1.3 — Shuffle partitions sizing Link to heading

You have a job whose shuffle stage will move 800 GB of data. You want each shuffle partition to land around 200 MB (a healthy target to avoid small-file problems downstream). What value should you set spark.sql.shuffle.partitions to?

Exercise 1.4 — Too many small partitions Link to heading

A DataFrame has 40,000 partitions, but the total data size is only 10 GB. What’s the average partition size? What problem does this cause, and what’s the fix (name the config or technique)?

Exercise 1.5 — Repartition vs coalesce cost Link to heading

You have a DataFrame with 2,000 partitions and want to reduce it to 200 for writing output. a) If you use coalesce(200), does this trigger a full shuffle? b) If you use repartition(200), does this trigger a full shuffle? c) Given 2,000 partitions merging into 200, roughly how many source partitions feed each target partition under coalesce, and why does this matter for skew?

Section 2 — Memory Management Link to heading

Exercise 2.1 — Unified memory pool Link to heading

An executor is launched with --executor-memory 20g. Default settings: spark.memory.fraction = 0.6, spark.memory.storageFraction = 0.5. a) How much memory is reserved for “reserved memory” (assume the standard 300 MB reserved)? b) How much is allocated to the unified memory pool (execution + storage)? c) How much is the minimum guaranteed storage memory within that pool?

Exercise 2.2 — Executor memory overhead Link to heading

You request --executor-memory 16g. Spark’s default overhead formula is max(384 MB, 0.10 * executorMemory). a) What is the overhead in this case? b) What is the total memory YARN/Kubernetes will actually reserve per executor? c) If your cluster node has 64 GB total RAM and you want to fit 4 executors per node, does 16 GB executor memory fit? Show the math including overhead.

Exercise 2.3 — Broadcast join threshold Link to heading

spark.sql.autoBroadcastJoinThreshold is set to the default 10 MB. You’re joining a fact table (2 TB) with a dimension table. The dimension table’s Parquet file is 8 MB on disk but has a 3:1 in-memory expansion ratio when deserialized. a) Will Spark broadcast it automatically? What number does Spark actually compare against the threshold — file size or in-memory size? b) If broadcasting causes an executor OOM despite passing the threshold check, what’s the likely explanation?

Exercise 2.4 — Spill estimation Link to heading

A single executor has 4 cores and 8 GB usable execution memory (after overhead/reserved memory is subtracted). It’s running a groupByKey aggregation where each of the 4 concurrent tasks needs to hold roughly 3 GB of shuffle data in memory to avoid spilling. a) Will this configuration spill to disk? Do the math. b) Name two ways to fix it without adding hardware.

Section 3 — Cluster Sizing Link to heading

Exercise 3.1 — Executors per node Link to heading

You have worker nodes with 128 GB RAM and 32 cores each. Best practice caps executor cores at 5 (to avoid HDFS I/O throughput issues) and leaves 1 core + ~1 GB per node for the OS/daemons. a) How many executors can you fit per node (core-wise)? b) Given that core constraint, how much memory should each executor get (ignore overhead for this step)? c) Recompute part (b) accounting for the default executor memory overhead (10%, min 384 MB).

Exercise 3.2 — Total parallelism Link to heading

Using your answer from 3.1, if the cluster has 20 worker nodes, what is the total number of concurrent tasks the cluster can run at once?

Exercise 3.3 — Right-sizing for a job Link to heading

A batch job reads 3 TB of input data. You want each task to process ~256 MB of input (a reasonable target for CPU-bound transforms). a) How many tasks/partitions do you need? b) Using the cluster from 3.1/3.2 (total concurrent task slots), how many “waves” of tasks will this job run in? c) If each wave takes ~40 seconds, what’s your rough total runtime estimate for just the map stage (ignore shuffle/scheduling overhead)?

Exercise 3.4 — Skewed partition, back-of-envelope Link to heading

A shuffle stage has 500 output partitions. 499 of them contain 400 MB each; one partition (due to a hot key) contains 60 GB. a) What’s the total shuffle output size? b) If every other stage bottleneck is solved, roughly how much longer will the skewed task take vs. a normal task (assume throughput is constant per core)? c) Name two concrete techniques to fix this (one config-based via AQE, one manual).

Section 4 — Mixed / Rapid-fire (good for whiteboard drills) Link to heading

A DataFrame has 1 billion rows, average row size 200 bytes. Estimate total in-memory size (uncompressed) and a reasonable partition count if targeting 128 MB/partition.
spark.default.parallelism is 200, but you’re reading from a source with 1,000 input splits. Which value actually controls the number of partitions for the initial read stage?
Your job has spark.executor.instances=50, spark.executor.cores=4. What’s the max number of tasks that can run simultaneously cluster-wide (ignoring dynamic allocation)?
You increase spark.sql.shuffle.partitions from 200 to 2000 on a job that shuffles only 5 GB. What’s the average partition size now, and why might this hurt performance despite “more parallelism”?
A 100 GB table is broadcast-joined against a 50 TB table. Each executor has 8 GB memory. If there are 200 executors, what’s the memory overhead as a percentage of total cluster memory just from holding a copy of the broadcast table on every executor?

Solutions Link to heading

1.1 Link to heading

500 GB / 128 MB ≈ 500,000 MB / 128 MB ≈ 3,907 partitions (Spark rounds up, so ~3,908).

1.2 Link to heading

Spark’s maxPartitionBytes operates on the on-disk file size, not the decompressed in-memory size, because partition planning happens before decompression. So targeting 128 MB per partition on-disk with a 4:1 expansion ratio means each partition actually holds ~512 MB in memory once decompressed and deserialized. Multiply across many concurrent tasks per executor and you can blow past execution memory — this is a classic cause of Parquet/Snappy-related OOMs. Fix: lower maxPartitionBytes (e.g., to 32 MB) so the decompressed size per partition stays near the real target.

1.3 Link to heading

800 GB / 200 MB ≈ 800,000 MB / 200 MB = 4,000 shuffle partitions. Set spark.sql.shuffle.partitions = 4000 (or let AQE’s coalescePartitions handle it dynamically instead of hardcoding).

1.4 Link to heading

10 GB / 40,000 = 0.25 MB per partition on average. This is the “small files / small partitions” problem: task scheduling overhead (each task has fixed overhead of a few ms to tens of ms for scheduling, serialization, etc.) dominates actual compute time, so the job spends more time on overhead than work. Fix: coalesce() to reduce partition count, or enable AQE with spark.sql.adaptive.coalescePartitions.enabled=true to merge small shuffle partitions automatically.

1.5 Link to heading

a) No — coalesce() avoids a full shuffle by combining existing partitions locally where possible (no shuffle stage boundary). b) Yes — repartition() always triggers a full shuffle since it needs to redistribute data evenly (uses round-robin or hash partitioning). c) Roughly 2,000 / 200 = 10 source partitions per target partition. Because coalesce just groups existing partitions without rebalancing data volume, if the source partitions are unevenly sized, the merged output partitions inherit that imbalance — you can end up with skewed output partitions since there’s no shuffle to even things out.

2.1 Link to heading

a) Reserved memory: 300 MB (fixed). b) Usable memory = 20 GB − 300 MB ≈ 19.7 GB. Unified pool = 0.6 × 19.7 GB ≈ 11.82 GB. c) Minimum guaranteed storage memory = storageFraction × unified pool = 0.5 × 11.82 GB ≈ 5.91 GB.

2.2 Link to heading

a) Overhead = max(384 MB, 0.10 × 16 GB) = max(384 MB, 1.6 GB) = 1.6 GB. b) Total reserved = 16 GB + 1.6 GB = 17.6 GB. c) 4 executors × 17.6 GB = 70.4 GB > 64 GB node capacity. It does not fit — you’d need to drop to 3 executors/node (52.8 GB, leaving headroom for OS) or reduce executor memory.

2.3 Link to heading

a) Spark compares the in-memory (deserialized) estimated size, not the on-disk file size, when deciding whether to broadcast (it uses stats from the query plan / table statistics). At 3:1 expansion, 8 MB on disk ≈ 24 MB in memory, which exceeds the 10 MB threshold — so Spark would not auto-broadcast it in this case (this is the reverse of the classic mistake of assuming file size = broadcast size). b) If it does get broadcast (e.g., via explicit broadcast() hint) and OOMs anyway, likely causes: the estimate/statistics were stale or wrong (e.g., no recent ANALYZE TABLE), or the broadcast is being held on every executor simultaneously while other memory-heavy operations are also running, exceeding the driver’s or executor’s available memory — broadcast variables also first collect to the driver, which can OOM there before it even reaches executors.

2.4 Link to heading

a) 4 concurrent tasks × 3 GB each = 12 GB needed, but only 8 GB execution memory available. 12 GB > 8 GB → yes, it will spill to disk. b) Options: (1) reduce spark.executor.cores (fewer concurrent tasks per executor, e.g., 2 instead of 4, so only 6 GB is needed concurrently), or (2) increase spark.memory.fraction / reduce storage usage to give execution memory more room, or (3) use reduceByKey/aggregateByKey instead of groupByKey to reduce the amount of data held in memory per key via map-side combining.

3.1 Link to heading

a) 32 cores total, minus 1 for OS = 31 available. 31 / 5 cores per executor = 6 executors per node (with 1 core left idle/spare). b) 128 GB − ~1 GB OS reserve = 127 GB / 6 executors ≈ 21.16 GB per executor (before overhead). c) Solve for executorMemory where executorMemory + max(384MB, 0.10 × executorMemory) = 21.16 GB. Approximate: 1.10 × executorMemory ≈ 21.16 GB → executorMemory ≈ 19.2 GB, overhead ≈ 1.92 GB, total ≈ 21.1 GB. (In practice you’d round down to something clean like 19 GB.)

3.2 Link to heading

6 executors/node × 5 cores/executor × 20 nodes = 600 concurrent task slots.

3.3 Link to heading

a) 3 TB / 256 MB = 3,000,000 MB / 256 MB ≈ 11,719 tasks/partitions. b) 11,719 / 600 slots ≈ 19.5 → 20 waves. c) 20 waves × 40 sec ≈ 800 seconds (~13.3 minutes) for the map stage alone.

3.4 Link to heading

a) 499 × 400 MB + 60 GB = 199.6 GB + 60 GB = ~259.6 GB total shuffle output. b) Normal task processes 400 MB; skewed task processes 60 GB = 60,000 MB. Ratio = 60,000 / 400 = 150x longer — the skewed task alone could dominate the entire stage’s wall-clock time since all other tasks finish and the job waits on this one straggler. c) (1) AQE skew join optimization — spark.sql.adaptive.skewJoin.enabled=true, which automatically splits oversized partitions into smaller sub-partitions during a join. (2) Manual salting — add a random “salt” key to the skewed join key, splitting the hot key into N sub-keys distributed across partitions, then aggregate/de-salt afterward.

Rapid-fire answers Link to heading

1e9 rows × 200 bytes = 200 GB. 200,000 MB / 128 MB ≈ 1,563 partitions.
For file-based sources, the input split count / maxPartitionBytes governs initial partitions — not spark.default.parallelism (that config mainly affects RDD operations without explicit partitioning, like parallelize, and non-file-based shuffles when AQE/shuffle partition count isn’t otherwise set).
50 × 4 = 200 concurrent tasks.
5 GB / 2000 ≈ 2.5 MB/partition — far too small. This causes task scheduling overhead to dominate: thousands of tiny tasks each pay fixed per-task overhead (serialization, scheduler latency, executor communication), so total overhead time can exceed actual compute time despite nominal “parallelism.”
Broadcast table held on every executor: 100 GB × 200 = 20 TB, but that’s clearly unrealistic for 8 GB executors (100 GB wouldn’t even broadcast — it’s way above threshold and wouldn’t fit in one executor’s memory anyway). The real lesson: total cluster memory = 200 × 8 GB = 1.6 TB. Broadcasting any table means every executor pays that memory cost simultaneously, so broadcast is only viable for genuinely small dimension tables (typically under a few hundred MB) — this exercise is a trick question to test whether you catch that 100 GB is never a realistic broadcast candidate regardless of cluster size.

Exercise List 2 Link to heading

Part 1 — Partition Calculations Link to heading

Exercise 1 — Initial Partitions Link to heading

You have a CSV file of 1.2 TB stored in HDFS. HDFS block size = 256 MB. You load it with spark.read.csv(...).

How many input partitions will Spark create?
If you have 80 executors with 5 cores each, how many tasks can run simultaneously?
How many waves of tasks are needed?
If every task takes 45 seconds, estimate the total stage duration (ignore scheduling overhead).

Exercise 2 — repartition() Link to heading

A DataFrame currently has 180 partitions. You execute df2 = df.repartition(600).

Will this trigger a shuffle?
How many output partitions exist?
If the dataset size is 900 GB, what is the average partition size?
Why might this be a bad idea?

Exercise 3 — coalesce() Link to heading

A DataFrame has 1200 partitions. You execute df2 = df.coalesce(300).

Does Spark shuffle?
Why is coalesce faster than repartition here?
Under what situation would coalesce produce skewed partitions?

Part 2 — Shuffle Calculations Link to heading

Exercise 4 — GroupBy Shuffle Link to heading

Dataset: 2 TB, 500 partitions. You execute df.groupBy("country").count() with spark.sql.shuffle.partitions = 400.

Is a shuffle required?
Approximately how much data moves across the network?
What determines the number of reduce tasks?
How many output partitions exist?

Exercise 5 — Join Shuffle Link to heading

Table A: 400 GB, 200 partitions. Table B: 600 GB, 300 partitions. You perform A.join(B, "customer_id").

Does Spark shuffle both tables?
Approximately how much data is shuffled?
How many shuffle stages exist?

Part 3 — Broadcast Join Link to heading

Exercise 6 Link to heading

Table A = 800 GB, Table B = 8 MB, broadcast threshold = 10 MB.

Which join strategy should Spark use?
Is there a shuffle?
How many copies of Table B exist if there are 100 executors?

Exercise 7 Link to heading

Table A = 300 GB, Table B = 25 MB, broadcast threshold = 10 MB.

Will Spark broadcast automatically?
Could you force broadcasting?
What are the risks?

Part 4 — Memory Link to heading

Exercise 8 Link to heading

Executor config: 8 GB executor memory, 4 cores. Spark storage fraction = 50%.

How much memory is available for storage?
How much for execution?
If cached data occupies 5 GB, what happens?

Exercise 9 Link to heading

Executor: 16 GB, 8 cores. Task memory usage: 900 MB per task.

Can all 8 tasks run simultaneously?
Estimate total task memory required.
What happens if memory exceeds available execution memory?

Part 5 — Cluster Utilization Link to heading

Exercise 10 Link to heading

Cluster: 20 workers, each with 16 cores / 64 GB RAM. Executors: 4 cores, 16 GB.

How many executors per worker?
Total executors?
Total parallel tasks?
Total executor memory?

Part 6 — Stage Counting Link to heading

Exercise 11 Link to heading

df = spark.read.parquet(...)
df = df.filter(...)
df = df.select(...)
df = df.groupBy(...).sum()
df.write.parquet(...)

How many stages?
Where does the shuffle happen?
Which transformations are narrow?
Which are wide?

Exercise 12 Link to heading

df1.join(df2, "id") \
   .groupBy("country") \
   .count() \
   .sort("count")

How many shuffles?
How many stages?
Which operation is the most expensive?

Part 7 — Data Skew Link to heading

Exercise 13 Link to heading

Dataset: 100 million rows. Distribution: USA 90M, Canada 5M, Mexico 5M.

Why is this skew?
Which partition becomes the bottleneck?
Suggest three ways to fix it.

Exercise 14 Link to heading

After a join, you observe these partition sizes: 300 MB, 290 MB, 310 MB, 305 MB, 7.8 GB, 295 MB.

What is happening?
Why will one task run much longer?
Suggest two Spark techniques to mitigate this.

Part 8 — File Size Problems Link to heading

Exercise 15 Link to heading

A job writes 18,000 parquet files, each 3 MB.

Why is this inefficient?
How many partitions likely existed before writing?
How could you produce ~256 MB output files?

Exercise 16 Link to heading

Dataset size: 640 GB. Desired output file size: 256 MB.

How many output partitions should you use?
Which transformation would you use before writing?

Part 9 — DAG Reasoning Link to heading

Exercise 17 Link to heading

df = spark.read.parquet(...)
df = df.filter(df.age > 20)
df = df.filter(df.country == "US")
df = df.withColumn(...)
df = df.groupBy("city").count()
df = df.orderBy("count")
df.show()

Draw the DAG.
Where do stage boundaries occur?
Which operations are pipelined?
Which operations force a shuffle?

Part 10 — Advanced Interview Math Link to heading

Exercise 18 Link to heading

Cluster: 25 executors, 6 cores each. Dataset: 1.5 TB. Partition size: 128 MB. Each task: 55 seconds.

Number of partitions?
Maximum parallelism?
Number of task waves?
Approximate execution time?

Exercise 19 Link to heading

You have 600 partitions, average size 1 GB.

Is this a good partition size?
Recommend a better partition count.
Explain why.

Exercise 20 Link to heading

Fact table: 8 TB. Dimension table: 150 MB.

Would Spark broadcast automatically?
Should you force a broadcast?
Estimate network traffic with and without broadcasting.
Which solution scales better?

Bonus: Real Interview Challenge Link to heading

A company has:

12-node cluster
Each node: 32 cores, 128 GB RAM
Executor configuration: 8 cores, 24 GB RAM
Dataset: 9.6 TB
HDFS block size: 256 MB
spark.sql.shuffle.partitions = 200
A groupBy(user_id) causes a shuffle
The output is written to Parquet with one file per partition

Without running the job, answer:

How many executors run in the cluster?
What is the maximum task parallelism?
How many input partitions are created from the dataset?
How many waves of map tasks are required?
How many reduce tasks will execute after the shuffle?
Approximately how large is each output file?
What performance issues do you anticipate?
How would you tune the job to improve throughput?
If one user_id accounts for 40% of all rows, what issue could arise?
How would you detect and mitigate that issue?