How do you decide the number of executors, cores, and memory?

Rule-of-thumb (for a node with N cores, M GB RAM):

• Leave 1 core + ~1–2 GB for OS/overhead.
• Target 4–5 cores per executor (to limit GC overhead).
• Memory per executor: (node_memory - OS_reserve) / num_executors_per_node.
• Total executors = (#nodes * executors_per_node).
  Fine-tune by monitoring Spark UI: adjust if tasks are slow (need more cores/executors) or OOM (need more memory/fewer cores per executor).

 

What are the main components of a Spark cluster and how do they interact?

• Driver: Runs your main program, builds logical plans, coordinates tasks, holds metadata, sometimes collects results.
• Executors: JVM processes on worker nodes that run tasks, store cached data, and write shuffle files.
• Cluster manager (YARN / Kubernetes / Databricks / Standalone): Allocates resources (containers/pods/VMs) for driver and executors.
• Flow: Driver requests resources from cluster manager → cluster manager starts executors → driver sends tasks to executors and tracks progress.