Saturday, January 31, 2026

 

How do you decide the number of executors, cores, and memory?


Rule-of-thumb (for a node with N cores, M GB RAM):

  • Leave 1 core + ~1–2 GB for OS/overhead.
  • Target 4–5 cores per executor (to limit GC overhead).
  • Memory per executor: (node_memory - OS_reserve) / num_executors_per_node.
  • Total executors = (#nodes * executors_per_node).
    Fine-tune by monitoring Spark UI: adjust if tasks are slow (need more cores/executors) or OOM (need more memory/fewer cores per executor).

No comments: