How do you decide the number of executors, cores, and memory?
Rule-of-thumb (for a node with N cores, M
GB RAM):
- Leave 1 core + ~1–2 GB for OS/overhead.
- Target 4–5 cores per executor (to limit GC overhead).
- Memory per executor: (node_memory - OS_reserve) /
num_executors_per_node.
- Total executors = (#nodes * executors_per_node).
Fine-tune by monitoring Spark UI: adjust if tasks are slow (need more cores/executors) or OOM (need more memory/fewer cores per executor).

No comments:
Post a Comment