I am currently working on optimizing the configuration of a Ray cluster for running large sets of remote-tasks while leveraging NUMA domains for enhance performance. My goal is to segment each node I acquired into 4 distinct NUMA domains in order to improve memory access and overall computation efficiency. Each node, for my use-case, has 96 cores and total of 4 NUMA domains (24 cores per NUMA ).
My current setup involves me setting
numactl before calling
ray start and properly pinning each worker-node to correct numa domain. I followed this post for guidance. This approach works for me, however, my questions is:
Does Ray have a way to restrict work (remote tasks) to run on specific NUMA domains without need to set
numactl at start? If so , how can I achieve that and verify it is actually doing the right thing?
Thank you in advance! Any examples would be greatly appreciated !
For reference I am using:
- Ray 2.8.1
- CentOS 7