Setting Ray worker niceness

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hi, I’m trying to change the worker niceness as we are seeing some contention with other processes running on the same system (we use Kuberay so anything outside of Ray ends up with significantly higher priority than the workers). This is bad enough that it’s significantly degrading the throughput of our jobs.

I see there is a configuration environment variable defined here https://github.com/ray-project/ray/blob/ca0e04994edcbbced2a2b18215b7ebf8d47c7bce/src/ray/common/ray_config_def.h#L318

and used in various places: https://github.com/ray-project/ray/blob/master/python/ray/_private/ray_constants.py#L419

How does this setting actually work? Where would I set it for my own actors?

2

Hi there, I ran into the same issue and just solved it. The file you mentioned (https://github.com/ray-project/ray/blob/ca0e04994edcbbced2a2b18215b7ebf8d47c7bce/src/ray/common/ray_config_def.h#L318) contains all environment variables. You can change it by calling this at the start of your script.

import os
os.environ["RAY_worker_niceness"] = str(0)

For more infos on niceness values (maybe not you personally, but other readers), see e.g. here : UNIX / Linux : How to change the niceness (priority) of a process – The Geek Diary. In the above case, I set niceness to 0, which would be the lowest possible niceness value without sudo rights (the lower, the more important the process).

Same logic for all other variables mentioned in the ray_config_def.h file. E.g.the out of memory threshold (default 0.95) is called “memory_usage_threshold”. So to change it you have to call os.environ[“RAY_memory_usage_threshold”].

Hope this helps !

Best, Lasse

1 Like