Disable ray low memory monitoring

Ray Core
1

I’m using RAY+RLlib for research and I have access to a server with a lot of resources and too many people using it at the same time. The admins told me that the access to the resources on that server is not fair and I need to grab what I can as soon as they are freed. I’m the only RAY user on that server.

I saw this issue https://github.com/ray-project/ray/issues/4638 and I was hoping that there is a way to disable the low memory check.

What I would like to achieve is to have ray starting and just wait, even if the RAM and Swap are 100% full. Eventually, the RAM gets freed, and then it’s when I need all my experiments to start.
Otherwise, I need to stay there and monitor the RAM myself to restart my experiments.
Would this be possible? Would this work?

I understand that this goes against good practice, but I have no real alternatives on that server.

2

Hey @codeca thanks for bringing this up! Do you mind creating a feature request to our Github and tag @rkooo567 there?

3

Done, thank you.

[edit] I found RAY_DISABLE_MEMORY_MONITOR but it’s not in the release yet.

4

Can you use RAY_MEMORY_MONITOR_ERROR_THRESHOLD=9999 to effectively disable it?

5

I’m trying with os.environ["RAY_DEBUG_DISABLE_MEMORY_MONITOR"] = str(1).
Testing it now.