I’m using RAY+RLlib for research and I have access to a server with a lot of resources and too many people using it at the same time. The admins told me that the access to the resources on that server is not fair and I need to grab what I can as soon as they are freed. I’m the only RAY user on that server.
I saw this issue https://github.com/ray-project/ray/issues/4638 and I was hoping that there is a way to disable the low memory check.
What I would like to achieve is to have ray starting and just wait, even if the RAM and Swap are 100% full. Eventually, the RAM gets freed, and then it’s when I need all my experiments to start.
Otherwise, I need to stay there and monitor the RAM myself to restart my experiments.
Would this be possible? Would this work?
I understand that this goes against good practice, but I have no real alternatives on that server.