Ray_disable_memory_monitor

1

Hi!
I am having problems with RAM. When my computer’s capacity is full, Ray stops running and gives a warning. Also, it recommends making RAY_DISABLE_MEMORY_MONITOR=1.

My question is: where do I assign RAY_DISABLE_MEMORY_MONITOR=1? I tried assigning it as a Ray Tune variable, but it doesn’t identify it as such. I’ve been reading other discussions and one of them says it’s an environment variable, but I’m not sure it is.

Anyone who can help me?
Thanks in advance!

== Status ==
Current time: 2022-04-18 17:40:59 (running for 08:16:21.92)
Memory usage on this node: 15.5/15.7 GiB: ***LOW MEMORY*** less than 10% of the memory on this node is available for use. This can cause unexpected crashes. Consider reducing the memory used by your application or reducing the Ray object store size by setting `object_store_memory` when calling `ray.init`.
Using FIFO scheduling algorithm.
Resources requested: 5.0/8 CPUs, 0/0 GPUs, 0.0/4.46 GiB heap, 0.0/2.23 GiB objects
Result logdir: C:\Users\grhen\ray_results\DQN
Number of trials: 1/1 (1 RUNNING)


 pid=25088) 2022-04-18 17:41:01,494     ERROR worker.py:84 -- Unhandled error (suppress with RAY_IGNORE_UNHANDLED_ERRORS=1): ray::RolloutWorker.set_weights() (pid=3148, ip=127.0.0.1, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x0000024ADAC45AC0>)
 pid=25088)   File "python\ray\_raylet.pyx", line 585, in ray._raylet.execute_task
 pid=25088)   File "C:\Users\grhen\AppData\Local\Programs\Python\Python39\lib\site-packages\ray\_private\memory_monitor.py", line 156, in raise_if_low_memory
 pid=25088)     raise RayOutOfMemoryError(
 pid=25088) ray._private.memory_monitor.RayOutOfMemoryError: More than 95% of the memory on node DESKTOP-DQA5FE6 is used (15.33 / 15.71 GB). The top 10 memory consumers are:
 pid=25088)
 pid=25088) PID MEM     COMMAND
 pid=25088) 22280       8.32GiB C:\Users\grhen\AppData\Local\Programs\Python\Python39\python.exe c:\Users\grhen\Documents\GitHub\EP_
 pid=25088) 3152        0.69GiB C:\Users\grhen\AppData\Local\Programs\Microsoft VS Code\Code.exe --ms-enable-electron-run-as-node c:
 pid=25088) 25088       0.33GiB C:\Users\grhen\AppData\Local\Programs\Python\Python39\python.exe C:\Users\grhen\AppData\Local\Progra
 pid=25088) 14712       0.28GiB C:\Users\grhen\AppData\Local\Programs\Python\Python39\python.exe C:\Users\grhen\AppData\Local\Progra
 pid=25088) 20120       0.25GiB C:\Program Files\Google\Chrome\Application\chrome.exe --type=renderer --display-capture-permissions-
 pid=25088) 3148        0.24GiB C:\Users\grhen\AppData\Local\Programs\Python\Python39\python.exe C:\Users\grhen\AppData\Local\Progra
 pid=25088) 1160        0.23GiB C:\Program Files\Google\Chrome\Application\chrome.exe
 pid=25088) 7324        0.23GiB C:\Users\grhen\AppData\Local\Programs\Python\Python39\python.exe C:\Users\grhen\AppData\Local\Progra
 pid=25088) 6708        0.22GiB C:\Users\grhen\AppData\Local\Microsoft\Teams\current\Teams.exe --type=renderer --autoplay-policy=no-
 pid=25088) 23648       0.22GiB C:\Users\grhen\AppData\Local\Programs\Python\Python39\python.exe C:\Users\grhen\AppData\Local\Progra
 pid=25088)
 pid=25088) In addition, up to 0.0 GiB of shared memory is currently being used by the Ray object store.
 pid=25088) ---
 pid=25088) --- Tip: Use the `ray memory` command to list active objects in the cluster.
 pid=25088) --- To disable OOM exceptions, set RAY_DISABLE_MEMORY_MONITOR=1.
2

Yeah, it’s an env variable.

RAY_DISABLE_MEMORY_MONITOR=1 python script.py
1 Like
3

I’m using ray-tune in jupyter notebook and when I tried:
os.environ['RAY_DISABLE_MEMORY_MONITOR'] = 1
I got an error that it expects a string, not an integer

os.environ['RAY_DISABLE_MEMORY_MONITOR'] = '1'
seems to have done the trick

1 Like
4

So , how to set the RAY_DISABLE_MEMORY_MONITOR?

5

Hi,
as @RNarayan show. In your script you must import os and then write the comand os.environ['RAY_DISABLE_MEMORY_MONITOR']='1'. This would be like:

import os
os.environ['RAY_DISABLE_MEMORY_MONITOR']='1'

This worked for me.