Ray command-line parameter to turn OFF monitoring completely?

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

This is somewhat of a repeat / cross-post, extracting a question I had embedded in a recent post to the Ray Clusters category.

Question is: What is the command-line parameter to completely turn OFF anything to do with dashboard, monitoring etc.?

I am starting ray as:

$ ray --logging-level debug start --head --object-manager-port=8076 --include-dashboard=false --disable-usage-stats --object-store-memory=1073741824

I was under the impression that the combination of --include-dashboard=false --disable-usage-stats should turn OFF dashboard and relevant monitoring completely.

But I do see logs/dashboard.log and logs/dashboard_agent.log being created.

The 2nd file has several messages indicating some monitoring processes are being executed.

Due to some limitations on the platform I am trying this on, I get errors like so:

 48 2023-12-13 03:08:54,755»INFO event_utils.py:132 -- Monitor events logs modified after 1702435134.608772 on /tmp/ray/session_2023-12-13_03-08-42_327199_1/logs/events, the source types     are all.
 49 2023-12-13 03:08:54,757»ERROR reporter_agent.py:1149 -- Error publishing node physical stats.
 50 Traceback (most recent call last):
 51   File "/home/sgx/.local/lib/python3.8/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 1132, in _perform_iteration
 52     stats = self._get_all_stats()
 53   File "/home/sgx/.local/lib/python3.8/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 634, in _get_all_stats
 54     disk_stats = self._get_disk_io_stats()
 55   File "/home/sgx/.local/lib/python3.8/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 468, in _get_disk_io_stats
 56     stats = psutil.disk_io_counters()
 57   File "/home/sgx/.local/lib/python3.8/site-packages/ray/thirdparty_files/psutil/__init__.py", line 2072, in disk_io_counters
 58     rawdict = _psplatform.disk_io_counters(**kwargs)
 59   File "/home/sgx/.local/lib/python3.8/site-packages/ray/thirdparty_files/psutil/_pslinux.py", line 1161, in disk_io_counters
 60     raise NotImplementedError(
 61 NotImplementedError: /proc/diskstats nor /sys/block filesystem are available on this system

Basically, I want to disable completely the execution of monitoring stuff like dashboard/modules/reporter/reporter_agent.py.

What is the command-line option to turn these monitoring modules OFF (disable) completely?

Why aren’t the pair of --include-dashboard=false --disable-usage-stats enough to disable all this monitoring activity?

Thanks, in advance,
–AdityA>

I don’t think there is a way to completely disable dashboard agent at the moment.

@sangcho to confirm.

Hmm if you turn off dashboard & usage stats, agent is alive, but it doesn’t really do anything meaningful, so technically you can simply ignore it.

Doing no monitoring at all is unfortunately not possible now. You should create an Github issue for this one