How severe does this issue affect your experience of using Ray?
- High: It blocks me to complete my task.
This is somewhat of a repeat / cross-post, extracting a question I had embedded in a recent post to the Ray Clusters category.
Question is: What is the command-line parameter to completely turn OFF anything to do with dashboard, monitoring etc.?
I am starting ray as:
$ ray --logging-level debug start --head --object-manager-port=8076 --include-dashboard=false --disable-usage-stats --object-store-memory=1073741824
I was under the impression that the combination of --include-dashboard=false --disable-usage-stats should turn OFF dashboard and relevant monitoring completely.
But I do see logs/dashboard.log and logs/dashboard_agent.log being created.
The 2nd file has several messages indicating some monitoring processes are being executed.
Due to some limitations on the platform I am trying this on, I get errors like so:
48 2023-12-13 03:08:54,755»INFO event_utils.py:132 -- Monitor events logs modified after 1702435134.608772 on /tmp/ray/session_2023-12-13_03-08-42_327199_1/logs/events, the source types are all.
49 2023-12-13 03:08:54,757»ERROR reporter_agent.py:1149 -- Error publishing node physical stats.
50 Traceback (most recent call last):
51 File "/home/sgx/.local/lib/python3.8/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 1132, in _perform_iteration
52 stats = self._get_all_stats()
53 File "/home/sgx/.local/lib/python3.8/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 634, in _get_all_stats
54 disk_stats = self._get_disk_io_stats()
55 File "/home/sgx/.local/lib/python3.8/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 468, in _get_disk_io_stats
56 stats = psutil.disk_io_counters()
57 File "/home/sgx/.local/lib/python3.8/site-packages/ray/thirdparty_files/psutil/__init__.py", line 2072, in disk_io_counters
58 rawdict = _psplatform.disk_io_counters(**kwargs)
59 File "/home/sgx/.local/lib/python3.8/site-packages/ray/thirdparty_files/psutil/_pslinux.py", line 1161, in disk_io_counters
60 raise NotImplementedError(
61 NotImplementedError: /proc/diskstats nor /sys/block filesystem are available on this system
Basically, I want to disable completely the execution of monitoring stuff like dashboard/modules/reporter/reporter_agent.py.
What is the command-line option to turn these monitoring modules OFF (disable) completely?
Why aren’t the pair of --include-dashboard=false --disable-usage-stats enough to disable all this monitoring activity?
Thanks, in advance,
–AdityA>