Here is the bug log:
/home/ec2-user/conda/envs/grounded-rl/lib/python3.10/site-packages/_distutils_hack/__init__.py:53: UserWarning: Reliance on distutils f
rom stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools
first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=
distutils-deprecation.yml
warnings.warn(
INFO 07-19 18:48:12 [__init__.py:239] Automatically detected platform cuda.
File limit: (65536, 1048576)
2025-07-19 18:48:38,306 ERROR services.py:1362 -- Failed to start the dashboard
2025-07-19 18:48:38,306 ERROR services.py:1387 -- Error should be written to 'dashboard.log' or 'dashboard.err'. We are printing the la
st 20 lines for you. See 'https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#logging-directory-structur
e' to find where the log file is.
2025-07-19 18:48:38,306 ERROR services.py:1431 --
The last 20 lines of /tmp/ray/session_2025-07-19_18-48-17_473287_12724/logs/dashboard.log (it contains the error message from the dashb
oard):
2025-07-19 18:48:28,036 INFO utils.py:336 -- Available modules: [<class 'ray.dashboard.modules.usage_stats.usage_stats_head.UsageStatsH
ead'>]
2025-07-19 18:48:28,037 INFO head.py:231 -- DashboardHeadModules to load: None.
2025-07-19 18:48:28,037 INFO head.py:234 -- Loading DashboardHeadModule: <class 'ray.dashboard.modules.usage_stats.usage_stats_head.Usa
geStatsHead'>.
2025-07-19 18:48:28,037 INFO head.py:238 -- Loaded 1 dashboard head modules: [<ray.dashboard.modules.usage_stats.usage_stats_head.Usage
StatsHead object at 0x7f40b7a9fe50>].
2025-07-19 18:48:28,037 INFO utils.py:303 -- Get all modules by type: SubprocessModule
2025-07-19 18:48:28,038 INFO utils.py:336 -- Available modules: [<class 'ray.dashboard.modules.metrics.metrics_head.MetricsHead'>, <cla
ss 'ray.dashboard.modules.data.data_head.DataHead'>, <class 'ray.dashboard.modules.event.event_head.EventHead'>, <class 'ray.dashboard.
modules.job.job_head.JobHead'>, <class 'ray.dashboard.modules.node.node_head.NodeHead'>, <class 'ray.dashboard.modules.reporter.reporte
r_head.ReportHead'>, <class 'ray.dashboard.modules.serve.serve_head.ServeHead'>, <class 'ray.dashboard.modules.state.state_head.StateHe
ad'>, <class 'ray.dashboard.modules.train.train_head.TrainHead'>]
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.metrics.metrics_head.MetricsHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.data.data_head.DataHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.event.event_head.EventHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.job.job_head.JobHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.node.node_head.NodeHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.reporter.reporter_head.ReportHead'>
.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.serve.serve_head.ServeHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.state.state_head.StateHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.train.train_head.TrainHead'>.
2025-07-19 18:48:28,039 INFO head.py:292 -- Loaded 9 subprocess modules: [<ray.dashboard.subprocesses.handle.SubprocessModuleHandle obj
ect at 0x7f40a1968880>, <ray.dashboard.subprocesses.handle.SubprocessModuleHandle object at 0x7f409dd5c0d0>, <ray.dashboard.subprocesse
s.handle.SubprocessModuleHandle object at 0x7f409df021d0>, <ray.dashboard.subprocesses.handle.SubprocessModuleHandle object at 0x7f409d
c26bf0>, <ray.dashboard.subprocesses.handle.SubprocessModuleHandle object at 0x7f409dc26410>, <ray.dashboard.subprocesses.handle.Subpro
cessModuleHandle object at 0x7f409dc264a0>, <ray.dashboard.subprocesses.handle.SubprocessModuleHandle object at 0x7f409dc26cb0>, <ray.d
ashboard.subprocesses.handle.SubprocessModuleHandle object at 0x7f409dc25510>, <ray.dashboard.subprocesses.handle.SubprocessModuleHandl
e object at 0x7f409dc26bc0>].
2025-07-19 18:48:38,852 INFO worker.py:1888 -- Started a local Ray instance.
[2025-07-19 18:49:09,842 E 12724 12724] core_worker.cc:513: Failed to register worker to Raylet: IOError: [RayletClient] Unable to regi
ster worker with raylet. Failed to read data from the socket: End of file worker_id=01000000fffffffffffffffffffffffffffffffffffffffffff
fffff
I think this error is related to the latest update to the 2.48. Given the same environment is fine last week. But it does not help when I try to revert it back to 2.47.1 or even 2.46
