Bug may related to the 2.48

Ray Core
1

Here is the bug log:

/home/ec2-user/conda/envs/grounded-rl/lib/python3.10/site-packages/_distutils_hack/__init__.py:53: UserWarning: Reliance on distutils f
rom stdlib is deprecated. Users must rely on setuptools to provide the distutils module. Avoid importing distutils or import setuptools
 first, and avoid setting SETUPTOOLS_USE_DISTUTILS=stdlib. Register concerns at https://github.com/pypa/setuptools/issues/new?template=
distutils-deprecation.yml                                                                                                              
  warnings.warn(                                                                                                                       
INFO 07-19 18:48:12 [__init__.py:239] Automatically detected platform cuda.                                                            
File limit: (65536, 1048576)                                                                                                           
2025-07-19 18:48:38,306 ERROR services.py:1362 -- Failed to start the dashboard                                                        
2025-07-19 18:48:38,306 ERROR services.py:1387 -- Error should be written to 'dashboard.log' or 'dashboard.err'. We are printing the la
st 20 lines for you. See 'https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#logging-directory-structur
e' to find where the log file is.                                                                                                      
2025-07-19 18:48:38,306 ERROR services.py:1431 --                                                                                      
The last 20 lines of /tmp/ray/session_2025-07-19_18-48-17_473287_12724/logs/dashboard.log (it contains the error message from the dashb
oard):                                                                                                                                 
2025-07-19 18:48:28,036 INFO utils.py:336 -- Available modules: [<class 'ray.dashboard.modules.usage_stats.usage_stats_head.UsageStatsH
ead'>]                                                                                                                                 
2025-07-19 18:48:28,037 INFO head.py:231 -- DashboardHeadModules to load: None.                                                        
2025-07-19 18:48:28,037 INFO head.py:234 -- Loading DashboardHeadModule: <class 'ray.dashboard.modules.usage_stats.usage_stats_head.Usa
geStatsHead'>.                                                                                                                         
2025-07-19 18:48:28,037 INFO head.py:238 -- Loaded 1 dashboard head modules: [<ray.dashboard.modules.usage_stats.usage_stats_head.Usage
StatsHead object at 0x7f40b7a9fe50>].                                                                                                  
2025-07-19 18:48:28,037 INFO utils.py:303 -- Get all modules by type: SubprocessModule                                                 
2025-07-19 18:48:28,038 INFO utils.py:336 -- Available modules: [<class 'ray.dashboard.modules.metrics.metrics_head.MetricsHead'>, <cla
ss 'ray.dashboard.modules.data.data_head.DataHead'>, <class 'ray.dashboard.modules.event.event_head.EventHead'>, <class 'ray.dashboard.
modules.job.job_head.JobHead'>, <class 'ray.dashboard.modules.node.node_head.NodeHead'>, <class 'ray.dashboard.modules.reporter.reporte
r_head.ReportHead'>, <class 'ray.dashboard.modules.serve.serve_head.ServeHead'>, <class 'ray.dashboard.modules.state.state_head.StateHe
ad'>, <class 'ray.dashboard.modules.train.train_head.TrainHead'>]
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.metrics.metrics_head.MetricsHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.data.data_head.DataHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.event.event_head.EventHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.job.job_head.JobHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.node.node_head.NodeHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.reporter.reporter_head.ReportHead'>
.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.serve.serve_head.ServeHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.state.state_head.StateHead'>.
2025-07-19 18:48:28,038 INFO head.py:288 -- Loading SubprocessModule: <class 'ray.dashboard.modules.train.train_head.TrainHead'>.
2025-07-19 18:48:28,039 INFO head.py:292 -- Loaded 9 subprocess modules: [<ray.dashboard.subprocesses.handle.SubprocessModuleHandle obj
ect at 0x7f40a1968880>, <ray.dashboard.subprocesses.handle.SubprocessModuleHandle object at 0x7f409dd5c0d0>, <ray.dashboard.subprocesse
s.handle.SubprocessModuleHandle object at 0x7f409df021d0>, <ray.dashboard.subprocesses.handle.SubprocessModuleHandle object at 0x7f409d
c26bf0>, <ray.dashboard.subprocesses.handle.SubprocessModuleHandle object at 0x7f409dc26410>, <ray.dashboard.subprocesses.handle.Subpro
cessModuleHandle object at 0x7f409dc264a0>, <ray.dashboard.subprocesses.handle.SubprocessModuleHandle object at 0x7f409dc26cb0>, <ray.d
ashboard.subprocesses.handle.SubprocessModuleHandle object at 0x7f409dc25510>, <ray.dashboard.subprocesses.handle.SubprocessModuleHandl
e object at 0x7f409dc26bc0>].

2025-07-19 18:48:38,852 INFO worker.py:1888 -- Started a local Ray instance.
[2025-07-19 18:49:09,842 E 12724 12724] core_worker.cc:513: Failed to register worker to Raylet: IOError: [RayletClient] Unable to regi
ster worker with raylet. Failed to read data from the socket: End of file worker_id=01000000fffffffffffffffffffffffffffffffffffffffffff
fffff

I think this error is related to the latest update to the 2.48. Given the same environment is fine last week. But it does not help when I try to revert it back to 2.47.1 or even 2.46

2

Hi @zhuoming_liu , welcome to the Ray community! Is it possible for you to file a bug on our GitHub so we can track this issue? Thank you for raising awareness :slight_smile:

3

Hi @zhuoming_liu I have met the same problem. I met the problem with ray==2.50.0, when I done grade ray to 2.49.2, the problem disappeared.

4

This issue appears to be a regression or breaking change introduced in Ray 2.50.0, as several users have reported that downgrading to 2.49.2 resolves the problem. The error is likely related to Ray’s dashboard or worker registration, as similar issues have been discussed recently, especially after upgrading Ray versions. No official fix or workaround for 2.50.0 is documented yet, so downgrading to 2.49.2 is currently the recommended solution if you encounter this problem. Would you like more technical details or links to related discussions?

Sources:

Hint: Mention @RunLLM in the post for followups.