I am running a K8s cluster and I deployed the ray image rayproject/ray-ml:2.5.1.164b25-py38-gpu. When I run the ray status command, I am getting the error:
No cluster status.
The autoscaler failed with the following error:
Traceback (most recent call last):
File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 594, in run
self._initialize_autoscaler()
File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/autoscaler/_private/monitor.py", line 239, in _initialize_autoscaler
self.autoscaler = StandardAutoscaler(
File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/autoscaler/_private/autoscaler.py", line 251, in __init__
self.reset(errors_fatal=True)
File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/autoscaler/_private/autoscaler.py", line 1115, in reset
raise e
File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/autoscaler/_private/autoscaler.py", line 1059, in reset
self.provider = _get_node_provider(
File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/autoscaler/_private/providers.py", line 228, in _get_node_provider
provider_cls = _get_node_provider_cls(provider_config)
File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/autoscaler/_private/providers.py", line 204, in _get_node_provider_cls
return importer(provider_config)
File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/autoscaler/_private/providers.py", line 82, in _import_kubernetes
from ray.autoscaler._private._kubernetes.node_provider import KubernetesNodeProvider
ModuleNotFoundError: No module named 'ray.autoscaler._private._kubernetes'