Autoscaler failing on minikube

Ray Clusters Kubernetes
1

Hello,

I get the below exception with autoscaler:

2021-04-22 15:06:23,806 ERROR autoscaler.py:142 – StandardAutoscaler: Error during autoscaling.
Traceback (most recent call last):
File “/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py”, line 140, in update
self._update()
File “/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py”, line 191, in _update
if (self._keep_min_worker_of_node_type(node_id, node_type_counts)
File “/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py”, line 443, in _keep_min_worker_of_node_type
min_workers = self.available_node_types[node_type].get(
KeyError: ‘worker-node’
2021-04-22 15:06:28,923 ERROR autoscaler.py:142 – StandardAutoscaler: Error during autoscaling.
Traceback (most recent call last):
File “/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py”, line 140, in update
self._update()
File “/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py”, line 191, in _update
if (self._keep_min_worker_of_node_type(node_id, node_type_counts)
File “/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py”, line 443, in _keep_min_worker_of_node_type
min_workers = self.available_node_types[node_type].get(
KeyError: ‘worker-node’
2021-04-22 15:06:28,923 CRITICAL autoscaler.py:152 – StandardAutoscaler: Too many errors, abort.
2021-04-22 15:06:28,923 ERROR monitor.py:253 – Error in monitor loop
Traceback (most recent call last):
File “/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/monitor.py”, line 285, in run
self._run()
File “/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/monitor.py”, line 187, in _run
self.autoscaler.update()
File “/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py”, line 154, in update
raise e
File “/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py”, line 140, in update
self._update()
File “/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py”, line 191, in _update
if (self._keep_min_worker_of_node_type(node_id, node_type_counts)
File “/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py”, line 443, in _keep_min_worker_of_node_type
min_workers = self.available_node_types[node_type].get(
KeyError: ‘worker-node’

I used ray up to launch the example cluster and then did ray.init(address=“auto”) which caused the above exception

2

What version of Ray is being used locally (i.e. outside of minikube)?
You can check with ray --version.

3

Thank you for your reply. Sorry I do not have the setup, I stopped all container instances in minikube, restarted minikube and deployed again, surprisingly it worked after restart