# Autoscaler on K8s import client error

**URL:** <https://discuss.ray.io/t/autoscaler-on-k8s-import-client-error/1529>\
**Category:** Kubernetes\
**Created:** [April 2, 2021, 2:35am UTC](https://discuss.ray.io/t/autoscaler-on-k8s-import-client-error/1529 "2021-04-02T02:35:41Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![wkerr-riot](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/wkerr-riot/32/735_2.png) [@wkerr-riot](https://discuss.ray.io/u/wkerr-riot)\
**Post date:** [April 2, 2021, 2:35am UTC](https://discuss.ray.io/t/autoscaler-on-k8s-import-client-error/1529/1 "2021-04-02T02:35:41Z")

</div>

I have a K8s cluster running and trying to deploy rayproject/ray-ml:nightly nodes for both head and workers. Currently when I run `ray monitor cluster.yaml` I’m seeing the following stack trace after `ray up` completes

```auto
2021-04-01 19:28:44,147 ERROR monitor.py:245 -- Error in monitor loop
Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/monitor.py", line 276, in run
    self._initialize_autoscaler()
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/monitor.py", line 125, in _initialize_autoscaler
    event_summarizer=self.event_summarizer)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py", line 86, in __init__
    self.reset(errors_fatal=True)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py", line 521, in reset
    raise e
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/autoscaler.py", line 479, in reset
    self.config["cluster_name"])
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/providers.py", line 186, in _get_node_provider
    provider_cls = _get_node_provider_cls(provider_config)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/providers.py", line 162, in _get_node_provider_cls
    return importer(provider_config)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/providers.py", line 54, in _import_kubernetes
    from ray.autoscaler._private.kubernetes.node_provider import \
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/kubernetes/ __init__.py", line 1, in <module>
    import kubernetes
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/kubernetes/ __init__.py", line 2, in <module>
    from kubernetes.config.config_exception import ConfigException
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/kubernetes/config.py", line 6, in <module>
    from kubernetes import client
ImportError: cannot import name 'client' from 'kubernetes' (/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/kubernetes/ __init__.py)

```

I’ve connected to the container, run python and was able to import both kubernetes and able to run `from kubernetes import client`, so I’m fresh out of options what to do next.

The installed kubernetes version is

```auto
>>> kubernetes. __version__'12.0.1'

```

If I drop back to `rayproject/ray-ml:1.3.0` I don’t see this error, but I run into different rllib errors.

---

<div class="post-metadata">

**Author:** ![wkerr-riot](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/wkerr-riot/32/735_2.png) [@wkerr-riot](https://discuss.ray.io/u/wkerr-riot)\
**Post date:** [April 2, 2021, 3:14am UTC](https://discuss.ray.io/t/autoscaler-on-k8s-import-client-error/1529/2 "2021-04-02T03:14:41Z")

</div>

Follow up -

Tried going down to `rayproject/ray-ml:latest` and it doesn’t have the `tune.durable` that I’m trying to leverage. Next step I’m going to try going to a build from a few days ago.

Edit: Update for tonight `FROM rayproject/ray-ml:73fb5d` going all the way back to this one works.

---

<div class="post-metadata">

**Author:** ![yic](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/yic/32/437_2.png) [@yic](https://discuss.ray.io/u/yic)\
**Post date:** [April 2, 2021, 8:06pm UTC](https://discuss.ray.io/t/autoscaler-on-k8s-import-client-error/1529/3 "2021-04-02T20:06:05Z")

</div>

@Ameer_Haj_Ali it looks like autoscaler can’t find some library. Maybe due to env issues. Could you have a look at it?

---

<div class="post-metadata">

**Author:** ![Ameer\_Haj\_Ali](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ameer_haj_ali/32/279_2.png) [@Ameer\_Haj\_Ali](https://discuss.ray.io/u/Ameer_Haj_Ali)\
**Post date:** [April 2, 2021, 8:19pm UTC](https://discuss.ray.io/t/autoscaler-on-k8s-import-client-error/1529/4 "2021-04-02T20:19:42Z")

</div>

CC @Dmitri , can you please help?

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [April 2, 2021, 9:06pm UTC](https://discuss.ray.io/t/autoscaler-on-k8s-import-client-error/1529/5 "2021-04-02T21:06:47Z")

</div>

I’m looking into it.

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [April 2, 2021, 9:10pm UTC](https://discuss.ray.io/t/autoscaler-on-k8s-import-client-error/1529/6 "2021-04-02T21:10:16Z")

</div>

Yep, replicated the error. That’s very odd – will open an issue on github and immediately start working on a fix.

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [April 2, 2021, 9:15pm UTC](https://discuss.ray.io/t/autoscaler-on-k8s-import-client-error/1529/7 "2021-04-02T21:15:08Z")

</div>

Also, I recommend using the Ray Kubernetes Operator – you can use the default `rayproject/ray:nightly` image in the operator pod (which runs the autoscaler in this setup) and use the `rayproject/ray-ml:nightly` for the head and workers.

The operator is documented here:  
[https://docs.ray.io/en/master/cluster/kubernetes.html](https://docs.ray.io/en/master/cluster/kubernetes.html)

Issue tracked here:

> <https://github.com/ray-project/ray/issues/15095>
>
> What is the problem?
> Autoscaler running in ray-ml:nightly image on Kubernetes fails, with a Kubernetes import error.
> ImportError: cannot import name 'client' from...

---

<div class="post-metadata">

**Author:** ![wkerr-riot](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/wkerr-riot/32/735_2.png) [@wkerr-riot](https://discuss.ray.io/u/wkerr-riot)\
**Post date:** [April 2, 2021, 9:59pm UTC](https://discuss.ray.io/t/autoscaler-on-k8s-import-client-error/1529/8 "2021-04-02T21:59:53Z")

</div>

Thanks, working through the operator examples and I don’t see a great way to run a more complex workload that has multiple python files importing each other. Is the thinking that we build our own docker containers that extend `nightly` and put the code we need onto it and submit K8s jobs?

---

<div class="post-metadata">

**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\
**Post date:** [April 3, 2021, 3:07am UTC](https://discuss.ray.io/t/autoscaler-on-k8s-import-client-error/1529/9 "2021-04-03T03:07:28Z")

</div>

Yep, that’s currently the intended workflow.

There’s work in progress that will allow Ray to automatically handle syncing of modules between nodes – see the discussion here: [[autoscaler] [kubernetes] Calling ray down does not remove Kubernetes services · Issue #14700 · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues/14700#issuecomment-800508290) .