# Exception when ray up

**URL:** <https://discuss.ray.io/t/exception-when-ray-up/11756>\
**Category:** Ray Clusters\
**Created:** [August 11, 2023, 9:37am UTC](https://discuss.ray.io/t/exception-when-ray-up/11756 "2023-08-11T09:37:35Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![KelvinKarRoy](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kelvinkarroy/32/4910_2.png) [@KelvinKarRoy](https://discuss.ray.io/u/KelvinKarRoy)\
**Post date:** [August 11, 2023, 9:37am UTC](https://discuss.ray.io/t/exception-when-ray-up/11756/1 "2023-08-11T09:37:35Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- None: Just asking a question out of curiosity
- Low: It annoys or frustrates me for a moment.
- Medium: It contributes to significant difficulty to complete my task, but I can work around it.
- High: It blocks me to complete my task.  
High

I run the “ray up -y default-full.yaml”. Exception occurs When it is initializing command runner [5/7].  
Unable to deserialize `image_env` to Python object. The `image_env` is:  
Good morning centos

- Hostname …: sh-prod-aigame-gpu-1
- Release …: CentOS Linux release 7.9.2009 (Core)
- Users …: Currently 2 user(s) logged on  
===========================================================================
- Current user …: centos
- CPU usage …: 0.02, 0.05, 0.05 (1, 5, 15 min)
- Memory used …: 1809 MB / 32011 MB
- Swap in use …: 0 MB
- Processes …: 185 running
- System uptime …: 4 days 0 hours 38 minutes 16 seconds
- Disk space SYS …: remaining
- Disk space DATA …: 499G remaining  
===========================================================================

[“PATH=/home/ray/anaconda3/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin”,“CUDA\_VERSION=11.0.3”,“LD\_LIBRARY\_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64”,“NVIDIA\_VISIBLE\_DEVICES=all”,“NVIDIA\_DRIVER\_CAPABILITIES=compute,utility”,“NVIDIA\_REQUIRE\_CUDA=cuda\>=11.0 brand=tesla,driver\>=418,driver\<419 brand=tesla,driver\>=440,driver\<441 brand=tesla,driver\>=450,driver\<451”,“NCCL\_VERSION=2.7.8”,“LIBRARY\_PATH=/usr/local/cuda/lib64/stubs”,“CUDNN\_VERSION=8.0.4.30”,“TZ=America/Los\_Angeles”,“HOME=/home/ray”,“LC\_ALL=C.UTF-8”,“LANG=C.UTF-8”]  
2023-08-11 09:32:57,103 INFO node\_provider.py:116 – ClusterState: Writing cluster state: [‘172.23.1.175’, ‘172.23.0.224’]  
New status: update-failed  
!!!  
Expecting value: line 1 column 1 (char 0)  
!!!

Exception in thread Thread-1:  
Traceback (most recent call last):  
File “/usr/lib64/python3.6/threading.py”, line 916, in \_bootstrap\_inner  
self.run()  
File “/usr/local/lib/python3.6/site-packages/ray/autoscaler/\_private/updater.py”, line 153, in run  
self.do\_update()  
File “/usr/local/lib/python3.6/site-packages/ray/autoscaler/\_private/updater.py”, line 445, in do\_update  
sync\_run\_yet=True,  
File “/usr/local/lib/python3.6/site-packages/ray/autoscaler/\_private/command\_runner.py”, line 781, in run\_init  
raise e  
File “/usr/local/lib/python3.6/site-packages/ray/autoscaler/\_private/command\_runner.py”, line 772, in run\_init  
for env\_var in json.loads(image\_env):  
File “/usr/lib64/python3.6/json/ **init**.py”, line 354, in loads  
return \_default\_decoder.decode(s)  
File “/usr/lib64/python3.6/json/decoder.py”, line 339, in decode  
obj, end = self.raw\_decode(s, idx=\_w(s, 0).end())  
File “/usr/lib64/python3.6/json/decoder.py”, line 357, in raw\_decode  
raise JSONDecodeError(“Expecting value”, s, err.value) from None  
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Failed to setup head node.

---

<div class="post-metadata">

**Author:** ![jbcdnr](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jbcdnr/32/4535_2.png) [@jbcdnr](https://discuss.ray.io/u/jbcdnr)\
**Post date:** [August 17, 2023, 8:28am UTC](https://discuss.ray.io/t/exception-when-ray-up/11756/2 "2023-08-17T08:28:16Z")

</div>

I am also getting this error, any update on this?

---

<div class="post-metadata">

**Author:** ![isaacrob](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/isaacrob/32/5100_2.png) [@isaacrob](https://discuss.ray.io/u/isaacrob)\
**Post date:** [September 25, 2023, 7:03pm UTC](https://discuss.ray.io/t/exception-when-ray-up/11756/3 "2023-09-25T19:03:31Z")

</div>

I am also getting this error, any updates?

---

<div class="post-metadata">

**Author:** ![ben-directai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ben-directai/32/5102_2.png) [@ben-directai](https://discuss.ray.io/u/ben-directai)\
**Post date:** [September 25, 2023, 9:37pm UTC](https://discuss.ray.io/t/exception-when-ray-up/11756/4 "2023-09-25T21:37:37Z")

</div>

Also getting this error. Updates would be extremely helpful!

---

<div class="post-metadata">

**Author:** ![ben-directai](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ben-directai/32/5102_2.png) [@ben-directai](https://discuss.ray.io/u/ben-directai)\
**Post date:** [September 25, 2023, 11:19pm UTC](https://discuss.ray.io/t/exception-when-ray-up/11756/5 "2023-09-25T23:19:36Z")

</div>

Note that:

```auto
Good morning centos

Hostname …: sh-prod-aigame-gpu-1
Release …: CentOS Linux release 7.9.2009 (Core)
Users …: Currently 2 user(s) logged on
===========================================================================
Current user …: centos
CPU usage …: 0.02, 0.05, 0.05 (1, 5, 15 min)
Memory used …: 1809 MB / 32011 MB
Swap in use …: 0 MB
Processes …: 185 running
System uptime …: 4 days 0 hours 38 minutes 16 seconds
Disk space SYS …: remaining
Disk space DATA …: 499G remaining
===========================================================================

```

is stated to be part of `image_env`. For some reason, the inspection of the environment variables is yielding centos boilerplate. You want `image_env` to exclusively be

```auto
[“PATH=/home/ray/anaconda3/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin”,“CUDA_VERSION=11.0.3”,“LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64”,“NVIDIA_VISIBLE_DEVICES=all”,“NVIDIA_DRIVER_CAPABILITIES=compute,utility”,“NVIDIA_REQUIRE_CUDA=cuda>=11.0 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=440,driver<441 brand=tesla,driver>=450,driver<451”,“NCCL_VERSION=2.7.8”,“LIBRARY_PATH=/usr/local/cuda/lib64/stubs”,“CUDNN_VERSION=8.0.4.30”,“TZ=America/Los_Angeles”,“HOME=/home/ray”,“LC_ALL=C.UTF-8”,“LANG=C.UTF-8”]

```

I was running into a similar problem that was resolved by adding lines to the `initialization_commands` of my yaml to ensure that there were no unexpected prefixes to the JSON string.

---

<div class="post-metadata">

**Author:** ![bjornah](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/bjornah/32/5304_2.png) [@bjornah](https://discuss.ray.io/u/bjornah)\
**Post date:** [November 7, 2023, 6:13pm UTC](https://discuss.ray.io/t/exception-when-ray-up/11756/6 "2023-11-07T18:13:03Z")

</div>

I had the same issue. It turns out that my problem was that the image\_env got some error messages from the bash shell prepended to it, which caused the json decoder to fail. I resolved it by following [command line - Error message when opening the terminal "-bash: /usr/bin/tclsh: No such file or directory" - Ask Ubuntu](https://askubuntu.com/questions/1269024/error-message-when-opening-the-terminal-bash-usr-bin-tclsh-no-such-file-or)

---

<div class="post-metadata">

**Author:** ![sjperkins](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sjperkins/32/3544_2.png) [@sjperkins](https://discuss.ray.io/u/sjperkins)\
**Post date:** [March 25, 2026, 10:19am UTC](https://discuss.ray.io/t/exception-when-ray-up/11756/7 "2026-03-25T10:19:37Z")

</div>

I also ran into this on ray 2.54 with a docker cluster created via ssh. As mentioned previously in the thread, the immediate cause is unexpected output (i.e. MOTD/quota scripts in /etc/profile.d) on login. As ray is trying to parse json produced by commands such as `docker inspect`, this causes ray up to fail

It seems to be possible to work around this with `ray up ... –use-normal-shells` but the problem also manifests when the ray autoscaler tries to bring up worker nodes via ssh. This shows up deep in `monitor.log` and is took some effort to diagnose. Even more problematic is that there isn’t a way to tell the autoscaler to avoid login shells.

However, it _is_ possible to patch ray 2.54 in the ray cluster yaml setup\_command like so:

```yaml
setup_commands:
    # List of shell commands to run to set up each nodes.
    # By default Ray uses login shells (bash --login -i) and pseudo-TTYs (ssh -tt)
    # when SSHing into nodes. This causes .bash_profile output and terminal
    # progress bars to pollute stdout, breaking Ray's JSON parsing of command
    # output. These patches switch the autoscaler to plain non-interactive shells.
    # Two flags must be patched together: use_login_shells (controls -tt and
    # bash --login) and _allow_interactive (controls whether stdin is a pipe or
    # inherited from the parent). If only use_login_shells is patched, the Popen
    # path tries to close p.stdin which is None, causing SSH readiness checks to
    # never succeed.
    - |
        python3 -c "
        import pathlib;
        import ray.autoscaler._private.command_runner as cr;
        import ray.autoscaler._private.subprocess_output_util as su;
        p = pathlib.Path(cr. __file__ );
        p.write_text(p.read_text().replace('\"use_login_shells\": True', '\"use_login_shells\": False'));
        p = pathlib.Path(su. __file__ );
        p.write_text(p.read_text().replace('_allow_interactive = True', '_allow_interactive = False'))"

```