# \`ray debug --address=\\` results in UnicodeDecodeError on k8s cluster **URL:** \ **Category:** Dashboard, Monitoring & Debugging\ **Created:** [February 2, 2023, 3:02pm UTC](https://discuss.ray.io/t/ray-debug-address-ip-port-results-in-unicodedecodeerror-on-k8s-cluster/9208 "2023-02-02T15:02:53Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Zac](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zac/32/3822_2.png) [@Zac](https://discuss.ray.io/u/Zac)\ **Post date:** [February 2, 2023, 3:02pm UTC](https://discuss.ray.io/t/ray-debug-address-ip-port-results-in-unicodedecodeerror-on-k8s-cluster/9208/1 "2023-02-02T15:02:53Z")
- Medium: It contributes to significant difficulty to complete my task, but I can work around it. Ray2.2 cluster on k8s - `ray debug --address=` results in UnicodeDecodeError on k8s cluster - works locally Steps to reproduce: 1. copy simple\_task.py code into a file of the same name 2. terminal 1: `python simple_task.py` 3. terminal 2: `ray debug --address=` Using the example from the Ray docs: simple\_task.py ```auto import ray ray.init("ray://ray-examples-head-svc:10001") @ray.remote def f(x): breakpoint() return x * x futures = [f.remote(i) for i in range(2)] print(ray.get(futures)) ``` terminal 1 ```auto (freenome-ray-examples-py3.10) ➜ ray-examples git:(rpx-3397-add-sens-spec) ✗ python ray_examples/simple_task.py (f pid=4037, ip=10.192.6.5) RemotePdb session open at 10.192.6.5:44835, use 'ray debug' to connect... (f pid=4036, ip=10.192.6.5) RemotePdb session open at 10.192.6.5:38031, use 'ray debug' to connect... ``` terminal 2 ```auto (freenome-ray-examples-py3.10) ➜ ray-examples git:(rpx-3397-add-sens-spec) ✗ ray debug --address=10.192.6.5:38031 2023-02-02 14:58:05,783 INFO scripts.py:206 -- Connecting to Ray instance at 10.192.6.5:38031. 2023-02-02 14:58:05,783 INFO worker.py:1352 -- Connecting to existing Ray cluster at address: 10.192.6.5:38031... ``` terminal 1 ```auto (freenome-ray-examples-py3.10) ➜ ray-examples git:(rpx-3397-add-sens-spec) ✗ python ray_examples/simple_task.py (f pid=4037, ip=10.192.6.5) RemotePdb session open at 10.192.6.5:44835, use 'ray debug' to connect... (f pid=4036, ip=10.192.6.5) RemotePdb session open at 10.192.6.5:38031, use 'ray debug' to connect... Traceback (most recent call last): File "/home/zcarrico/ray-examples/ray_examples/simple_task.py", line 13, in print(ray.get(futures)) File "/home/zcarrico/.virtualenvs/freenome-ray-examples-QntpqrFU-py3.10/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper return getattr(ray, func. __name__ )(*args, **kwargs) File "/home/zcarrico/.virtualenvs/freenome-ray-examples-QntpqrFU-py3.10/lib/python3.10/site-packages/ray/util/client/api.py", line 42, in get return self.worker.get(vals, timeout=timeout) File "/home/zcarrico/.virtualenvs/freenome-ray-examples-QntpqrFU-py3.10/lib/python3.10/site-packages/ray/util/client/worker.py", line 434, in get res = self._get(to_get, op_timeout) File "/home/zcarrico/.virtualenvs/freenome-ray-examples-QntpqrFU-py3.10/lib/python3.10/site-packages/ray/util/client/worker.py", line 462, in _get raise err types.RayTaskError(UnicodeDecodeError): ray::f() (pid=4036, ip=10.192.6.5) File "/home/zcarrico/ray-examples/ray_examples/simple_task.py", line 9, in f File "/home/zcarrico/ray-examples/ray_examples/simple_task.py", line 9, in f File "/fn/lib/python3.10/bdb.py", line 90, in trace_dispatch return self.dispatch_line(frame) File "/fn/lib/python3.10/bdb.py", line 114, in dispatch_line self.user_line(frame) File "/fn/lib/python3.10/pdb.py", line 262, in user_line self.interaction(frame, None) File "/fn/lib/python3.10/pdb.py", line 357, in interaction self._cmdloop() File "/fn/lib/python3.10/pdb.py", line 322, in _cmdloop self.cmdloop() File "/fn/lib/python3.10/cmd.py", line 132, in cmdloop line = self.stdin.readline() File "/fn/lib/python3.10/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 49: invalid start byte ``` Anyone have any questions, ideas what’s going on here, or something to try? ---
**Author:** ![Huaiwei\_Sun](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/huaiwei_sun/32/1316_2.png) [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)\ **Post date:** [February 3, 2023, 2:19am UTC](https://discuss.ray.io/t/ray-debug-address-ip-port-results-in-unicodedecodeerror-on-k8s-cluster/9208/2 "2023-02-03T02:19:10Z")
@sangcho @ClarenceNg for thoughts ---
**Author:** ![Zac](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zac/32/3822_2.png) [@Zac](https://discuss.ray.io/u/Zac)\ **Post date:** [February 13, 2023, 5:43pm UTC](https://discuss.ray.io/t/ray-debug-address-ip-port-results-in-unicodedecodeerror-on-k8s-cluster/9208/3 "2023-02-13T17:43:13Z")
@sangcho @ClarenceNg , following up on the above comment, did you have any thoughts about things to try to fix this? Ray’s debugging features were one of its more attractive features for us using it, so hopefully there’s a way to get this to work on k8s 🙂 ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [February 13, 2023, 11:13pm UTC](https://discuss.ray.io/t/ray-debug-address-ip-port-results-in-unicodedecodeerror-on-k8s-cluster/9208/4 "2023-02-13T23:13:50Z")
Are you using a cluster? Wonder if it is because you don’t use [Ray Debugger — Ray 3.0.0.dev0](https://docs.ray.io/en/master/ray-observability/ray-debugging.html#running-on-a-cluster)? ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [February 13, 2023, 11:13pm UTC](https://discuss.ray.io/t/ray-debug-address-ip-port-results-in-unicodedecodeerror-on-k8s-cluster/9208/5 "2023-02-13T23:13:58Z")
If it is not the case, please create an issue! ---
**Author:** ![Zac](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zac/32/3822_2.png) [@Zac](https://discuss.ray.io/u/Zac)\ **Post date:** [February 14, 2023, 1:09am UTC](https://discuss.ray.io/t/ray-debug-address-ip-port-results-in-unicodedecodeerror-on-k8s-cluster/9208/6 "2023-02-14T01:09:51Z")
Thank you @sangcho. Yes, using a cluster, and using ```auto rayStartParams: ray-debugger-external: "true" ``` in the CRD ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [February 15, 2023, 11:49pm UTC](https://discuss.ray.io/t/ray-debug-address-ip-port-results-in-unicodedecodeerror-on-k8s-cluster/9208/7 "2023-02-15T23:49:33Z")
What’s the address you are using? Do you use `ray://ray-examples-head-svc:10001`? Also is terminal 1 & 2 in the same machine? ---
**Author:** ![Zac](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zac/32/3822_2.png) [@Zac](https://discuss.ray.io/u/Zac)\ **Post date:** [February 16, 2023, 12:38am UTC](https://discuss.ray.io/t/ray-debug-address-ip-port-results-in-unicodedecodeerror-on-k8s-cluster/9208/8 "2023-02-16T00:38:52Z")
Yes to both questions ---
**Author:** ![Zac](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/zac/32/3822_2.png) [@Zac](https://discuss.ray.io/u/Zac)\ **Post date:** [February 17, 2023, 7:46pm UTC](https://discuss.ray.io/t/ray-debug-address-ip-port-results-in-unicodedecodeerror-on-k8s-cluster/9208/9 "2023-02-17T19:46:12Z")
> > > \### Search before asking > > \- \[X\] I searched the \[issues\](https://github.com/ray-p…roject/kuberay/issues) and found no similar issues. > > > \### KubeRay Component > > ray-operator > > \### What happened + What you expected to happen > > Observation > \* \`ray debug --address=\\` results in UnicodeDecodeError on kubernetes cluster > > Expectation > \* \`ray debug --address=\\` would enable remote debugging the way it is shown to in the \[Ray documentation\](https://docs.ray.io/en/latest/ray-observability/ray-debugging.html) > > Useful Information > \* ray debug on a local ray.init() session works > > \### Reproduction script > > simple\_task.py > \`\`\` > import ray > ray.init("ray://ray-examples-head-svc:10001") > > @ray.remote > def f(x): > breakpoint() > return x \* x > > futures = \[f.remote(i) for i in range(2)\] > print(ray.get(futures)) > \`\`\` > > terminal 1 > \`\`\` > ~ python ray\_examples/simple\_task.py > (f pid=4037, ip=10.192.6.5) RemotePdb session open at 10.192.6.5:44835, use 'ray debug' to connect... > (f pid=4036, ip=10.192.6.5) RemotePdb session open at 10.192.6.5:38031, use 'ray debug' to connect... > \`\`\` > > terminal 2 > \`\`\` > ~ ray debug --address=10.192.6.5:38031 > 2023-02-02 14:58:05,783 INFO scripts.py:206 -- Connecting to Ray instance at 10.192.6.5:38031. > 2023-02-02 14:58:05,783 INFO worker.py:1352 -- Connecting to existing Ray cluster at address: 10.192.6.5:38031... > \`\`\` > > after running \`ray debug --address=10.192.6.5:38031\` terminal 1 shows this > \`\`\` > ~ python ray\_examples/simple\_task.py > (f pid=4037, ip=10.192.6.5) RemotePdb session open at 10.192.6.5:44835, use 'ray debug' to connect... > (f pid=4036, ip=10.192.6.5) RemotePdb session open at 10.192.6.5:38031, use 'ray debug' to connect... > Traceback (most recent call last): > File "/home/zcarrico/ray-examples/ray\_examples/simple\_task.py", line 13, in \ > print(ray.get(futures)) > File "/home/zcarrico/.virtualenvs/freenome-ray-examples-QntpqrFU-py3.10/lib/python3.10/site-packages/ray/\_private/client\_mode\_hook.py", line 104, in wrapper > return getattr(ray, func.\_\_name\_\_)(\*args, \*\*kwargs) > File "/home/zcarrico/.virtualenvs/freenome-ray-examples-QntpqrFU-py3.10/lib/python3.10/site-packages/ray/util/client/api.py", line 42, in get > return self.worker.get(vals, timeout=timeout) > File "/home/zcarrico/.virtualenvs/freenome-ray-examples-QntpqrFU-py3.10/lib/python3.10/site-packages/ray/util/client/worker.py", line 434, in get > res = self.\_get(to\_get, op\_timeout) > File "/home/zcarrico/.virtualenvs/freenome-ray-examples-QntpqrFU-py3.10/lib/python3.10/site-packages/ray/util/client/worker.py", line 462, in \_get > raise err > types.RayTaskError(UnicodeDecodeError): ray::f() (pid=4036, ip=10.192.6.5) > File "/home/zcarrico/ray-examples/ray\_examples/simple\_task.py", line 9, in f > File "/home/zcarrico/ray-examples/ray\_examples/simple\_task.py", line 9, in f > File "/fn/lib/python3.10/bdb.py", line 90, in trace\_dispatch > return self.dispatch\_line(frame) > File "/fn/lib/python3.10/bdb.py", line 114, in dispatch\_line > self.user\_line(frame) > File "/fn/lib/python3.10/pdb.py", line 262, in user\_line > self.interaction(frame, None) > File "/fn/lib/python3.10/pdb.py", line 357, in interaction > self.\_cmdloop() > File "/fn/lib/python3.10/pdb.py", line 322, in \_cmdloop > self.cmdloop() > File "/fn/lib/python3.10/cmd.py", line 132, in cmdloop > line = self.stdin.readline() > File "/fn/lib/python3.10/codecs.py", line 322, in decode > (result, consumed) = self.\_buffer\_decode(data, self.errors, final) > UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 49: invalid start byte > > \`\`\` > > \### Anything else > > \_No response\_ > > \### Are you willing to submit a PR? > > \- \[X\] Yes I am willing to submit a PR!