### Author:  [@Francis\_Boon](https://discuss.ray.io/u/Francis_Boon)
#### Post date: [September 5, 2024, 1:12pm UTC](https://discuss.ray.io/t/debugging-inside-cv-wait-for/15732/1 "2024-09-05T13:12:43Z")
**How severe does this issue affect your experience of using Ray?**
- ~~High: It blocks me to complete my task.~~ Downgraded…now have a workaround
There is a task that I send to a ray cluster via `ray.remote(callable).remote(**kwargs)`
All works fine until I do an import from a module within the module where this callable resides.
If I do the import then I get this Traceback:
```auto
File "/home/fboon/code/app/ray/python/ray/remote_function.py", line 250, in remote
return func_cls._remote(args=args, kwargs=kwargs, **updated_options)
File "/home/fboon/code/app/ray/python/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/home/fboon/code/app/ray/python/ray/util/tracing/tracing_helper.py", line 310, in _invocation_remote_span
return method(self, args, kwargs, *_args, **_kwargs)
File "/home/fboon/code/app/ray/python/ray/remote_function.py", line 272, in _remote
return client_mode_convert_function(self, args, kwargs, **task_options)
File "/home/fboon/code/app/ray/python/ray/_private/client_mode_hook.py", line 164, in client_mode_convert_function
return client_func._remote(in_args, in_kwargs, **kwargs)
File "/home/fboon/code/app/ray/python/ray/util/client/common.py", line 308, in _remote
return self.options(**option_args).remote(*args,**kwargs)
File "/home/fboon/code/app/ray/python/ray/util/client/common.py", line 599, in remote
return return_refs(ray.call_remote(self, *args, **kwargs))
File "/home/fboon/code/app/ray/python/ray/util/client/api.py", line 100, in call_remote
return self.worker.call_remote(instance, *args, **kwargs)
File "/home/fboon/code/app/ray/python/ray/util/client/worker.py", line 555, in call_remote
task = instance._prepare_client_task()
File "/home/fboon/code/app/ray/python/ray/util/client/common.py", line 605, in _prepare_client_task
task = self._remote_stub._prepare_client_task()
File "/home/fboon/code/app/ray/python/ray/util/client/common.py", line 334, in _prepare_client_task
self._ensure_ref()
File "/home/fboon/code/app/ray/python/ray/util/client/common.py", line 329, in _ensure_ref
self._ref = ray.worker._put_pickled(
File "/home/fboon/code/app/ray/python/ray/util/client/worker.py", line 506, in _put_pickled
resp = self.data_client.PutObject(req)
File "/home/fboon/code/app/ray/python/ray/util/client/dataclient.py", line 568, in PutObject
resp = self._blocking_send(datareq)
File "/home/fboon/code/app/ray/python/ray/util/client/dataclient.py", line 458, in _blocking_send
self._check_shutdown()
File "/home/fboon/code/app/ray/python/ray/util/client/dataclient.py", line 511, in _check_shutdown
raise ConnectionError(msg)
ConnectionError: Request can't be sent because the Ray client has already been disconnected due to an error. Last exception: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.NOT_FOUND
details = "Attempted to reconnect a session that has already been cleaned up"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Attempted to reconnect a session that has already been cleaned up", grpc_status:5, created_time:"2024-09-05T14:03:36.081482247+01:00"}"
```
The error happens even if I put the import within a try/except
I was initially using 2.9.3 but I see the exact same issue with 2.35.0.
I did an editable install of a local version of 2.35.0 to add some debugging, but it hasn’t helped isolate beyond that it happens in this line:
>
So within `cv.wait_for()`
I can’t see how to debug inside that.
`RAY_PDB=1` isn’t opening a debugger on this crash
Pointers on how to debug very welcome!
---
### Author:  [@Huaiwei\_Sun](https://discuss.ray.io/u/Huaiwei_Sun)
#### Post date: [September 5, 2024, 5:02pm UTC](https://discuss.ray.io/t/debugging-inside-cv-wait-for/15732/2 "2024-09-05T17:02:15Z")
Seems a ray core question. Moving it to ray core category.
You could also try the vscode debugger integration [Easily Debug Ray Applications with Ray Distributed Debugger](https://www.anyscale.com/blog/ray-distributed-debugger) for debugging.
---
### Author:  [@Francis\_Boon](https://discuss.ray.io/u/Francis_Boon)
#### Post date: [September 5, 2024, 5:36pm UTC](https://discuss.ray.io/t/debugging-inside-cv-wait-for/15732/3 "2024-09-05T17:36:35Z")
> [@Huaiwei\_Sun](#):
>
> You could also try the vscode debugger integration
Thanks, a handy thing to know about for sure, however not useful here as I’ve tried a `breakpoint()`…this is outside that…
---
### Author:  [@Francis\_Boon](https://discuss.ray.io/u/Francis_Boon)
#### Post date: [September 5, 2024, 5:46pm UTC](https://discuss.ray.io/t/debugging-inside-cv-wait-for/15732/4 "2024-09-05T17:46:48Z")
Intrerestingly I can do the import inside the function just fine…unfortunately that’s no use in this case as the function is to add decorators to the function.
---
### Author:  [@Francis\_Boon](https://discuss.ray.io/u/Francis_Boon)
#### Post date: [September 5, 2024, 6:46pm UTC](https://discuss.ray.io/t/debugging-inside-cv-wait-for/15732/5 "2024-09-05T18:46:51Z")
This may be related…this is Python inside the Docker image which runs the cluster nodes:
```auto
# python
>>> from hamilton import function_modifiers
>>> import ray
>>> ray.init()
ERROR: Flag 'grpc_experiments' was defined more than once but with differing types. Defined in files 'home/coder/grpc/src/core/lib/config/config_vars.cc' and 'src/core/lib/config/config_vars.cc'.
#
```
i.e. Python just exits with no message
---
### Author:  [@Francis\_Boon](https://discuss.ray.io/u/Francis_Boon)
#### Post date: [September 5, 2024, 6:57pm UTC](https://discuss.ray.io/t/debugging-inside-cv-wait-for/15732/6 "2024-09-05T18:57:47Z")
Tried `pip uninstall xgboost` (I don’t need it, it came with the base image)
See this link for xgboost hint:
>
>
> It seems like the xgboost Python package now comes with grpc statically linked i…n.
> This causes a clash in my environment with the \`grpcio\` Python package:
>
> \`\`\`
> Python 3.10.13 (main, Sep 5 2023, 06:03:44) \[GCC 11.4.0\] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> \>\>\> import grpc
> \>\>\> import xgboost
> ERROR: Flag 'grpc\_experiments' was defined more than once but with differing types. Defined in files 'src/core/lib/config/config\_vars.cc' and 'home/coder/grpc/src/core/lib/config/config\_vars.cc'.
> \`\`\`
>
> This is with the following grpc packages installed:
> \`\`\`
> pip3 list | grep grpc
> grpcio 1.62.1
> grpcio-status 1.48.2
> grpcio-tools 1.48.2
> opentelemetry-exporter-otlp-proto-grpc 1.24.0
> opentelemetry-instrumentation-grpc 0.45b0
> \`\`\`
>
>
> This is the same issue as was raised before in grpc: https://github.com/grpc/grpc/issues/36648 It's caused by having two versions of the same symbols loaded at runtime.
>
> I am not familiar with the xgboost codebase, but it looks like grpc is only used for federated learning https://github.com/dmlc/xgboost/blob/513d7a7d84678fc4e75c18899fe52263258f0290/doc/build.rst#federated-learning. Could this be disabled in the xgboost pip distribution? Perhaps it oculd be deployed as a pip "extra" https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-extras
---
### Author:  [@Francis\_Boon](https://discuss.ray.io/u/Francis_Boon)
#### Post date: [September 5, 2024, 7:00pm UTC](https://discuss.ray.io/t/debugging-inside-cv-wait-for/15732/7 "2024-09-05T19:00:58Z")
I still think that this is some problem with the base image: `nvcr.io/nvidia/pytorch:24.07-py3`
Although why importing this module would trigger that I don’t know
---
### Author:  [@Francis\_Boon](https://discuss.ray.io/u/Francis_Boon)
#### Post date: [September 5, 2024, 7:01pm UTC](https://discuss.ray.io/t/debugging-inside-cv-wait-for/15732/8 "2024-09-05T19:01:58Z")
This is the module being imported:
> **[hamilton/hamilton/function\_modifiers at main · DAGWorks-Inc/hamilton](https://github.com/DAGWorks-Inc/hamilton/tree/main/hamilton/function_modifiers)**
>
> Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does. - DAGWorks-Inc/h...
---
### Author:  [@Francis\_Boon](https://discuss.ray.io/u/Francis_Boon)
#### Post date: [September 5, 2024, 7:13pm UTC](https://discuss.ray.io/t/debugging-inside-cv-wait-for/15732/9 "2024-09-05T19:13:00Z")
OK, all works if I disable hamilton plugins before importing it:
```auto
>>> import os
>>> os.environ["HAMILTON_AUTOLOAD_EXTENSIONS"] = "0"
```
---
### Author:  [@Francis\_Boon](https://discuss.ray.io/u/Francis_Boon)
#### Post date: [September 5, 2024, 7:26pm UTC](https://discuss.ray.io/t/debugging-inside-cv-wait-for/15732/10 "2024-09-05T19:26:27Z")
I don’t know if Ray can catch this error & work around it, or at least make it display in the logs…