Upgrading from ver 2.4.0 to ver 2.8.0 breaks tracing hook

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I’ve tried upgrading the ray version on my system and i got the following error:

File “/usr/local/lib/python3.10/dist-packages/ray/util/client/common.py”, line 544, in remote
return return_refs(ray.call_remote(self, *args, **kwargs))
File “/usr/local/lib/python3.10/dist-packages/ray/util/client/api.py”, line 100, in call_remote
return self.worker.call_remote(instance, *args, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/ray/util/client/worker.py”, line 558, in call_remote
task.data = dumps_from_client((args, kwargs), self._client_id)
File “/usr/local/lib/python3.10/dist-packages/ray/util/client/client_pickler.py”, line 159, in dumps_from_client
File “/usr/local/lib/python3.10/dist-packages/ray/cloudpickle/cloudpickle_fast.py”, line 733, in dump
return Pickler.dump(self, obj)
File “/usr/local/lib/python3.10/dist-packages/ray/util/client/client_pickler.py”, line 84, in persistent_id
File “/usr/local/lib/python3.10/dist-packages/ray/util/client/common.py”, line 136, in id
return self.binary()
File “/usr/local/lib/python3.10/dist-packages/ray/util/client/common.py”, line 115, in binary
File “/usr/local/lib/python3.10/dist-packages/ray/util/client/common.py”, line 192, in _wait_for_id
File “/usr/lib/python3.10/concurrent/futures/_base.py”, line 451, in result
return self.__get_result()
File “/usr/lib/python3.10/concurrent/futures/_base.py”, line 403, in __get_result
raise self._exception
TypeError: got an unexpected keyword argument ‘_ray_trace_ctx’

after a bit of digging and trial i found that removing the hook call from the start solved that problem.

i tried going over the hook according to the current docs and it still seems like it should work.
i know tracing is currently an experimental feature but what could change for it to break in this way ?
the hook worked in 2.4.0

@sangcho @rickyyx could you help on this one? Seems related to observability.

just updating,
i tried commenting out the hook content and it still didn’t work,
the only thing that works was removing the call…

can you create an issue with a repro script?

the trace feature still should work yes.

we upgraded our code in areas regarding opentelemetry and the issue disappeared, not sure what caused it. so its working now