Problem with worker_process_setup_hook

Ray Core
1

I’m trying to use the new worker_process_setup_hook and for me it fails with:

  File "/home/milad/micromamba/envs/dev/lib/python3.8/site-packages/ray/actor.py", line 645, in remote
    return actor_cls._remote(args=args, kwargs=kwargs, **updated_options)
  File "/home/milad/micromamba/envs/dev/lib/python3.8/site-packages/ray/util/tracing/tracing_helper.py", line 385, in _invocation_actor_class_remote_span
    return method(self, args, kwargs, *_args, **_kwargs)
  File "/home/milad/micromamba/envs/dev/lib/python3.8/site-packages/ray/actor.py", line 929, in _remote
    serialized_runtime_env_info = get_runtime_env_info(
  File "/home/milad/micromamba/envs/dev/lib/python3.8/site-packages/ray/_private/utils.py", line 1591, in get_runtime_env_info
    proto_runtime_env_info.serialized_runtime_env = runtime_env.serialize()
  File "/home/milad/micromamba/envs/dev/lib/python3.8/site-packages/ray/runtime_env/runtime_env.py", line 389, in serialize
    return json.dumps(
  File "/home/milad/micromamba/envs/dev/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/home/milad/micromamba/envs/dev/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/milad/micromamba/envs/dev/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/milad/micromamba/envs/dev/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type function is not JSON serializable

I’ve tried various functions but it even fails with:

def foo():
  pass

Is this a known issue? Happens on both 2.5.1 and 2.6.1

1 Like
2

actually, seems to work when I pass it in ray.init() but not when I pass it in worker options. Is this intentional?

3

cc: @sangcho could you take a quick look? Thanks.

4

Yeah this is only working with ray.init right now. We are planning to support for worker options too, but I think it won’t be merged by Ray 2.8. I will raise a better error message in Ray 2.7.

5

Yeah this is only working with ray.init right now. We are planning to support for worker options too, but I think it won’t be merged by Ray 2.8. I will raise a better error message in Ray 2.7.

FYI, it also appears to not work if local_mode is enabled in ray.init, so debugging the worker(s) doesn’t work but it will run when local_mode is off and I’m not trying to debug.

6

local_mode is actually deprecated https://github.com/ray-project/ray/blob/cae62298ae59b92698e441b1be1f85da38ee47d0/python/ray/_private/worker.py#L1233 from Ray 2.0, so it is expected the feature is not working with it now.