ModuleNotFoundError for custom env import when num_workers > 0

@klausk55 I think I run into a similar problem.

Remote workers would fail to deserialize data from modules imported from within the project. In my case, it was due to the deserialization process being attempted from the workers’ initial directory instead of the working dir. I could fix this issue by running a worker_process_setup_hook function before submitting tasks to the workers. This function would perform an os.chdir so that the (relative) imports work well:

ray.init(runtime_env={"worker_process_setup_hook": logging_setup_func})

I got the idea from Execute function on each worker process in the cluster

I didn’t find a explicit solution to my problem, so maybe this will help others as well.