How to set working directory for each worker node?

Ray Core
1
  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

I am running ray cluster using cluster.yaml file. In a cluster I have 3 local worker nodes. Now, my project pipeline has different path in all three worker node. So, how can I set different working directory for each worker node?

Note:

  • I don’t want to send files from head node to worker node. When I submit job, head node should run the command in specific working directory in worker node.
2

cc @architkulkarni, @Dmitri Is it possible?

3

Hi @shyampatel , unfortunately Ray doesn’t support this, please feel free to file a feature request on the Ray Github if this is important to you. There is ongoing work to support specifying an existing working directory without sending files between nodes, but we don’t have plans to specify a different directory for each node.

4

Is this functionality added in nightly/beta version?

5

This functionality isn’t available in the nightlies yet unfortunately. If you’re only interested in doing Python imports from the directory, one workaround is to use the env_vars field of runtime_env to set the environment variable PYTHONPATH to the desired directory.