Hello, I have a question regarding dependency management: if a user spawns a task on the ray remote cluster, how are the binaries required to run the task copied from the local laptop to the remote ray cluster? is there any documentation from a dependency management perspective for remote tasks?
Right now, all ray nodes should be homogeneous, which means that you need them running in the same environment.
For script functions, it’ll be picked to run in remote node, but for other dependency it’s not done automatically, which means you need to copy it there by yourself.
We are putting an effort to let user specify a runtime environment for each job so this can be done automatically. It’s not ready yet, but should be ready to use within April. You can check it later if you are interested in it.
Thank you @yic, April is not too far! I am looking forward to trying out this feature.
Hello @yic , can you please share if dependency management with conda is supported on ray?
Hi @asm582 sorry for the delay!
The project got delayed. It’ll come with the 1.4.0 release which is planed for the end of this month (or early next month).
This thread seems like a good place to chat about runtime envs and their relationship to Ray deployments on k8s – lots of questions from k8s users on dependency management recently.
cc @yic @eoakes @architkulkarni
Hello, is this feature available in ray 1.4?