Hello, I have a question regarding dependency management: if a user spawns a task on the ray remote cluster, how are the binaries required to run the task copied from the local laptop to the remote ray cluster? is there any documentation from a dependency management perspective for remote tasks?
Right now, all ray nodes should be homogeneous, which means that you need them running in the same environment.
For script functions, it’ll be picked to run in remote node, but for other dependency it’s not done automatically, which means you need to copy it there by yourself.
We are putting an effort to let user specify a runtime environment for each job so this can be done automatically. It’s not ready yet, but should be ready to use within April. You can check it later if you are interested in it.
Thank you @yic, April is not too far! I am looking forward to trying out this feature.
Hello @yic , can you please share if dependency management with conda is supported on ray?
Hi @asm582 sorry for the delay!
The project got delayed. It’ll come with the 1.4.0 release which is planed for the end of this month (or early next month).
This thread seems like a good place to chat about runtime envs and their relationship to Ray deployments on k8s – lots of questions from k8s users on dependency management recently.
cc @yic @eoakes @architkulkarni
Hello, is this feature available in ray 1.4?
Note, this is still an experimental feature, and welcome any feedback. cc @architkulkarni
Question with respect to dependency management of ray operator and ray nodes, do we have some sort of compatibility check if operator image is compatible with ray nodes when users build images from ray source?
There’s no explicit check — but as of Ray 1.2.0 the interfaces are designed for cross-compatibility of scaling functionality with core ray functionality.
You’d have to modify very specific code to break compatibility — I think the relevant protobufs are labelled with “do not change” warnings.