Trying to find examples of submitting a k8s job that runs a custom ray program using multiprocessing. Getting ModuleNotFoundError on ‘mycustompackage’ due to package not on worker nodes. Seeing mentions of file mounts but examples would be appreciated. Also how do we solve for if 'mycustompackage` is updated in future?
Hi! For now, the best officially available solution is to build your dependencies into the image you’re using.
In the near future, the preferred way to do this will be via a feature called runtime environments
An experimental version of this functionality exists in Ray 1.4.0 but is not yet documented.
I recommend bugging the maintainers of runtime envs on this thread: Dependency management with ray for remote task execution - #3 by asm582
Thanks Dmitri!
We have the dependency in the image for the k8s job but how do we get that propagated to worker nodes. Or do you mean we add the dependencies to the ray image we use in the cluster?
yes, exactly – until runtime envs are available, client and server ends for Ray client must have matching dependencies installed, so you’d need to add the dependencies to the image used in the cluster