When a Ray cluster is created, it creates two types of nodes: head and workers. When I am submitting a job to Ray, is it by default using either head or worker node? Or it is always routed to worker node?
I assumed the latter and installed additional software only on worker nodes (via setup commands). But when I submitted my job it failed due to unavailability of the library. When I installed it on the head pod, everything worked.
Also, is runtime install as described pip - Installing python module within code - Stack Overflow is a viable option for Ray?
Ray tasks and actors may be scheduled on either head or workers. There are means of controlling where stuff gets executed. The Ray head is a worker with some extra global control processes running on it.
Probably installing stuff from a python script at runtime is not the best idea.
When using Ray on Kubernetes, I’d recommend a building a Docker image based on one of the rayproject/ray images. You can specify the additional stuff to install in a Dockerfile.