We have a ray cluster running but the required packages for a particular ML project may not be installed there already.
Now, I want to run a hyper parameter tuning experiment using this Ray cluster. However, before I make a call to tune.run, I would need to prepare the cluster so that all the prerequisites and the dependencies are installed.
I was wondering what would be a good way to achieve that? We use poetry for package management but pip is also an option. I am having trouble figuring out how to get my dependencies on all the ray nodes so that the tuning experiments can run fine.
EDIT So runtime environment can be specified directly in the ray init command. I think this will suffice for now.