We have a ray cluster running but the required packages for a particular ML project may not be installed there already.
Now, I want to run a hyper parameter tuning experiment using this Ray cluster. However, before I make a call to
tune.run, I would need to prepare the cluster so that all the prerequisites and the dependencies are installed.
I was wondering what would be a good way to achieve that? We use
poetry for package management but
pip is also an option. I am having trouble figuring out how to get my dependencies on all the
ray nodes so that the tuning experiments can run fine.
EDIT So runtime environment can be specified directly in the ray init command. I think this will suffice for now.