Ray launches processes with wrong virtual environment

Ray Clusters
1

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

From the command line I call “ray start --address=address_of_slurm_ray_head_node” to connect a local machine to a slurm ray cluster.
I then check the processes started on the local machine associated with ray and I see 5 python processes. The virtual environment being used by those processes is not the same one as the virtual environment used in the shell or my default. Is there a config file ray could refer to that sets the venv it is using when launching processes?

2

the path to the python used is hardcoded in the first line of $VENV/bin/ray. That path doesn’t have to match the venv one is currently using.