Adding a node to a cluster while running tune experiments

1. Severity of the issue: (select one)
None: I’m just curious or want clarification.
Low: Annoying but doesn’t hinder my work.
Medium: Significantly affects my productivity but can find a workaround.
High: Completely blocks me.

2. Environment:

  • Ray version: 2.31.0
  • Python version: 3.10
  • OS: Ubuntu 22.04
  • Cloud/Infrastructure: on-premise
  • Other libs/tools (if relevant): N/A

I manually created a cluster using on-premise nodes to run a tuning experiment with Ray Tune. If a node is manually added to the cluster with ray start --address='<ADDRESS>':

  • Will the on-going tuning experiment start using the new node?
  • Will the tuning experiment crash?

I did not test it yet, as I am currently running a large experiment and would not like to have issues. However, I have a machine that became available and I would like to use it in the current experiment. I am asking as other people may end up in the same situation.