Ray.init() max num clients

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.

Problem
I am using Ray on Kubernetes and was running into some problems with the number of times I can successfully do ray.init(...). I see that ray.init(...) returns something like this:

ClientContext(dashboard_url='10.1.4.142:8265', python_version='3.9.10', ray_version='1.9.2', 
ray_commit='ef593fe5d3c864836b80ae77be32635cef42b537', protocol_version='2021-09-22', 
_num_clients=3, _context_to_restore=<ray.util.client._ClientContext object at 0x7f53bca77700>)

I am wondering if there is a max limit for _num_clients here? I cannot seem to go above 50.

Update
We found from the source code that this is why we cannot exceed 50:

CLIENT_SERVER_MAX_THREADS = float(
    os.getenv("RAY_CLIENT_SERVER_MAX_THREADS", 100))
threshold = int(CLIENT_SERVER_MAX_THREADS / 2)
if self.num_clients >= threshold:
  logger.warning(
    f"[Data Servicer]: Num clients {self.num_clients} "
    f"has reached the threshold {threshold}. "
    f"Rejecting client: {client_id}. ")

Any reason they are set to this amount and is it safe for me to change the env variable here?

Yes it’s safe to change the env variable as long as your container has sufficient resources.