How severe does this issue affect your experience of using Ray?
- Medium: It contributes to significant difficulty to complete my task, but I can work around it.
Problem
I am using Ray on Kubernetes and was running into some problems with the number of times I can successfully do ray.init(...)
. I see that ray.init(...)
returns something like this:
ClientContext(dashboard_url='10.1.4.142:8265', python_version='3.9.10', ray_version='1.9.2',
ray_commit='ef593fe5d3c864836b80ae77be32635cef42b537', protocol_version='2021-09-22',
_num_clients=3, _context_to_restore=<ray.util.client._ClientContext object at 0x7f53bca77700>)
I am wondering if there is a max limit for _num_clients
here? I cannot seem to go above 50.
Update
We found from the source code that this is why we cannot exceed 50:
CLIENT_SERVER_MAX_THREADS = float(
os.getenv("RAY_CLIENT_SERVER_MAX_THREADS", 100))
threshold = int(CLIENT_SERVER_MAX_THREADS / 2)
if self.num_clients >= threshold:
logger.warning(
f"[Data Servicer]: Num clients {self.num_clients} "
f"has reached the threshold {threshold}. "
f"Rejecting client: {client_id}. ")
Any reason they are set to this amount and is it safe for me to change the env variable here?