# Error Scaling Ray Serve to 2 Replicas **URL:** \ **Category:** Ray Serve\ **Created:** [August 9, 2021, 10:27pm UTC](https://discuss.ray.io/t/error-scaling-ray-serve-to-2-replicas/3181 "2021-08-09T22:27:17Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![omnific9](https://avatars.discourse-cdn.com/v4/letter/o/3d9bf3/32.png) [@omnific9](https://discuss.ray.io/u/omnific9)\ **Post date:** [August 9, 2021, 10:27pm UTC](https://discuss.ray.io/t/error-scaling-ray-serve-to-2-replicas/3181/1 "2021-08-09T22:27:17Z")
Hello community. I’m new to both Ray and Kubernetes, and yet here I am, trying to set up a Ray Serve product on Kubernetes. I managed to wade through some of the learning challenges and set up a Kubernetes cluster using Minikube locally, with 2 nodes. I then set up a simple Ray Serve deployment following this quickstart: [Deploying Ray Serve — Ray v2.0.0.dev0](https://docs.ray.io/en/master/serve/deployment.html) and it worked well with just 1 replica. When I added “num\_replicas=2” to the serve.deployment clause, all of a sudden I’m getting this message: WARNING backend\_state.py:928 – Backend ‘hello’ has 1 replicas that have taken more than 30s to start up. This may be caused by waiting for the cluster to auto-scale or because the constructor is slow. Resources required for each replica: {‘CPU’: 1}, resources available: {} I couldn’t find any other error messages but maybe there are logs that I’m not aware of? I tried running another Minikube example and was able to spin up 2 replicas of an API endpoint on the 2 nodes, so it looks like other programs are able to function properly on my cluster. I’m confused with this message and will appreciate if you could help me understand what’s going on. Thanks! ---
**Author:** ![eoakes](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eoakes/32/7825_2.png) [@eoakes](https://discuss.ray.io/u/eoakes)\ **Post date:** [August 10, 2021, 4:33pm UTC](https://discuss.ray.io/t/error-scaling-ray-serve-to-2-replicas/3181/2 "2021-08-10T16:33:57Z")
@omnific9 this error message means that Serve is trying to create the requested replica but there aren’t currently enough resources in the cluster (given that you’re running on k8s, this means there aren’t enough Pods created). Assuming you’re using the Kubernetes operator / helm chart, the autoscaler should be creating more Pods to satisfy this request but it seems like it might not be. One thing you can do to figure out what’s going on is run `ray status` on the head node, that should give you a periodic log from the autoscaler with what it’s trying to do. You can also see the autoscaler logs in `/tmp/ray/session_latest/logs/autoscaler.{log,err}`. If you post the output of either of those, I should be able to help more! ---
**Author:** ![omnific9](https://avatars.discourse-cdn.com/v4/letter/o/3d9bf3/32.png) [@omnific9](https://discuss.ray.io/u/omnific9)\ **Post date:** [August 10, 2021, 5:09pm UTC](https://discuss.ray.io/t/error-scaling-ray-serve-to-2-replicas/3181/3 "2021-08-10T17:09:43Z")
@eoakes thanks for the response! I ran the following command to get the status: (venv) **➜** **ray** **git:(** **releases/1.4.1** **)** **✗** ray status --address localhost:6379 --redis\_password “” No cluster status. So a couple of things here. My local redis cluster doesn’t have a password: is this the correct parameter to pass to the status command? Secondly, there seems to be no cluster status, and I wonder why that is. Also, I’m on Macbook and there’s no “/tmp/ray” folder. Are these things that I should’ve set up manually, or is there something wrong with my ray setup? ---
**Author:** ![architkulkarni](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/architkulkarni/32/10_2.png) [@architkulkarni](https://discuss.ray.io/u/architkulkarni)\ **Post date:** [August 10, 2021, 7:49pm UTC](https://discuss.ray.io/t/error-scaling-ray-serve-to-2-replicas/3181/4 "2021-08-10T19:49:26Z")
I think `/tmp/ray/session_latest/logs/autoscaler.{log,err}` would be in the filesystem of the nodes in the Kubernetes cluster, do you have a way of accessing that? cc’ing @Dmitri for Kubernetes/autoscaler help ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [August 10, 2021, 8:04pm UTC](https://discuss.ray.io/t/error-scaling-ray-serve-to-2-replicas/3181/5 "2021-08-10T20:04:54Z")
On K8s, to get autoscaling status, it’s `kubectl -n exec -- ray status` If using the helm chart / operator, to get the full autoscaling logs, it’s `kubectl -n logs ` ---
**Author:** ![omnific9](https://avatars.discourse-cdn.com/v4/letter/o/3d9bf3/32.png) [@omnific9](https://discuss.ray.io/u/omnific9)\ **Post date:** [August 10, 2021, 11:46pm UTC](https://discuss.ray.io/t/error-scaling-ray-serve-to-2-replicas/3181/6 "2021-08-10T23:46:44Z")
Thanks for getting back @architkulkarni and @Dmitri I attached onto the pod but there doesn’t seem to be an autoscaler.\* file. I’ve copied the list of all the files under /tmp/ray/session\_latest/logs/ below: (base) ray@example-cluster-ray-head-j2bqx:~$ vi /tmp/ray/session\_latest/logs/ dashboard.log redis-shard\_0.out dashboard\_agent.log redis.err gcs\_server.err redis.out gcs\_server.out worker-10e4f82206fd98f0eb80337c7959b55ce475b5459223841dc5dfb0b2-01000000-552.err log\_monitor.log worker-10e4f82206fd98f0eb80337c7959b55ce475b5459223841dc5dfb0b2-01000000-552.out monitor.err worker-15f2b2a5f40a30c9fd730facd5d02a1463d42b086fb3c8e9c5067755-01000000-887.err monitor.log worker-15f2b2a5f40a30c9fd730facd5d02a1463d42b086fb3c8e9c5067755-01000000-887.out monitor.out worker-1d42d3ae8f262bf75e4cdc02ea834d7726d1a7b43471c4e7ad80f76b-01000000-368.err old/ worker-1d42d3ae8f262bf75e4cdc02ea834d7726d1a7b43471c4e7ad80f76b-01000000-368.out python-core-driver-01000000ffffffffffffffffffffffffffffffffffffffffffffffff\_334.log worker-20120211afcd1281c52f4cae64d2493ba251a112d484f8444d4c7608-01000000-755.err python-core-driver-02000000ffffffffffffffffffffffffffffffffffffffffffffffff\_519.log worker-20120211afcd1281c52f4cae64d2493ba251a112d484f8444d4c7608-01000000-755.out python-core-driver-03000000ffffffffffffffffffffffffffffffffffffffffffffffff\_687.log worker-3484d04f1f74910e7c225783f4ac12547d7b60d69d2c7965ddd45761-01000000-447.err python-core-driver-04000000ffffffffffffffffffffffffffffffffffffffffffffffff\_854.log worker-3484d04f1f74910e7c225783f4ac12547d7b60d69d2c7965ddd45761-01000000-447.out python-core-worker-10e4f82206fd98f0eb80337c7959b55ce475b5459223841dc5dfb0b2\_552.log worker-55ad6b7f62cb49237812276a3d96afc175566dabc3471b9e78bc7da2-01000000-888.err python-core-worker-15f2b2a5f40a30c9fd730facd5d02a1463d42b086fb3c8e9c5067755\_887.log worker-55ad6b7f62cb49237812276a3d96afc175566dabc3471b9e78bc7da2-01000000-888.out python-core-worker-1d42d3ae8f262bf75e4cdc02ea834d7726d1a7b43471c4e7ad80f76b\_368.log worker-5eb829eaeacced6d9f9ca863a2b4951d6da7472c56361889592f9418-01000000-614.err python-core-worker-20120211afcd1281c52f4cae64d2493ba251a112d484f8444d4c7608\_755.log worker-5eb829eaeacced6d9f9ca863a2b4951d6da7472c56361889592f9418-01000000-614.out python-core-worker-3484d04f1f74910e7c225783f4ac12547d7b60d69d2c7965ddd45761\_447.log worker-61e1b794bb4aae224d03ab1c28802f5bef0a8b3174c3be664c65a988-01000000-982.err python-core-worker-55ad6b7f62cb49237812276a3d96afc175566dabc3471b9e78bc7da2\_888.log worker-61e1b794bb4aae224d03ab1c28802f5bef0a8b3174c3be664c65a988-01000000-982.out python-core-worker-5eb829eaeacced6d9f9ca863a2b4951d6da7472c56361889592f9418\_614.log worker-7a7889e06bebf2494651bd1a586892f01f8f31d26792b5511d65e94c-01000000-555.err python-core-worker-61e1b794bb4aae224d03ab1c28802f5bef0a8b3174c3be664c65a988\_982.log worker-7a7889e06bebf2494651bd1a586892f01f8f31d26792b5511d65e94c-01000000-555.out python-core-worker-7a7889e06bebf2494651bd1a586892f01f8f31d26792b5511d65e94c\_555.log worker-92060c48bf0e3414f036b5092bdf6928c064d43d827dacf5c7c9dbbe-01000000-889.err python-core-worker-92060c48bf0e3414f036b5092bdf6928c064d43d827dacf5c7c9dbbe\_889.log worker-92060c48bf0e3414f036b5092bdf6928c064d43d827dacf5c7c9dbbe-01000000-889.out python-core-worker-92c19dc12e47d97c46259208152f8204fe22ac5ff1055e7e2369b288\_752.log worker-92c19dc12e47d97c46259208152f8204fe22ac5ff1055e7e2369b288-01000000-752.err python-core-worker-c57e0eba806f6e3b7edf903eec28466dd57d52ea78b209043d276a54\_395.log worker-92c19dc12e47d97c46259208152f8204fe22ac5ff1055e7e2369b288-01000000-752.out python-core-worker-dfea5b24e8dd6d2029b091119b37cfcb4c6537a9ed217f7e451febab\_890.log worker-c57e0eba806f6e3b7edf903eec28466dd57d52ea78b209043d276a54-01000000-395.err python-core-worker-f5e3465db81b4d0b69fb85881ae55723e51c3ea707382b77b059e89a\_720.log worker-c57e0eba806f6e3b7edf903eec28466dd57d52ea78b209043d276a54-01000000-395.out python-core-worker-f6d3eb7eeeb69c8b014e28db15e0d2e82bad337e67416f55504b96e0\_396.log worker-dfea5b24e8dd6d2029b091119b37cfcb4c6537a9ed217f7e451febab-01000000-890.err ray\_client\_server.err worker-dfea5b24e8dd6d2029b091119b37cfcb4c6537a9ed217f7e451febab-01000000-890.out ray\_client\_server.out worker-f5e3465db81b4d0b69fb85881ae55723e51c3ea707382b77b059e89a-01000000-720.err raylet.err worker-f5e3465db81b4d0b69fb85881ae55723e51c3ea707382b77b059e89a-01000000-720.out raylet.out worker-f6d3eb7eeeb69c8b014e28db15e0d2e82bad337e67416f55504b96e0-01000000-396.err redis-shard\_0.err worker-f6d3eb7eeeb69c8b014e28db15e0d2e82bad337e67416f55504b96e0-01000000-396.out ---
**Author:** ![omnific9](https://avatars.discourse-cdn.com/v4/letter/o/3d9bf3/32.png) [@omnific9](https://discuss.ray.io/u/omnific9)\ **Post date:** [August 10, 2021, 11:48pm UTC](https://discuss.ray.io/t/error-scaling-ray-serve-to-2-replicas/3181/7 "2021-08-10T23:48:02Z")
As for @Dmitri 's method, here’s the status right after I tried running 5 replicas: ➜ ray git:(releases/1.4.1) ✗ kubectl -n ray exec example-cluster-ray-head-j2bqx – ray status ======== Autoscaler status: 2021-08-10 16:41:14.985306 ======== Node status * * * Healthy: 1 head\_node Pending: (no pending nodes) Recent failures: (no failures) Resources * * * Usage: 4.0/4.0 CPU 0.0/1.0 CPU\_group\_0\_1a05a614d31404430b219ade936e9e03 0.0/1.0 CPU\_group\_0\_68f029c34779e27ecdc0ab25c8c5c6ea 0.0/1.0 CPU\_group\_0\_a56f2612e58b97d6310a9db96c6aa948 0.0/1.0 CPU\_group\_0\_ea8b9e5878f0d39ea69a6cea41f1d919 1.0/1.0 CPU\_group\_1a05a614d31404430b219ade936e9e03 1.0/1.0 CPU\_group\_68f029c34779e27ecdc0ab25c8c5c6ea 1.0/1.0 CPU\_group\_a56f2612e58b97d6310a9db96c6aa948 1.0/1.0 CPU\_group\_ea8b9e5878f0d39ea69a6cea41f1d919 0.00/1.400 GiB memory 0.00/0.585 GiB object\_store\_memory Demands: {‘CPU\_group\_41c2a9cef5e5273468fe536c57e34ffa’: 1.0}: 1+ pending tasks/actors {‘CPU’: 1.0} \* 1 (PACK): 1+ pending placement groups Although I did run the ray helm chart to set it up, running “kubectl -n ray logs example-cluster-ray-head-j2bqx” showed nothing. Did I get that command right? ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [August 11, 2021, 3:30am UTC](https://discuss.ray.io/t/error-scaling-ray-serve-to-2-replicas/3181/8 "2021-08-11T03:30:07Z")
It’s a little confusing – for the logs command, your target should be the operator pod which by default is in namespace `default`. ---
**Author:** ![Dmitri](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/dmitri/32/657_2.png) [@Dmitri](https://discuss.ray.io/u/Dmitri)\ **Post date:** [August 11, 2021, 3:31am UTC](https://discuss.ray.io/t/error-scaling-ray-serve-to-2-replicas/3181/9 "2021-08-11T03:31:39Z")
It appears from the `ray status` output that there’s only one ray node (the head) up. ---
**Author:** ![omnific9](https://avatars.discourse-cdn.com/v4/letter/o/3d9bf3/32.png) [@omnific9](https://discuss.ray.io/u/omnific9)\ **Post date:** [August 11, 2021, 11:44pm UTC](https://discuss.ray.io/t/error-scaling-ray-serve-to-2-replicas/3181/10 "2021-08-11T23:44:14Z")
@Dmitri thanks for the clarification. So I thought maybe it had to do with me running helm chart before starting two nodes in Minikube, so I did the following: kubectl -n ray delete raycluster example-cluster helm -n ray uninstall example-cluster And then, with the two Minikube nodes, I ran the following commands: ```auto (venv) ➜ ray git:(releases/1.4.1) ✗ kubectl -n ray delete raycluster example-cluster raycluster.cluster.ray.io "example-cluster" deleted (venv) ➜ ray git:(releases/1.4.1) ✗ helm -n ray uninstall example-cluster release "example-cluster" uninstalled (venv) ➜ ray git:(releases/1.4.1) ✗ kubectl -n ray get rayclusters NAME STATUS RESTARTS AGE example-cluster Running 0 26s (venv) ➜ ray git:(releases/1.4.1) ✗ kubectl -n ray get pods NAME READY STATUS RESTARTS AGE example-cluster-ray-head-type-rw89l 1/1 Running 0 29s example-cluster-ray-worker-type-7jn6p 0/1 Terminating 0 17s example-cluster-ray-worker-type-kj4mq 0/1 Terminating 0 17s (venv) ➜ ray git:(releases/1.4.1) ✗ kubectl -n ray get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE example-cluster-ray-head ClusterIP 10.96.164.162 10001/TCP,8265/TCP,8000/TCP 45s (venv) ➜ ray git:(releases/1.4.1) ✗ kubectl get deployment ray-operator NAME READY UP-TO-DATE AVAILABLE AGE ray-operator 1/1 1 1 57s (venv) ➜ ray git:(releases/1.4.1) ✗ kubectl get pod -l cluster.ray.io/component=operator NAME READY STATUS RESTARTS AGE ray-operator-55c86765c9-ml9l7 1/1 Running 0 64s (venv) ➜ ray git:(releases/1.4.1) ✗ kubectl get crd rayclusters.cluster.ray.io NAME CREATED AT rayclusters.cluster.ray.io 2021-08-11T23:20:20Z ``` Here I noticed two worker nodes were terminating. Not sure why. I proceeded to run the following: ray up -y python/ray/autoscaler/kubernetes/example-full.yaml --no-config-cache ray submit python/ray/autoscaler/kubernetes/example-full.yaml deploy.py Again it showed me the message ```auto (pid=326) 2021-08-11 16:42:05,282 WARNING backend_state.py:928 -- Backend 'hello' has 1 replicas that have taken more than 30s to start up. This may be caused by waiting for the cluster to auto-scale or because the constructor is slow. Resources required for each replica: {'CPU': 1}, resources available: {}. ``` I then looked at the log. It’s breaking the character limit, so I’ll be posting it in the next few comments. Please let me know if this tells us anything useful, or if I can dig up anything else. ---
**Author:** ![omnific9](https://avatars.discourse-cdn.com/v4/letter/o/3d9bf3/32.png) [@omnific9](https://discuss.ray.io/u/omnific9)\ **Post date:** [August 11, 2021, 11:49pm UTC](https://discuss.ray.io/t/error-scaling-ray-serve-to-2-replicas/3181/13 "2021-08-11T23:49:32Z")
Hmm. The forum wouldn’t let me post a long message even if I limit the post length. Here’s to try to post a small portion of the error message again from the logs: ```auto ======== Autoscaler status: 2021-08-11 16:34:22.421266 ======== Node status --------------------------------------------------------------- Healthy: 1 rayHeadType Pending: 172.17.0.5: rayWorkerType, setting-up 172.17.0.7: rayWorkerType, setting-up Recent failures: (no failures) Resources --------------------------------------------------------------- Usage: 0.0/1.0 CPU 0.00/0.350 GiB memory 0.00/0.135 GiB object_store_memory Demands: (no resource demands) example-cluster,ray:2021-08-11 16:34:22,444 DEBUG legacy_info_string.py:24 -- Cluster status: 2 nodes (2 updating) (4 failed to update) - MostDelayedHeartbeats: {'172.17.0.4': 0.254608154296875} - NodeIdleSeconds: Min=54 Mean=54 Max=54 - ResourceUsage: 0.0/1.0 CPU, 0.0 GiB/0.35 GiB memory, 0.0 GiB/0.13 GiB object_store_memory - TimeSinceLastHeartbeat: Min=0 Mean=0 Max=0 Worker node types: - rayWorkerType: 2 Traceback (most recent call last): File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 559, in connect sock = self._connect() File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 615, in _connect raise err File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 603, in _connect sock.connect(socket_address) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/services.py", line 666, in wait_for_redis_to_start redis_client.client_list() File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/client.py", line 1194, in client_list return self.execute_command('CLIENT LIST') File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/client.py", line 898, in execute_command conn = self.connection or pool.get_connection(command_name, **options) File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 1192, in get_connection connection.connect() File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 563, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to 172.17.0.4:6379. Connection refused. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/ray/anaconda3/bin/ray", line 8, in sys.exit(main()) File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/scripts/scripts.py", line 1808, in main return cli() File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 763, in invoke return __callback(*args, **kwargs) File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/scripts/scripts.py", line 656, in start redis_address_ip, redis_address_port, password=redis_password) File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/services.py", line 688, in wait_for_redis_to_start " attempts to ping the Redis server.") from connEx RuntimeError: Unable to connect to Redis at 172.17.0.4:6379 after 12 retries. Check that 172.17.0.4:6379 is reachable from this machine. If it is not, your firewall may be blocking this port. If the problem is a flaky connection, try setting the environment variable `RAY_START_REDIS_WAIT_RETRIES` to increase the number of attempts to ping the Redis server. Traceback (most recent call last): File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 559, in connect sock = self._connect() File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 615, in _connect raise err File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 603, in _connect sock.connect(socket_address) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/services.py", line 666, in wait_for_redis_to_start redis_client.client_list() File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/client.py", line 1194, in client_list return self.execute_command('CLIENT LIST') File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/client.py", line 898, in execute_command conn = self.connection or pool.get_connection(command_name, **options) File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 1192, in get_connection connection.connect() File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 563, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to 172.17.0.4:6379. Connection refused. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/ray/anaconda3/bin/ray", line 8, in sys.exit(main()) File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/scripts/scripts.py", line 1808, in main return cli() File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1668, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 763, in invoke return __callback(*args, **kwargs) File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/scripts/scripts.py", line 656, in start redis_address_ip, redis_address_port, password=redis_password) File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/services.py", line 688, in wait_for_redis_to_start " attempts to ping the Redis server.") from connEx RuntimeError: Unable to connect to Redis at 172.17.0.4:6379 after 12 retries. Check that 172.17.0.4:6379 is reachable from this machine. If it is not, your firewall may be blocking this port. If the problem is a flaky connection, try setting the environment variable `RAY_START_REDIS_WAIT_RETRIES` to increase the number of attempts to ping the Redis server. command terminated with exit code 1 command terminated with exit code 1 Exception in thread Thread-15: Traceback (most recent call last): File "/home/ray/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/updater.py", line 134, in run self.do_update() File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/updater.py", line 468, in do_update run_env="auto") File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/command_runner.py", line 178, in run self.process_runner.check_call(final_cmd, shell=True) File "/home/ray/anaconda3/lib/python3.7/subprocess.py", line 363, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'kubectl -n ray exec -it example-cluster-ray-worker-type-fnftz -- bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (export RAY_OVERRIDE_RESOURCES='"'"'{"CPU":1,"memory":375809638}'"'"';export RAY_HEAD_IP=172.17.0.4; ulimit -n 65536; ray start --address=$RAY_HEAD_IP:6379)'' returned non-zero exit status 1. Exception in thread Thread-16: Traceback (most recent call last): File "/home/ray/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/updater.py", line 134, in run self.do_update() File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/updater.py", line 468, in do_update run_env="auto") File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/command_runner.py", line 178, in run self.process_runner.check_call(final_cmd, shell=True) File "/home/ray/anaconda3/lib/python3.7/subprocess.py", line 363, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'kubectl -n ray exec -it example-cluster-ray-worker-type-ptmtt -- bash --login -c -i 'true && source ~/.bashrc && export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore && (export RAY_OVERRIDE_RESOURCES='"'"'{"CPU":1,"memory":375809638}'"'"';export RAY_HEAD_IP=172.17.0.4; ulimit -n 65536; ray start --address=$RAY_HEAD_IP:6379)'' returned non-zero exit status 1. example-cluster,ray:2021-08-11 16:34:27,660 DEBUG resource_demand_scheduler.py:160 -- Cluster resources: [{'object_store_memory': 144745267.0, 'node:172.17.0.4': 1.0, 'CPU': 1.0, 'memory': 375809638.0}, {'CPU': 1, 'memory': 375809638}, {'CPU': 1, 'memory': 375809638}] example-cluster,ray:2021-08-11 16:34:27,661 DEBUG resource_demand_scheduler.py:161 -- Node counts: defaultdict(, {'rayHeadType': 1, 'rayWorkerType': 2}) example-cluster,ray:2021-08-11 16:34:27,661 DEBUG resource_demand_scheduler.py:172 -- Placement group demands: [] example-cluster,ray:2021-08-11 16:34:27,661 DEBUG resource_demand_scheduler.py:218 -- Resource demands: [] example-cluster,ray:2021-08-11 16:34:27,661 DEBUG resource_demand_scheduler.py:219 -- Unfulfilled demands: [] example-cluster,ray:2021-08-11 16:34:27,693 DEBUG resource_demand_scheduler.py:241 -- Node requests: {} example-cluster,ray:2021-08-11 16:34:27,702 ERROR autoscaler.py:306 -- StandardAutoscaler: example-cluster-ray-worker-type-fnftz: Terminating. Failed to setup/initialize node. example-cluster,ray:2021-08-11 16:34:27,708 ERROR autoscaler.py:306 -- StandardAutoscaler: example-cluster-ray-worker-type-ptmtt: Terminating. Failed to setup/initialize node. example-cluster,ray:2021-08-11 16:34:27,718 INFO node_provider.py:171 -- KubernetesNodeProvider: calling delete_namespaced_pod example-cluster,ray:2021-08-11 16:34:27,731 INFO node_provider.py:171 -- KubernetesNodeProvider: calling delete_namespaced_pod example-cluster,ray:2021-08-11 16:34:27,815 INFO autoscaler.py:354 -- ======== Autoscaler status: 2021-08-11 16:34:27.815676 ======== Node status --------------------------------------------------------------- Healthy: 1 rayHeadType Pending: (no pending nodes) Recent failures: (no failures) Resources --------------------------------------------------------------- Usage: 0.0/1.0 CPU 0.00/0.350 GiB memory 0.00/0.135 GiB object_store_memory Demands: (no resource demands) example-cluster,ray:2021-08-11 16:34:27,816 DEBUG legacy_info_string.py:24 -- Cluster status: 0 nodes (6 failed to update) - MostDelayedHeartbeats: {'172.17.0.4': 0.3220522403717041} - NodeIdleSeconds: Min=59 Mean=59 Max=59 - ResourceUsage: 0.0/1.0 CPU, 0.0 GiB/0.35 GiB memory, 0.0 GiB/0.13 GiB object_store_memory - TimeSinceLastHeartbeat: Min=0 Mean=0 Max=0 Worker node types: example-cluster,ray:2021-08-11 16:34:27,853 INFO monitor.py:224 -- :event_summary:Removing 2 nodes of type rayWorkerType (launch failed). example-cluster,ray:2021-08-11 16:34:32,828 ERROR monitor.py:285 -- Error in monitor loop Traceback (most recent call last): File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/monitor.py", line 317, in run self._run() File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/monitor.py", line 207, in _run self.update_load_metrics() File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/monitor.py", line 170, in update_load_metrics request, timeout=4) File "/home/ray/anaconda3/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/home/ray/anaconda3/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "Socket closed" debug_error_string = "{"created":"@1628724872.826523900","description":"Error received from peer ipv4:172.17.0.4:38277","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"Socket closed","grpc_status":14}" > Process example-cluster,ray: Traceback (most recent call last): File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/monitor.py", line 317, in run self._run() File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/monitor.py", line 207, in _run self.update_load_metrics() File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/monitor.py", line 170, in update_load_metrics request, timeout=4) File "/home/ray/anaconda3/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/home/ray/anaconda3/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "Socket closed" debug_error_string = "{"created":"@1628724872.826523900","description":"Error received from peer ipv4:172.17.0.4:38277","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"Socket closed","grpc_status":14}" > During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 1198, in get_connection if connection.can_read(): File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 734, in can_read return self._parser.can_read(timeout) File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 417, in can_read raise_on_timeout=False) File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 429, in read_from_socket raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR) redis.exceptions.ConnectionError: Connection closed by server. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 559, in connect sock = self._connect() File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 615, in _connect raise err File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 603, in _connect sock.connect(socket_address) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ray/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/home/ray/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/ray_operator/operator.py", line 87, in _create_or_update self.start_monitor() File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/ray_operator/operator.py", line 125, in start_monitor self.mtr.run() File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/monitor.py", line 319, in run self._handle_failure(traceback.format_exc()) File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/monitor.py", line 296, in _handle_failure _internal_kv_put(DEBUG_AUTOSCALING_ERROR, message, overwrite=True) File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 62, in wrapper return func(*args, **kwargs) File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/experimental/internal_kv.py", line 57, in _internal_kv_put key, "value", value) File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/client.py", line 3050, in hset return self.execute_command('HSET', name, *items) File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/client.py", line 898, in execute_command conn = self.connection or pool.get_connection(command_name, **options) File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 1202, in get_connection connection.connect() File "/home/ray/anaconda3/lib/python3.7/site-packages/redis/connection.py", line 563, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to 172.17.0.4:6379. Connection refused. ``` ---
**Author:** ![omnific9](https://avatars.discourse-cdn.com/v4/letter/o/3d9bf3/32.png) [@omnific9](https://discuss.ray.io/u/omnific9)\ **Post date:** [August 11, 2021, 11:52pm UTC](https://discuss.ray.io/t/error-scaling-ray-serve-to-2-replicas/3181/14 "2021-08-11T23:52:23Z")
Also, here’s what I see about the two minikube nodes: ```auto ➜ charts git:(master) ✗ minikube status -p minikube minikube type: Control Plane host: Running kubelet: Running apiserver: Running kubeconfig: Configured minikube-m02 type: Worker host: Running kubelet: Running ➜ charts git:(master) ✗ kubectl -n ray get nodes NAME STATUS ROLES AGE VERSION minikube Ready master 300d v1.19.2 minikube-m02 Ready 2d2h v1.19.2 ```