When running 100k no-op tasks and blocking for them to come back, it takes with 10 8x32 machines running on k8s about 25 seconds. In essence, this breaks down 10k per machine in 25 seconds, or 400 per machine per 1 sec, or 50 per core per 1 sec.
Just as a point of comparison I’d be very interested in what others measured. Are your numbers in the same ballpark?
Below I included the small test program I used
import ray
import time
import datetime
import os
import datetime
ray.init(address='<ip-adress>:6379', _redis_password='5241590000000000')
@ray.remote
def f(i):
return i
begin = datetime.datetime.now()
print(", cluster resources: " + str(ray.cluster_resources()))
print(", available resources: " + str(ray.available_resources()))
futures = [f.remote(i) for i in range(100000)]
print(", cluster resources: " + str(ray.cluster_resources()))
print(", available resources: " + str(ray.available_resources()))
ray.get(futures)
end = datetime.datetime.now()
duration = end-begin
print("Duration: " + str(duration))
Hey @mbehrendt this is interesting! I wonder if you add this in the beginning of the code, it’ll still take 25 seconds.
(basically this wait until every node in the cluster is registered to ray cluster)
num_nodes = 0
while num_nodes < 32:
num_alive_nodes = 0
nodes = ray.nodes()
for node in nodes:
if node["Alive"]:
num_alive_nodes += 1
num_nodes = num_alive_nodes
to illustrate what is happening when i configure min_workers to a larger value, i included the information below.
there i simple ran ray up ..., and had the command below running in parallel. You see there is a larger number of pods provisioned & deprovisioned, until after a while the system only has the head node running. (split this into multiple posts since i’m constrained in size.
Does that help?
for i in {1..1000}; do kubectl -n ray get pods; sleep 1; done
No resources found in ray namespace.
No resources found in ray namespace.
No resources found in ray namespace.
No resources found in ray namespace.
No resources found in ray namespace.
No resources found in ray namespace.
No resources found in ray namespace.
No resources found in ray namespace.
No resources found in ray namespace.
No resources found in ray namespace.
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 0/1 ContainerCreating 0 1s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 4s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 6s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 7s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 9s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 11s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 13s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 14s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 16s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 17s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 19s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 21s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 22s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 24s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 25s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 27s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 29s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 30s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 32s
example-cluster-ray-worker-2fkhb 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-5wpt8 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-67d8n 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-8fbxt 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-8p7jj 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-8qhdz 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-8tz77 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-8xkjs 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-9wxkw 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-bnbng 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-cdr5x 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-d6ttm 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-drj42 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-g5k7f 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-h2xcl 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-jwtb7 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-k64f7 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-lfjzm 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-nv9zg 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-pmfc4 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-q7r5k 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-rkwqr 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-w696b 0/1 ContainerCreating 0 1s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 33s
example-cluster-ray-worker-2fkhb 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-5wpt8 1/1 Running 0 2s
example-cluster-ray-worker-67d8n 1/1 Running 0 2s
example-cluster-ray-worker-8fbxt 1/1 Running 0 2s
example-cluster-ray-worker-8p7jj 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-8qhdz 1/1 Running 0 2s
example-cluster-ray-worker-8tz77 1/1 Running 0 2s
example-cluster-ray-worker-8xkjs 1/1 Running 0 2s
example-cluster-ray-worker-9wxkw 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-bnbng 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-cdr5x 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-d6ttm 1/1 Running 0 2s
example-cluster-ray-worker-drj42 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-g5k7f 1/1 Running 0 2s
example-cluster-ray-worker-h2xcl 1/1 Running 0 2s
example-cluster-ray-worker-jwtb7 1/1 Running 0 2s
example-cluster-ray-worker-k64f7 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-lfjzm 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-nv9zg 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-pmfc4 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-q7r5k 1/1 Running 0 2s
example-cluster-ray-worker-rkwqr 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-w696b 0/1 ContainerCreating 0 2s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 35s
example-cluster-ray-worker-2fkhb 1/1 Running 0 4s
example-cluster-ray-worker-5wpt8 1/1 Running 0 4s
example-cluster-ray-worker-67d8n 1/1 Running 0 4s
example-cluster-ray-worker-8fbxt 1/1 Running 0 4s
example-cluster-ray-worker-8p7jj 1/1 Running 0 4s
example-cluster-ray-worker-8qhdz 1/1 Running 0 4s
example-cluster-ray-worker-8tz77 1/1 Running 0 4s
example-cluster-ray-worker-8xkjs 1/1 Running 0 4s
example-cluster-ray-worker-9wxkw 1/1 Running 0 4s
example-cluster-ray-worker-bnbng 1/1 Running 0 4s
example-cluster-ray-worker-cdr5x 1/1 Running 0 4s
example-cluster-ray-worker-d6ttm 1/1 Running 0 4s
example-cluster-ray-worker-drj42 0/1 ContainerCreating 0 4s
example-cluster-ray-worker-g5k7f 1/1 Running 0 4s
example-cluster-ray-worker-h2xcl 1/1 Running 0 4s
example-cluster-ray-worker-jwtb7 1/1 Running 0 4s
example-cluster-ray-worker-k64f7 1/1 Running 0 4s
example-cluster-ray-worker-lfjzm 1/1 Running 0 4s
example-cluster-ray-worker-nv9zg 1/1 Running 0 4s
example-cluster-ray-worker-pmfc4 1/1 Running 0 4s
example-cluster-ray-worker-q7r5k 1/1 Running 0 4s
example-cluster-ray-worker-rkwqr 1/1 Running 0 4s
example-cluster-ray-worker-w696b 1/1 Running 0 4s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 37s
example-cluster-ray-worker-2fkhb 1/1 Running 0 6s
example-cluster-ray-worker-5wpt8 1/1 Running 0 6s
example-cluster-ray-worker-67d8n 1/1 Running 0 6s
example-cluster-ray-worker-8fbxt 1/1 Running 0 6s
example-cluster-ray-worker-8p7jj 1/1 Running 0 6s
example-cluster-ray-worker-8qhdz 1/1 Running 0 6s
example-cluster-ray-worker-8tz77 1/1 Running 0 6s
example-cluster-ray-worker-8xkjs 1/1 Running 0 6s
example-cluster-ray-worker-9wxkw 1/1 Running 0 6s
example-cluster-ray-worker-bnbng 1/1 Running 0 6s
example-cluster-ray-worker-cdr5x 1/1 Running 0 6s
example-cluster-ray-worker-d6ttm 1/1 Running 0 6s
example-cluster-ray-worker-drj42 1/1 Running 0 6s
example-cluster-ray-worker-g5k7f 1/1 Running 0 6s
example-cluster-ray-worker-h2xcl 1/1 Running 0 6s
example-cluster-ray-worker-jwtb7 1/1 Running 0 6s
example-cluster-ray-worker-k64f7 1/1 Running 0 6s
example-cluster-ray-worker-lfjzm 1/1 Running 0 6s
example-cluster-ray-worker-nv9zg 1/1 Running 0 6s
example-cluster-ray-worker-pmfc4 1/1 Running 0 6s
example-cluster-ray-worker-q7r5k 1/1 Running 0 6s
example-cluster-ray-worker-rkwqr 1/1 Running 0 6s
example-cluster-ray-worker-w696b 1/1 Running 0 6s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 39s
example-cluster-ray-worker-2fkhb 1/1 Running 0 8s
example-cluster-ray-worker-5wpt8 1/1 Running 0 8s
example-cluster-ray-worker-67d8n 1/1 Running 0 8s
example-cluster-ray-worker-8fbxt 1/1 Running 0 8s
example-cluster-ray-worker-8p7jj 1/1 Running 0 8s
example-cluster-ray-worker-8qhdz 1/1 Running 0 8s
example-cluster-ray-worker-8tz77 1/1 Running 0 8s
example-cluster-ray-worker-8xkjs 1/1 Running 0 8s
example-cluster-ray-worker-9wxkw 1/1 Running 0 8s
example-cluster-ray-worker-bnbng 1/1 Running 0 8s
example-cluster-ray-worker-cdr5x 1/1 Running 0 8s
example-cluster-ray-worker-d6ttm 1/1 Running 0 8s
example-cluster-ray-worker-drj42 1/1 Running 0 8s
example-cluster-ray-worker-g5k7f 1/1 Running 0 8s
example-cluster-ray-worker-h2xcl 1/1 Running 0 8s
example-cluster-ray-worker-jwtb7 1/1 Running 0 8s
example-cluster-ray-worker-k64f7 1/1 Running 0 8s
example-cluster-ray-worker-lfjzm 1/1 Running 0 8s
example-cluster-ray-worker-nv9zg 1/1 Running 0 8s
example-cluster-ray-worker-pmfc4 1/1 Running 0 8s
example-cluster-ray-worker-q7r5k 1/1 Running 0 8s
example-cluster-ray-worker-rkwqr 1/1 Running 0 8s
example-cluster-ray-worker-w696b 1/1 Running 0 8s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 41s
example-cluster-ray-worker-2fkhb 0/1 Terminating 0 10s
example-cluster-ray-worker-5wpt8 1/1 Running 0 10s
example-cluster-ray-worker-67d8n 0/1 Terminating 0 10s
example-cluster-ray-worker-8fbxt 1/1 Terminating 0 10s
example-cluster-ray-worker-8p7jj 0/1 Terminating 0 10s
example-cluster-ray-worker-8qhdz 1/1 Terminating 0 10s
example-cluster-ray-worker-8tz77 0/1 Terminating 0 10s
example-cluster-ray-worker-8xkjs 0/1 Terminating 0 10s
example-cluster-ray-worker-9wxkw 0/1 Terminating 0 10s
example-cluster-ray-worker-bnbng 0/1 Terminating 0 10s
example-cluster-ray-worker-cdr5x 1/1 Terminating 0 10s
example-cluster-ray-worker-d6ttm 0/1 Terminating 0 10s
example-cluster-ray-worker-drj42 0/1 Terminating 0 10s
example-cluster-ray-worker-g5k7f 1/1 Terminating 0 10s
example-cluster-ray-worker-h2xcl 0/1 Terminating 0 10s
example-cluster-ray-worker-jwtb7 1/1 Terminating 0 10s
example-cluster-ray-worker-k64f7 1/1 Terminating 0 10s
example-cluster-ray-worker-lfjzm 0/1 Terminating 0 10s
example-cluster-ray-worker-nv9zg 0/1 Terminating 0 10s
example-cluster-ray-worker-pmfc4 1/1 Terminating 0 10s
example-cluster-ray-worker-q7r5k 1/1 Terminating 0 10s
example-cluster-ray-worker-rkwqr 1/1 Terminating 0 10s
example-cluster-ray-worker-w696b 1/1 Terminating 0 10s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 43s
example-cluster-ray-worker-2fkhb 0/1 Terminating 0 12s
example-cluster-ray-worker-5wpt8 1/1 Running 0 12s
example-cluster-ray-worker-67d8n 0/1 Terminating 0 12s
example-cluster-ray-worker-8fbxt 0/1 Terminating 0 12s
example-cluster-ray-worker-8p7jj 0/1 Terminating 0 12s
example-cluster-ray-worker-8qhdz 0/1 Terminating 0 12s
example-cluster-ray-worker-9wxkw 0/1 Terminating 0 12s
example-cluster-ray-worker-cdr5x 0/1 Terminating 0 12s
example-cluster-ray-worker-d6ttm 0/1 Terminating 0 12s
example-cluster-ray-worker-drj42 0/1 Terminating 0 12s
example-cluster-ray-worker-g5k7f 0/1 Terminating 0 12s
example-cluster-ray-worker-h2xcl 0/1 Terminating 0 12s
example-cluster-ray-worker-jwtb7 0/1 Terminating 0 12s
example-cluster-ray-worker-k64f7 0/1 Terminating 0 12s
example-cluster-ray-worker-lfjzm 0/1 Terminating 0 12s
example-cluster-ray-worker-nv9zg 0/1 Terminating 0 12s
example-cluster-ray-worker-pmfc4 0/1 Terminating 0 12s
example-cluster-ray-worker-q7r5k 0/1 Terminating 0 12s
example-cluster-ray-worker-rkwqr 0/1 Terminating 0 12s
example-cluster-ray-worker-w696b 0/1 Terminating 0 12s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 45s
example-cluster-ray-worker-2fkhb 0/1 Terminating 0 14s
example-cluster-ray-worker-5wpt8 1/1 Running 0 14s
example-cluster-ray-worker-67d8n 0/1 Terminating 0 14s
example-cluster-ray-worker-8qhdz 0/1 Terminating 0 14s
example-cluster-ray-worker-9wxkw 0/1 Terminating 0 14s
example-cluster-ray-worker-cdr5x 0/1 Terminating 0 14s
example-cluster-ray-worker-drj42 0/1 Terminating 0 14s
example-cluster-ray-worker-g5k7f 0/1 Terminating 0 14s
example-cluster-ray-worker-h2xcl 0/1 Terminating 0 14s
example-cluster-ray-worker-jwtb7 0/1 Terminating 0 14s
example-cluster-ray-worker-k64f7 0/1 Terminating 0 14s
example-cluster-ray-worker-lfjzm 0/1 Terminating 0 14s
example-cluster-ray-worker-nv9zg 0/1 Terminating 0 14s
example-cluster-ray-worker-pmfc4 0/1 Terminating 0 14s
example-cluster-ray-worker-q7r5k 0/1 Terminating 0 14s
example-cluster-ray-worker-rkwqr 0/1 Terminating 0 14s
example-cluster-ray-worker-w696b 0/1 Terminating 0 14s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 47s
example-cluster-ray-worker-5wpt8 1/1 Running 0 16s
example-cluster-ray-worker-8qhdz 0/1 Terminating 0 16s
example-cluster-ray-worker-cdr5x 0/1 Terminating 0 16s
example-cluster-ray-worker-g5k7f 0/1 Terminating 0 16s
example-cluster-ray-worker-h2xcl 0/1 Terminating 0 16s
example-cluster-ray-worker-jwtb7 0/1 Terminating 0 16s
example-cluster-ray-worker-k64f7 0/1 Terminating 0 16s
example-cluster-ray-worker-lfjzm 0/1 Terminating 0 16s
example-cluster-ray-worker-nv9zg 0/1 Terminating 0 16s
example-cluster-ray-worker-pmfc4 0/1 Terminating 0 16s
example-cluster-ray-worker-q7r5k 0/1 Terminating 0 16s
example-cluster-ray-worker-rkwqr 0/1 Terminating 0 16s
example-cluster-ray-worker-w696b 0/1 Terminating 0 16s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 49s
example-cluster-ray-worker-5wpt8 1/1 Running 0 18s
example-cluster-ray-worker-8qhdz 0/1 Terminating 0 18s
example-cluster-ray-worker-cdr5x 0/1 Terminating 0 18s
example-cluster-ray-worker-g5k7f 0/1 Terminating 0 18s
example-cluster-ray-worker-h2xcl 0/1 Terminating 0 18s
example-cluster-ray-worker-jwtb7 0/1 Terminating 0 18s
example-cluster-ray-worker-kp425 0/1 Pending 0 0s
example-cluster-ray-worker-lfjzm 0/1 Terminating 0 18s
example-cluster-ray-worker-pmfc4 0/1 Terminating 0 18s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 51s
example-cluster-ray-worker-4sg6z 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-5t56c 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-5wpt8 1/1 Running 0 20s
example-cluster-ray-worker-6cfbx 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-6d4fs 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-6vrgd 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-8jnpp 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-b5s5l 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-cdr5x 0/1 Terminating 0 20s
example-cluster-ray-worker-crfs6 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-dvbpz 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-g5k7f 0/1 Terminating 0 20s
example-cluster-ray-worker-h2xcl 0/1 Terminating 0 20s
example-cluster-ray-worker-hgvls 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-jmw7q 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-jwtb7 0/1 Terminating 0 20s
example-cluster-ray-worker-kp425 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-lfjzm 0/1 Terminating 0 20s
example-cluster-ray-worker-pbvwv 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-pmfc4 0/1 Terminating 0 20s
example-cluster-ray-worker-rt4l2 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-sfg7n 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-wkjxr 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-wlcmt 0/1 ContainerCreating 0 2s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 52s
example-cluster-ray-worker-4sg6z 1/1 Running 0 3s
example-cluster-ray-worker-5t56c 1/1 Running 0 2s
example-cluster-ray-worker-5wpt8 1/1 Running 0 21s
example-cluster-ray-worker-6cfbx 1/1 Running 0 3s
example-cluster-ray-worker-6d4fs 1/1 Running 0 3s
example-cluster-ray-worker-6vrgd 1/1 Running 0 3s
example-cluster-ray-worker-8jnpp 1/1 Running 0 3s
example-cluster-ray-worker-b5s5l 1/1 Running 0 3s
example-cluster-ray-worker-cdr5x 0/1 Terminating 0 21s
example-cluster-ray-worker-crfs6 1/1 Running 0 2s
example-cluster-ray-worker-dvbpz 1/1 Running 0 3s
example-cluster-ray-worker-hgvls 1/1 Running 0 3s
example-cluster-ray-worker-jmw7q 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-kp425 1/1 Running 0 3s
example-cluster-ray-worker-lfjzm 0/1 Terminating 0 21s
example-cluster-ray-worker-pbvwv 1/1 Running 0 3s
example-cluster-ray-worker-pmfc4 0/1 Terminating 0 21s
example-cluster-ray-worker-rt4l2 1/1 Running 0 3s
example-cluster-ray-worker-sfg7n 1/1 Running 0 3s
example-cluster-ray-worker-wkjxr 1/1 Running 0 2s
example-cluster-ray-worker-wlcmt 1/1 Running 0 3s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 54s
example-cluster-ray-worker-4sg6z 1/1 Running 0 5s
example-cluster-ray-worker-5t56c 1/1 Running 0 4s
example-cluster-ray-worker-5wpt8 1/1 Running 0 23s
example-cluster-ray-worker-6cfbx 1/1 Running 0 5s
example-cluster-ray-worker-6d4fs 1/1 Running 0 5s
example-cluster-ray-worker-6vrgd 1/1 Running 0 5s
example-cluster-ray-worker-8jnpp 1/1 Running 0 5s
example-cluster-ray-worker-b5s5l 1/1 Running 0 5s
example-cluster-ray-worker-crfs6 1/1 Running 0 4s
example-cluster-ray-worker-dvbpz 1/1 Running 0 5s
example-cluster-ray-worker-hgvls 1/1 Running 0 5s
example-cluster-ray-worker-jmw7q 1/1 Running 0 4s
example-cluster-ray-worker-kp425 1/1 Running 0 5s
example-cluster-ray-worker-pbvwv 1/1 Running 0 5s
example-cluster-ray-worker-rt4l2 1/1 Running 0 5s
example-cluster-ray-worker-sfg7n 1/1 Running 0 5s
example-cluster-ray-worker-wkjxr 1/1 Running 0 4s
example-cluster-ray-worker-wlcmt 1/1 Running 0 5s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 56s
example-cluster-ray-worker-4sg6z 1/1 Running 0 7s
example-cluster-ray-worker-5t56c 1/1 Running 0 6s
example-cluster-ray-worker-5wpt8 1/1 Running 0 25s
example-cluster-ray-worker-6cfbx 1/1 Running 0 7s
example-cluster-ray-worker-6d4fs 1/1 Running 0 7s
example-cluster-ray-worker-6vrgd 1/1 Running 0 7s
example-cluster-ray-worker-8jnpp 1/1 Running 0 7s
example-cluster-ray-worker-b5s5l 1/1 Running 0 7s
example-cluster-ray-worker-crfs6 1/1 Running 0 6s
example-cluster-ray-worker-dvbpz 1/1 Running 0 7s
example-cluster-ray-worker-hgvls 1/1 Running 0 7s
example-cluster-ray-worker-jmw7q 1/1 Running 0 6s
example-cluster-ray-worker-kp425 1/1 Running 0 7s
example-cluster-ray-worker-pbvwv 1/1 Running 0 7s
example-cluster-ray-worker-rt4l2 1/1 Running 0 7s
example-cluster-ray-worker-sfg7n 1/1 Running 0 7s
example-cluster-ray-worker-wkjxr 1/1 Running 0 6s
example-cluster-ray-worker-wlcmt 1/1 Running 0 7s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 58s
example-cluster-ray-worker-4sg6z 1/1 Running 0 9s
example-cluster-ray-worker-5t56c 1/1 Running 0 8s
example-cluster-ray-worker-5wpt8 1/1 Running 0 27s
example-cluster-ray-worker-6cfbx 1/1 Running 0 9s
example-cluster-ray-worker-6d4fs 1/1 Running 0 9s
example-cluster-ray-worker-6vrgd 1/1 Running 0 9s
example-cluster-ray-worker-8jnpp 1/1 Running 0 9s
example-cluster-ray-worker-b5s5l 1/1 Running 0 9s
example-cluster-ray-worker-crfs6 1/1 Running 0 8s
example-cluster-ray-worker-dvbpz 1/1 Running 0 9s
example-cluster-ray-worker-hgvls 1/1 Running 0 9s
example-cluster-ray-worker-jmw7q 1/1 Running 0 8s
example-cluster-ray-worker-kp425 1/1 Running 0 9s
example-cluster-ray-worker-pbvwv 1/1 Running 0 9s
example-cluster-ray-worker-rt4l2 1/1 Running 0 9s
example-cluster-ray-worker-sfg7n 1/1 Running 0 9s
example-cluster-ray-worker-wkjxr 1/1 Running 0 8s
example-cluster-ray-worker-wlcmt 1/1 Running 0 9s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 60s
example-cluster-ray-worker-4sg6z 1/1 Running 0 11s
example-cluster-ray-worker-5t56c 1/1 Terminating 0 10s
example-cluster-ray-worker-5wpt8 1/1 Running 0 29s
example-cluster-ray-worker-6cfbx 1/1 Terminating 0 11s
example-cluster-ray-worker-6d4fs 1/1 Running 0 11s
example-cluster-ray-worker-6vrgd 0/1 Terminating 0 11s
example-cluster-ray-worker-8jnpp 1/1 Terminating 0 11s
example-cluster-ray-worker-b5s5l 0/1 Terminating 0 11s
example-cluster-ray-worker-crfs6 1/1 Terminating 0 10s
example-cluster-ray-worker-dvbpz 0/1 Terminating 0 11s
example-cluster-ray-worker-hgvls 1/1 Terminating 0 11s
example-cluster-ray-worker-jmw7q 1/1 Terminating 0 10s
example-cluster-ray-worker-kp425 1/1 Running 0 11s
example-cluster-ray-worker-pbvwv 1/1 Terminating 0 11s
example-cluster-ray-worker-rt4l2 1/1 Terminating 0 11s
example-cluster-ray-worker-sfg7n 1/1 Terminating 0 11s
example-cluster-ray-worker-wkjxr 1/1 Terminating 0 10s
example-cluster-ray-worker-wlcmt 1/1 Terminating 0 11s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 62s
example-cluster-ray-worker-4sg6z 1/1 Running 0 13s
example-cluster-ray-worker-5t56c 0/1 Terminating 0 12s
example-cluster-ray-worker-5wpt8 1/1 Running 0 31s
example-cluster-ray-worker-6cfbx 0/1 Terminating 0 13s
example-cluster-ray-worker-6d4fs 1/1 Running 0 13s
example-cluster-ray-worker-6vrgd 0/1 Terminating 0 13s
example-cluster-ray-worker-8jnpp 0/1 Terminating 0 13s
example-cluster-ray-worker-bvpzm 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-crfs6 0/1 Terminating 0 12s
example-cluster-ray-worker-ds6vg 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-hgvls 0/1 Terminating 0 13s
example-cluster-ray-worker-jmw7q 0/1 Terminating 0 12s
example-cluster-ray-worker-kp425 1/1 Running 0 13s
example-cluster-ray-worker-pbvwv 0/1 Terminating 0 13s
example-cluster-ray-worker-rt4l2 0/1 Terminating 0 13s
example-cluster-ray-worker-s6zxb 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-sfg7n 0/1 Terminating 0 13s
example-cluster-ray-worker-wkjxr 0/1 Terminating 0 12s
example-cluster-ray-worker-xdpx8 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-xgs2v 0/1 ContainerCreating 0 1s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 64s
example-cluster-ray-worker-4sg6z 1/1 Running 0 15s
example-cluster-ray-worker-5t56c 0/1 Terminating 0 14s
example-cluster-ray-worker-5wpt8 1/1 Running 0 33s
example-cluster-ray-worker-6cfbx 0/1 Terminating 0 15s
example-cluster-ray-worker-6d4fs 1/1 Running 0 15s
example-cluster-ray-worker-6vrgd 0/1 Terminating 0 15s
example-cluster-ray-worker-bvpzm 0/1 ContainerCreating 0 3s
example-cluster-ray-worker-ds6vg 0/1 ContainerCreating 0 3s
example-cluster-ray-worker-kp425 1/1 Running 0 15s
example-cluster-ray-worker-rt4l2 0/1 Terminating 0 15s
example-cluster-ray-worker-s6zxb 0/1 ContainerCreating 0 3s
example-cluster-ray-worker-sfg7n 0/1 Terminating 0 15s
example-cluster-ray-worker-wkjxr 0/1 Terminating 0 14s
example-cluster-ray-worker-xdpx8 0/1 ContainerCreating 0 3s
example-cluster-ray-worker-xgs2v 0/1 ContainerCreating 0 3s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 66s
example-cluster-ray-worker-4sg6z 1/1 Running 0 17s
example-cluster-ray-worker-5t56c 0/1 Terminating 0 16s
example-cluster-ray-worker-5wpt8 1/1 Running 0 35s
example-cluster-ray-worker-6cfbx 0/1 Terminating 0 17s
example-cluster-ray-worker-6d4fs 1/1 Running 0 17s
example-cluster-ray-worker-6vrgd 0/1 Terminating 0 17s
example-cluster-ray-worker-bvpzm 0/1 ContainerCreating 0 5s
example-cluster-ray-worker-ds6vg 0/1 ContainerCreating 0 5s
example-cluster-ray-worker-kp425 1/1 Running 0 17s
example-cluster-ray-worker-rt4l2 0/1 Terminating 0 17s
example-cluster-ray-worker-s6zxb 0/1 ContainerCreating 0 5s
example-cluster-ray-worker-xdpx8 0/1 ContainerCreating 0 5s
example-cluster-ray-worker-xgs2v 1/1 Running 0 5s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 68s
example-cluster-ray-worker-4sg6z 1/1 Running 0 19s
example-cluster-ray-worker-5wpt8 1/1 Running 0 37s
example-cluster-ray-worker-6cfbx 0/1 Terminating 0 19s
example-cluster-ray-worker-6d4fs 1/1 Running 0 19s
example-cluster-ray-worker-6vrgd 0/1 Terminating 0 19s
example-cluster-ray-worker-bvpzm 1/1 Running 0 7s
example-cluster-ray-worker-ds6vg 1/1 Running 0 7s
example-cluster-ray-worker-kp425 1/1 Running 0 19s
example-cluster-ray-worker-s6zxb 1/1 Running 0 7s
example-cluster-ray-worker-xdpx8 1/1 Running 0 7s
example-cluster-ray-worker-xgs2v 1/1 Running 0 7s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 70s
example-cluster-ray-worker-4sg6z 1/1 Running 0 21s
example-cluster-ray-worker-5wpt8 1/1 Running 0 39s
example-cluster-ray-worker-6d4fs 1/1 Running 0 21s
example-cluster-ray-worker-bvpzm 1/1 Running 0 9s
example-cluster-ray-worker-ds6vg 1/1 Running 0 9s
example-cluster-ray-worker-kp425 1/1 Running 0 21s
example-cluster-ray-worker-s6zxb 1/1 Running 0 9s
example-cluster-ray-worker-xdpx8 1/1 Running 0 9s
example-cluster-ray-worker-xgs2v 1/1 Running 0 9s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 71s
example-cluster-ray-worker-4ktnh 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-4msq8 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-4sg6z 1/1 Running 0 22s
example-cluster-ray-worker-52qk8 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-5wpt8 1/1 Running 0 40s
example-cluster-ray-worker-6d4fs 1/1 Running 0 22s
example-cluster-ray-worker-87p5k 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-87z52 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-92jrb 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-9zthk 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-bv94v 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-bvpzm 1/1 Running 0 10s
example-cluster-ray-worker-ds6vg 1/1 Running 0 10s
example-cluster-ray-worker-g9t4q 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-hltxf 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-ht9zj 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-j85vx 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-kp425 1/1 Running 0 22s
example-cluster-ray-worker-rvxbr 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-s6zxb 1/1 Running 0 10s
example-cluster-ray-worker-twmbf 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-xdpx8 1/1 Running 0 10s
example-cluster-ray-worker-xgs2v 1/1 Running 0 10s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 73s
example-cluster-ray-worker-4ktnh 1/1 Running 0 3s
example-cluster-ray-worker-4msq8 1/1 Running 0 3s
example-cluster-ray-worker-4sg6z 1/1 Running 0 24s
example-cluster-ray-worker-52qk8 0/1 ContainerCreating 0 3s
example-cluster-ray-worker-5wpt8 1/1 Running 0 42s
example-cluster-ray-worker-6d4fs 1/1 Running 0 24s
example-cluster-ray-worker-87p5k 1/1 Running 0 3s
example-cluster-ray-worker-87z52 1/1 Running 0 3s
example-cluster-ray-worker-92jrb 1/1 Running 0 3s
example-cluster-ray-worker-9zthk 0/1 ContainerCreating 0 3s
example-cluster-ray-worker-bv94v 1/1 Running 0 3s
example-cluster-ray-worker-bvpzm 1/1 Running 0 12s
example-cluster-ray-worker-ds6vg 1/1 Running 0 12s
example-cluster-ray-worker-g9t4q 0/1 ContainerCreating 0 3s
example-cluster-ray-worker-hltxf 1/1 Running 0 3s
example-cluster-ray-worker-ht9zj 0/1 ContainerCreating 0 3s
example-cluster-ray-worker-j85vx 0/1 ContainerCreating 0 3s
example-cluster-ray-worker-kp425 1/1 Running 0 24s
example-cluster-ray-worker-rvxbr 1/1 Running 0 3s
example-cluster-ray-worker-s6zxb 1/1 Running 0 12s
example-cluster-ray-worker-twmbf 0/1 ContainerCreating 0 3s
example-cluster-ray-worker-xdpx8 1/1 Running 0 12s
example-cluster-ray-worker-xgs2v 1/1 Running 0 12s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 75s
example-cluster-ray-worker-4ktnh 1/1 Running 0 5s
example-cluster-ray-worker-4msq8 1/1 Running 0 5s
example-cluster-ray-worker-4sg6z 1/1 Running 0 26s
example-cluster-ray-worker-52qk8 1/1 Running 0 5s
example-cluster-ray-worker-5wpt8 1/1 Running 0 44s
example-cluster-ray-worker-6d4fs 1/1 Running 0 26s
example-cluster-ray-worker-87p5k 1/1 Running 0 5s
example-cluster-ray-worker-87z52 1/1 Running 0 5s
example-cluster-ray-worker-92jrb 1/1 Running 0 5s
example-cluster-ray-worker-9zthk 1/1 Running 0 5s
example-cluster-ray-worker-bv94v 1/1 Running 0 5s
example-cluster-ray-worker-bvpzm 1/1 Running 0 14s
example-cluster-ray-worker-ds6vg 1/1 Running 0 14s
example-cluster-ray-worker-g9t4q 1/1 Running 0 5s
example-cluster-ray-worker-hltxf 1/1 Running 0 5s
example-cluster-ray-worker-ht9zj 1/1 Running 0 5s
example-cluster-ray-worker-j85vx 1/1 Running 0 5s
example-cluster-ray-worker-kp425 1/1 Running 0 26s
example-cluster-ray-worker-rvxbr 1/1 Running 0 5s
example-cluster-ray-worker-s6zxb 1/1 Running 0 14s
example-cluster-ray-worker-twmbf 1/1 Running 0 5s
example-cluster-ray-worker-xdpx8 1/1 Running 0 14s
example-cluster-ray-worker-xgs2v 1/1 Running 0 14s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 77s
example-cluster-ray-worker-4ktnh 1/1 Running 0 7s
example-cluster-ray-worker-4msq8 1/1 Running 0 7s
example-cluster-ray-worker-4sg6z 1/1 Running 0 28s
example-cluster-ray-worker-52qk8 1/1 Running 0 7s
example-cluster-ray-worker-5wpt8 1/1 Running 0 46s
example-cluster-ray-worker-6d4fs 1/1 Running 0 28s
example-cluster-ray-worker-87p5k 1/1 Running 0 7s
example-cluster-ray-worker-87z52 1/1 Running 0 7s
example-cluster-ray-worker-92jrb 1/1 Running 0 7s
example-cluster-ray-worker-9zthk 1/1 Running 0 7s
example-cluster-ray-worker-bv94v 1/1 Running 0 7s
example-cluster-ray-worker-bvpzm 1/1 Running 0 16s
example-cluster-ray-worker-ds6vg 1/1 Running 0 16s
example-cluster-ray-worker-g9t4q 1/1 Running 0 7s
example-cluster-ray-worker-hltxf 1/1 Running 0 7s
example-cluster-ray-worker-ht9zj 1/1 Running 0 7s
example-cluster-ray-worker-j85vx 1/1 Running 0 7s
example-cluster-ray-worker-kp425 1/1 Running 0 28s
example-cluster-ray-worker-rvxbr 1/1 Running 0 7s
example-cluster-ray-worker-s6zxb 1/1 Running 0 16s
example-cluster-ray-worker-twmbf 1/1 Running 0 7s
example-cluster-ray-worker-xdpx8 1/1 Running 0 16s
example-cluster-ray-worker-xgs2v 1/1 Running 0 16s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 79s
example-cluster-ray-worker-4ktnh 1/1 Running 0 9s
example-cluster-ray-worker-4msq8 1/1 Running 0 9s
example-cluster-ray-worker-4sg6z 1/1 Running 0 30s
example-cluster-ray-worker-52qk8 1/1 Running 0 9s
example-cluster-ray-worker-5wpt8 1/1 Running 0 48s
example-cluster-ray-worker-6d4fs 1/1 Running 0 30s
example-cluster-ray-worker-87p5k 1/1 Running 0 9s
example-cluster-ray-worker-87z52 1/1 Running 0 9s
example-cluster-ray-worker-92jrb 1/1 Running 0 9s
example-cluster-ray-worker-9zthk 1/1 Running 0 9s
example-cluster-ray-worker-bv94v 1/1 Running 0 9s
example-cluster-ray-worker-bvpzm 1/1 Running 0 18s
example-cluster-ray-worker-ds6vg 1/1 Running 0 18s
example-cluster-ray-worker-g9t4q 1/1 Running 0 9s
example-cluster-ray-worker-hltxf 1/1 Running 0 9s
example-cluster-ray-worker-ht9zj 1/1 Running 0 9s
example-cluster-ray-worker-j85vx 1/1 Running 0 9s
example-cluster-ray-worker-kp425 1/1 Running 0 30s
example-cluster-ray-worker-rvxbr 1/1 Running 0 9s
example-cluster-ray-worker-s6zxb 1/1 Running 0 18s
example-cluster-ray-worker-twmbf 1/1 Running 0 9s
example-cluster-ray-worker-xdpx8 1/1 Running 0 18s
example-cluster-ray-worker-xgs2v 1/1 Running 0 18s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 81s
example-cluster-ray-worker-4ktnh 1/1 Running 0 11s
example-cluster-ray-worker-4msq8 1/1 Running 0 11s
example-cluster-ray-worker-4sg6z 1/1 Running 0 32s
example-cluster-ray-worker-52qk8 1/1 Running 0 11s
example-cluster-ray-worker-5wpt8 1/1 Running 0 50s
example-cluster-ray-worker-6d4fs 1/1 Running 0 32s
example-cluster-ray-worker-87p5k 1/1 Running 0 11s
example-cluster-ray-worker-87z52 1/1 Running 0 11s
example-cluster-ray-worker-92jrb 1/1 Running 0 11s
example-cluster-ray-worker-9zthk 1/1 Running 0 11s
example-cluster-ray-worker-bv94v 1/1 Running 0 11s
example-cluster-ray-worker-bvpzm 1/1 Running 0 20s
example-cluster-ray-worker-ds6vg 1/1 Running 0 20s
example-cluster-ray-worker-g9t4q 1/1 Running 0 11s
example-cluster-ray-worker-hltxf 1/1 Running 0 11s
example-cluster-ray-worker-ht9zj 1/1 Running 0 11s
example-cluster-ray-worker-j85vx 1/1 Running 0 11s
example-cluster-ray-worker-kp425 1/1 Running 0 32s
example-cluster-ray-worker-rvxbr 1/1 Running 0 11s
example-cluster-ray-worker-s6zxb 1/1 Running 0 20s
example-cluster-ray-worker-twmbf 1/1 Running 0 11s
example-cluster-ray-worker-xdpx8 1/1 Running 0 20s
example-cluster-ray-worker-xgs2v 1/1 Running 0 20s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 83s
example-cluster-ray-worker-4ktnh 1/1 Terminating 0 13s
example-cluster-ray-worker-4msq8 1/1 Running 0 13s
example-cluster-ray-worker-4sg6z 1/1 Running 0 34s
example-cluster-ray-worker-52qk8 1/1 Terminating 0 13s
example-cluster-ray-worker-5wpt8 1/1 Running 0 52s
example-cluster-ray-worker-6d4fs 1/1 Running 0 34s
example-cluster-ray-worker-87p5k 1/1 Terminating 0 13s
example-cluster-ray-worker-87z52 1/1 Terminating 0 13s
example-cluster-ray-worker-92jrb 1/1 Terminating 0 13s
example-cluster-ray-worker-9zthk 1/1 Terminating 0 13s
example-cluster-ray-worker-bv94v 1/1 Terminating 0 13s
example-cluster-ray-worker-bvpzm 1/1 Running 0 22s
example-cluster-ray-worker-ds6vg 1/1 Running 0 22s
example-cluster-ray-worker-g9t4q 1/1 Terminating 0 13s
example-cluster-ray-worker-hltxf 1/1 Terminating 0 13s
example-cluster-ray-worker-ht9zj 1/1 Terminating 0 13s
example-cluster-ray-worker-j85vx 1/1 Terminating 0 13s
example-cluster-ray-worker-kp425 1/1 Running 0 34s
example-cluster-ray-worker-rvxbr 1/1 Terminating 0 13s
example-cluster-ray-worker-s6zxb 1/1 Running 0 22s
example-cluster-ray-worker-twmbf 1/1 Terminating 0 13s
example-cluster-ray-worker-xdpx8 1/1 Running 0 22s
example-cluster-ray-worker-xgs2v 1/1 Running 0 22s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 85s
example-cluster-ray-worker-4ktnh 0/1 Terminating 0 15s
example-cluster-ray-worker-4msq8 1/1 Running 0 15s
example-cluster-ray-worker-4sg6z 1/1 Running 0 36s
example-cluster-ray-worker-52qk8 0/1 Terminating 0 15s
example-cluster-ray-worker-5wpt8 1/1 Running 0 54s
example-cluster-ray-worker-6d4fs 1/1 Running 0 36s
example-cluster-ray-worker-87p5k 0/1 Terminating 0 15s
example-cluster-ray-worker-87z52 0/1 Terminating 0 15s
example-cluster-ray-worker-9zthk 0/1 Terminating 0 15s
example-cluster-ray-worker-bv94v 0/1 Terminating 0 15s
example-cluster-ray-worker-bvpzm 1/1 Running 0 24s
example-cluster-ray-worker-ds6vg 1/1 Running 0 24s
example-cluster-ray-worker-g9t4q 0/1 Terminating 0 15s
example-cluster-ray-worker-hltxf 0/1 Terminating 0 15s
example-cluster-ray-worker-ht9zj 0/1 Terminating 0 15s
example-cluster-ray-worker-j85vx 0/1 Terminating 0 15s
example-cluster-ray-worker-kp425 1/1 Running 0 36s
example-cluster-ray-worker-s6zxb 1/1 Running 0 24s
example-cluster-ray-worker-twmbf 0/1 Terminating 0 15s
example-cluster-ray-worker-xdpx8 1/1 Running 0 24s
example-cluster-ray-worker-xgs2v 1/1 Running 0 24s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 86s
example-cluster-ray-worker-4ktnh 0/1 Terminating 0 16s
example-cluster-ray-worker-4msq8 1/1 Running 0 16s
example-cluster-ray-worker-4sg6z 1/1 Running 0 37s
example-cluster-ray-worker-52qk8 0/1 Terminating 0 16s
example-cluster-ray-worker-5wpt8 1/1 Running 0 55s
example-cluster-ray-worker-6d4fs 1/1 Running 0 37s
example-cluster-ray-worker-87p5k 0/1 Terminating 0 16s
example-cluster-ray-worker-87z52 0/1 Terminating 0 16s
example-cluster-ray-worker-9zthk 0/1 Terminating 0 16s
example-cluster-ray-worker-bv94v 0/1 Terminating 0 16s
example-cluster-ray-worker-bvpzm 1/1 Running 0 25s
example-cluster-ray-worker-ds6vg 1/1 Running 0 25s
example-cluster-ray-worker-g9t4q 0/1 Terminating 0 16s
example-cluster-ray-worker-hltxf 0/1 Terminating 0 16s
example-cluster-ray-worker-ht9zj 0/1 Terminating 0 16s
example-cluster-ray-worker-j85vx 0/1 Terminating 0 16s
example-cluster-ray-worker-kp425 1/1 Running 0 37s
example-cluster-ray-worker-s6zxb 1/1 Running 0 25s
example-cluster-ray-worker-twmbf 0/1 Terminating 0 16s
example-cluster-ray-worker-xdpx8 1/1 Running 0 25s
example-cluster-ray-worker-xgs2v 1/1 Running 0 25s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 88s
example-cluster-ray-worker-4msq8 1/1 Running 0 18s
example-cluster-ray-worker-4sg6z 1/1 Running 0 39s
example-cluster-ray-worker-52qk8 0/1 Terminating 0 18s
example-cluster-ray-worker-5wpt8 1/1 Running 0 57s
example-cluster-ray-worker-6d4fs 1/1 Running 0 39s
example-cluster-ray-worker-87z52 0/1 Terminating 0 18s
example-cluster-ray-worker-9zthk 0/1 Terminating 0 18s
example-cluster-ray-worker-bv94v 0/1 Terminating 0 18s
example-cluster-ray-worker-bvpzm 1/1 Running 0 27s
example-cluster-ray-worker-ds6vg 1/1 Running 0 27s
example-cluster-ray-worker-g9t4q 0/1 Terminating 0 18s
example-cluster-ray-worker-ht9zj 0/1 Terminating 0 18s
example-cluster-ray-worker-j85vx 0/1 Terminating 0 18s
example-cluster-ray-worker-kp425 1/1 Running 0 39s
example-cluster-ray-worker-s6zxb 1/1 Running 0 27s
example-cluster-ray-worker-twmbf 0/1 Terminating 0 18s
example-cluster-ray-worker-xdpx8 1/1 Running 0 27s
example-cluster-ray-worker-xgs2v 1/1 Running 0 27s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 91s
example-cluster-ray-worker-4msq8 1/1 Running 0 21s
example-cluster-ray-worker-4sg6z 1/1 Running 0 42s
example-cluster-ray-worker-5wpt8 1/1 Running 0 60s
example-cluster-ray-worker-6d4fs 1/1 Running 0 42s
example-cluster-ray-worker-87z52 0/1 Terminating 0 21s
example-cluster-ray-worker-9zthk 0/1 Terminating 0 21s
example-cluster-ray-worker-bvpzm 1/1 Running 0 30s
example-cluster-ray-worker-ds6vg 1/1 Running 0 30s
example-cluster-ray-worker-g9t4q 0/1 Terminating 0 21s
example-cluster-ray-worker-ht9zj 0/1 Terminating 0 21s
example-cluster-ray-worker-j85vx 0/1 Terminating 0 21s
example-cluster-ray-worker-kp425 1/1 Running 0 42s
example-cluster-ray-worker-s6zxb 1/1 Running 0 30s
example-cluster-ray-worker-twmbf 0/1 Terminating 0 21s
example-cluster-ray-worker-xdpx8 1/1 Running 0 30s
example-cluster-ray-worker-xgs2v 1/1 Running 0 30s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 92s
example-cluster-ray-worker-4msq8 1/1 Running 0 22s
example-cluster-ray-worker-4sg6z 1/1 Running 0 43s
example-cluster-ray-worker-5wpt8 1/1 Running 0 61s
example-cluster-ray-worker-6d4fs 1/1 Running 0 43s
example-cluster-ray-worker-87z52 0/1 Terminating 0 22s
example-cluster-ray-worker-9zthk 0/1 Terminating 0 22s
example-cluster-ray-worker-bvpzm 1/1 Running 0 31s
example-cluster-ray-worker-ds6vg 1/1 Running 0 31s
example-cluster-ray-worker-g9t4q 0/1 Terminating 0 22s
example-cluster-ray-worker-ht9zj 0/1 Terminating 0 22s
example-cluster-ray-worker-kp425 1/1 Running 0 43s
example-cluster-ray-worker-s6zxb 1/1 Running 0 31s
example-cluster-ray-worker-twmbf 0/1 Terminating 0 22s
example-cluster-ray-worker-xdpx8 1/1 Running 0 31s
example-cluster-ray-worker-xgs2v 1/1 Running 0 31s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 94s
example-cluster-ray-worker-4msq8 1/1 Running 0 24s
example-cluster-ray-worker-4sg6z 1/1 Running 0 45s
example-cluster-ray-worker-5wpt8 1/1 Running 0 63s
example-cluster-ray-worker-6d4fs 1/1 Running 0 45s
example-cluster-ray-worker-bvpzm 1/1 Running 0 33s
example-cluster-ray-worker-ds6vg 1/1 Running 0 33s
example-cluster-ray-worker-ht9zj 0/1 Terminating 0 24s
example-cluster-ray-worker-kp425 1/1 Running 0 45s
example-cluster-ray-worker-s6zxb 1/1 Running 0 33s
example-cluster-ray-worker-twmbf 0/1 Terminating 0 24s
example-cluster-ray-worker-xdpx8 1/1 Running 0 33s
example-cluster-ray-worker-xgs2v 1/1 Running 0 33s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 96s
example-cluster-ray-worker-4msq8 1/1 Running 0 26s
example-cluster-ray-worker-4sg6z 1/1 Running 0 47s
example-cluster-ray-worker-5wpt8 1/1 Running 0 65s
example-cluster-ray-worker-6d4fs 1/1 Running 0 47s
example-cluster-ray-worker-bvpzm 1/1 Running 0 35s
example-cluster-ray-worker-ds6vg 1/1 Running 0 35s
example-cluster-ray-worker-kp425 1/1 Running 0 47s
example-cluster-ray-worker-s6zxb 1/1 Running 0 35s
example-cluster-ray-worker-xdpx8 1/1 Running 0 35s
example-cluster-ray-worker-xgs2v 1/1 Running 0 35s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 98s
example-cluster-ray-worker-4msq8 1/1 Running 0 28s
example-cluster-ray-worker-4sg6z 1/1 Running 0 49s
example-cluster-ray-worker-5wpt8 1/1 Running 0 67s
example-cluster-ray-worker-6d4fs 1/1 Running 0 49s
example-cluster-ray-worker-bvpzm 1/1 Running 0 37s
example-cluster-ray-worker-ds6vg 1/1 Running 0 37s
example-cluster-ray-worker-kp425 1/1 Running 0 49s
example-cluster-ray-worker-s6zxb 1/1 Running 0 37s
example-cluster-ray-worker-xdpx8 1/1 Running 0 37s
example-cluster-ray-worker-xgs2v 1/1 Running 0 37s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 100s
example-cluster-ray-worker-4msq8 1/1 Running 0 30s
example-cluster-ray-worker-4sg6z 1/1 Running 0 51s
example-cluster-ray-worker-5wpt8 1/1 Running 0 69s
example-cluster-ray-worker-6d4fs 1/1 Running 0 51s
example-cluster-ray-worker-bvpzm 1/1 Running 0 39s
example-cluster-ray-worker-ds6vg 1/1 Running 0 39s
example-cluster-ray-worker-kp425 1/1 Running 0 51s
example-cluster-ray-worker-s6zxb 1/1 Running 0 39s
example-cluster-ray-worker-xdpx8 1/1 Running 0 39s
example-cluster-ray-worker-xgs2v 1/1 Running 0 39s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 102s
example-cluster-ray-worker-4msq8 1/1 Running 0 32s
example-cluster-ray-worker-4nrvc 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-4qjtv 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-4sg6z 1/1 Running 0 53s
example-cluster-ray-worker-5wpt8 1/1 Running 0 71s
example-cluster-ray-worker-6d4fs 1/1 Running 0 53s
example-cluster-ray-worker-brlxx 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-bvpzm 1/1 Running 0 41s
example-cluster-ray-worker-cv4gn 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-ds6vg 1/1 Running 0 41s
example-cluster-ray-worker-grjwc 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-h4c6r 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-k8j66 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-kp425 1/1 Running 0 53s
example-cluster-ray-worker-lkvf7 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-ppbq2 0/1 ContainerCreating 0 1s
example-cluster-ray-worker-s6zxb 1/1 Running 0 41s
example-cluster-ray-worker-t6ksh 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-vvfv7 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-vzbfr 0/1 ContainerCreating 0 2s
example-cluster-ray-worker-xdpx8 1/1 Running 0 41s
example-cluster-ray-worker-xgs2v 1/1 Running 0 41s
example-cluster-ray-worker-ztxcc 0/1 ContainerCreating 0 2s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 104s
example-cluster-ray-worker-4msq8 1/1 Running 0 34s
example-cluster-ray-worker-4nrvc 0/1 ContainerCreating 0 4s
example-cluster-ray-worker-4qjtv 0/1 ContainerCreating 0 3s
example-cluster-ray-worker-4sg6z 1/1 Running 0 55s
example-cluster-ray-worker-5wpt8 1/1 Running 0 73s
example-cluster-ray-worker-6d4fs 1/1 Running 0 55s
example-cluster-ray-worker-brlxx 1/1 Running 0 4s
example-cluster-ray-worker-bvpzm 1/1 Running 0 43s
example-cluster-ray-worker-cv4gn 0/1 ContainerCreating 0 4s
example-cluster-ray-worker-ds6vg 1/1 Running 0 43s
example-cluster-ray-worker-grjwc 0/1 ContainerCreating 0 4s
example-cluster-ray-worker-h4c6r 1/1 Running 0 4s
example-cluster-ray-worker-k8j66 1/1 Running 0 4s
example-cluster-ray-worker-kp425 1/1 Running 0 55s
example-cluster-ray-worker-lkvf7 0/1 ContainerCreating 0 3s
example-cluster-ray-worker-ppbq2 1/1 Running 0 3s
example-cluster-ray-worker-s6zxb 1/1 Running 0 43s
example-cluster-ray-worker-t6ksh 1/1 Running 0 4s
example-cluster-ray-worker-vvfv7 0/1 ContainerCreating 0 4s
example-cluster-ray-worker-vzbfr 1/1 Running 0 4s
example-cluster-ray-worker-xdpx8 1/1 Running 0 43s
example-cluster-ray-worker-xgs2v 1/1 Running 0 43s
example-cluster-ray-worker-ztxcc 1/1 Running 0 4s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 106s
example-cluster-ray-worker-4msq8 1/1 Running 0 36s
example-cluster-ray-worker-4nrvc 1/1 Running 0 6s
example-cluster-ray-worker-4qjtv 1/1 Running 0 5s
example-cluster-ray-worker-4sg6z 1/1 Running 0 57s
example-cluster-ray-worker-5wpt8 1/1 Running 0 75s
example-cluster-ray-worker-6d4fs 1/1 Running 0 57s
example-cluster-ray-worker-brlxx 1/1 Running 0 6s
example-cluster-ray-worker-bvpzm 1/1 Running 0 45s
example-cluster-ray-worker-cv4gn 1/1 Running 0 6s
example-cluster-ray-worker-ds6vg 1/1 Running 0 45s
example-cluster-ray-worker-grjwc 1/1 Running 0 6s
example-cluster-ray-worker-h4c6r 1/1 Running 0 6s
example-cluster-ray-worker-k8j66 1/1 Running 0 6s
example-cluster-ray-worker-kp425 1/1 Running 0 57s
example-cluster-ray-worker-lkvf7 1/1 Running 0 5s
example-cluster-ray-worker-ppbq2 1/1 Running 0 5s
example-cluster-ray-worker-s6zxb 1/1 Running 0 45s
example-cluster-ray-worker-t6ksh 1/1 Running 0 6s
example-cluster-ray-worker-vvfv7 1/1 Running 0 6s
example-cluster-ray-worker-vzbfr 1/1 Running 0 6s
example-cluster-ray-worker-xdpx8 1/1 Running 0 45s
example-cluster-ray-worker-xgs2v 1/1 Running 0 45s
example-cluster-ray-worker-ztxcc 1/1 Running 0 6s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 108s
example-cluster-ray-worker-4msq8 1/1 Running 0 38s
example-cluster-ray-worker-4nrvc 1/1 Running 0 8s
example-cluster-ray-worker-4qjtv 1/1 Running 0 7s
example-cluster-ray-worker-4sg6z 1/1 Running 0 59s
example-cluster-ray-worker-5wpt8 1/1 Running 0 77s
example-cluster-ray-worker-6d4fs 1/1 Running 0 59s
example-cluster-ray-worker-brlxx 1/1 Running 0 8s
example-cluster-ray-worker-bvpzm 1/1 Running 0 47s
example-cluster-ray-worker-cv4gn 1/1 Running 0 8s
example-cluster-ray-worker-ds6vg 1/1 Running 0 47s
example-cluster-ray-worker-grjwc 1/1 Running 0 8s
example-cluster-ray-worker-h4c6r 1/1 Running 0 8s
example-cluster-ray-worker-k8j66 1/1 Running 0 8s
example-cluster-ray-worker-kp425 1/1 Running 0 59s
example-cluster-ray-worker-lkvf7 1/1 Running 0 7s
example-cluster-ray-worker-ppbq2 1/1 Running 0 7s
example-cluster-ray-worker-s6zxb 1/1 Running 0 47s
example-cluster-ray-worker-t6ksh 1/1 Running 0 8s
example-cluster-ray-worker-vvfv7 1/1 Running 0 8s
example-cluster-ray-worker-vzbfr 1/1 Running 0 8s
example-cluster-ray-worker-xdpx8 1/1 Running 0 47s
example-cluster-ray-worker-xgs2v 1/1 Running 0 47s
example-cluster-ray-worker-ztxcc 1/1 Running 0 8s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 110s
example-cluster-ray-worker-4msq8 1/1 Running 0 40s
example-cluster-ray-worker-4nrvc 1/1 Running 0 10s
example-cluster-ray-worker-4qjtv 1/1 Running 0 9s
example-cluster-ray-worker-4sg6z 1/1 Running 0 61s
example-cluster-ray-worker-5wpt8 1/1 Running 0 79s
example-cluster-ray-worker-6d4fs 1/1 Running 0 61s
example-cluster-ray-worker-brlxx 1/1 Running 0 10s
example-cluster-ray-worker-bvpzm 1/1 Running 0 49s
example-cluster-ray-worker-cv4gn 1/1 Running 0 10s
example-cluster-ray-worker-ds6vg 1/1 Running 0 49s
example-cluster-ray-worker-grjwc 1/1 Running 0 10s
example-cluster-ray-worker-h4c6r 1/1 Running 0 10s
example-cluster-ray-worker-k8j66 1/1 Running 0 10s
example-cluster-ray-worker-kp425 1/1 Running 0 61s
example-cluster-ray-worker-lkvf7 1/1 Running 0 9s
example-cluster-ray-worker-ppbq2 1/1 Running 0 9s
example-cluster-ray-worker-s6zxb 1/1 Running 0 49s
example-cluster-ray-worker-t6ksh 1/1 Running 0 10s
example-cluster-ray-worker-vvfv7 1/1 Running 0 10s
example-cluster-ray-worker-vzbfr 1/1 Running 0 10s
example-cluster-ray-worker-xdpx8 1/1 Running 0 49s
example-cluster-ray-worker-xgs2v 1/1 Running 0 49s
example-cluster-ray-worker-ztxcc 1/1 Running 0 10s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 111s
example-cluster-ray-worker-4msq8 1/1 Running 0 41s
example-cluster-ray-worker-4nrvc 1/1 Running 0 11s
example-cluster-ray-worker-4qjtv 1/1 Running 0 10s
example-cluster-ray-worker-4sg6z 1/1 Running 0 62s
example-cluster-ray-worker-5wpt8 1/1 Running 0 80s
example-cluster-ray-worker-6d4fs 1/1 Running 0 62s
example-cluster-ray-worker-brlxx 1/1 Running 0 11s
example-cluster-ray-worker-bvpzm 1/1 Running 0 50s
example-cluster-ray-worker-cv4gn 1/1 Running 0 11s
example-cluster-ray-worker-ds6vg 1/1 Running 0 50s
example-cluster-ray-worker-grjwc 1/1 Running 0 11s
example-cluster-ray-worker-h4c6r 1/1 Running 0 11s
example-cluster-ray-worker-k8j66 1/1 Running 0 11s
example-cluster-ray-worker-kp425 1/1 Running 0 62s
example-cluster-ray-worker-lkvf7 1/1 Running 0 10s
example-cluster-ray-worker-ppbq2 1/1 Running 0 10s
example-cluster-ray-worker-s6zxb 1/1 Running 0 50s
example-cluster-ray-worker-t6ksh 1/1 Running 0 11s
example-cluster-ray-worker-vvfv7 1/1 Running 0 11s
example-cluster-ray-worker-vzbfr 1/1 Running 0 11s
example-cluster-ray-worker-xdpx8 1/1 Running 0 50s
example-cluster-ray-worker-xgs2v 1/1 Running 0 50s
example-cluster-ray-worker-ztxcc 1/1 Running 0 11s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 113s
example-cluster-ray-worker-4msq8 1/1 Running 0 43s
example-cluster-ray-worker-4nrvc 1/1 Terminating 0 13s
example-cluster-ray-worker-4qjtv 0/1 Terminating 0 12s
example-cluster-ray-worker-4sg6z 1/1 Running 0 64s
example-cluster-ray-worker-5wpt8 1/1 Running 0 82s
example-cluster-ray-worker-6d4fs 1/1 Running 0 64s
example-cluster-ray-worker-brlxx 1/1 Running 0 13s
example-cluster-ray-worker-bvpzm 1/1 Running 0 52s
example-cluster-ray-worker-cv4gn 1/1 Terminating 0 13s
example-cluster-ray-worker-ds6vg 1/1 Running 0 52s
example-cluster-ray-worker-grjwc 1/1 Terminating 0 13s
example-cluster-ray-worker-h4c6r 0/1 Terminating 0 13s
example-cluster-ray-worker-k8j66 0/1 Terminating 0 13s
example-cluster-ray-worker-kp425 1/1 Running 0 64s
example-cluster-ray-worker-lkvf7 0/1 Terminating 0 12s
example-cluster-ray-worker-ppbq2 1/1 Terminating 0 12s
example-cluster-ray-worker-s6zxb 1/1 Running 0 52s
example-cluster-ray-worker-t6ksh 1/1 Terminating 0 13s
example-cluster-ray-worker-vvfv7 0/1 Terminating 0 13s
example-cluster-ray-worker-vzbfr 0/1 Terminating 0 13s
example-cluster-ray-worker-xdpx8 1/1 Running 0 52s
example-cluster-ray-worker-xgs2v 1/1 Running 0 52s
example-cluster-ray-worker-ztxcc 0/1 Terminating 0 13s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 115s
example-cluster-ray-worker-4msq8 1/1 Running 0 45s
example-cluster-ray-worker-4nrvc 0/1 Terminating 0 15s
example-cluster-ray-worker-4qjtv 0/1 Terminating 0 14s
example-cluster-ray-worker-4sg6z 1/1 Running 0 66s
example-cluster-ray-worker-5wpt8 1/1 Running 0 84s
example-cluster-ray-worker-6d4fs 1/1 Running 0 66s
example-cluster-ray-worker-brlxx 1/1 Running 0 15s
example-cluster-ray-worker-bvpzm 1/1 Running 0 54s
example-cluster-ray-worker-cv4gn 0/1 Terminating 0 15s
example-cluster-ray-worker-ds6vg 1/1 Running 0 54s
example-cluster-ray-worker-grjwc 0/1 Terminating 0 15s
example-cluster-ray-worker-h4c6r 0/1 Terminating 0 15s
example-cluster-ray-worker-k8j66 0/1 Terminating 0 15s
example-cluster-ray-worker-kp425 1/1 Running 0 66s
example-cluster-ray-worker-lkvf7 0/1 Terminating 0 14s
example-cluster-ray-worker-ppbq2 0/1 Terminating 0 14s
example-cluster-ray-worker-s6zxb 1/1 Running 0 54s
example-cluster-ray-worker-t6ksh 0/1 Terminating 0 15s
example-cluster-ray-worker-vvfv7 0/1 Terminating 0 15s
example-cluster-ray-worker-vzbfr 0/1 Terminating 0 15s
example-cluster-ray-worker-xdpx8 1/1 Running 0 54s
example-cluster-ray-worker-xgs2v 1/1 Running 0 54s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 117s
example-cluster-ray-worker-4msq8 1/1 Terminating 0 47s
example-cluster-ray-worker-4qjtv 0/1 Terminating 0 16s
example-cluster-ray-worker-4sg6z 0/1 Terminating 0 68s
example-cluster-ray-worker-5wpt8 0/1 Terminating 0 86s
example-cluster-ray-worker-6d4fs 0/1 Terminating 0 68s
example-cluster-ray-worker-brlxx 0/1 Terminating 0 17s
example-cluster-ray-worker-bvpzm 0/1 Terminating 0 56s
example-cluster-ray-worker-ds6vg 0/1 Terminating 0 56s
example-cluster-ray-worker-grjwc 0/1 Terminating 0 17s
example-cluster-ray-worker-h4c6r 0/1 Terminating 0 17s
example-cluster-ray-worker-k8j66 0/1 Terminating 0 17s
example-cluster-ray-worker-kp425 1/1 Terminating 0 68s
example-cluster-ray-worker-ppbq2 0/1 Terminating 0 16s
example-cluster-ray-worker-s6zxb 1/1 Terminating 0 56s
example-cluster-ray-worker-t6ksh 0/1 Terminating 0 17s
example-cluster-ray-worker-vvfv7 0/1 Terminating 0 17s
example-cluster-ray-worker-vzbfr 0/1 Terminating 0 17s
example-cluster-ray-worker-xdpx8 0/1 Terminating 0 56s
example-cluster-ray-worker-xgs2v 1/1 Terminating 0 56s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 119s
example-cluster-ray-worker-4msq8 0/1 Terminating 0 49s
example-cluster-ray-worker-4qjtv 0/1 Terminating 0 18s
example-cluster-ray-worker-4sg6z 0/1 Terminating 0 70s
example-cluster-ray-worker-5wpt8 0/1 Terminating 0 88s
example-cluster-ray-worker-6d4fs 0/1 Terminating 0 70s
example-cluster-ray-worker-brlxx 0/1 Terminating 0 19s
example-cluster-ray-worker-bvpzm 0/1 Terminating 0 58s
example-cluster-ray-worker-ds6vg 0/1 Terminating 0 58s
example-cluster-ray-worker-grjwc 0/1 Terminating 0 19s
example-cluster-ray-worker-h4c6r 0/1 Terminating 0 19s
example-cluster-ray-worker-k8j66 0/1 Terminating 0 19s
example-cluster-ray-worker-kp425 0/1 Terminating 0 70s
example-cluster-ray-worker-ppbq2 0/1 Terminating 0 18s
example-cluster-ray-worker-s6zxb 0/1 Terminating 0 58s
example-cluster-ray-worker-t6ksh 0/1 Terminating 0 19s
example-cluster-ray-worker-vvfv7 0/1 Terminating 0 19s
example-cluster-ray-worker-vzbfr 0/1 Terminating 0 19s
example-cluster-ray-worker-xdpx8 0/1 Terminating 0 58s
example-cluster-ray-worker-xgs2v 0/1 Terminating 0 58s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m1s
example-cluster-ray-worker-4msq8 0/1 Terminating 0 51s
example-cluster-ray-worker-4qjtv 0/1 Terminating 0 20s
example-cluster-ray-worker-4sg6z 0/1 Terminating 0 72s
example-cluster-ray-worker-6d4fs 0/1 Terminating 0 72s
example-cluster-ray-worker-brlxx 0/1 Terminating 0 21s
example-cluster-ray-worker-bvpzm 0/1 Terminating 0 60s
example-cluster-ray-worker-ds6vg 0/1 Terminating 0 60s
example-cluster-ray-worker-h4c6r 0/1 Terminating 0 21s
example-cluster-ray-worker-k8j66 0/1 Terminating 0 21s
example-cluster-ray-worker-kp425 0/1 Terminating 0 72s
example-cluster-ray-worker-ppbq2 0/1 Terminating 0 20s
example-cluster-ray-worker-s6zxb 0/1 Terminating 0 60s
example-cluster-ray-worker-t6ksh 0/1 Terminating 0 21s
example-cluster-ray-worker-vvfv7 0/1 Terminating 0 21s
example-cluster-ray-worker-vzbfr 0/1 Terminating 0 21s
example-cluster-ray-worker-xdpx8 0/1 Terminating 0 60s
example-cluster-ray-worker-xgs2v 0/1 Terminating 0 60s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m3s
example-cluster-ray-worker-4msq8 0/1 Terminating 0 53s
example-cluster-ray-worker-4qjtv 0/1 Terminating 0 22s
example-cluster-ray-worker-4sg6z 0/1 Terminating 0 74s
example-cluster-ray-worker-6d4fs 0/1 Terminating 0 74s
example-cluster-ray-worker-brlxx 0/1 Terminating 0 23s
example-cluster-ray-worker-ds6vg 0/1 Terminating 0 62s
example-cluster-ray-worker-h4c6r 0/1 Terminating 0 23s
example-cluster-ray-worker-k8j66 0/1 Terminating 0 23s
example-cluster-ray-worker-kp425 0/1 Terminating 0 74s
example-cluster-ray-worker-ppbq2 0/1 Terminating 0 22s
example-cluster-ray-worker-vzbfr 0/1 Terminating 0 23s
example-cluster-ray-worker-xdpx8 0/1 Terminating 0 62s
example-cluster-ray-worker-xgs2v 0/1 Terminating 0 62s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m5s
example-cluster-ray-worker-4msq8 0/1 Terminating 0 55s
example-cluster-ray-worker-6d4fs 0/1 Terminating 0 76s
example-cluster-ray-worker-ds6vg 0/1 Terminating 0 64s
example-cluster-ray-worker-k8j66 0/1 Terminating 0 25s
example-cluster-ray-worker-kp425 0/1 Terminating 0 76s
example-cluster-ray-worker-xdpx8 0/1 Terminating 0 64s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m6s
example-cluster-ray-worker-4msq8 0/1 Terminating 0 56s
example-cluster-ray-worker-ds6vg 0/1 Terminating 0 65s
example-cluster-ray-worker-kp425 0/1 Terminating 0 77s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m8s
example-cluster-ray-worker-ds6vg 0/1 Terminating 0 67s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m10s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m11s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m13s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m15s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m16s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m18s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m19s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m21s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m22s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m24s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m26s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m27s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m29s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m30s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m32s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m34s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m35s
NAME READY STATUS RESTARTS AGE
example-cluster-ray-head-q7hb8 1/1 Running 0 2m37s
Hey just a note that this could be actually a regression from our recent changes. What version of Ray did you use when you ran the experiment?
cc @Alex So this seems to be a separate issue from what we discovered.
just re-reading this – the lengthy nlogs i included above actually illustrate what Dmitri also described here: [k8s] min_workers doesn't seem to be honored · Issue #14400 · ray-project/ray · GitHub
So probably not relevant to the topic of this thread around processing throughput of threads.
Doe you have nay benchmark values for that to compare against?
Alex
March 3, 2021, 6:49pm
11
@mbehrendt can you double check the version of ray running on the cluster?
You mentioned using 1.2 and the example yaml, but the example uses nightly wheels. I also can’t reproduce your issue in 1.2, but can reproduce a similar issue on the latest nightly.
The best way to confirm the ray version of the cluster is to run ray --version from the head node of the cluster. Note this can be different from your local ray version.
good question. Frankly, I’m a bit confused about the versioning scheme – in pip it says 1.2, while ray --version shows 2.0.0. . Does that help?
@Alex are there any such numbers you have benchmarked and which you could be sharing?
Alex
March 8, 2021, 6:23pm
14
With versioning: 1.2.0 is the latest release version of Ray. We use 2.0.0.dev0 to denote nightly wheels. We only publish the release wheels to pip. This may be where the confusion is coming from?
In terms of benchmarks, I ran your script (I don’t have easy k8s access, so I used m5.2xlarge AWS EC2 instances which have the same number of cores as your setup) and I saw it take 14s.
The related problem I was mentioning happens if you put a sleep in the remote function.
Alex
March 8, 2021, 6:54pm
15
Btw, in my run I think we’re limited by the number of tasks that the driver process can submit not the number that the worker nodes can process. (This is part of why very small tasks is an antipattern Ray Design Patterns - Google Docs )