Autoscaler scale down slow

[KurtT]

Hi all,
when I used the Autoscaler(v1.2) in GCP, I found when scale down the instances, it’s slow, approximately 1 and 2 instance get terminated at a time, is it normal? Do we have any parameter can control the instance termination speed? Like “upscaling_speed” can control the upscale speed. The slowness will not make it production possible when we have big cluster.

[rliaw]

@KurtT

when I used the Autoscaler(v1.2) in GCP, I found when scale down the instances, it’s slow, approximately 1 and 2 instance get terminated at a time, is it normal? Do we have any parameter can control the instance termination speed? Like “upscaling_speed” can control the upscale speed. The slowness will not make it production possible when we have big cluster.

@Ameer_Haj_Ali do you have any tips here?

[Ameer_Haj_Ali]

Hi @kurtT, the scale down is actually immediate, any idle worker that has been idle for more than idle_timeout_minutes (Cluster YAML Configuration Options — Ray v2.0.0.dev0) and the number of workers of that node type is larger than min_workers (Cluster YAML Configuration Options — Ray v2.0.0.dev0) is automatically terminated.

[KurtT]

hi @Ameer_Haj_Ali , Thanks for the response. is Autoscaler possibly doing the node terminate in sequence & wait mode? That means wait until the last instance got terminated and then go to the next one for terminate? That could be the reason why I’m seeing the slowly deletion when it needs to scale down many nodes?