I have this Azure cluster that uses Spot Machines as workers. I am trying to create some actors that “hot start” nodes before doing any computation. What is weird is that it doesn’t seems spawn enough worker Pods. Each to be spawned node has 8 vCPUS and 32Gi - Azure Standard_D8ds_v5 If you refer…

Solution (also in GH ticket) is this: [image]

Interesting > is this happening consistently where there’s a 1-2 resource gap?

Yes, I’ve tried a couple times. The only moment it seem to execute correctly is when I spawn only 1 Actor. Which it auto scales correctly to 1 worker

I’ve made a change and made the Head node possible to schedule stuff. So, addednum-cpus=8 to the head node spec. Seems to have fixed. My Impression is that the printing and task scheduling is correct, but the auto-scaling is considering the head node resources as available. Edit: it has not

@Sam_Chan Did some extra investigation. The issue kept happening. Instead of using the actors to “hot start” the nodes, I tried using the placement groups and it had the same behavior. But now I had a small insight. If I add a sleep timer when I want to start the actors on the python script above…

Oh interesting - so you’re running a Ray Cluster on top of a K8s clusters sitting on Azure? Are you using AKS?

Yes, Ray cluster on top of AKS. Issue still happening, sometimes worse depending on the amount of nodes I request. The issue also happens if I try to spawn new pods while some previous provisioned ones are in “Terminating state” (but not yet fully terminated)

I had opened an issue on git, [Core] Ray Worker stuck in launching state - Azure AKS · Issue #45775 · ray-project/ray · GitHub . Looks like the issue is a problem with the autoscale V2, I didn’t notice I was using an alpha version.

Ray Cluster seem to be spawning less nodes than it should

Ray Clusters

Sam_Chan August 28, 2024, 6:42pm 9

Solution (also in GH ticket) is this:

Topic		Replies	Views
Ray starts head node succesfully but no workers (Azure) Ray Clusters	2	605	June 29, 2022
Ray cluster is stuck in creating worker nodes Ray Clusters	0	425	August 27, 2021
Create a cluster on azure the workers node no available when resource request Ray Clusters	0	96	April 4, 2024
Scale up from 0 Ray Clusters	7	624	July 15, 2021
RayServe Autoscaling not creating Ray Pods Ray Serve	3	354	March 29, 2024

Ray Cluster seem to be spawning less nodes than it should

Related topics