How ray memory works on clusters?

Ray Clusters
1

I have a project which need 800G memory.

I used 5 servers which provides 200G memory (no spill to disk), then I have 1000G in total

When I run my project, if servers 1 reach 200G, and other servers is empty, would new nodes store in other servers, or it return a error?

2

is there someone can help me ?

3

hey Asuka do you have some examples on how exactly your code uses memory? If you simply using ray.put to put objects into Ray you might end up storing them into the same server.

4

In you case, ray.put() will use the same server(server 1). When server 1 is filled up, would it use server 2 or just return a error?

In my case , I use ray_dask_get to run a big DAG.

5

ah I think for dask-on-ray, it will spread the memory load across the cluster.

6

so In common case, like ray.put().
Would it return a error?

I got a memory error as I use dask_on_ray.
I looked into the ray dashboard, server1 is filled up but server 2,3,4,5 have lots of empty.