At what size will objects no longer use local storage

marsupialtail · February 17, 2022, 3:44am

When I do ray.put, what sized objects should I expect to stay local? Is there a way to control that explicitly?

I would like the object to stay on one machine for locality.

yic · February 17, 2022, 9:13am

When you call ray.put, from my memory, it always stays on the node of the worker where it’s executed. @sangcho or @Chen_Shen may double confirm that.

marsupialtail · February 17, 2022, 4:20pm

From the Ray arch google doc, it seems to suggest that large objects get stored in the “distributed object store”. I don’t know what “large” means

yic · February 17, 2022, 9:33pm

Even it’s put into distributed object store, it’s still local and there is no copy that will happen.

yic · February 17, 2022, 10:01pm

To answer your original question, the threshold is defined here:

github.com

ray-project/ray/blob/master/src/ray/common/ray_config_def.h#L147

      
        
            /// even balancing of load. Low values (min 0.0) encourage more load spreading.
            RAY_CONFIG(float, scheduler_spread_threshold, 0.5);
            
            
/// Whether to only report the usage of pinned copies of objects in the
            /// object_store_memory resource. This means nodes holding secondary copies only
            /// will become eligible for removal in the autoscaler.
            RAY_CONFIG(bool, scheduler_report_pinned_bytes_only, false)
            
            
// The max allowed size in bytes of a return object from direct actor calls.
            // Objects larger than this size will be spilled/promoted to plasma.
            RAY_CONFIG(int64_t, max_direct_call_object_size, 100 * 1024)
            
            
// The max gRPC message size (the gRPC internal default is 4MB). We use a higher
            // limit in Ray to avoid crashing with many small inlined task arguments.
            RAY_CONFIG(int64_t, max_grpc_message_size, 100 * 1024 * 1024)
            
            
// Retry timeout for trying to create a gRPC server. Only applies if the number
            // of retries is non zero.
            RAY_CONFIG(int64_t, grpc_server_retry_timeout_milliseconds, 1000)
            
            
// The min number of retries for direct actor creation tasks. The actual number

which is 100K

sangcho · February 19, 2022, 2:16pm

When you call ray.put, it is always put into the object store, and they are always local to nodes that call ray.put.

Topic		Replies	Views
In-process object store of the driver process Ray Core	8	412	January 28, 2021
Will it be stored in plasma regardless of the size of the data? Ray Core	4	589	November 25, 2021
How does Ray handle when a single object exceeds the Plasma capacity? Ray Core	2	436	November 29, 2021
Ray.put from Ray task - local or not Ray Core	9	553	March 15, 2021
[Core] Controlling object store size? Ray Core	2	1674	November 20, 2021

At what size will objects no longer use local storage

Related topics