How do they work together without canceling each other? I see memory monitor has RAY_memory_usage_threshold, how is it calculated in terms of object store memory and worker’s heap space? Does Object Spilling have any similar threshold to configure?
hi @dirtyValera
memory_usage_threshold controls when Ray starts killing workers to prevent running into out-of-memory problems. It’s triggered when all the Ray processes’ memory usages (including heap space, object store memory) exceed the memory_usage_threshold limit.
You can refer to Out-Of-Memory Prevention — Ray 2.2.0 for more details.
For object spilling, yes, there is a configuration as well RAY_object_spilling_threshold, where the Ray object store will start spilling objects if the object store’s usage over this threshold. For more on object_spilling you might want to refer to Ray white paper