Autoscaler node type specific object storage amount

Ray Core
1

in autoscaler.py, only worker_setup_commands are node type specific.

setup_commands = self._get_node_type_specific_fields(
     node_id, "worker_setup_commands")
ray_start_commands = self.config["worker_start_ray_commands"]

different aws instance type have different amount of RAM and I would like to specify the object storage amount differently.

ulimit -n 65535; ray start --address=$RAY_HEAD_IP:6379 --object-manager-port=8076 --object-store-memory=142807662592
ulimit -n 65535; ray start --address=$RAY_HEAD_IP:6379 --object-manager-port=8076 --object-store-memory=348966092800

how could I achieve this given that start_ray_commands are not node specific?

thank you!

2

cc @Ameer_Haj_Ali @Alex Can you guys address this question?

3 
    cpu_48_spot:
        node_config:
            InstanceType: c5.24xlarge
    ... 
    worker_setup_commands:
       - ulimit -n 65535; ray start --address=$RAY_HEAD_IP:6379 --object-manager-port=8076 --object-store-memory=142807662592

worker_start_ray_commands: []

would this work? explicitly leaving worker_start_ray_commands empty so that ray actually start using worker_setup_commands

4

Thanks for bringing this up
This feature is tracked here: [Autoscaler] cleanup the cluster yaml · Issue #12960 · ray-project/ray · GitHub

I am not sure that using it in the worker_setup_commands would work, did you try?

5

@Jennice_Tao you should be able to directly specify this resource already without modifying the start command.

available_node_types:
    more_memory:
        resources:
            object_store_memory: 20000
        node_config: {}
    less_memory: 
         resources:
             object_store_memory: 10000
1 Like
6

that looks great. the example yaml uses dictionary. I assume that this is correct too.
resources: { "CPU": 48, "is_spot": 1, "small_memory": 1, "object_store_memory": 142807662592}

7

@Alex what’s the unit of object_store_memory when passed through resource config? same as the option to ray start?

8

It is different. They are currently 50MB chunks. (ray.ray_constants.MEMORY_RESOURCE_UNIT_BYTES).

9

Yeah, to be clear, every place a user should specify memory, should be specified in bytes. It’s only the displayed values in ray.cluster_resources() and available_resources() is in 50MB chunks.

1 Like