Is ray setting memory resource?

Ray Core
1

I got this warning message. I only set the CPU requirement, and it seems to consider memory resources. So is there a default memory resource setting for one actor?

2021-05-22 11:20:03,844	WARNING worker.py:1115 -- The actor or task with ID d4cbda1bcbc948bdffffffffffffffffffffffff01000000 cannot be scheduled right now. It requires {CPU: 1.000000} for placement, but this node only has remaining {387.000000/460.000000 CPU, 797.301085 GiB/797.301085 GiB memory, 186.264515 GiB/186.264515 GiB object_store_memory, 387.000000/460.000000 scheduler-0, 387.000000/460.000000 scheduler, 1.000000/1.000000 node:10.132.167.88}
. In total there are 1 pending tasks and 0 pending actors on this node. This is likely due to all cluster resources being claimed by actors. To resolve the issue, consider creating fewer actors or increase the resources available to this Ray cluster. You can ignore this message if this Ray cluster is expected to auto-scale.
2

This issue doesn’t exist when we were using 1.0.0. It appears when we update ray to 1.3.0.

3

It doesn’t consider memory resources actually. Usually after some time, your task or actor should be scheduled after you see this message, Are your tasks/actors hanging?

4

(I think we updated the condition to print this log message, and that could be a reason why you are seeing it).

5

Okay, thanks for clarifying this issue. I was just worried about there was a hidden bug.

1 Like
6

[i have 8gb of ram]
in ray dashboard it shows allocated memory:1.5gb (so i thought it will automatically use the resources when it is needed, but then i tried setting the memory=7gb for a task ,at the time of running task it throwed a error saying no node is available with the required resources)

how to change 1.5gb to 8gb

7

Can you try setting something up a bit lower from the top threshold ie: 5GB and see if it schedules successfully?

8

is there a way to change the allocated memory ,cpu,gpu etc…

AND one more doubt is there a way to get the meomery_uage,cpu_usage,gpu_usage percentage for a particular task through CLI

Thank You…

9

You can see it per Actor but not at the Task level. See here for how to change mem, cpu, and gpu for Resource Allocation — Ray 2.34.0