Confused on the behavior of "RAY_task_oom_retries" in Actor restarts

cemk · June 23, 2023, 9:22pm

How severe does this issue affect your experience of using Ray?

None: Just asking a question out of curiosity

Hi all, while perusing the config.h file here ray/src/ray/common/ray_config_def.h at ray-2.3.0 · ray-project/ray · GitHub

I stumbled upon this environment flag RAY_task_oom_retries which is set to -1 by default. I am confused as to the implications of this flag on actor restarts.

In the description this seems to imply that when an actor is killed by the memory monitor due to OOM (hence the actor also dies) dose this not count as a restart in the max_restarts? In which case I think this is not documented in the actor fault tolerance or actor page on the Ray docs because I was under the impression that any actor restart will decrease the max_restarts counter.

yic · June 26, 2023, 9:34pm

@cemk OOM retries and the max_retries are two separate counters. If the task failed due to OOM, it’ll use the oom counter. So if it’s set to 1, and it OOMed twice, it won’t retry.

github.com

ray-project/ray/blob/master/src/ray/core_worker/task_manager.cc#L795


      
          bool task_failed_due_to_oom = error_info.error_type() == rpc::ErrorType::OUT_OF_MEMORY;
          {
            absl::MutexLock lock(&mu_);
            auto it = submissible_tasks_.find(task_id);
            RAY_CHECK(it != submissible_tasks_.end())
                << "Tried to retry task that was not pending " << task_id;
            RAY_CHECK(it->second.IsPending())
                << "Tried to retry task that was not pending " << task_id;
            spec = it->second.spec;
            num_retries_left = it->second.num_retries_left;
            num_oom_retries_left = it->second.num_oom_retries_left;
            if (task_failed_due_to_oom) {
              if (num_oom_retries_left > 0) {
                will_retry = true;
                it->second.num_oom_retries_left--;
              } else if (num_oom_retries_left == -1) {
                will_retry = true;
              } else {
                RAY_CHECK(num_oom_retries_left == 0);
              }
            } else {

Topic		Replies	Views
How to prevent ray from retrying an actor task while the actor is restarting? Ray Core	1	208	October 31, 2023
Restarting task that was running before Actor killed for OOM Ray Core	3	53	June 25, 2024
Newbi Question: Worker Fault Tolerance?	4	544	February 28, 2022
Memory Scheduled Tasks OOM Ray Clusters	1	34	August 9, 2024
(raylet) node_manager.cc Workers (tasks / actors) killed due to memory pressure (OOM)	2	268	March 6, 2024

Confused on the behavior of "RAY_task_oom_retries" in Actor restarts

Related topics