Confused on the behavior of "RAY_task_oom_retries" in Actor restarts

cemk · June 23, 2023, 9:22pm

How severe does this issue affect your experience of using Ray?

None: Just asking a question out of curiosity

Hi all, while perusing the config.h file here ray/src/ray/common/ray_config_def.h at ray-2.3.0 · ray-project/ray · GitHub

I stumbled upon this environment flag RAY_task_oom_retries which is set to -1 by default. I am confused as to the implications of this flag on actor restarts.

In the description this seems to imply that when an actor is killed by the memory monitor due to OOM (hence the actor also dies) dose this not count as a restart in the max_restarts? In which case I think this is not documented in the actor fault tolerance or actor page on the Ray docs because I was under the impression that any actor restart will decrease the max_restarts counter.

yic · June 26, 2023, 9:34pm

@cemk OOM retries and the max_retries are two separate counters. If the task failed due to OOM, it’ll use the oom counter. So if it’s set to 1, and it OOMed twice, it won’t retry.

github.com

ray-project/ray/blob/master/src/ray/core_worker/task_manager.cc#L795


      
          bool task_failed_due_to_oom = error_info.error_type() == rpc::ErrorType::OUT_OF_MEMORY;
          {
            absl::MutexLock lock(&mu_);
            auto it = submissible_tasks_.find(task_id);
            RAY_CHECK(it != submissible_tasks_.end())
                << "Tried to retry task that was not pending " << task_id;
            RAY_CHECK(it->second.IsPending())
                << "Tried to retry task that was not pending " << task_id;
            spec = it->second.spec;
            num_retries_left = it->second.num_retries_left;
            num_oom_retries_left = it->second.num_oom_retries_left;
            if (task_failed_due_to_oom) {
              if (num_oom_retries_left > 0) {
                will_retry = true;
                it->second.num_oom_retries_left--;
              } else if (num_oom_retries_left == -1) {
                will_retry = true;
              } else {
                RAY_CHECK(num_oom_retries_left == 0);
              }
            } else {

Topic		Replies	Views
How to prevent ray from retrying an actor task while the actor is restarting? Ray Core	1	149	October 31, 2023
Newbi Question: Worker Fault Tolerance?	4	506	February 28, 2022
Is ray setting memory resource? Ray Core	4	306	May 24, 2021
Although node memory usage is high, I don't want to kill my actor Ray Train	3	428	February 2, 2023
Best practice for custom actor recovery Ray Core	1	285	May 23, 2022

Confused on the behavior of "RAY_task_oom_retries" in Actor restarts

Related Topics