How can i get the current retry time and max retry times within task

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • None: Just asking a question out of curiosity

I am using ray as the backend of the task scheduling system, and I would like to obtain the maximum number of retries for task execution in the task , as well as the current number of retries. According to my research documents, I have found that the relevant API has not yet been exposed. How should I obtain it? Is it safe to do so

2

Hi @AndreKuu,

You can set the number of retries for each task via the max_retries option.

@ray.remote(max_retries=5)
def func():
   pass

You cannot know the current number of retries since retry is transparent to the end user.

3

Thank you, jjyao. Because of the usage scenario, i need to wrap a layer of packaging on the outer layer of ray. When the task was retring, i want to get the current number of retries to do some recording. Now I have confirmed that there are no publicly exposed APIs that can obtain this information here. Thanks!

1 Like
4

The information is technically available from the dashboard (for the observability purpose) in the latest Ray versions (I believe from ray 2.3).

There’s no programmatic way to obtain the information or print since as Jiajun said, the semantics is to transparently retry. It is not a bad idea to use your way if you’d like to add more information.

1 Like
5

Got it. Thanks, sangcho.