# @ray.remote(max\_calls=1) executing too early

**URL:** <https://discuss.ray.io/t/ray-remote-max-calls-1-executing-too-early/354>\
**Category:** Uncategorized\
**Created:** [December 30, 2020, 4:46pm UTC](https://discuss.ray.io/t/ray-remote-max-calls-1-executing-too-early/354 "2020-12-30T16:46:53Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Erik\_Brakkee](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/erik_brakkee/32/198_2.png) [@Erik\_Brakkee](https://discuss.ray.io/u/Erik_Brakkee)\
**Post date:** [December 30, 2020, 4:46pm UTC](https://discuss.ray.io/t/ray-remote-max-calls-1-executing-too-early/354/1 "2020-12-30T16:46:53Z")

</div>

Hi,

I have run into a subtle issue with @ray.remote. I am running a keras model on a server with a single GPU as part of a grid search. I am using max\_calls=1 with @ray.remote to make sure that all resources are cleaned up between runs. Otherwise, I get OOM errors. Now, what I am seeing is that after setting up the nvidia persistence daemon, the 2nd tasks starts too quickly after the first and this leads to an OOM error. I can fix this problem by simply sleeping for a few seconds between runs. So it seems that the next task is executing too quickly after the first and starts while the cleanup of the previous task is not yet finished.

This is what I am doing now:

```
@ray.remote(num_gpus = 1, max_calls = 1)
def ray_train_model(*, model_name, loss_name):
    import time
    time.sleep(5)
    return train_model(model_name, loss_name)

```

Note that I am doing a sleep at the beginning to wait for cleanups from the previous task. Doing a sleep at the end would not work of course.

Cheers  
Erik

---

<div class="post-metadata">

**Author:** ![simon-mo](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/simon-mo/32/9_2.png) [@simon-mo](https://discuss.ray.io/u/simon-mo)\
**Post date:** [December 30, 2020, 10:38pm UTC](https://discuss.ray.io/t/ray-remote-max-calls-1-executing-too-early/354/2 "2020-12-30T22:38:28Z")

</div>

In ray the effect of `max_calls` is actually manifested as the worker process exit. There might be some cleanup that happens before the worker process properly exits. I think what happened was:

```auto
first worker finishes
first worker start exit seqeuence
second worker starts
second worker starts executing task
[but first worker still haven't finish cleanup up (e.g. freeing CUDA memory)]
OOM

```

I don’t think there’s a direct workaround other than the sleep. Thanks for bringing it up though!