Spammy warnings on not registered functions

Ray Core
1

I’m using autoscaler and placement group features. I do get the below warnings pretty often. I ended up ignoring them as it does not actually cause any harm. However, it would be better to remove false positives like these. @sangcho

2021-02-25 22:34:49,091 WARNING worker.py:1107 -- This worker was asked to execute a function that it does not have registered. You may have to restart Ray.
2021-02-25 22:35:02,183 WARNING worker.py:1107 -- This worker was asked to execute a function that it does not have registered. You may have to restart Ray.
2021-02-25 22:35:49,241 WARNING worker.py:1107 -- This worker was asked to execute a function that it does not have registered. You may have to restart Ray.
2021-02-25 22:36:01,951 WARNING worker.py:1107 -- This worker was asked to execute a function that it does not have registered. You may have to restart Ray.
2021-02-25 22:36:12,829 WARNING worker.py:1107 -- This worker was asked to execute a function that it does not have registered. You may have to restart Ray.
2021-02-25 22:36:55,279 WARNING worker.py:1107 -- This worker was asked to execute a function that it does not have registered. You may have to restart Ray.
2021-02-25 22:37:00,874 WARNING worker.py:1107 -- This worker was asked to execute a function that it does not have registered. You may have to restart Ray.
2021-02-25 22:37:10,154 WARNING worker.py:1107 -- This worker was asked to execute a function that it does not have registered. You may have to restart Ray.
2021-02-25 22:37:42,660 WARNING worker.py:1107 -- This worker was asked to execute a function that it does not have registered. You may have to restart Ray.
2021-02-25 22:38:29,663 WARNING worker.py:1107 -- This worker was asked to execute a function that it does not have registered. You may have to restart Ray.
2021-02-25 22:39:58,818 WARNING worker.py:1107 -- This worker was asked to execute a function that it does not have registered. You may have to restart Ray.
2021-02-25 22:40:10,884 WARNING worker.py:1107 -- This worker was asked to execute a function that it does not have registered. You may have to restart Ray.
2021-02-25 22:40:41,026 WARNING worker.py:1107 -- This worker was asked to execute a function that it does not have registered. You may have to restart Ray.
2

Looking at code, this seems to happen due to the pressure to Redis. have you ever seen the Redis cpu usage when you are seeing these error messages?

3

I generally run at 100% cpus, so I assume that redis would have been under pressure for sure.

4

I see. It seems like it is actually already suppressed (one per worker). Maybe I can improve the error message though. Not sure if removing it is a good idea (unless we can have a better way to handle this).

1 Like
5 
There might be resource pressure on this node.

you could add this?

6

I think it is the resource pressure on Redis. Anyway, I will modify the error msg.