Error while fetching data from Object Store

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

I am putting image into object store via ray.put() and passing the reference to another actor. When I tried to access it, for some frames I am receiving following error:

Failed to retrieve object 001d81a1e0baa01dade71591650d689f2dec4a52010000008ae3f505. To see information about where this ObjectRef was created in Python, set the environment variable RAY_record_ref_creation_sites=1 during `ray start` and `ray.init()`.
(ResultExtractor pid=2803196) 
(ResultExtractor pid=2803196) The object has already been deleted by the reference counting protocol. This should not happen.

Can anyone please suggest, what could be the issue?

2

Hi @shyampatel,

Could you share the repro script? How do you pass the reference to another actor?

3

Thanks for your reply @jjyao.

I could not reproduce the same via simple script, but it’s occurring in our pipeline.

What we are doing is, to pass image from one actor to another, first I am putting image into object store via img_ref = ray.put(img), and passing this reference with some other information via a class instance of Frame to another actor:

class Frame:
    def __init__(self, img_ref, dets):
        self.img_ref = img_ref
        self.dets = dets
4

How do you pass Frame to the other actor?

5

We are passing Frame instance to another actor via ray queue.

6

@jjyao Can you please suggest, where things could be going wrong ?