By cascading processes i meant to say a sequence operations performed by individual actors on a data and the each of the actor’s output reference is fed as input to the next actor in the sequence, here all the actor are created once at the beginning of the pipeline by a main supervising actor and handles the object/output reference passing between actors.
Not sure whether a reproducible scripts is possible as the original pipeline involves a much more complex cascading and parallel connection for data process flow, but will try if it is possible to reproduce the error with simple pipeline with less actor processes within short run duration.
The error appears very random,
1st instance came after running the pipeline for 7 days, 2nd instance came up after 1 hour and 3rd instance appeared after 1 day. The pipeline also had a change to run for more than 7 days (approx. 2 weeks) at a stretch without this error appearing. This is another reason for uncertainty in creating a reproducible script
will check if it is reproducing with latest ray version, but may take a bit of time to check depending on the amount of changes needed to the whole pipeline and the sporadic nature of the error.