Ok I got an answer to this question on the Ray slack. I don’t want to give names of who said this because I don’t have permission (didn’t ask), but basically:
“You should think of a Ray cluster as basically flammable. In production scenarios anything that uses Ray should be wrapped in external retries and durable external stores.”
So with that I’m going to proceed with saving result state to cloud storage and I’ll figure out some signalling mechanism to let my calling code know that the job is finished. Probably polling job status but it could be something else.