Ray.cloudpickle error while pickling ray.dataset

Ray Core
1

I am trying to pickle a python dictionary object containing ray dataset. The purpose of doing this is to checkpoint the dataset so that it can be used later for fault tolerance.

Here the code which I am using to do so:

import ray
import ray.cloudpickle as cloudpickle
# ray.init("ray://localhost:10001")
ray_dataset = ray.data.range(10)
map = {'key1': 'value1', 'ray_dataset' : ray_dataset}
pickled_map = cloudpickle.dumps(map)
print("pickled_map:" , pickled_map)

map1 = ray.cloudpickle.loads(pickled_map)
print("map1:" , map1)

The above code works when run in local standalone mode.
But when I uncomment in line: ray.init("ray://localhost:10001") to connect to a ray cluster run using local k8s setup, it fails with the below mentioned error:

    pickled_map = cloudpickle.dumps(map)
  File "<python site packages dir>/ray/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "<python site packages dir>/ray/cloudpickle/cloudpickle_fast.py", line 620, in dump
    return Pickler.dump(self, obj)
  File "<python site packages dir>/ray/util/client/common.py", line 340, in __call__
    raise TypeError("Actor methods cannot be called directly. Instead "
TypeError: Actor methods cannot be called directly. Instead of running 'object.__getstate__()', try 'object.__getstate__.remote()'.

Please let me know how can we get the pickling work here.

How severe does this issue affect your experience of using Ray?

  • Medium: It contributes to significant difficulty to complete my task, but I can work around it.
2

@Clark_Zinzow could you answer this question?

3

Hi @Preeti_Joshi, this works for me on latest master, what version of Ray are you using?

4

I am using ray 1.11.0. Let me try the latest dev 2.0 build though.

5

Hello @Clark_Zinzow , I tried with latest 2.0.0.dev0 (Linux Python 3.7 wheel) but still getting the same error. Again not in local standalone mode, but in k8s based cluster mode.

6

Thanks for checking @Preeti_Joshi and apologies for the late response, I momentarily lost track of this thread, but I’ll dig into this on Monday!

7

Hello @Clark_Zinzow, did you get a chance to look into this issue?

8

Hello @Clark_Zinzow, following up on this again. Did you get a chance to check the pickling error mentioned in this thread?

9

Hi @Preeti_Joshi, I was able to reproduce this on latest master when using Ray Client, doing a quick debugging push to see if I can determine the underlying issue.

10

It looks like this is an underlying issue with Ray Client, where direct serialization of an actor handle created under Ray Client fails due to a direct method call check on the actor handle. The below minimal reproduction also fails:

import ray
import ray.cloudpickle as cloudpickle
ray.init("ray://localhost:10001")

@ray.remote
class Foo:
    def ping(self):
        return "ok"

f = Foo.remote()
s = cloudpickle.dumps(f)  # This raises the same error.

Interestingly, in-band serialization of the actor handle by passing it to another task does not fail, so Ray task arguments must go through a slightly different serialization path than direct out-of-band serialization with our cloudpickle:

@ray.remote
def bar(h):
    return ray.get(h.ping.remote())

ray.get(bar.remote(f))  # This works just fine.

cc @ckw017 for Ray Client-specific things, do we expect direct, out-of-band serialization of actor handles to work under Ray Client?

11

Hi @Clark_Zinzow and team,
I have the same problem with Ray 2.2.0. What is the status of this issue?

12

There was a recent fix to Ray Client that should fix actor handle serialization here: [Client] Fix __getattr__ behavior for ClientActorHandles by ckw017 · Pull Request #31586 · ray-project/ray · GitHub that should be ready in 2.3.0. Additionally it looks like the actor dependency is planned to be removed from datasets eventually

1 Like