Small Ray objects

Ray Core
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hi there,

By reading the Ray arch document I got confused by the following paragraph.

For remote tasks, the object value is computed by the executing worker. If the value is small, the worker replies directly to the owner with the value, which is copied into the owner’s in-process store.

Let’s say we have an example code like this.

import ray

ray.init()

@ray.remote
def foo():
    return 1

o_ref = foo.remote()

Upon competion of foo will the returned value be copied directly to the driver process even without calling ray.get? Or is the returned value just stored in-process memory of the executing worker?

2

@sangcho, could you shed the light on this?

3

So, if the returned data is smaller than 100KB, it is always copied. if it is bigger, it is put into the object store (which supports zero-copy for numpy array).

4

@sangcho, yes, I got the point when the data is copied and when it is not. My questions is if the “small” object is stored in the executing worker or it is right away copied into the worker that called the remote function without calling ray.get?

5

@sangcho, just a friendly reminder.

6

Sorry for the delay.

My questions is if the “small” object is stored in the executing worker or it is right away copied into the worker that called the remote function without calling ray.get?

Both worker (the caller and the owner) will have the copy

7

So the second copy happens when calling ray.get() in the caller, right?

9

I think it happens when you call a remote method.

10

@sangcho, this behaviour is not obvious. I wonder if you know someone who has more insight on this to be sure about the behaviour? Is it highlighted somewhere else in docs?

11

that should be me. I just checked code, and it should be copied when a task is finished (regardless of ray.get). ray/src/ray/protobuf/core_worker.proto at ccfcb21c9ad9960604a349155300c55698e2e971 · ray-project/ray · GitHub

if is_plasma == true, it won’t return the copy. Otherwise, it includes the copy

12

Okay, then I accept the fact the callee replies to the caller with returned data copy. Please correct if this is wrong. Thanks for answering my questions!