While examining of how my ray computations work using
ray.timeline I found out that unexpectedly long deserialization takes place.
The prelude is as follows.
There is a remote function
f that yields output which then goes to function
f yields dictionary like as follows:
class FOutoput(TypedDict): key1: np.ndarray # Big array. key2: List[float] key3: Optional[List[float]] key4: bool key5: "CustomClass"
On a timeline I see that inside
f there happened
task:store_outputs and inside
Following this and this I hesitated that ray can properly handle NumPy array that lies in a dictionary with custom classes or optional lists. So I implemented encoder and decor from
RayFriendlyFOutput which is like this:
class RayFriendlyFOutoput(TypedDict): key1: np.ndarray # Big array. key2: np.ndarray key3: np.ndarray key4: bool key5: Dict[str, np.ndarray]
g gets it as well, but internally there happens a conversion
FOutoput. Deserealization time remained the same.
I checked that all arrays had flag
writable equal to
False which may indicate that .
My questions are:
task:deserialize_argsalways mean that arguments was not plasma stored properly?
- What may be the reason that
- Is that right that in case of
key1value cannot be stored to Plasma store as not all values are primitive or NumPy arrays?
- How is it possible to establish the reason of deserialization?
It also may be worth mentioning that in function
key1 value is copied to perform
- This copy is counted in