Facing issue over objects when using ray multinode cluster

Ray Core
1

Ray version: 1.9.2
pandas: 1.1.5
OS: Ubuntu

Getting below error trace when executing my code., This exception triggers from worker node.

At least one of the input arguments for this task could not be computed:
ray.exceptions.RaySystemError: System error: cannot import name ‘maybe_mangle_lambdas’
traceback: Traceback (most recent call last):
File “/home/rayuser/.local/lib/python3.6/site-packages/ray/serialization.py”, line 281, in deserialize_objects
obj = self._deserialize_object(data, metadata, object_ref)
File “/home/rayuser/.local/lib/python3.6/site-packages/ray/serialization.py”, line 194, in _deserialize_object
return self._deserialize_msgpack_data(data, metadata_fields)
File “/home/rayuser/.local/lib/python3.6/site-packages/ray/serialization.py”, line 172, in _deserialize_msgpack_data
python_objects = self._deserialize_pickle5_data(pickle5_data)
File “/home/rayuser/.local/lib/python3.6/site-packages/ray/serialization.py”, line 160, in _deserialize_pickle5_data
obj = pickle.loads(in_band, buffers=buffers)
File “/home/rayuser/.local/lib/python3.6/site-packages/pandas/core/frame.py”, line 118, in
from pandas.core.aggregation import reconstruct_func, relabel_result
File “/home/rayuser/.local/lib/python3.6/site-packages/pandas/core/aggregation.py”, line 25, in
import pandas.core.common as com
File “/home/rayuser/.local/lib/python3.6/site-packages/pandas/ init .py”, line 52, in
from pandas.core.api import (
File “/home/rayuser/.local/lib/python3.6/site-packages/pandas/core/api.py”, line 29, in
from pandas.core.groupby import Grouper, NamedAgg
File “/home/rayuser/.local/lib/python3.6/site-packages/pandas/core/groupby/ init .py”, line 1, in
from pandas.core.groupby.generic import DataFrameGroupBy, NamedAgg, SeriesGroupBy
File “/home/rayuser/.local/lib/python3.6/site-packages/pandas/core/groupby/generic.py”, line 57, in
from pandas.core.aggregation import (
ImportError: cannot import name ‘maybe_mangle_lambdas’ - Stack Trace - Traceback (most recent call last):

Kindly help me to resolve this asap., Pandas actually has this mentioned function(maybe_mangle_lambdas). Tried reinstalling packages.

Faced the same in Ray version 1.10.0 too…

2

Seems like it is the dependency import related issue. Is it possible to provide me some smaller script that causes the same issue?

3

Hi @sangcho , Thanks for replying. Following is my import list in source file

xyz.py (driver)
import logging
import pandas
import numpy
import ray
import time

my internal code snippet here. Have called a function from abc.py with remote keyword. abc.py contains only ray functions which I need to distribute among worker nodes.

abc.py
import ray
import logging
import pandas as pd

Please help me out to solve this. Importing pandas in both will cause any issue?
I’m totally new to python and ray environment.

4

Is it possible to actually try the newest version of ray (2.4)?