TypeError: Could not serialize the function Ray-Cassandra

Ray Core
1

Hello,

I have a issue.
TypeError: Could not serialize the function main.read_cassandra. Check https://docs.ray.io/en/master/serialization.html#troubleshooting for more information.
There is no information anymore.

I’m reading data from cassandra. I added @ray.remote decorator. I also to invoke the remote function, I used the remote method.

Reproduction script

@ray.remote
def read_cassandra(assetx,start_date,end_date):
query = “SELECT * from table_name where sensor_id=”+str(assetx)+" and sensor_time>=‘“+str(start_date)+”‘and sensor_time<=’“+str(end_date)+”’ ALLOW FILTERING"
rows = session.execute(query)
df = pd.DataFrame(list(rows))
print(df)

x = [read_cassandra.remote(asset,start_date,end_date) for asset in assets]
start_time = time.time()
ray.get(x)
print(“time=”+str(time.time()-start_time))

Versions

Ray:1.10.0
Cassandra-driver : 3.25.0
Python: 3.7

2

hi @cansucandanx

Ray has changed a lot since 1.10.0; have you tried the latest version (1.13.0) to see if it works?

3

What’s session here? I feel it might not be pickle-able here. Could you try to create session object inside read_cassandra?

4

I created session object inside read_cassandra but I got another error.

Reproduction script

start_date = "2021-02-28"
end_date = "2021-03-26"
daily_date = "2021-03-25"

@ray.remote
def read_cassandra(assetx,start_date,end_date):
          auth_provider = PlainTextAuthProvider(username='x', password='x')
          cluster = Cluster(['x', 'x', 'x'], auth_provider=auth_provider)
          session = cluster.connect('keyspace')
          query = “SELECT * from table_name where sensor_id=”+str(assetx)+" and sensor_time>=‘“+str(start_date)+”‘and sensor_time<=’“+str(end_date)+”’ ALLOW FILTERING"
          rows = session.execute(query)
          df = pd.DataFrame((rows))
          print(df)


x = [read_cassandra.remote(asset,start_date,end_date) for asset in assets]
start_time = time.time()
ray.get(x)

I am getting this error:

RaySystemError: System error: Failed to unpickle serialized exception
traceback: Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site-packages/ray/exceptions.py", line 32, in from_bytes
    return pickle.loads(ray_exception.serialized_exception)
  File "cassandra/protocol.py", line 123, in cassandra.protocol.ErrorMessage.__init__
TypeError: __init__() takes exactly 4 positional arguments (1 given)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.7/site-packages/ray/serialization.py", line 287, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
  File "/opt/anaconda3/lib/python3.7/site-packages/ray/serialization.py", line 217, in _deserialize_object
    return RayError.from_bytes(obj)
  File "/opt/anaconda3/lib/python3.7/site-packages/ray/exceptions.py", line 35, in from_bytes
    raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception
5

seems read_cassandra is throwing some exceptions. Could you try catch the code and print it out?