[tune] AttributeError: Can't get attribute '_object_ref_deserializer' on module 'ray.serialization'

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

Hello,

I running ray tune on a cluster and I get the following error in one of the three nodes. The other nodes run experiments correctly

Traceback (most recent call last):            
  File ".venv/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 678, in _process_trial     
    results = self.trial_executor.fetch_result(trial)                                                                               
  File ".venv/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 597, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)        
  File ".venv/lib/python3.6/site-packages/ray/_private/client_mode_hook.py", line 47, in wrapper
    return func(*args, **kwargs)                                          
  File ".venv/lib/python3.6/site-packages/ray/worker.py", line 1458, in get
    raise value.as_instanceof_cause()                                     
ray.exceptions.RayTaskError(AttributeError): ray::TrainableDecorator.train_buffered() (pid=197547, ip=<head-ip>)
  File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task    
  File "python/ray/_raylet.pyx", line 442, in ray._raylet.execute_task    
  File "python/ray/_raylet.pyx", line 459, in ray._raylet.execute_task    
  File "venv/lib/python3.6/site-packages/ray/serialization.py", line 245, in deserialize_objects
    self._deserialize_object(data, metadata, object_ref))                 
  File "venv/lib/python3.6/site-packages/ray/serialization.py", line 192, in _deserialize_object
    return self._deserialize_msgpack_data(data, metadata_fields)          
  File "venv/lib/python3.6/site-packages/ray/serialization.py", line 170, in _deserialize_msgpack_data
    python_objects = self._deserialize_pickle5_data(pickle5_data)         
  File "venv/lib/python3.6/site-packages/ray/serialization.py", line 160, in _deserialize_pickle5_data
    obj = pickle.loads(in_band)                                           
AttributeError: Can't get attribute '_object_ref_deserializer' on <module 'ray.serialization' from 'venv/lib/python3.6/site-packages/ray/serialization.py'>

Any ideas for debugging?

2

Are there any other logs posted?

3

Excuse me for the long silence. Yes I can provide more logs, can you please clarify what type of logs would be useful?

4

Hmm, I guess maybe if you could log onto the node and get /tmp/ray/session_latest/*?