Ray : ModuleNotFoundError coupled with Serialization issue

Ray Core
1

Hi Team,

Another ModuleNotFoundError coupled with Serialization issue.

Here is code structure -

  1. CounterUser.py
#CounterUser.py

from Counter import Counter
import ray
from CounterUtil import CounterUtil

ray.init(address='ray://127.0.0.1:10001')



class Consume:
    def __init__(self, val) -> None:
        self.counter_actor = Counter.remote(val)

    def consume_counter(self, val):
        fut = self.counter_actor.increment.remote(val)
        resp = ray.get(fut)
        return resp



if __name__ == "__main__":
    num=5
    consumer = Consume(num)
    resp = consumer.consume_counter(1)
    print("got reponse as", resp)
  1. Counter.py
# Counter.py

import ray
from CounterUtil import num_dict

@ray.remote
class Counter(object):
    def __init__(self, val):
        ## Model is loaded here, instead of val
        self.value = val

    def increment(self, inc_by):
        self.value += inc_by
        return self.value

    def get_counter(self):
        return self.value

Above demo code works but when I try to fit this into working project, with Counter.py having a model loaded(instead of self.value = val) then I am getting this error.

Here is what traceback look like…

(TemporaryActor pid=52997) 2021-12-16 18:09:23,125	ERROR serialization.py:283 -- No module named 'src'
(TemporaryActor pid=52997) Traceback (most recent call last):
(TemporaryActor pid=52997)   File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/serialization.py", line 281, in deserialize_objects
(TemporaryActor pid=52997)     obj = self._deserialize_object(data, metadata, object_ref)
(TemporaryActor pid=52997)   File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/serialization.py", line 194, in _deserialize_object
(TemporaryActor pid=52997)     return self._deserialize_msgpack_data(data, metadata_fields)
(TemporaryActor pid=52997)   File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/serialization.py", line 172, in _deserialize_msgpack_data
(TemporaryActor pid=52997)     python_objects = self._deserialize_pickle5_data(pickle5_data)
(TemporaryActor pid=52997)   File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/serialization.py", line 162, in _deserialize_pickle5_data
(TemporaryActor pid=52997)     obj = pickle.loads(in_band)
(TemporaryActor pid=52997) ModuleNotFoundError: No module named 'src'
[2021-12-16 18:09:25,568] ERROR in app: Exception on /api/v2/handler/inspect [POST]
Traceback (most recent call last):
  File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/flask/app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/flask/app.py", line 1515, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/flask/app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/flask/app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
    
  File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return getattr(ray, func.__name__)(*args, **kwargs)
  File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/util/client/api.py", line 42, in get
    return self.worker.get(vals, timeout=timeout)
  File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/util/client/worker.py", line 359, in get
    res = self._get(to_get, op_timeout)
  File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/util/client/worker.py", line 386, in _get
    raise err
ray.exceptions.RayActorError: The actor died because of an error raised in its creation task, ray::Title.__init__() (pid=52997, ip=127.0.0.1)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RaySystemError: System error: No module named 'src'
traceback: Traceback (most recent call last):
  File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/serialization.py", line 281, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
  File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/serialization.py", line 194, in _deserialize_object
    return self._deserialize_msgpack_data(data, metadata_fields)
  File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/serialization.py", line 172, in _deserialize_msgpack_data
    python_objects = self._deserialize_pickle5_data(pickle5_data)
  File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/serialization.py", line 162, in _deserialize_pickle5_data
    obj = pickle.loads(in_band)
ModuleNotFoundError: No module named 'src'
(TemporaryActor pid=52997) 2021-12-16 18:09:25,554	ERROR worker.py:431 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::Name.__init__() (pid=52997, ip=127.0.0.1)
(TemporaryActor pid=52997)   At least one of the input arguments for this task could not be computed:
(TemporaryActor pid=52997) ray.exceptions.RaySystemError: System error: No module named 'src'
(TemporaryActor pid=52997) traceback: Traceback (most recent call last):
(TemporaryActor pid=52997)   File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/serialization.py", line 281, in deserialize_objects
(TemporaryActor pid=52997)     obj = self._deserialize_object(data, metadata, object_ref)
(TemporaryActor pid=52997)   File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/serialization.py", line 194, in _deserialize_object
(TemporaryActor pid=52997)     return self._deserialize_msgpack_data(data, metadata_fields)
(TemporaryActor pid=52997)   File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/serialization.py", line 172, in _deserialize_msgpack_data
(TemporaryActor pid=52997)     python_objects = self._deserialize_pickle5_data(pickle5_data)
(TemporaryActor pid=52997)   File "/Users/macpro/anaconda3/envs/demo3.9.7/lib/python3.9/site-packages/ray/serialization.py", line 162, in _deserialize_pickle5_data
(TemporaryActor pid=52997)     obj = pickle.loads(in_band)
(TemporaryActor pid=52997) ModuleNotFoundError: No module named 'src
2

Hi @lihost, it looks like this might be an issue with your src package being accessible to the actor, i.e. that src package needs to be available on the node that the actor runs on. Have you checked out our docs on handling dependencies? Handling Dependencies — Ray v2.0.0.dev0