# RuntimeError: The actor with name TrainTrainable failed to import on the worker
**URL:** https://discuss.ray.io/t/runtimeerror-the-actor-with-name-traintrainable-failed-to-import-on-the-worker/8145
**Category:** Ray Core
**Created:** [November 5, 2022, 12:19am UTC](https://discuss.ray.io/t/runtimeerror-the-actor-with-name-traintrainable-failed-to-import-on-the-worker/8145 "2022-11-05T00:19:57Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@3rdgenta](https://discuss.ray.io/u/3rdgenta)
#### Post date: [November 5, 2022, 12:19am UTC](https://discuss.ray.io/t/runtimeerror-the-actor-with-name-traintrainable-failed-to-import-on-the-worker/8145/1 "2022-11-05T00:19:57Z")
Hello. I’m trying to add a custom training backend, similar to how Ray integrates PyTorch. The issue is, when I define a network model (which is different from PyTorch’s nn.Module), is train\_loop\_per\_worker(), it has the following error message:
File “python/ray/\_raylet.pyx”, line 655, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 696, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 662, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 666, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 613, in ray.\_raylet.execute\_task.function\_executor
File “/home/wjeon/ray/python/ray/\_private/function\_manager.py”, line 674, in actor\_method\_executor
return method(\_\_ray\_actor, \*args, \*\*kwargs)
File “/home/wjeon/ray/python/ray/\_private/function\_manager.py”, line 586, in temporary\_actor\_method
raise RuntimeError(
RuntimeError: The actor with name TrainTrainable failed to import on the worker. This may be because needed library dependencies are not installed in the worker environment:
Traceback (most recent call last):
File “/home/wjeon/ray/python/ray/\_private/function\_manager.py”, line 625, in \_load\_actor\_class\_from\_gcs
actor\_class = pickle.loads(pickled\_class)
File “/home/wjeon/ray/python/ray/cloudpickle/cloudpickle.py”, line 872, in \_make\_skeleton\_class
skeleton\_class = types.new\_class(
File “/usr/lib/python3.8/types.py”, line 72, in new\_class
meta, ns, kwds = prepare\_class(name, resolved\_bases, kwds)
File “/usr/lib/python3.8/types.py”, line 125, in prepare\_class
meta = \_calculate\_meta(meta, bases)
File “/usr/lib/python3.8/types.py”, line 143, in \_calculate\_meta
raise TypeError("metaclass conflict: "
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
Could you guide me how to tackle this issue? If my custom object for the network is implemented in C++ and provided to Python via pybind11, do I need to take care of native libraries accordingly, other than just importing the module of the corresponding Python objects?
Thanks.
---
### Author:  [@xwjiang2010](https://discuss.ray.io/u/xwjiang2010)
#### Post date: [November 8, 2022, 5:09pm UTC](https://discuss.ray.io/t/runtimeerror-the-actor-with-name-traintrainable-failed-to-import-on-the-worker/8145/2 "2022-11-08T17:09:02Z")
Hi,
I don’t really have any experience with this. But my hunch is for this to work, you need to basically make sure that you can pickle the function and ship it to a remote actor and unpickle it successfully.
To test that directly and debug, you can try `ray.cloudpickle.dumps(f)` and `ray.cloudpickle.loads()` on your function and see if it works.
cc @sangcho
---
### Author:  [@3rdgenta](https://discuss.ray.io/u/3rdgenta)
#### Post date: [November 8, 2022, 8:18pm UTC](https://discuss.ray.io/t/runtimeerror-the-actor-with-name-traintrainable-failed-to-import-on-the-worker/8145/3 "2022-11-08T20:18:34Z")
@xwjiang2010 Thanks for your response. The detailed error message from the beginning:
Exception has occurred: TrainingFailedError
Failure # 1 (occurred at 2022-11-08\_11-58-53)
Traceback (most recent call last):
File “/home/wjeon/ray/python/ray/tune/execution/ray\_trial\_executor.py”, line 996, in get\_next\_executor\_event
future\_result = ray.get(ready\_future)
File “/home/wjeon/ray/python/ray/\_private/client\_mode\_hook.py”, line 105, in wrapper
return func(\*args, \*\*kwargs)
File “/home/wjeon/ray/python/ray/\_private/worker.py”, line 2282, in get
raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
class\_name: BaseTrainer.as\_trainable..TrainTrainable
actor\_id: 89986ce05f4d7d344c68da4c01000000
pid: 19275
namespace: 41c04ed9-1bdb-464c-93bf-e12649db5c7f
ip:
The actor is dead because its worker process has died. Worker exit type: SYSTEM\_ERROR Worker exit detail: Worker exits unexpectedly. Worker exits with an exit code None.
Traceback (most recent call last):
File “python/ray/\_raylet.pyx”, line 655, in ray.\_raylet.execute\_task
with core\_worker.profile\_event(b"task:execute"):
File “python/ray/\_raylet.pyx”, line 696, in ray.\_raylet.execute\_task
raise e
File “python/ray/\_raylet.pyx”, line 662, in ray.\_raylet.execute\_task
with ray.\_private.worker.\_changeproctitle(title, next\_title):
File “python/ray/\_raylet.pyx”, line 666, in ray.\_raylet.execute\_task
outputs = function\_executor(\*args, \*\*kwargs)
File “python/ray/\_raylet.pyx”, line 613, in ray.\_raylet.execute\_task.function\_executor
return function(actor, \*arguments, \*\*kwarguments)
File “/home/wjeon/ray/python/ray/\_private/function\_manager.py”, line 674, in actor\_method\_executor
return method(\_\_ray\_actor, \*args, \*\*kwargs)
File “/home/wjeon/ray/python/ray/\_private/function\_manager.py”, line 586, in temporary\_actor\_method
raise RuntimeError(
RuntimeError: The actor with name TrainTrainable failed to import on the worker. This may be because needed library dependencies are not installed in the worker environment:
Is there any way to figure out why the actor with name TrainTrainable failed to import on the worker? The only difference from PyTorch’s training example is that the PyTorch’s network model is inherited from torch.nn.Module class (of ‘type’ class type) whereas my custom test code uses a network class inherited from a base class of ‘pybind11\_builtins.pybind11\_type’ type. Thank you.
---
### Author:  [@xwjiang2010](https://discuss.ray.io/u/xwjiang2010)
#### Post date: [November 8, 2022, 10:25pm UTC](https://discuss.ray.io/t/runtimeerror-the-actor-with-name-traintrainable-failed-to-import-on-the-worker/8145/4 "2022-11-08T22:25:38Z")
Yeah, I think this is saying pickle is not working as intended. You probably have to dig into how to make it work with a C++ impl + python binding.
Just a random search on the web, I find [this](https://github.com/ray-project/pygloo). As stated, Pygloo provides python bindings for gloo which is a collective communication library implemented in C++. And as you can see in their example, `pygloo.xxx` can be put inside a remote function (decorated with ray.remote) without a problem. This means pickling works for pygloo’s case. I would suggest you taking a look there and see how that is done.
---
### Author:  [@ClarenceNg](https://discuss.ray.io/u/ClarenceNg)
#### Post date: [November 15, 2022, 6:58am UTC](https://discuss.ray.io/t/runtimeerror-the-actor-with-name-traintrainable-failed-to-import-on-the-worker/8145/5 "2022-11-15T06:58:15Z")
Is there a minimal code example that will reproduce the failure? Its best to file a bug so the team can address accordingly : [Sign in to GitHub · GitHub](https://github.com/ray-project/ray/issues/new?assignees=&labels=bug%2Ctriage&template=bug-report.yml&title=%5B%3CRay+component%3A+Core%7CRLlib%7Cetc...%3E%5D+)