AttributeError: module '__main__' has no attribute 'test_func'

Ray Core
1

Hi, I’m facing some problems when trying to use some of my custom modules in Ray.

python main.py

main.py:

import sys

sys.path.extend(['/home/auderson/my_packages/TestModule'])
from test_import import *

import ray
from ray.job_config import JobConfig

job_config = JobConfig(code_search_path=['/home/auderson/my_packages/TestModule'])
ray.init(address='auto', _redis_password='5241590000000000', dashboard_host='0.0.0.0', job_config=job_config)


@ray.remote
def test_func():
    return np.random.random(10)


future = test_func.remote()
result = ray.get(future)

my_packages

  • TestModule

    • test_import.py:
      import numpy as np

this produces the following errors:

2021-04-19 11:14:22,369 WARNING worker.py:1107 – Traceback (most recent call last):
File “/home/auderson/miniconda3/lib/python3.8/site-packages/ray/function_manager.py”, line 271, in _load_function_from_local
function = getattr(module, function_name)._function
AttributeError: module ‘__main__’ has no attribute ‘test_func’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “python/ray/_raylet.pyx”, line 570, in ray._raylet.task_execution_handler
File “python/ray/_raylet.pyx”, line 381, in ray._raylet.execute_task
File “/home/auderson/miniconda3/lib/python3.8/site-packages/ray/function_manager.py”, line 239, in get_execution_info
self._load_function_from_local(job_id, function_descriptor)
File “/home/auderson/miniconda3/lib/python3.8/site-packages/ray/function_manager.py”, line 280, in _load_function_from_local
raise RuntimeError(f"Function {function_descriptor} failed "
RuntimeError: Function {type=PythonFunctionDescriptor, module_name=__main__, class_name=, function_name=test_func, function_hash=24f1202743b6ec54ee995f52be5364ec053f81e10eb6ac09756c0809} failed to be loaded from local code. Error message: module ‘__main__’ has no attribute ‘test_func’
An unexpected internal error occurred while the worker was executing a task.
Traceback (most recent call last):
File “main.py”, line 27, in
2021-04-19 11:14:22,369 WARNING worker.py:1107 – A worker died or was killed while executing task a67dc375e60ddd1affffffffffffffffffffffff01000000.
result = ray.get(future)
File “/home/auderson/miniconda3/lib/python3.8/site-packages/ray/_private/client_mode_hook.py”, line 47, in wrapper
return func(args, **kwargs)
File “/home/auderson/miniconda3/lib/python3.8/site-packages/ray/worker.py”, line 1458, in get
raise value
ray.exceptions.WorkerCrashedError: The worker died unexpectedly while executing this task. Check python-core-worker-.log files for more information.
(pid=671135) 2021-04-19 11:14:22,362 ERROR worker.py:390 – SystemExit was raised from the worker
(pid=671135) Traceback (most recent call last):
(pid=671135) File “/home/auderson/miniconda3/lib/python3.8/site-packages/ray/function_manager.py”, line 271, in _load_function_from_local
(pid=671135) function = getattr(module, function_name)._function
(pid=671135) AttributeError: module ‘__main__’ has no attribute ‘test_func’
(pid=671135)
(pid=671135) During handling of the above exception, another exception occurred:
(pid=671135)
(pid=671135) Traceback (most recent call last):
(pid=671135) File “python/ray/_raylet.pyx”, line 570, in ray._raylet.task_execution_handler
(pid=671135) File “python/ray/_raylet.pyx”, line 381, in ray._raylet.execute_task
(pid=671135) File “/home/auderson/miniconda3/lib/python3.8/site-packages/ray/function_manager.py”, line 239, in get_execution_info
(pid=671135) self._load_function_from_local(job_id, function_descriptor)
(pid=671135) File “/home/auderson/miniconda3/lib/python3.8/site-packages/ray/function_manager.py”, line 280, in _load_function_from_local
(pid=671135) raise RuntimeError(f"Function {function_descriptor} failed "
(pid=671135) RuntimeError: Function {type=PythonFunctionDescriptor, module_name=__main__, class_name=, function_name=test_func, function_hash=24f1202743b6ec54ee995f52be5364ec053f81e10eb6ac09756c0809} failed to be loaded from local code. Error message: module ‘__main__’ has no attribute ‘test_func’
(pid=671135)
(pid=671135) During handling of the above exception, another exception occurred:
(pid=671135)
(pid=671135) Traceback (most recent call last):
(pid=671135) File “python/ray/_raylet.pyx”, line 582, in ray._raylet.task_execution_handler
(pid=671135) SystemExit: 1

If I do not set job_config and import numpy in main.py, then it works fine.

2

here is the log found in tmp/ray

python-core-worker-33441e6c35f12c9dc9e8cbc4f8690dbeca94aeb45aa9a18eafe87c11_671135.log

[2021-04-19 11:14:04,592 I 671135 671135] core_worker.cc:136: Constructing CoreWorkerProcess. pid: 671135
[2021-04-19 11:14:04,593 I 671135 671135] grpc_server.cc:75: worker server started, listening on port 10002.
[2021-04-19 11:14:04,597 I 671135 671135] core_worker.cc:364: Initializing worker at address: 172.17.2.3:10002, worker ID 33441e6c35f12c9dc9e8cbc4f8690dbeca94aeb45aa9a18eafe87c11, raylet 8224d39ab72e19dbb4b9852eae6c24f41bbc6dc831436a4bc73dbe23
[2021-04-19 11:14:04,608 I 671135 671135] core_worker.cc:231: Worker 33441e6c35f12c9dc9e8cbc4f8690dbeca94aeb45aa9a18eafe87c11 is created.
[2021-04-19 11:14:04,608 I 671135 671135] io_service_pool.cc:36: IOServicePool is running with 1 io_service.
[2021-04-19 11:14:04,609 I 671135 671160] service_based_accessor.cc:559: Received notification for node id = 8224d39ab72e19dbb4b9852eae6c24f41bbc6dc831436a4bc73dbe23, IsAlive = 1
[2021-04-19 11:14:04,609 I 671135 671160] service_based_accessor.cc:559: Received notification for node id = c5f586637fc500b97ad8d8144367699fd8e83bd7776bbcfbfdbccbbc, IsAlive = 1
[2021-04-19 11:14:22,362 I 671135 671135] core_worker.cc:591: Exit signal received, this process will exit after all outstanding tasks have finished
[2021-04-19 11:14:22,363 I 671135 671135] core_worker.cc:254: Removed worker 33441e6c35f12c9dc9e8cbc4f8690dbeca94aeb45aa9a18eafe87c11
[2021-04-19 11:14:22,366 I 671135 671135] core_worker.cc:167: Destructing CoreWorkerProcess. pid: 671135
[2021-04-19 11:14:22,366 I 671135 671135] io_service_pool.cc:48: IOServicePool is stopped.