Hello,
ray Version 1.6.
python 3.7
Linux
We train model with rllib, tune, ray. At the end of the training a sys.exit(1) call pops up.
e.g.
(pid=30781)
(pid=30781) During handling of the above exception, another exception occurred:
(pid=30781)
(pid=30781) Traceback (most recent call last):
(pid=30781) File “python/ray/_raylet.pyx”, line 640, in ray._raylet.task_execution_handler
(pid=30781) File “python/ray/_raylet.pyx”, line 488, in ray._raylet.execute_task
(pid=30781) File “python/ray/_raylet.pyx”, line 525, in ray._raylet.execute_task
(pid=30781) File “python/ray/includes/libcoreworker.pxi”, line 33, in ray._raylet.ProfileEvent.exit
(pid=30781) File “/usr/lib64/python3.7/traceback.py”, line 167, in format_exc
(pid=30781) return “”.join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
(pid=30781) File “/usr/lib64/python3.7/traceback.py”, line 121, in format_exception
(pid=30781) type(value), value, tb, limit=limit).format(chain=chain))
(pid=30781) File “/usr/lib64/python3.7/traceback.py”, line 508, in init
(pid=30781) capture_locals=capture_locals)
(pid=30781) File “/usr/lib64/python3.7/traceback.py”, line 363, in extract
(pid=30781) f.line
(pid=30781) File “/usr/lib64/python3.7/traceback.py”, line 285, in line
(pid=30781) self._line = linecache.getline(self.filename, self.lineno).strip()
(pid=30781) File “/usr/lib64/python3.7/linecache.py”, line 16, in getline
(pid=30781) lines = getlines(filename, module_globals)
(pid=30781) File “/usr/lib64/python3.7/linecache.py”, line 47, in getlines
(pid=30781) return updatecache(filename, module_globals)
(pid=30781) File “/usr/lib64/python3.7/linecache.py”, line 136, in updatecache
(pid=30781) with tokenize.open(fullname) as fp:
(pid=30781) File “/usr/lib64/python3.7/tokenize.py”, line 449, in open
(pid=30781) encoding, lines = detect_encoding(buffer.readline)
(pid=30781) File “/usr/lib64/python3.7/tokenize.py”, line 426, in detect_encoding
(pid=30781) encoding = find_cookie(first)
(pid=30781) File “/usr/lib64/python3.7/tokenize.py”, line 392, in find_cookie
(pid=30781) match = cookie_re.match(line_string)
(pid=30781) File “/venv/lib64/python3.7/site-packages/ray/worker.py”, line 425, in sigterm_handler
(pid=30781) sys.exit(1)
(pid=30781) SystemExit: 1
(pid=30784) 2021-09-16 11:48:14,671 ERROR worker.py:428 – SystemExit was raised from the worker
(pid=30784) Traceback (most recent call last):
(pid=30784) File “python/ray/_raylet.pyx”, line 532, in ray._raylet.execute_task
(pid=30784) File “python/ray/_raylet.pyx”, line 536, in ray._raylet.execute_task
(pid=30784) File “python/ray/_raylet.pyx”, line 486, in ray._raylet.execute_task.function_executor
(pid=30784) File “/venv/lib64/python3.7/site-packages/ray/_private/function_manager.py”, line 563, in actor_method_executor
(pid=30784) return method(__ray_actor, *args, **kwargs)
(pid=30784) File “/venv/lib64/python3.7/site-packages/ray/actor.py”, line 1047, in ray_terminate
(pid=30784) ray.actor.exit_actor()
(pid=30784) File “/venv/lib64/python3.7/site-packages/ray/actor.py”, line 1123, in exit_actor
(pid=30784) raise exit
(pid=30784) SystemExit: 0
(pid=30784)
(pid=30784) During handling of the above exception, another exception occurred:
(pid=30784)
(pid=30784) Traceback (most recent call last):
(pid=30784) File “python/ray/_raylet.pyx”, line 640, in ray._raylet.task_execution_handler
(pid=30784) File “python/ray/_raylet.pyx”, line 488, in ray._raylet.execute_task
(pid=30784) File “python/ray/_raylet.pyx”, line 525, in ray._raylet.execute_task
(pid=30784) File “python/ray/includes/libcoreworker.pxi”, line 33, in ray._raylet.ProfileEvent.exit
(pid=30784) File “/usr/lib64/python3.7/traceback.py”, line 167, in format_exc
(pid=30784) return “”.join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
(pid=30784) File “/usr/lib64/python3.7/traceback.py”, line 121, in format_exception
(pid=30784) type(value), value, tb, limit=limit).format(chain=chain))
(pid=30784) File “/usr/lib64/python3.7/traceback.py”, line 508, in init
(pid=30784) capture_locals=capture_locals)
(pid=30784) File “/usr/lib64/python3.7/traceback.py”, line 363, in extract
(pid=30784) f.line
(pid=30784) File “/usr/lib64/python3.7/traceback.py”, line 285, in line
(pid=30784) self._line = linecache.getline(self.filename, self.lineno).strip()
(pid=30784) File “/usr/lib64/python3.7/linecache.py”, line 16, in getline
(pid=30784) lines = getlines(filename, module_globals)
(pid=30784) File “/usr/lib64/python3.7/linecache.py”, line 47, in getlines
(pid=30784) return updatecache(filename, module_globals)
(pid=30784) File “/usr/lib64/python3.7/linecache.py”, line 136, in updatecache
(pid=30784) with tokenize.open(fullname) as fp:
(pid=30784) File “/usr/lib64/python3.7/tokenize.py”, line 449, in open
(pid=30784) encoding, lines = detect_encoding(buffer.readline)
(pid=30784) File “/usr/lib64/python3.7/tokenize.py”, line 426, in detect_encoding
(pid=30784) encoding = find_cookie(first)
(pid=30784) File “/usr/lib64/python3.7/tokenize.py”, line 392, in find_cookie
(pid=30784) match = cookie_re.match(line_string)
(pid=30784) File “/venv/lib64/python3.7/site-packages/ray/worker.py”, line 425, in sigterm_handler
(pid=30784) sys.exit(1)
(pid=30784) SystemExit: 1
What is the cause of that sys.exit, any ideas ?
TIA
Michael