SystemExit error in Tune experiment

Ray Libraries (Data, Train, Tune, Serve) Ray Tune
1

[originally posted using the rllib tag, but Sven1977 told me to post it here so it can get the proper attention]

Hi all, I’m getting back into reinforcement learning so I thought I’d try some basic algorithms again. However, I’m getting an error that I have never found before, and no one else seems to have dealt with it either. All I could find is a similar error that ray actor’s throw on purpose, but then it doesn’t make sense that an error that is thrown by design is not handled properly. The error occurs after training is complete, it seems, and the training results seem fine.

Here’s my code and a screenshot of the error:

analysis = tune.run(
    rllib.agents.ppo.PPOTrainer,
    name='trainingwheels',
    config={ 'env': 'CartPole-v0' },
    stop={
        'episode_reward_mean': 195,
        # fallback in case mean episode length does not go high enough
        'training_iteration': 15
    },
    local_dir='ray_results',
    verbose=1
)

Screenshot from 2021-05-02 12-59-25
Screenshot from 2021-05-02 12-59-251920×1080 560 KB

2

hmm, that seems odd, could you post the full stacktrace?

3

Sure thing, here is all of it:

(pid=14319) 2021-05-07 16:39:27,402 ERROR worker.py:382 – SystemExit was raised from the worker
(pid=14319) Traceback (most recent call last):
(pid=14319) File “python/ray/_raylet.pyx”, line 495, in ray._raylet.execute_task
(pid=14319) File “python/ray/_raylet.pyx”, line 505, in ray._raylet.execute_task
(pid=14319) File “python/ray/_raylet.pyx”, line 449, in ray._raylet.execute_task.function_executor
(pid=14319) File “/home/rick/Museum/mllab/venv/lib/python3.8/site-packages/ray/_private/function_manager.py”, line 556, in actor_method_executor
(pid=14319) return method(__ray_actor, *args, **kwargs)
(pid=14319) File “/home/rick/Museum/mllab/venv/lib/python3.8/site-packages/ray/actor.py”, line 1001, in ray_terminate
(pid=14319) ray.actor.exit_actor()
(pid=14319) File “/home/rick/Museum/mllab/venv/lib/python3.8/site-packages/ray/actor.py”, line 1077, in exit_actor
(pid=14319) raise exit
(pid=14319) SystemExit: 0
(pid=14319)
(pid=14319) During handling of the above exception, another exception occurred:
(pid=14319)
(pid=14319) Traceback (most recent call last):
(pid=14319) File “python/ray/_raylet.pyx”, line 599, in ray._raylet.task_execution_handler
(pid=14319) File “python/ray/_raylet.pyx”, line 451, in ray._raylet.execute_task
(pid=14319) File “python/ray/_raylet.pyx”, line 488, in ray._raylet.execute_task
(pid=14319) File “python/ray/includes/libcoreworker.pxi”, line 33, in ray._raylet.ProfileEvent.exit
(pid=14319) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 167, in format_exc
(pid=14319) return “”.join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
(pid=14319) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 120, in format_exception
(pid=14319) return list(TracebackException(
(pid=14319) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 509, in init
(pid=14319) self.stack = StackSummary.extract(
(pid=14319) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 366, in extract
(pid=14319) f.line
(pid=14319) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 288, in line
(pid=14319) self._line = linecache.getline(self.filename, self.lineno).strip()
(pid=14319) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/linecache.py”, line 16, in getline
(pid=14319) lines = getlines(filename, module_globals)
(pid=14319) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/linecache.py”, line 47, in getlines
(pid=14319) return updatecache(filename, module_globals)
(pid=14319) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/linecache.py”, line 136, in updatecache
(pid=14319) with tokenize.open(fullname) as fp:
(pid=14319) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/tokenize.py”, line 394, in open
(pid=14319) encoding, lines = detect_encoding(buffer.readline)
(pid=14319) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/tokenize.py”, line 363, in detect_encoding
(pid=14319) first = read_or_stop()
(pid=14319) File “/home/rick/Museum/mllab/venv/lib/python3.8/site-packages/ray/worker.py”, line 379, in sigterm_handler
(pid=14319) sys.exit(1)
(pid=14319) SystemExit: 1
(pid=14309) 2021-05-07 16:39:27,403 ERROR worker.py:382 – SystemExit was raised from the worker
(pid=14309) Traceback (most recent call last):
(pid=14309) File “python/ray/_raylet.pyx”, line 495, in ray._raylet.execute_task
(pid=14309) File “python/ray/_raylet.pyx”, line 505, in ray._raylet.execute_task
(pid=14309) File “python/ray/_raylet.pyx”, line 449, in ray._raylet.execute_task.function_executor
(pid=14309) File “/home/rick/Museum/mllab/venv/lib/python3.8/site-packages/ray/_private/function_manager.py”, line 556, in actor_method_executor
(pid=14309) return method(__ray_actor, *args, **kwargs)
(pid=14309) File “/home/rick/Museum/mllab/venv/lib/python3.8/site-packages/ray/actor.py”, line 1001, in ray_terminate
(pid=14309) ray.actor.exit_actor()
(pid=14309) File “/home/rick/Museum/mllab/venv/lib/python3.8/site-packages/ray/actor.py”, line 1077, in exit_actor
(pid=14309) raise exit
(pid=14309) SystemExit: 0
(pid=14309)
(pid=14309) During handling of the above exception, another exception occurred:
(pid=14309)
(pid=14309) Traceback (most recent call last):
(pid=14309) File “python/ray/_raylet.pyx”, line 599, in ray._raylet.task_execution_handler
(pid=14309) File “python/ray/_raylet.pyx”, line 451, in ray._raylet.execute_task
(pid=14309) File “python/ray/_raylet.pyx”, line 488, in ray._raylet.execute_task
(pid=14309) File “python/ray/includes/libcoreworker.pxi”, line 33, in ray._raylet.ProfileEvent.exit
(pid=14309) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 167, in format_exc
(pid=14309) return “”.join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
(pid=14309) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 120, in format_exception
(pid=14309) return list(TracebackException(
(pid=14309) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 509, in init
(pid=14309) self.stack = StackSummary.extract(
(pid=14309) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 366, in extract
(pid=14309) f.line
(pid=14309) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 288, in line
(pid=14309) self._line = linecache.getline(self.filename, self.lineno).strip()
(pid=14309) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/linecache.py”, line 16, in getline
(pid=14309) lines = getlines(filename, module_globals)
(pid=14309) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/linecache.py”, line 47, in getlines
(pid=14309) return updatecache(filename, module_globals)
(pid=14309) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/linecache.py”, line 137, in updatecache
(pid=14309) lines = fp.readlines()
(pid=14309) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/codecs.py”, line 319, in decode
(pid=14309) def decode(self, input, final=False):
(pid=14309) File “/home/rick/Museum/mllab/venv/lib/python3.8/site-packages/ray/worker.py”, line 379, in sigterm_handler
(pid=14309) sys.exit(1)
(pid=14309) SystemExit: 1
(pid=14318) 2021-05-07 16:39:27,402 ERROR worker.py:382 – SystemExit was raised from the worker
(pid=14318) Traceback (most recent call last):
(pid=14318) File “python/ray/_raylet.pyx”, line 495, in ray._raylet.execute_task
(pid=14318) File “python/ray/_raylet.pyx”, line 505, in ray._raylet.execute_task
(pid=14318) File “python/ray/_raylet.pyx”, line 449, in ray._raylet.execute_task.function_executor
(pid=14318) File “/home/rick/Museum/mllab/venv/lib/python3.8/site-packages/ray/_private/function_manager.py”, line 556, in actor_method_executor
(pid=14318) return method(__ray_actor, *args, **kwargs)
(pid=14318) File “/home/rick/Museum/mllab/venv/lib/python3.8/site-packages/ray/actor.py”, line 1001, in ray_terminate
(pid=14318) ray.actor.exit_actor()
(pid=14318) File “/home/rick/Museum/mllab/venv/lib/python3.8/site-packages/ray/actor.py”, line 1077, in exit_actor
(pid=14318) raise exit
(pid=14318) SystemExit: 0
(pid=14318)
(pid=14318) During handling of the above exception, another exception occurred:
(pid=14318)
(pid=14318) Traceback (most recent call last):
(pid=14318) File “python/ray/_raylet.pyx”, line 599, in ray._raylet.task_execution_handler
(pid=14318) File “python/ray/_raylet.pyx”, line 451, in ray._raylet.execute_task
(pid=14318) File “python/ray/_raylet.pyx”, line 488, in ray._raylet.execute_task
(pid=14318) File “python/ray/includes/libcoreworker.pxi”, line 33, in ray._raylet.ProfileEvent.exit
(pid=14318) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 167, in format_exc
(pid=14318) return “”.join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
(pid=14318) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 120, in format_exception
(pid=14318) return list(TracebackException(
(pid=14318) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 509, in init
(pid=14318) self.stack = StackSummary.extract(
(pid=14318) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 366, in extract
(pid=14318) f.line
(pid=14318) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/traceback.py”, line 288, in line
(pid=14318) self._line = linecache.getline(self.filename, self.lineno).strip()
(pid=14318) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/linecache.py”, line 16, in getline
(pid=14318) lines = getlines(filename, module_globals)
(pid=14318) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/linecache.py”, line 47, in getlines
(pid=14318) return updatecache(filename, module_globals)
(pid=14318) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/linecache.py”, line 136, in updatecache
(pid=14318) with tokenize.open(fullname) as fp:
(pid=14318) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/tokenize.py”, line 394, in open
(pid=14318) encoding, lines = detect_encoding(buffer.readline)
(pid=14318) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/tokenize.py”, line 363, in detect_encoding
(pid=14318) first = read_or_stop()
(pid=14318) File “/home/rick/Museum/.pyenv/versions/3.8.5/lib/python3.8/tokenize.py”, line 321, in read_or_stop
(pid=14318) return readline()
(pid=14318) File “/home/rick/Museum/mllab/venv/lib/python3.8/site-packages/ray/worker.py”, line 379, in sigterm_handler
(pid=14318) sys.exit(1)
(pid=14318) SystemExit: 1
2021-05-07 16:39:27,492 INFO tune.py:549 – Total run time: 35.02 seconds (34.24 seconds for the tuning loop).

4

Ah I was able to reproduce this. Let me file an issue.

I think this is largely harmless.