i am experiencing crashes happening with multiple machines after 4-5 days of training, with the following stack trace, on ray 2.30.0
2024-07-12 19:48:19,937 ERROR tune_controller.py:1331 -- Trial task failed for trial bb20cc69
Traceback (most recent call last):
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/air/execution/_internal/event_manager.py", line 110, in resolve_future
result = ray.get(future)
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/_private/worker.py", line 2630, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/_private/worker.py", line 863, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(KeyError): ^[[36mray::ImplicitFunc.train()^[[39m (pid=1665315, ip=192.168.0.4, actor_id=d7e98aaece62544336a6306801000000, repr=single_agent_train)
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/tune/trainable/trainable.py", line 331, in train
raise skipped from exception_cause(skipped)
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/air/_internal/util.py", line 98, in run
self._ret = self._target(*self._args, **self._kwargs)
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/tune/trainable/function_trainable.py", line 45, in <lambda>
training_func=lambda: self._trainable_func(self.config),
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/tune/trainable/function_trainable.py", line 248, in _trainable_func
output = fn()
File "/home/mfioravanti/example/.venv/bin/impl/learn.py", line 110, in single_agent_train
train.report(model.train())
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/tune/trainable/trainable.py", line 331, in train
raise skipped from exception_cause(skipped)
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/tune/trainable/trainable.py", line 328, in train
result = self.step()
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/rllib/algorithms/algorithm.py", line 867, in step
) = self._run_one_training_iteration_and_evaluation_in_parallel()
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/rllib/algorithms/algorithm.py", line 3239, in _run_one_training_iteration_and_evaluation_in_parallel
evaluation_results = self._run_one_evaluation(
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/rllib/algorithms/algorithm.py", line 3183, in _run_one_evaluation
eval_results = self.evaluate(parallel_train_future=parallel_train_future)
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/rllib/algorithms/algorithm.py", line 988, in evaluate
self.evaluation_workers.sync_env_runner_states(
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/rllib/env/env_runner_group.py", line 484, in sync_env_runner_states
self.foreach_worker(
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/rllib/env/env_runner_group.py", line 836, in foreach_worker
local_result = [func(self.local_worker())]
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/rllib/env/env_runner_group.py", line 468, in _update
_env_runner._env_to_module.set_state(
File "/home/mfioravanti/example/.venv/lib/python3.10/site-packages/ray/rllib/connectors/connector_pipeline_v2.py", line 231, in set_state
raise KeyError(f"No state found in `state` for connector piece: {key}!")
KeyError: 'No state found in `state` for connector piece: 000_AddObservationsFromEpisodesToBatch!'
i tried to understand what is going on, but as far as i can tell, there was no other issue before this one, and this one seems non deterministic. If there a known way i can work around it at least?
since it is just the evaluation crashing, i would be fine with disabling ray tune evaluation