# Ray tune trials fail due to unexpected worker exit
**URL:** https://discuss.ray.io/t/ray-tune-trials-fail-due-to-unexpected-worker-exit/14192
**Category:** Ray Train
**Created:** [March 28, 2024, 6:21pm UTC](https://discuss.ray.io/t/ray-tune-trials-fail-due-to-unexpected-worker-exit/14192 "2024-03-28T18:21:15Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@KSimi7](https://discuss.ray.io/u/KSimi7)
#### Post date: [March 28, 2024, 6:21pm UTC](https://discuss.ray.io/t/ray-tune-trials-fail-due-to-unexpected-worker-exit/14192/1 "2024-03-28T18:21:15Z")
**How severe does this issue affect your experience of using Ray?**
- High: It blocks me to complete my task.
Hi, I am new to using ray tune for parameter optimization. I followed this example - [How to use Tune with PyTorch — Ray 2.10.0](https://docs.ray.io/en/latest/tune/examples/tune-pytorch-cifar.html) to write my own code to perform a hyperparameter sweep.
I also tested this tutorial in my own environment and it worked fine so there is some issue when I try to replicate the above with my own code for training.
Here is error log -
2024-03-28 13:08:49,895 ERROR tune\_controller.py:1332 – Trial task failed for trial train\_raytune\_36d7f\_00000
Traceback (most recent call last):
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/air/execution/\_internal/event\_manager.py”, line 110, in resolve\_future
result = ray.get(future)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/auto\_init\_hook.py”, line 21, in auto\_init\_wrapper
return fn(\*args, \*\*kwargs)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/client\_mode\_hook.py”, line 103, in wrapper
return func(\*args, \*\*kwargs)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/worker.py”, line 2667, in get
values, debugger\_breakpoint = worker.get\_objects(object\_refs, timeout=timeout)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/worker.py”, line 866, in get\_objects
raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
class\_name: ImplicitFunc
actor\_id: 4d5b5d4750a02b9e81d49ecc01000000
pid: 468929
namespace: 5fec80a5-be18-4e76-b070-24d50c52c44a
ip: 10.141.26.182
The actor is dead because its worker process has died. Worker exit type: SYSTEM\_ERROR Worker exit detail: Worker exits unexpectedly. Worker exits with an exit code None. Traceback (most recent call last):
File “python/ray/\_raylet.pyx”, line 1883, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 1984, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 1889, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 1830, in ray.\_raylet.execute\_task.function\_executor
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/function\_manager.py”, line 724, in actor\_method\_executor
return method(\_\_ray\_actor, \*args, \*\*kwargs)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/util/tracing/tracing\_helper.py”, line 467, in \_resume\_span
return method(self, \*\_args, \*\*\_kwargs)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/tune/trainable/trainable.py”, line 334, in train
raise skipped from exception\_cause(skipped)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/air/\_internal/util.py”, line 88, in run
self.\_ret = self.\_target(\*self.\_args, \*\*self.\_kwargs)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/tune/trainable/function\_trainable.py”, line 53, in
training\_func=lambda: self.\_trainable\_func(self.config),
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/util/tracing/tracing\_helper.py”, line 467, in \_resume\_span
return method(self, \*\_args, \*\*\_kwargs)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/tune/trainable/function\_trainable.py”, line 261, in \_trainable\_func
output = fn()
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/tune/trainable/util.py”, line 130, in inner
return trainable(config, \*\*fn\_kwargs)
File “/tmp/ipykernel\_459985/2726713512.py”, line 91, in train\_raytune
AttributeError: ‘Tensor’ object has no attribute ‘numpu’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/cloudpickle/cloudpickle.py”, line 1245, in dump
return super().dump(obj)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/tblib/pickling\_support.py”, line 46, in pickle\_exception
rv = obj. **reduce\_ex** (3)
RecursionError: maximum recursion depth exceeded while calling a Python object
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “python/ray/\_raylet.pyx”, line 2281, in ray.\_raylet.task\_execution\_handler
File “python/ray/\_raylet.pyx”, line 2177, in ray.\_raylet.execute\_task\_with\_cancellation\_handler
File “python/ray/\_raylet.pyx”, line 1832, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 1833, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 2071, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 1089, in ray.\_raylet.store\_task\_errors
File “python/ray/\_raylet.pyx”, line 4575, in ray.\_raylet.CoreWorker.store\_task\_outputs
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/serialization.py”, line 494, in serialize
return self.\_serialize\_to\_msgpack(value)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/serialization.py”, line 449, in \_serialize\_to\_msgpack
value = value.to\_bytes()
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/exceptions.py”, line 32, in to\_bytes
serialized\_exception=pickle.dumps(self),
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/cloudpickle/cloudpickle.py”, line 1479, in dumps
cp.dump(obj)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/cloudpickle/cloudpickle.py”, line 1249, in dump
raise pickle.PicklingError(msg) from e
\_pickle.PicklingError: Could not pickle object as excessively deep recursion required.
An unexpected internal error occurred while the worker was executing a task.
2024-03-28 13:08:50,108 ERROR tune\_controller.py:1332 – Trial task failed for trial train\_raytune\_36d7f\_00001
Traceback (most recent call last):
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/air/execution/\_internal/event\_manager.py”, line 110, in resolve\_future
result = ray.get(future)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/auto\_init\_hook.py”, line 21, in auto\_init\_wrapper
return fn(\*args, \*\*kwargs)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/client\_mode\_hook.py”, line 103, in wrapper
return func(\*args, \*\*kwargs)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/worker.py”, line 2667, in get
values, debugger\_breakpoint = worker.get\_objects(object\_refs, timeout=timeout)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/worker.py”, line 866, in get\_objects
raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
class\_name: ImplicitFunc
actor\_id: a09a9258d0748b83315d473c01000000
pid: 468930
namespace: 5fec80a5-be18-4e76-b070-24d50c52c44a
ip: 10.141.26.182
The actor is dead because its worker process has died. Worker exit type: SYSTEM\_ERROR Worker exit detail: Worker exits unexpectedly. Worker exits with an exit code None. Traceback (most recent call last):
File “python/ray/\_raylet.pyx”, line 1883, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 1984, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 1889, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 1830, in ray.\_raylet.execute\_task.function\_executor
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/function\_manager.py”, line 724, in actor\_method\_executor
return method(\_\_ray\_actor, \*args, \*\*kwargs)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/util/tracing/tracing\_helper.py”, line 467, in \_resume\_span
return method(self, \*\_args, \*\*\_kwargs)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/tune/trainable/trainable.py”, line 334, in train
raise skipped from exception\_cause(skipped)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/air/\_internal/util.py”, line 88, in run
self.\_ret = self.\_target(\*self.\_args, \*\*self.\_kwargs)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/tune/trainable/function\_trainable.py”, line 53, in
training\_func=lambda: self.\_trainable\_func(self.config),
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/util/tracing/tracing\_helper.py”, line 467, in \_resume\_span
return method(self, \*\_args, \*\*\_kwargs)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/tune/trainable/function\_trainable.py”, line 261, in \_trainable\_func
output = fn()
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/tune/trainable/util.py”, line 130, in inner
return trainable(config, \*\*fn\_kwargs)
File “/tmp/ipykernel\_459985/2726713512.py”, line 91, in train\_raytune
AttributeError: ‘Tensor’ object has no attribute ‘numpu’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/cloudpickle/cloudpickle.py”, line 1245, in dump
return super().dump(obj)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/tblib/pickling\_support.py”, line 46, in pickle\_exception
rv = obj. **reduce\_ex** (3)
RecursionError: maximum recursion depth exceeded while calling a Python object
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “python/ray/\_raylet.pyx”, line 2281, in ray.\_raylet.task\_execution\_handler
File “python/ray/\_raylet.pyx”, line 2177, in ray.\_raylet.execute\_task\_with\_cancellation\_handler
File “python/ray/\_raylet.pyx”, line 1832, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 1833, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 2071, in ray.\_raylet.execute\_task
File “python/ray/\_raylet.pyx”, line 1089, in ray.\_raylet.store\_task\_errors
File “python/ray/\_raylet.pyx”, line 4575, in ray.\_raylet.CoreWorker.store\_task\_outputs
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/serialization.py”, line 494, in serialize
return self.\_serialize\_to\_msgpack(value)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/\_private/serialization.py”, line 449, in \_serialize\_to\_msgpack
value = value.to\_bytes()
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/exceptions.py”, line 32, in to\_bytes
serialized\_exception=pickle.dumps(self),
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/cloudpickle/cloudpickle.py”, line 1479, in dumps
cp.dump(obj)
File “/home/simran/anaconda3/envs/deepcrc/lib/python3.9/site-packages/ray/cloudpickle/cloudpickle.py”, line 1249, in dump
raise pickle.PicklingError(msg) from e
\_pickle.PicklingError: Could not pickle object as excessively deep recursion required.
An unexpected internal error occurred while the worker was executing a task.
2024-03-28 13:08:50,139 INFO tune.py:1016 – Wrote the latest version of all result files and experiment state to ‘/home/simran/ray\_results/train\_raytune\_2024-03-28\_13-08-43’ in 0.0254s.
2024-03-28 13:08:50,148 ERROR tune.py:1044 – Trials did not complete: [train\_raytune\_36d7f\_00000, train\_raytune\_36d7f\_00001]
2024-03-28 13:08:50,149 INFO tune.py:1048 – Total run time: 6.22 seconds (6.16 seconds for the tuning loop).
---
### Author:  [@matthewdeng](https://discuss.ray.io/u/matthewdeng)
#### Post date: [April 1, 2024, 8:28pm UTC](https://discuss.ray.io/t/ray-tune-trials-fail-due-to-unexpected-worker-exit/14192/2 "2024-04-01T20:28:49Z")
> [@KSimi7](#):
>
> AttributeError: ‘Tensor’ object has no attribute ‘numpu’
Is there a typo somewhere in your trainable definition? I’m guessing this is expected to be `numpy`.