- High: It blocks me to complete my task.
Hi,
I would like to ask about an issue that I encountered when I try to distribute my work on multiple cpu nodes using ray.
My input file is a simulation file consisting of multiple time frames, so I would like to distribute the calculation of one frame to one task. It works fine when I just used pool from the multiprocessing python library, where only one node (128 tasks in total) can be used. Since I have more than 2,000 time frames, I would like to use multiple nodes in this calculation, and the multiprocessing python library isn’t the best choice.
I created my code using this template: ray/simple-trainer.py at master · ray-project/ray · GitHub. Here’s a brief summary of my code:
import socket
import sys
import time
import ray
@ray.remote
def hydration_water_calculation2(t, u): # in one frame
return xyz
ray.init(address=os.environ[“ip_head”])
print(“Nodes in the Ray cluster:”)
print(ray.nodes())
for i in frame_values:
ip_addresses = ray.get([hydration_water_calculation2.remote(i, u0) for _ in range(1)])
print(Counter(ip_addresses))
But I got the following error:
Traceback (most recent call last):
File “…/…/hydration_whole_global2_ray.py”, line 269, in
ip_addresses = ray.get([hydration_water_calculation2.remote(i, u0) for _ in range(1)])
File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/_private/client_mode_hook.py”, line 105, in wrapper
return func(*args, **kwargs)
File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/worker.py”, line 1809, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError: ray::hydration_water_calculation2() (pid=27283, ip=10.8.9.236)
At least one of the input arguments for this task could not be computed:
ray.exceptions.RaySystemError: System error: assignment destination is read-only
traceback: Traceback (most recent call last):
File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 332, in deserialize_objects
obj = self._deserialize_object(data, metadata, object_ref)
File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 235, in _deserialize_object
return self._deserialize_msgpack_data(data, metadata_fields)
File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 190, in _deserialize_msgpack_data
python_objects = self._deserialize_pickle5_data(pickle5_data)
File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 178, in _deserialize_pickle5_data
obj = pickle.loads(in_band, buffers=buffers)
File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 2106, in setstate
self[self.ts.frame]
File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 1610, in getitem
return self._read_frame_with_aux(frame)
File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 1642, in _read_frame_with_aux
ts = self._read_frame(frame) # pylint: disable=assignment-from-no-return
File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/XDR.py”, line 255, in _read_frame
timestep = self._read_next_timestep()
File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/XDR.py”, line 273, in _read_next_timestep
self._frame_to_ts(frame, ts)
File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/XTC.py”, line 144, in _frame_to_ts
ts.dimensions = triclinic_box(*frame.box)
File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 810, in dimensions
self._unitcell[:] = box
ValueError: assignment destination is read-only
(hydration_water_calculation2 pid=27283) 2022-05-01 22:53:55,714 ERROR serialization.py:334 – assignment destination is read-only
(hydration_water_calculation2 pid=27283) Traceback (most recent call last):
(hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 332, in deserialize_objects
(hydration_water_calculation2 pid=27283) obj = self._deserialize_object(data, metadata, object_ref)
(hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 235, in _deserialize_object
(hydration_water_calculation2 pid=27283) return self._deserialize_msgpack_data(data, metadata_fields)
(hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 190, in _deserialize_msgpack_data
(hydration_water_calculation2 pid=27283) python_objects = self._deserialize_pickle5_data(pickle5_data)
(hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/ray/serialization.py”, line 178, in _deserialize_pickle5_data
(hydration_water_calculation2 pid=27283) obj = pickle.loads(in_band, buffers=buffers)
(hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 2106, in setstate
(hydration_water_calculation2 pid=27283) self[self.ts.frame]
(hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 1610, in getitem
(hydration_water_calculation2 pid=27283) return self._read_frame_with_aux(frame)
(hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 1642, in _read_frame_with_aux
(hydration_water_calculation2 pid=27283) ts = self._read_frame(frame) # pylint: disable=assignment-from-no-return
(hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/XDR.py”, line 255, in _read_frame
(hydration_water_calculation2 pid=27283) timestep = self._read_next_timestep()
(hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/XDR.py”, line 273, in _read_next_timestep
(hydration_water_calculation2 pid=27283) self._frame_to_ts(frame, ts)
(hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/XTC.py”, line 144, in _frame_to_ts
(hydration_water_calculation2 pid=27283) ts.dimensions = triclinic_box(*frame.box)
(hydration_water_calculation2 pid=27283) File “/jet/home/chy20004/.conda/envs/ray/lib/python3.8/site-packages/MDAnalysis/coordinates/base.py”, line 810, in dimensions
(hydration_water_calculation2 pid=27283) self._unitcell[:] = box
(hydration_water_calculation2 pid=27283) ValueError: assignment destination is read-only
Could anyone help me diagnose the issue? I’m new to ray and still learning why I was getting the “assignment destination is read-only” error. Many thanks in advance!