How severe does this issue affect your experience of using Ray?
- High: It blocks me to complete my task.
INSTALLATION INFORMATIONS:
OS: Ubuntu 20.04.5 LTS
Python: 3.8.13
Ray: 1.12.1
CONDA INFO
active environment : sisppeo-dev
active env location : /nfs/.../conda/.../envs/sisppeo-dev
shell level : 2
user config file : /local/.../.condarc
populated config files : /local/.../.condarc
conda version : 4.12.0
conda-build version : 3.21.5
python version : 3.9.7.final.0
virtual packages : __linux=5.4.0=0
__glibc=2.31=0
__unix=0=0
__archspec=1=x86_64
base environment : /opt/anaconda3 (read only)
conda av data dir : /opt/anaconda3/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://conda.anaconda.org/anaconda/linux-64
https://conda.anaconda.org/anaconda/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /nfs/.../conda/pkgs
envs directories : /nfs/.../conda/.../envs
/local/.../.conda/envs
/opt/anaconda3/envs
platform : linux-64
user-agent : conda/4.12.0 requests/2.26.0 CPython/3.9.7 Linux/5.4.0-122-generic ubuntu/20.04.5 glibc/2.31
UID:GID : 2457:9003
netrc file : None
offline mode : False
CONDA LIST
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
aenum 3.1.11 pyhd8ed1ab_0 conda-forge
affine 2.3.1 pyhd8ed1ab_0 conda-forge
aiosignal 1.2.0 pyhd8ed1ab_0 conda-forge
alabaster 0.7.12 pypi_0 pypi
aom 3.5.0 h27087fc_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
attrs 22.1.0 pyh71513ae_1 conda-forge
babel 2.10.3 pypi_0 pypi
beautifulsoup4 4.11.1 pypi_0 pypi
blessings 1.7 py38h578d9bd_1006 conda-forge
blinker 1.5 pyhd8ed1ab_0 conda-forge
blosc 1.21.1 h83bc5f7_3 conda-forge
bokeh 2.4.3 pypi_0 pypi
boost-cpp 1.78.0 h75c5d50_1 conda-forge
boto3 1.24.85 pyhd8ed1ab_0 conda-forge
botocore 1.27.85 pyhd8ed1ab_0 conda-forge
brotli 1.0.9 h166bdaf_7 conda-forge
brotli-bin 1.0.9 h166bdaf_7 conda-forge
brotlipy 0.7.0 py38h0a891b7_1004 conda-forge
brunsli 0.1 h9c3ff4c_0 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
c-blosc2 2.4.2 h7a311fb_0 conda-forge
ca-certificates 2022.9.24 ha878542_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cairo 1.16.0 ha61ee94_1014 conda-forge
certifi 2022.9.24 pyhd8ed1ab_0 conda-forge
cf_xarray 0.7.4 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py38h4a40e3a_0 conda-forge
cfgv 3.3.1 pyhd8ed1ab_0 conda-forge
cfitsio 4.1.0 hd9d235c_0 conda-forge
cftime 1.6.2 py38h26c90d9_0 conda-forge
charls 2.3.4 h9c3ff4c_0 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
click 8.1.3 py38h578d9bd_0 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
cloudpickle 2.2.0 pyhd8ed1ab_0 conda-forge
colorama 0.4.5 pyhd8ed1ab_0 conda-forge
colorcet 3.0.0 pyhd8ed1ab_0 conda-forge
cryptography 37.0.1 py38h9ce1e76_0 anaconda
curl 7.85.0 h2283fc2_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cytoolz 0.12.0 py38h0a891b7_0 conda-forge
dask-core 2022.9.2 pyhd8ed1ab_0 conda-forge
dataclasses-json 0.5.7 pyhd8ed1ab_0 conda-forge
datashader 0.14.2 pyh6c4a22f_0 conda-forge
datashape 0.5.4 py_1 conda-forge
dav1d 1.0.0 h166bdaf_1 conda-forge
distlib 0.3.5 pyhd8ed1ab_0 conda-forge
distributed 2022.8.1 pypi_0 pypi
docutils 0.17.1 pypi_0 pypi
ephem 4.1.3 py38h0a891b7_4 conda-forge
esmf 8.2.0 nompi_h1ee4d8d_2 conda-forge
esmpy 8.2.0 nompi_py38h6633fd3_1 conda-forge
expat 2.4.9 h27087fc_0 conda-forge
filelock 3.8.0 pyhd8ed1ab_0 conda-forge
fiona 1.8.21 py38hd65b8f4_2 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.0 hc2a2eb6_1 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.37.4 py38h0a891b7_0 conda-forge
freetype 2.12.1 hca18f0e_0 conda-forge
freexl 1.0.6 h166bdaf_1 conda-forge
frozenlist 1.3.1 py38h0a891b7_0 conda-forge
fsspec 2022.8.2 pyhd8ed1ab_0 conda-forge
gdal 3.5.2 py38h721c686_1 conda-forge
geos 3.11.0 h27087fc_0 conda-forge
geotiff 1.7.1 h4fc65e6_3 conda-forge
gettext 0.19.8.1 h27087fc_1009 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
grpcio 1.43.0 py38h2457d08_0 conda-forge
h5py 3.7.0 nompi_py38h045baee_101 conda-forge
hdf4 4.2.15 h9772cbc_4 conda-forge
hdf5 1.12.2 nompi_h4df4325_100 conda-forge
heapdict 1.0.1 pypi_0 pypi
icu 70.1 h27087fc_0 conda-forge
identify 2.5.6 pyhd8ed1ab_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
imagecodecs 2022.9.26 py38h0e951e9_0 conda-forge
imageio 2.22.0 pyhfa7a67d_0 conda-forge
imagesize 1.4.1 pypi_0 pypi
importlib-metadata 4.11.4 py38h578d9bd_0 conda-forge
iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge
jinja2 3.1.2 pypi_0 pypi
jmespath 1.0.1 pyhd8ed1ab_0 conda-forge
joblib 1.2.0 pyhd8ed1ab_0 conda-forge
jpeg 9e h166bdaf_2 conda-forge
json-c 0.16 hc379101_0 conda-forge
jxrlib 1.1 h7f98852_2 conda-forge
kealib 1.4.15 ha7026e8_1 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 py38h43d8883_0 conda-forge
krb5 1.19.3 h08a2579_0 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.38 h1181459_1 anaconda
lerc 4.0.0 h27087fc_0 conda-forge
libaec 1.0.6 h9c3ff4c_0 conda-forge
libavif 0.10.1 h5cdd6b5_2 conda-forge
libblas 3.9.0 16_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h166bdaf_7 conda-forge
libbrotlidec 1.0.9 h166bdaf_7 conda-forge
libbrotlienc 1.0.9 h166bdaf_7 conda-forge
libcblas 3.9.0 16_linux64_openblas conda-forge
libcurl 7.85.0 h2283fc2_0 conda-forge
libdap4 3.20.6 hd7c4107_2 conda-forge
libdeflate 1.14 h166bdaf_0 conda-forge
libedit 3.1.20210910 h7f8727e_0 anaconda
libev 4.33 h516909a_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 12.1.0 h8d9b700_16 conda-forge
libgdal 3.5.2 h84f2e82_1 conda-forge
libgfortran-ng 12.1.0 h69a702a_16 conda-forge
libgfortran5 12.1.0 hdcd56e2_16 conda-forge
libglib 2.74.0 h7a41b64_0 conda-forge
libgomp 12.1.0 h8d9b700_16 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libkml 1.3.0 h37653c0_1015 conda-forge
liblapack 3.9.0 16_linux64_openblas conda-forge
libllvm11 11.1.0 he0ac6c6_4 conda-forge
libnetcdf 4.8.1 nompi_h21705cb_104 conda-forge
libnghttp2 1.47.0 hff17c54_1 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge
libpng 1.6.38 h753d276_0 conda-forge
libpq 14.5 he2d8382_0 conda-forge
libprotobuf 3.21.7 h6239696_0 conda-forge
librttopo 1.1.0 hf730bdb_11 conda-forge
libspatialite 5.0.1 hd36657c_19 conda-forge
libsqlite 3.39.4 h753d276_0 conda-forge
libssh2 1.10.0 hf14f497_3 conda-forge
libstdcxx-ng 12.1.0 ha89aaad_16 conda-forge
libtiff 4.4.0 h55922b4_4 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libwebp-base 1.2.4 h166bdaf_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxml2 2.10.2 h4c7fe37_1 conda-forge
libxslt 1.1.35 h8affb1d_0 conda-forge
libzip 1.9.2 hc929e4a_1 conda-forge
libzlib 1.2.12 h166bdaf_4 conda-forge
libzopfli 1.0.3 h9c3ff4c_0 conda-forge
lightgbm 3.3.2 py38h709712a_0 conda-forge
llvmlite 0.39.1 py38h38d86a4_0 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lxml 4.9.1 py38h0a891b7_0 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
markupsafe 2.1.1 pypi_0 pypi
marshmallow 3.18.0 pyhd8ed1ab_0 conda-forge
marshmallow-enum 1.5.1 pyh9f0ad1d_3 conda-forge
matplotlib-base 3.5.3 py38h38b5ce0_2 conda-forge
msgpack-python 1.0.4 py38h43d8883_0 conda-forge
multipledispatch 0.6.0 py_0 conda-forge
munch 2.5.0 py_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
mypy_extensions 0.4.3 py38h578d9bd_5 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
netcdf-fortran 4.6.0 nompi_hc402ea5_101 conda-forge
netcdf4 1.6.0 nompi_py38h32db9c8_101 conda-forge
networkx 2.8.7 pyhd8ed1ab_0 conda-forge
nodeenv 1.7.0 pyhd8ed1ab_0 conda-forge
nspr 4.33 h295c915_0 anaconda
nss 3.78 h2350873_0 conda-forge
numba 0.56.2 py38h9a4aae9_1 conda-forge
numpy 1.23.2 py38h3a7f9d9_0 conda-forge
oauthlib 3.2.1 pyhd8ed1ab_0 conda-forge
openjpeg 2.5.0 h7d73246_1 conda-forge
openssl 3.0.5 h166bdaf_2 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.4.3 py38h47df419_0 conda-forge
param 1.12.2 pyh6c4a22f_0 conda-forge
partd 1.3.0 pyhd8ed1ab_0 conda-forge
patsy 0.5.2 pyhd8ed1ab_0 conda-forge
pcre 8.45 h9c3ff4c_0 conda-forge
pcre2 10.37 hc3806b6_1 conda-forge
pillow 8.4.0 pypi_0 pypi
pip 22.2.2 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
platformdirs 2.5.2 pyhd8ed1ab_1 conda-forge
plotly 5.10.0 pyhd8ed1ab_0 conda-forge
pluggy 1.0.0 py38h578d9bd_3 conda-forge
pooch 1.6.0 pyhd8ed1ab_0 conda-forge
poppler 22.04.0 h0733791_3 conda-forge
poppler-data 0.4.11 hd8ed1ab_0 conda-forge
postgresql 14.5 ha7cec9f_0 conda-forge
pre-commit 2.20.0 py38h578d9bd_0 conda-forge
proj 9.0.1 h93bde94_1 conda-forge
protobuf 4.21.7 py38hfa26641_0 conda-forge
psutil 5.9.1 py38h0a891b7_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
pvlib 0.9.2 pyha770c72_2 conda-forge
pvlib-python 0.9.2 hd8ed1ab_2 conda-forge
py 1.11.0 pyh6c4a22f_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyct 0.4.8 py38h06a4308_1 anaconda
pydata-sphinx-theme 0.9.0 pypi_0 pypi
pygments 2.13.0 pypi_0 pypi
pyhdf 0.10.5 py38hd2cc768_0 conda-forge
pyjwt 2.5.0 pyhd8ed1ab_0 conda-forge
pyopenssl 22.0.0 pyhd8ed1ab_1 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyproj 3.3.1 py38he1635e7_1 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
pytest 7.1.2 py38h578d9bd_0 conda-forge
python 3.8.13 ha86cf86_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python_abi 3.8 2_cp38 conda-forge
pytz 2022.4 pyhd8ed1ab_0 conda-forge
pywavelets 1.3.0 py38h71d37f0_1 conda-forge
pyyaml 6.0 py38h0a891b7_4 conda-forge
rasterio 1.3.2 py38h164e37e_0 conda-forge
ray-core 1.12.1 py38hb9c8dfe_0 conda-forge
readline 8.1.2 h0f457ee_0 conda-forge
requests 2.28.1 pyhd8ed1ab_1 conda-forge
requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge
rioxarray 0.12.0 pyhd8ed1ab_0 conda-forge
s2cloudless 1.5.0 pyhd8ed1ab_0 conda-forge
s3transfer 0.6.0 pyhd8ed1ab_0 conda-forge
scikit-image 0.19.3 py38h47df419_1 conda-forge
scikit-learn 1.1.2 py38h0b08f9b_0 conda-forge
scipy 1.9.0 py38hea3f02b_0 conda-forge
sentinelhub 3.4.4 pyhd8ed1ab_0 conda-forge
setproctitle 1.2.2 py38h0a891b7_2 conda-forge
setuptools 65.4.1 pyhd8ed1ab_0 conda-forge
shapely 1.8.4 py38h3b45516_0 conda-forge
sisppeo 1.3.0 pypi_0 pypi
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.9 hbd366e4_1 conda-forge
snowballstemmer 2.2.0 pypi_0 pypi
snuggs 1.4.7 py_0 conda-forge
sortedcontainers 2.4.0 pypi_0 pypi
soupsieve 2.3.2.post1 pypi_0 pypi
sparse 0.13.0 pyhd8ed1ab_0 conda-forge
sphinx 4.5.0 pypi_0 pypi
sphinx-copybutton 0.5.0 pypi_0 pypi
sphinx-panels 0.6.0 pypi_0 pypi
sphinxcontrib-applehelp 1.0.2 pypi_0 pypi
sphinxcontrib-devhelp 1.0.2 pypi_0 pypi
sphinxcontrib-htmlhelp 2.0.0 pypi_0 pypi
sphinxcontrib-jsmath 1.0.1 pypi_0 pypi
sphinxcontrib-qthelp 1.0.3 pypi_0 pypi
sphinxcontrib-serializinghtml 1.1.5 pypi_0 pypi
sqlite 3.39.4 h4ff8645_0 conda-forge
statsmodels 0.13.2 py38h71d37f0_0 conda-forge
stringcase 1.2.0 py_0 conda-forge
tblib 1.7.0 pypi_0 pypi
tenacity 8.1.0 pyhd8ed1ab_0 conda-forge
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tifffile 2022.8.12 pyhd8ed1ab_0 conda-forge
tiledb 2.11.3 h3f4058f_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
tomli 2.0.1 pyhd8ed1ab_0 conda-forge
toolz 0.12.0 pyhd8ed1ab_0 conda-forge
tornado 6.1 pypi_0 pypi
tqdm 4.64.0 pyhd8ed1ab_0 conda-forge
typing_extensions 4.3.0 pyha770c72_0 conda-forge
typing_inspect 0.7.1 pyh6c4a22f_0 conda-forge
tzcode 2022d h166bdaf_0 conda-forge
tzdata 2022d h191b570_0 conda-forge
ukkonen 1.0.1 py38h43d8883_2 conda-forge
unicodedata2 14.0.0 py38h0a891b7_1 conda-forge
urllib3 1.26.11 pyhd8ed1ab_0 conda-forge
utm 0.7.0 pyhd8ed1ab_0 conda-forge
virtualenv 20.16.5 py38h578d9bd_0 conda-forge
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
xarray 2022.6.0 pyhd8ed1ab_1 conda-forge
xerces-c 3.2.3 h55805fa_5 conda-forge
xesmf 0.6.3 pyhd8ed1ab_1 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.2 h7f98852_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
zfp 1.0.0 h27087fc_1 conda-forge
zict 2.2.0 pypi_0 pypi
zipp 3.8.1 pyhd8ed1ab_0 conda-forge
zlib 1.2.12 h166bdaf_4 conda-forge
zlib-ng 2.0.6 h166bdaf_0 conda-forge
zstd 1.5.2 h6239696_4 conda-forge
ISSUE
Description of the problem
I use of set of functions (which I simplified a lot and detailed below after traceback) to extract and transform some data into an xarray Dataset.
I have 3 functions which i’ll call A, B and C:
def A(**kwargs):
ray_init(input_products)
# make None parameters lists of same size as Input_products from kwargs
futures = []
for kwarg in zip(**kwargs):
futures.append(B.remote(**copy.copy(kwargs)))
res = ray_get(futures, 'A')
if save:
# produce filename then save
return res
@ray.remote
def B(**kwargs):
# tweak config from kwargs
return C(**kwargs, **config)
def C(**kwargs):
builder = ProductBuilder()
# processing steps calls from builder class
products = builder.get_products()
if save:
# produce filename then save
return products
- Function A initialize ray (shutdown if initalized then init with correct number of cpus) , then produces the futures list (append B.remote(**kwargs)) and finally retrieve the results with ray_get (custom function detailed after traceback)
- Function B do some tweaking to a configuration dictionnary then call function C
- Function C do the real work then returns the result
What makes me reallt think that this is a Ray problem is that up to the return of the result (last line of function C), everything goes well (I tested it with prints an debugger). But, as soon as I try to retrieve the result with yield ray.get(done) (in function ray_get) in the next line of function A (so the returns of function C and B are ok) I get the following error message:
Error message (simplified)
=> call to custom ray_get function (detailed below)
==> result = [_ for _ in tqdm(to_generator(futures), in ray_get
==> result = [_ for _ in tqdm(to_generator(futures), in <listcomp>
==> tqdm/std.py", line 1195, in __iter__
for obj in iterable:
==> yield ray.get(done) in to_generator
==> ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
==> ray/worker.py", line 1811, in get
raise value
ray.exceptions.RaySystemError: System error: can't set attribute
traceback: Traceback (most recent call last):
==> ray/serialization.py", line 332, in deserialize_objects
obj = self._deserialize_object(data, metadata, object_ref)
==> ray/serialization.py", line 235, in _deserialize_object
return self._deserialize_msgpack_data(data, metadata_fields)
==> ray/serialization.py", line 190, in _deserialize_msgpack_data
python_objects = self._deserialize_pickle5_data(pickle5_data)
==> ray/serialization.py", line 178, in _deserialize_pickle5_data
obj = pickle.loads(in_band, buffers=buffers)
AttributeError: can't set attribute
I tried to look for this error message in ray’s and pickle’s source code but could not find it. Also I tried to debug my program with pycharm’s debugger but since Objectref is obfuscated i couldn’t go further this way either.
Thus I’m asking in this forum the following:
- I don’t know what the cause of this error can be
- Error message should be clearer
def ray_init(iterable: Sized):
"""Get the number of available CPUs from any sized object."""
num_cpus = psutil.cpu_count(logical=False)
cpus = [psutil.cpu_count(logical=False), len(iterable), num_cpus]
num_cpus = min(val for val in cpus if val is not None)
if ray.is_initialized():
ray.shutdown()
ray.init(num_cpus=num_cpus)
def ray_get(futures: list, msg: str = 'example') -> list:
"""Process ray futures with a progress bar."""
def to_generator(remaining):
while remaining:
done, remaining = ray.wait(remaining)
yield ray.get(done)
result = [_ for _ in tqdm(to_generator(futures),
total=len(futures),
desc=f'Processing {msg}',
position=0)]
ray.shutdown()
return result
