import ray
import tensorflow as tf
class Foo:
def __init__(self):
self.x = tf.constant(1.0, dtype=tf.float32)
self.v = 123
def func(self):
@ray.remote
def g():
return self.v + 1
print("====================")
g.remote()
ray.init()
foo = Foo()
foo.func()
Running the above script gives the following log:
2021-05-22 00:55:28.420575: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-05-22 00:55:28.420626: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-05-22 00:55:30,036 INFO services.py:1174 -- View the Ray dashboard at http://127.0.0.1:8265
2021-05-22 00:55:32.527661: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-05-22 00:55:32.531571: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-05-22 00:55:32.531635: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-05-22 00:55:32.531676: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ThinkPad-Lin): /proc/driver/nvidia/version does not exist
2021-05-22 00:55:32.536871: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
====================
(pid=2475) 2021-05-22 00:55:32.724708: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
(pid=2475) 2021-05-22 00:55:32.724743: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
(pid=2473) 2021-05-22 00:55:32.724342: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
(pid=2473) 2021-05-22 00:55:32.724395: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
(pid=2472) 2021-05-22 00:55:32.731796: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
(pid=2472) 2021-05-22 00:55:32.731855: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
(pid=2474) 2021-05-22 00:55:32.761908: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
(pid=2474) 2021-05-22 00:55:32.761968: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
(pid=2472) 2021-05-22 00:55:35.307416: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
(pid=2472) 2021-05-22 00:55:35.307613: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
(pid=2472) 2021-05-22 00:55:35.307635: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
(pid=2472) 2021-05-22 00:55:35.307667: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ThinkPad-Lin): /proc/driver/nvidia/version does not exist
(pid=2472) 2021-05-22 00:55:35.311893: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
(pid=2475) 2021-05-22 00:55:35.367974: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
(pid=2475) 2021-05-22 00:55:35.368346: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
(pid=2475) 2021-05-22 00:55:35.368380: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
(pid=2475) 2021-05-22 00:55:35.368420: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ThinkPad-Lin): /proc/driver/nvidia/version does not exist
(pid=2475) 2021-05-22 00:55:35.368990: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
(pid=2473) 2021-05-22 00:55:35.389763: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
(pid=2473) 2021-05-22 00:55:35.389983: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
(pid=2473) 2021-05-22 00:55:35.390008: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
(pid=2473) 2021-05-22 00:55:35.390038: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ThinkPad-Lin): /proc/driver/nvidia/version does not exist
(pid=2473) 2021-05-22 00:55:35.391014: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
(pid=2474) 2021-05-22 00:55:35.576212: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
(pid=2474) 2021-05-22 00:55:35.576395: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
(pid=2474) 2021-05-22 00:55:35.576417: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
(pid=2474) 2021-05-22 00:55:35.576441: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ThinkPad-Lin): /proc/driver/nvidia/version does not exist
(pid=2474) 2021-05-22 00:55:35.577909: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
I have some questions:
(1) In the remote function g, we only use a scalar self.v. However, the remote workers also import the library tensorflow. I’m just wondering what ray ships to the remote workers internally.
(2) Following (1), if we first put self.v into the object store by v_id = ray.put(self.v) and use it in the remote function by return ray.get(v_id) + 1, the remote workers will not import tensorflow. Is this behavior intended? If so, it would be nice to give some tips about it like these.
(3) I see that the tensorflow is imported in multiple remote workers, namely pids 2472-2475 in the above log. However, g.remote() is only called once. Does ray internally do some initialization in other remote workers in advance?
Thanks for your help!