I am trying out ray with tensorflow. I use tensorflow datasets in my input pipeline with some custom mapping functions (passed to tf.data.Dataset.map()).
Now when running the model using ray I get several messages like this:
(pid=4770) 2020-12-28 19:03:37.641332: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1080, Compute Capability 6.1 (pid=4770) WARNING:tensorflow:AutoGraph could not transform <function Dataset.load_train_recording at 0x7f98b05d0ef0> and will run it as-is.
Apart from this, the model appears to work, but since autograph does not have the source code, it cannot analyse it to improve performance. This is understandable since ray serializes the code to execute remotely and does not send over the source.
Is there any workaround for this problem?
- OS: centos 7
- tensorflow: 2.3.0
- ray: 1.1.0