WARNING:tensorflow:AutoGraph could not transform

1

Hi,

I am trying out ray with tensorflow. I use tensorflow datasets in my input pipeline with some custom mapping functions (passed to tf.data.Dataset.map()).

Now when running the model using ray I get several messages like this:

(pid=4770) 2020-12-28 19:03:37.641332: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1080, Compute Capability 6.1
(pid=4770) WARNING:tensorflow:AutoGraph could not transform <function Dataset.load_train_recording at 0x7f98b05d0ef0> and will run it as-is.

Apart from this, the model appears to work, but since autograph does not have the source code, it cannot analyse it to improve performance. This is understandable since ray serializes the code to execute remotely and does not send over the source.

Is there any workaround for this problem?

  • OS: centos 7
  • tensorflow: 2.3.0
  • ray: 1.1.0

Cheers
Erik

2

A peculiarity of my environment is that I am running using an old CPU (Intel Xeon L5630). This means I have to compile tensorflow from source. Currently, I am building 2.4.0 from source to see if the error reoccurs but since the way ray works I would expect the same error to occur with 2.4.0 as well.