Variable-Sharing between policies

RLlib
1

Hello,

I’m trying to share layers of my custom ModelV2 NN model between two policies.
I tried as noted in the documentation https://docs.ray.io/en/master/rllib-env.html#variable-sharing-between-policies where is said I can just put layers in global variables and directly share those layer objects between policy models.

But already for the “first connection” between a local layer and a globally shared layer I get this error:
ValueError: Tensor("hoist1/dense_self_hoist/kernel/Read/ReadVariableOp:0", shape=(512, 256), dtype=float32) must be from the same graph as Tensor("hoist1/embedded_self_hoist/Relu:0", shape=(?, ?, 1, 512), dtype=float32) (graphs are <tensorflow.python.framework.ops.Graph object at 0x7f1e3033d9d0> and <tensorflow.python.framework.ops.Graph object at 0x7f1d2ef7f0a0>).

What am I doing wrong? I globally share some layers in the manner showed in the example:
https://github.com/ray-project/ray/blob/ef944bc5f0d7764cd99d50500e470eac005a3d01/rllib/examples/models/shared_weights_model.py#L20

2

I forgot to set framework='tf2' which seems to be mandatory for sharing layers as global variables between models.
But thereafter it occurs another value error:

ValueError: tf.enable_eager_execution must be called at program startup

I have opened an issue, see ValueError: tf.enable_eager_execution must be called at program startup · Issue #14533 · ray-project/ray · GitHub

3

Thanks for this issue @klausk55 and for the repro script! Yeah, sharing layers like this is only possible in torch or tf-eager. In static-graph tf, you would have to do this via (old-school) tf scopes (and var re-uses).
I’ll take a look …

4

Okay @sven1977, do you will take a look at why it’s working using the default (results = tune.run("PPO", stop=stop, config=config, verbose=1) in that example?

As you said, so far I have to use the old-school manner to run the PPOTrainer w/o including tune.