When using PPO with Curiosity exploration in conjunction with a fully discrete obs/action space external environment I get the error below. However, this issue seems like it is a mix-up due to exploration or fully discreten external environment, as per multiple sources (python - ValueError: Shape mismatch: The shape of labels (received (15,)) should equal the shape of logits except for the last dimension (received (5, 3)) - Stack Overflow and neural network - how can I solve label shape problem in tensorflow when using one-hot encoding? - Data Science Stack Exchange) it seems like sparse_softmax_cross_entropy_with_logits is called when instead tf.nn.softmax_cross_entropy_with_logits() should be.
I have also opened an issue on: sparse_softmax_cross_entropy_with_logits Should Be "tf.nn.softmax_cross_entropy_with_logits()" · Issue #15676 · ray-project/ray · GitHub