Not sure if you are still around but I ran into the _queue.Empty error myself and was stumped on it for months. The culprit is one of 2 things.
- The connection gets dropped for whatever reason (outside of RLlib) and the client can’t communicate with server and dies
- The observation your are passing in is not the right shape (this took me ages to find out). If you run into queue issue, I would recommend having a check as follows:
print(env.observation_space.contains(obs)```
If you are lucky, it will print false right before the queue.empty error, than you will know that its an obs error