This post is related to this previous question ; the gist is during training, my custom model was given empty batches (the first dimension of obs is 0). I found that this is caused by data arrays being shorter than the calculated slice map in _slice method of SampleBatch (e.g., data arrays are actual…

@Aceticia , I use tuple observation spaces in some of my environment fine so I would first suspect there is something off with either the custom env or model. That said you may have found a bug in rllib. Too soon to tell. Here is what I would do to test. First I would use the RandomEnv environment…

@mannyv Thanks for the prompt reply! I think your suggestion is more systematic, but I decided it was hard to replicate the behavior of my custom env. I did find a snippet of code in RLlib that looks very suspicious and should be the cause for my problem: # RNN, attention net, or multi-agent case.…

This is the space definitions: # Set obs and act space self.action_space = spaces.Tuple( [spaces.Discrete(self.actions), spaces.Discrete(config['n_vocab'])]) # Observation space is squished tuple. self.observation_space = spaces.Tuple( …

@Aceticia , Here is a colab that uses spaces similar to the one you posted with the random env. Episodes have an expected length of 20 with a probability of being done on any particular step of 5%. It also has an lstm using the auto_lstm feature with a max_seq_len of 10. [image] Google Co…

I’m not sure why exactly, but this seems to work for tf but not torch. Here is a reproduction script: import ray import gym from ray import tune import numpy as np from gym.spaces import Discrete, Tuple, Box class RandomEnv(gym.Env): """A randomly acting environment. Can be instantiated w…

@mannyv Hi, sorry about the spamming, I’ve found my misunderstanding about the code. Normally tuple spaces are handled by flattening, but I disabled my preprocessor because of this previous issue I had. Therefore the padding couldn’t be done correctly. The reproduction script is likely also due to t…

@Aceticia , I am also reproducing your issue with the example observation space you provided for both tf and torch. It is coming from the Box space being 2D. If I flatten it from Box(…,(5,5)) to Box(…,(25,)) then the error goes away. I am not sure why yet though.

When are MARL replay buffers zero padded?

RLlib

mannyv October 11, 2021, 11:04pm 4

Hi @Aceticia,

My suggestion was not to reproduce the logic of your custom env but merely the interface between components.

What are your obs_space and action spaces?

Topic		Replies	Views
How to using self-definded tensor as padding observations for LSTM/Attention models RLlib	8	669	May 19, 2021
[RLlib] Variable-length Observation Spaces without padding RLlib	7	2796	March 9, 2021
Preprocessor error on batches of observations RLlib	4	732	February 7, 2023
[RLlib] PPO custom model only get flattened observations RLlib	5	1640	May 5, 2021
Need help with debugging: Getting batch size of 0 RLlib	1	375	October 8, 2021

When are MARL replay buffers zero padded?

Related topics