Choice of parameters for "conv_filters"

RLlib
1

Hi!,

I am trying to use parameter sharing with the battlefield environment (of Pettingzoo) https://www.pettingzoo.ml/magent/battlefield and because the observation shape is (13,13,5) I have to specify “conv_filters”. After some trials a working combination is (the format is [out_channels, kernel, stride]):

"post_fcnet_hiddens": [256],
"conv_filters": [
    [32, [3, 3]], 2],
    [32, [3, 3]], 2],
    [32, [3, 3]], 2],
    ]

But it has a very negative convergence value for mean reward (-140), so I am not sure if I lose useful information with this setup. Is there any way to specify “better” filters setup or to check the performance?

Thanks in advance.
George

2

Hey @george_sk , yeah, it seems like you are losing some information there. The “image” space is very small already, just 13x13. Every pixel contains useful information in this env. So doing heavy convolutions with strides > 1 could lead to information loss that means that the agent is not able to “see” properly anymore.
I would first of all try a simple flatten solution, where you just flatten the input into a 1D tensor and try to learn w/o convolutions. This gives your network full access to all available information and still would not create a too large obs space. 13x13x5=845 inputs into a dense layer.
Just write an gym env wrapper like so and wrap it around your env, something like:

class FlattenWrapper(gym.core.ObservationWrapper):
    def __init__(self, env):
        super().__init__(env)
        self.observation_space = gym.spaces.Box(
            -10.0,
            10.0,
            shape=(13*13*5, ),
            dtype=np.float32)

    def observation(self, obs):
        return np.reshape(obs, [-1])