Use integer box space with tensorflow

My action space is a Box(-1, 1, (2,), I specifically want integers, and I don’t want to use a Discrete space because I don’t want to manage the interpretation between [0, n) and the potentially multi-dimensional Box space that my environment uses. When I try to train this in RLlib with tensorflow, I get the following error: ValueError: minval must be a scalar; got a tensor of shape [2]. After digging around a bit, I traced this to tensorflow random uniform; apparently, the minval can be broadcasted when the type is float, but not when it is integer. Too bad.

So my first thought is to change to a float type and round the actions in my step function to the nearest integer. However, I just wanted to post this on here to see if anyone else has had a similar issue and any corresponding advice.


Hey @rusu24edward , to my knowledge, we haven’t really tested any Box(int) action spaces. Could you perhaps do a PR that would fix this issue? Maybe just add an if block in the Random Exploration module that checks for the action space and then does a randint instead?

Hi @sven1977, sure I can do that. I think the underlying issue is with tensorflow. I don’t see why it cannot broadcast an integer, but I’ll ping them about this and see what comes up. Certainly, I can dig around a bit and see if I can bypass this for rllib when I have a few free cycles.

1 Like

Ok, cool, let me also quickly try this right now, adding an int box action space test case to RLlib. …

Here is a PR (still testing, but it works already with the PG algo):

Allows for arbitrary int-box action_space shapes.

@rusu24edward ^^

This looks great @sven1977!

1 Like