RLlib and gym.space

RLlib
1

I came across this point in a blog:

When setting up your action and observation spaces, stick to 
Box, Discrete, 
and Tuple. The MultiDiscrete and MultiBinary don't work 
(currently) and will cause the run to crash. Instead, wrap 
Box or Discrete spaces in the Tuple function.

from Ray and RLlib for Fast and Parallel Reinforcement Learning.

The observation space in my code consists of a dictionary of multidiscrete, multibinary, and discrete values, the action space is also a dictionary of multidiscrete values. Ray version is: (0.8.9).

Problems:
I have a dummy space in my environment code that invokes only two functions: contain() and sample() from gym.Space to ensure gym.spaces.Dict is derived from gym.Space. However, I keep getting errors related to the shape() function of gym.spaces although it is not invoked in the environment code. And, if I added it to the code, I would get a TypeError from ray/rllib/models/preprocesssors.py.

I am not sure if these errors have anything to do with the way the observation and action spaces are defined (in terms of complexity) as mentioned in the beginning.

2

Hi @Ascrypto,

Out of curiosity, is there a reason you are using such an old version of Ray?

You are missing quite a few major big fixes and enhancements with such an old version. Some of those fixes include handling complex spaces.

3

Hi @mannyv,

Yes, part of my environment is borrowed from an existing implementation, where they used the above version. I tried to upgrade it but it causes some functionalities in the existing implementation to fail (e.g., ray.rllib.optimizers). Is there a way to maintain the version while handling these complex spaces or is it just impossible to do so?

Thank you for your response.

4

I ended up updating the code and using the corresponding functions from the new Ray version.
Now, when executing ray.init():

  if sys.gettrace() is not None:
        print(
            "Debug mode detected through sys.gettrace(), turning on ray local mode. Saving"
            " experiment under ray_results/debug_experiment"
        )
        args.local_mode = True
    if args.multi_node and args.local_mode:
        sys.exit("You cannot have both local mode and multi node on at the same time")
    ray.init(
        address=args.address,
        local_mode=args.local_mode,
        memory=args.memory,
        object_store_memory=args.object_store_memory,
        redis_max_memory=args.redis_max_memory,
    )

I get RuntimeError: Unknown keyword argument(s): memory, redis_max_memory.

5

@Ascrypto,

Here is the api reference for ray.init

https://docs.ray.io/en/latest/package-ref.html#ray-init

1 Like