Trying to integrate gym environments with RLLIB

RLlib
1

Hi,

I am trying to integrate a simple gym environment with RLLIB and at the moment it looks like this:


import gym
import gym_anytrading
import ray
import pandas as pd
from ray.rllib.algorithms.dqn.dqn import DQNConfig


ray.init()  # Initializes Ray

# Load custom data
df = pd.read_csv('AAPL.CO.csv')
df = df.drop(columns=['Adj Close'])
df = df[['Date', 'Open', 'High', 'Low', 'Close', 'Volume']]
df['Date'] = pd.to_datetime(df['Date'])
df.set_index('Date', inplace=True)

# Gym environment
gym_env = gym.make('stocks-v0', df=df, frame_bound=(5,100), window_size=5)

# Ray config
config = DQNConfig()
print(config.replay_buffer_config)  
replay_config = config.replay_buffer_config.update( 
    {
        "capacity": 60000,
        "prioritized_replay_alpha": 0.5,
        "prioritized_replay_beta": 0.5,
        "prioritized_replay_eps": 3e-6,
    }
)
config = config.training(replay_buffer_config=replay_config)
config = config.resources(num_gpus=1)  
config = config.rollouts(num_rollout_workers=3)  
config = config.environment(gym_env)

print(config.exploration_config)  
explore_config = config.exploration_config.update( 
    {
        "initial_epsilon": 1.5,
        "final_epsilon": 0.01,
        "epsilone_timesteps": 5000,
    }
)
config.training(lr_schedule=[[1, 1e-3], [500, 5e-3]])\ 
      .exploration(exploration_config=explore_config)

algo = config.build()  
algo.train()

The error I get however, seems to be quite unique:

self.exploration_config[“type”] == “ParameterNoise”: Typerror: ‘NoneType’ object is not subscriptable.

Does anyone know something?

2

Could it be a typo - “epsilone_timesteps” probably should read “epsilon_timesteps” in your explore_config.

I wonder if you got the extra “e” from here, but SimpleQConfig, as DQNConfig’s parent class, has “epsilon_timesteps” here" and here.

Bumped into it earlier - see if it helps.

3

I think that you need to add "type": str into the update of explore_config.