How to set configuration in tune for sac algorithm?

RLlib
1

I try to set configuration for SAC algorithm in ray.tune.

My ray tune is configured:

analysis = tune.run(
run_or_experiment=“SAC”, # check if your environment is continuous or discrete before choosing training algorithm Algorithms — Ray 3.0.0.dev0
scheduler=asha_scheduler,
stop={“episode_reward_mean”: 200}, # stop training if this value is reached
mode=‘max’, # find maximum vale as a target
config=config,

My configuration dict is:

config = {
“env”: “CartPole-v0”, # name of environment from gym library, it can be defined by user, example:
“num_gpus”: 1, #number of GPUs for trainer, remember even with GPU trainer needs 1 CPU
“num_workers”: 7, # number of workers for one trainer
“lr”: tune.grid_search([0.01, 0.001, 0.0001]), # or “lr”: tune.uniform(0.0001, 0.01)
“framework”: “tf2”, # configuration to use tensorflow 2 as a main framework

# Model options for the Q network(s).
"Q_model": {
    "fcnet_activation": "relu",
    "fcnet_hiddens": [256, 256],
},
# Model options for the policy function.
"policy_model": {
    "fcnet_activation": "relu",
    "fcnet_hiddens": [256, 256],
},

}

But the error is:

When not using a state-preprocessor with SAC, fcnet_hiddens will be set to an empty list! Any hidden layer sizes are defined via policy_model.fcnet_hiddens and Q_model.fcnet_hiddens.

I don’t know how to solve it because hidden layers are defined in config dictionary, for PPO algorithm it worked fine.
I will be thankful for any suggestions.

Peter

2

If someone tested layers configurations for other algorithms it will be useful too. As far I tested tune with success only for PPO algorithm, I tuned only lr.
Maybe I define nested config parameters in wrong way??

"Q_model": {
    "fcnet_activation": "relu",
    "fcnet_hiddens": [256, 256],
}
3

Thanks for the question @Peter_Pirog .
This is just a warning, but yes, it’s a little confusing b/c you do seem to have the correct config, with both
Q_model.fcnet_hiddens and policy_model.fcnet_hiddens specified.
We should suppress this warning, if this is the case. Will provide a PR. …

4

Thank You @sven1977 for the answer. These warning was a little confusing for me. The configuration of some algorithms isn’t obvious but I think it’s worth of effort. Peter