Change learning rete for DQN

varunjammula · February 11, 2022, 10:49pm

Hi, is there a way to use a scheduler to change the learning rate for training DQN without using tune? Currently, the learning rate is constant throughout! Any tutorial links or resources would be great!

mannyv · February 12, 2022, 2:36am

@varunjammula,

Here is an example from apex_dqn but it would work the same for synchronous dqn.

github.com

ray-project/ray/blob/7f1bacc7dc9caf6d0ec042e39499bbf1d9a7d065/rllib/agents/dqn/tests/test_apex_dqn.py#L95-L100

      
        
            lr_schedule = [
                [0, 0.2],
                [100, 0.001],
            ]
            config["lr_schedule"] = lr_schedule

MRuiz · February 12, 2022, 8:22pm

Hello @mannyv , is it possible to see the change of lr in the metrics? I have an issue with the lr schedule and I’m trying to see if its really changing

mannyv · February 12, 2022, 8:40pm

@MRuiz,

I think it should be logged as “cur_lr”

MRuiz · February 12, 2022, 9:04pm

Thank you @mannyv , I’m having some issues so I will open a topic

varunjammula · February 25, 2022, 3:27am

Thanks @mannyv . Can you please explain what the values represent or a doc I can refer to!

Lars_Simon_Zehnder · February 25, 2022, 8:46pm

@varunjammula ,

you can find something about this for example here. Most agents in RLlib are inherited from Trainer and you can build a Trainer that uses a learning rate schedule by including the mixin LearningRateSchedule. Than you can use such a behavior as shown above by @mannyv .