How do I call my custom TorchPolicyV2 train with ray.tune?

RLlib
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hi,
I implemented my custom TorchPolicyV2.
How do I now create an Algorithm out of it, and use it with ray.tune?
I found this code in the documentation: Key Concepts — Ray 2.9.3
However, I don’t want to run it in a while loop, but with ray.tune?

  config = DQNConfig()
  env = gym.make("CartPole-v1")
  policy = MyCustomPolicy(observation_space=env.observation_space,
                                     action_space=env.action_space,
                                     config=config)  # implenents TorchPolicyV2
  
  
  config = DQNConfig().environment(env="CartPole-v1").training(train_batch_size=8)
  algo = config.build()
  algo = algo.add_policy(policy=policy, policy_id="my_custom_policy")
  print(algo.train())

I get the following errors:
Can not figure out a durable policy name for <class 'my_custom_policy.MyCustomPolicy'>. You are probably trying to checkpoint a custom policy. Raw policy class may cause problems when the checkpoint needs to be loaded in the future. To fix this, make sure you add your custom policy in rllib.algorithms.registry.POLICIES.

AttributeError: 'MyCustomPolicy' object has no attribute 'train'

Thanks in advance for your help

2

@ChiefAlu it looks like you try to add a policy that is not registered (default). Afaik there is no way to register a custom policy. Instead you might want to take a look at this doc section here.

Also, keep in mind: adding a policy to an algorithm requires you to have a policy mapping function defined in the configuration. This is used for MARL.