Can't pass over custom_model_config to custom_model

RLlib
1

How severe does this issue affect your experience of using Ray?

  • Low: It annoys or frustrates me for a moment.

Was expecting custom_model_config to be passed over to custom_model but it keeps giving me an error saying duplicated input:

from ray.rllib.algorithms.ppo import PPOConfig
from ray.tune.logger import pretty_print

from ray.rllib.models.torch.fcnet import FullyConnectedNetwork
from ray.rllib.models import ModelCatalog
ModelCatalog.register_custom_model("ray_torch_fcnet", FullyConnectedNetwork)

algo = (
    PPOConfig()
    .rollouts(num_rollout_workers=1)
    .resources(num_gpus=0)
    .environment(env='Pendulum-v1')
    .framework('torch')
    .rollouts(num_rollout_workers=1)
    .training(
        model={
            "custom_model": "ray_torch_fcnet",
            "custom_model_config": {'num_outputs':None},
            "vf_share_layers": True,
        }
        )
    .build()
)

for i in range(10):
    result = algo.train()
    print(pretty_print(result))
    if i % 5 == 0:
        checkpoint_dir = algo.save()
        print(f"Checkpoint saved in directory {checkpoint_dir}")

The error:

(RolloutWorker pid=32453) 2023-03-09 03:37:33,762       ERROR worker.py:763 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=32453, ip=10.215.144.97, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x7f0e30887100>)
(RolloutWorker pid=32453)   File "/home/ec2-user/SageMaker/envs/py38/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 625, in __init__
(RolloutWorker pid=32453)     self._build_policy_map(
(RolloutWorker pid=32453)   File "/home/ec2-user/SageMaker/envs/py38/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 1899, in _build_policy_map
(RolloutWorker pid=32453)     self.policy_map.create_policy(
(RolloutWorker pid=32453)   File "/home/ec2-user/SageMaker/envs/py38/lib/python3.8/site-packages/ray/rllib/policy/policy_map.py", line 134, in create_policy
(RolloutWorker pid=32453)     policy = create_policy_for_framework(
(RolloutWorker pid=32453)   File "/home/ec2-user/SageMaker/envs/py38/lib/python3.8/site-packages/ray/rllib/utils/policy.py", line 116, in create_policy_for_framework
(RolloutWorker pid=32453)     return policy_class(observation_space, action_space, merged_config)
(RolloutWorker pid=32453)   File "/home/ec2-user/SageMaker/envs/py38/lib/python3.8/site-packages/ray/rllib/algorithms/ppo/ppo_torch_policy.py", line 51, in __init__
(RolloutWorker pid=32453)     TorchPolicyV2.__init__(
(RolloutWorker pid=32453)   File "/home/ec2-user/SageMaker/envs/py38/lib/python3.8/site-packages/ray/rllib/policy/torch_policy_v2.py", line 84, in __init__
(RolloutWorker pid=32453)     model, dist_class = self._init_model_and_dist_class()
(RolloutWorker pid=32453)   File "/home/ec2-user/SageMaker/envs/py38/lib/python3.8/site-packages/ray/rllib/policy/torch_policy_v2.py", line 452, in _init_model_and_dist_class
(RolloutWorker pid=32453)     model = ModelCatalog.get_model_v2(
(RolloutWorker pid=32453)   File "/home/ec2-user/SageMaker/envs/py38/lib/python3.8/site-packages/ray/rllib/models/catalog.py", line 648, in get_model_v2
(RolloutWorker pid=32453)     raise e
(RolloutWorker pid=32453)   File "/home/ec2-user/SageMaker/envs/py38/lib/python3.8/site-packages/ray/rllib/models/catalog.py", line 622, in get_model_v2
(RolloutWorker pid=32453)     instance = model_cls(
(RolloutWorker pid=32453) TypeError: __init__() got multiple values for argument 'num_outputs'
2

Hi @zhh210,

The custom model config dictionary is passed in as kwargs to the model class init dunder. But that method already has a positional argument named num_outputs. This results in an argument with the same name passed twice which causes the error you see. You will need to rename your custom argument to have a unique name.

3

thanks @mannyv, what if I want to override the num_outputs in the fcnet model? Do I have to redefine a customized model that copies the built-in fcnet model?

4

Usually that is based on your hidden sizes and your action space so you would adjust one. of those but yes in general you could create a custom model and ignore the parameter.