# Passing 'custom\_action\_dist' **URL:** \ **Category:** RLlib\ **Created:** [April 6, 2023, 5:21pm UTC](https://discuss.ray.io/t/passing-custom-action-dist/10111 "2023-04-06T17:21:33Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Username1](https://avatars.discourse-cdn.com/v4/letter/u/ee59a6/32.png) [@Username1](https://discuss.ray.io/u/Username1)\ **Post date:** [April 6, 2023, 5:21pm UTC](https://discuss.ray.io/t/passing-custom-action-dist/10111/1 "2023-04-06T17:21:33Z")
**How severe does this issue affect your experience of using Ray?** - High: It blocks me to complete my task. I am on Ray 2.0 and training PPO with a Dirichlet action space. I am training my model like this: ```auto tuner = tune.Tuner("PPO", param_space=config, run_config=air.RunConfig( name = experiment_name, ) ) results = tuner.fit() ``` Where does the “custom\_action\_dict” parameter goes now? Since the new config dict has changed from the old examples on the website. To give more background, these are all the steps I performed: 1. Import the “Simplex” action space from RLLIB and use it in the init on `self_action_space` 2. Import the Dirichlet action space from RLLIB: `from ray.rllib.models.torch.torch_action_dist import TorchDirichlet as Dirichlet` 3. Register the new action space: `from ray.rllib.models import ModelCatalog` `ModelCatalog.register_custom_action_dist("Dirichlet", Dirichlet)` 4. Pass the “custom\_action\_dict” to the trainer. **This is the part that I don’t know how to do (when using Tune to train) since the config dict has changed on Ray 2.0 from the examples on the website.** ---
**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\ **Post date:** [April 6, 2023, 6:24pm UTC](https://discuss.ray.io/t/passing-custom-action-dist/10111/2 "2023-04-06T18:24:19Z")
> ---
**Author:** ![Username1](https://avatars.discourse-cdn.com/v4/letter/u/ee59a6/32.png) [@Username1](https://discuss.ray.io/u/Username1)\ **Post date:** [April 9, 2023, 2:08pm UTC](https://discuss.ray.io/t/passing-custom-action-dist/10111/3 "2023-04-09T14:08:11Z")
Hello @mannyv . Thank you very much for your pointer, but I guess there is something else going on. I am getting this error, which is usually a “catch all” (or “red herring”) for some other error somewhere else: `AttributeError: 'PPO' object has no attribute '_warmup_time'` The error above is missleading, as I believe this is the issue going on (see below). RLLIB is trying to calculate the KL divergence and is calling the Dirichlet Class for it. I am not sure whether I am doing the steps correctly and importing the right things ```auto File "/usr/local/lib/python3.9/dist-packages/ray/rllib/models/torch/torch_action_dist.py", line 643, in kl return self.dist.kl_divergence(other.dist) AttributeError: 'Dirichlet' object has no attribute 'kl_divergence' ``` I see on the official implementation [here](https://github.com/ray-project/ray/blob/840215bc09e942b50cad0ab2db96a8fdc79217c1/rllib/models/torch/torch_action_dist.py#L601) of the Dirichlet Class that the existing method is called **“kl”** and not **“kl\_divergence”** To me, in the official code [here](https://github.com/ray-project/ray/blob/840215bc09e942b50cad0ab2db96a8fdc79217c1/rllib/models/torch/torch_action_dist.py#L643) this line is missing: ```auto def kl(self, other): return torch.distributions.kl.kl_divergence(self.dist, other.dist) ``` I’ve created a minimal example of the error here: > ---
**Author:** ![Username1](https://avatars.discourse-cdn.com/v4/letter/u/ee59a6/32.png) [@Username1](https://discuss.ray.io/u/Username1)\ **Post date:** [April 10, 2023, 3:45pm UTC](https://discuss.ray.io/t/passing-custom-action-dist/10111/4 "2023-04-10T15:45:13Z")
To me, this is a bug. Either the KL-divergence is not correct, and should be amended as I propose. Or the option I am using now in my code is to just delete the KL method and have it retrieved from the parent class. ---
**Author:** ![kourosh](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/kourosh/32/2130_2.png) [@kourosh](https://discuss.ray.io/u/kourosh)\ **Post date:** [April 10, 2023, 5:44pm UTC](https://discuss.ray.io/t/passing-custom-action-dist/10111/5 "2023-04-10T17:44:50Z")
Hey @Username1, You are right. Thanks for bringing up the bug. I have just made a PR to fix this issue. Torch.Dirchelet is not something we have good test coverage for. The fix basically inherits the default kl computation logic from parent which is indeed what you suggested. > > > \## Why are these changes needed? > > > > \## Related issue number > > > > \## Checks… > > \- \[\] I've signed off every commit(by using the -s flag, i.e., \`git commit -s\`) in this PR. > \- \[\] I've run \`scripts/format.sh\` to lint the changes in this PR. > \- \[\] I've included any doc changes needed for https://docs.ray.io/en/master/. > - \[\] I've added any new APIs to the API Reference. For example, if I added a > method in Tune, I've added it in \`doc/source/tune/api/\` under the > corresponding \`.rst\` file. > \- \[\] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ > \- Testing Strategy > - \[\] Unit tests > - \[\] Release tests > - \[\] This PR is not tested :( ---
**Author:** ![Username1](https://avatars.discourse-cdn.com/v4/letter/u/ee59a6/32.png) [@Username1](https://discuss.ray.io/u/Username1)\ **Post date:** [April 10, 2023, 5:49pm UTC](https://discuss.ray.io/t/passing-custom-action-dist/10111/6 "2023-04-10T17:49:33Z")
Thank you angel for coming to my rescue! I’ve been scratching my head for a week! Cheers and case closed!