# Not able to locate rllib train function code

**URL:** <https://discuss.ray.io/t/not-able-to-locate-rllib-train-function-code/9230>\
**Category:** RLlib\
**Created:** [February 3, 2023, 11:24am UTC](https://discuss.ray.io/t/not-able-to-locate-rllib-train-function-code/9230 "2023-02-03T11:24:46Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Archana\_R](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/archana_r/32/3829_2.png) [@Archana\_R](https://discuss.ray.io/u/Archana_R)\
**Post date:** [February 3, 2023, 11:24am UTC](https://discuss.ray.io/t/not-able-to-locate-rllib-train-function-code/9230/1 "2023-02-03T11:24:46Z")

</div>

Below is the snapshot of my code ( I made a custom Gym Environment )

from ray.rllib.agents.ppo import PPOTrainer, DEFAULT\_CONFIG  
from ray.tune.logger import pretty\_print

agent = PPOTrainer(config, env=“fss-v1”)

for \_ in range(1):  
print(“Entered \_ :”,\_)  
result = agent.train()

print(pretty\_print(result))  
ray.shutdown()

Since my gym environment is custom , i would like to make a few changes in how Ray selects the actions ( currently i am guessing it is using the sample() function ) . To do so, I am not able to find the location of the train function that is connected to the Gym environment and calls the action and step function.

Can anyone please help me ?

---

<div class="post-metadata">

**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\
**Post date:** [February 3, 2023, 2:30pm UTC](https://discuss.ray.io/t/not-able-to-locate-rllib-train-function-code/9230/2 "2023-02-03T14:30:53Z")

</div>

Hi @Archana_R,  
It would probably be easier to ask what you want to change. Rllib is a large library with lots of abstraction. Looking at the train function is not likely to be useful. RLlib has lots of configuration and callback hooks that can be used to customize most aspects of the process.

Whether the action is a stochastic or deterministic sample depends on the configuration option “explore”.

> <https://github.com/ray-project/ray/blob/b31343a8afcafef0fbcf7e81f102aa947870265f/rllib/algorithms/algorithm_config.py#L1535>

---

<div class="post-metadata">

**Author:** ![Archana\_R](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/archana_r/32/3829_2.png) [@Archana\_R](https://discuss.ray.io/u/Archana_R)\
**Post date:** [February 3, 2023, 2:54pm UTC](https://discuss.ray.io/t/not-able-to-locate-rllib-train-function-code/9230/3 "2023-02-03T14:54:45Z")

</div>

Basically, i have 144 actions ( Multi discrete 12 , 12 ) and not all of them are legal actions . I would like to early on filter out the non legal actions , so that the agent can access the legal actions and optimise the solution.

My project is on job scheduling and hence my actions are → [Workstation , Job] since not all workstations can work with all jobs . Hence based on qualification, the action needs to be filtered out.

I understand that Action masking implies , making this change in the Neural network side . But the agent is keep on selecting Non legal actions

---

<div class="post-metadata">

**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\
**Post date:** [February 3, 2023, 3:01pm UTC](https://discuss.ray.io/t/not-able-to-locate-rllib-train-function-code/9230/4 "2023-02-03T15:01:22Z")

</div>

Hi @Archana_R,

What kind of algorithm are you using. If you are using DQN then action masking is not straightforward. If you are using a PG algorithm like A2C or PPO then you want to do action masking like this:

> <https://github.com/ray-project/ray/blob/master/rllib/examples/models/action_mask_model.py>

As for the action selection that will be a combination of an exploration algorithm which are found here:

> **[ray/rllib/utils/exploration at master · ray-project/ray](https://github.com/ray-project/ray/tree/master/rllib/utils/exploration)**
>
> master/rllib/utils/exploration

and an action distribution which is found here:

> <https://github.com/ray-project/ray/blob/master/rllib/models/torch/torch_action_dist.py>

> <https://github.com/ray-project/ray/blob/master/rllib/models/tf/tf_action_dist.py>

---

<div class="post-metadata">

**Author:** ![Archana\_R](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/archana_r/32/3829_2.png) [@Archana\_R](https://discuss.ray.io/u/Archana_R)\
**Post date:** [February 6, 2023, 11:42am UTC](https://discuss.ray.io/t/not-able-to-locate-rllib-train-function-code/9230/5 "2023-02-06T11:42:54Z")

</div>

Thank you for this. But in the files you have shared, these are empty functions. Can you please help me locate where does the action sample gets picked from the environment ?

---

<div class="post-metadata">

**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\
**Post date:** [February 6, 2023, 11:58am UTC](https://discuss.ray.io/t/not-able-to-locate-rllib-train-function-code/9230/6 "2023-02-06T11:58:59Z")

</div>

What do you mean by they are empty functions? Those are the functions that are called to convert the logits returned by your model into actions that are passed into your environment.

In particular, you probably want to look at the sample or deterministic\_sample methods in the Categorical action\_distribution.

---

<div class="post-metadata">

**Author:** ![Archana\_R](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/archana_r/32/3829_2.png) [@Archana\_R](https://discuss.ray.io/u/Archana_R)\
**Post date:** [March 22, 2023, 1:53pm UTC](https://discuss.ray.io/t/not-able-to-locate-rllib-train-function-code/9230/7 "2023-03-22T13:53:37Z")

</div>

How do we carry it out for DQN Algorithm ? Do you have any code snips for the same ?