# Sample Rule-Based Expert Demonstrations in Rllib

**URL:** <https://discuss.ray.io/t/sample-rule-based-expert-demonstrations-in-rllib/3065>\
**Category:** RLlib\
**Created:** [July 29, 2021, 5:52am UTC](https://discuss.ray.io/t/sample-rule-based-expert-demonstrations-in-rllib/3065 "2021-07-29T05:52:10Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![mickelliu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mickelliu/32/1347_2.png) [@mickelliu](https://discuss.ray.io/u/mickelliu)\
**Post date:** [July 29, 2021, 5:52am UTC](https://discuss.ray.io/t/sample-rule-based-expert-demonstrations-in-rllib/3065/1 "2021-07-29T05:52:10Z")

</div>

Hello there,

I made a custom environment (with Gym API) and I was able to use Rllib for training agents in this environment. I wrote a rule-based "expert " that does not utilize a neural network to sample its actions. I wish to sample trajectories from this “expert” then warm-start my RL agents using imitation learning based on the trajectories generated from this expert.

To this end, I probably need to build an offline dataset to do imitation learning (specifically using BC and MRWIL algorithms available in Rllib). How can I just do data sampling using Rllib without training any agent? Since I saw most of the example scripts use some sort of trainers. Do I need to instantiate a TFPolicy or TorchPolicy and fit my rule-based module into the policy (e.g. put the action-sampling rules in the “forward” function rather than using a NN)?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Lars\_Simon\_Zehnder](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lars_simon_zehnder/32/1185_2.png) [@Lars\_Simon\_Zehnder](https://discuss.ray.io/u/Lars_Simon_Zehnder)\
**Post date:** [July 29, 2021, 3:00pm UTC](https://discuss.ray.io/t/sample-rule-based-expert-demonstrations-in-rllib/3065/2 "2021-07-29T15:00:16Z")

</div>

Hi @mickelliu and welcome,

so to sample data from the expert interactions with your environment I would write a custom policy as shown in the [documentation](https://docs.ray.io/en/master/rllib-concepts.html).

Then to produce an offline dataset for training you need to use the [offline dataset API](https://docs.ray.io/en/master/rllib-concepts.html). Basically, you create from your Custom Policy a `Trainer`:

```auto
from ray.rllib.agents.trainer_template import build_trainer

MyTrainer = build_trainer(
    name="MyExpertPolicy",
    default_policy=MyExpertPolicy)

```

and with this `Trainer` you ` train()` your policy (of course it does not train as it is deterministic) and store the data to `path/to/folder/where/I/want/my/data`:

```auto
config = {
   "env": MyEnv, # class name of your gym environment
   "output": "path/to/folder/where/I/want/my/data",
   "framework": None,
}
ray.init(ignore_reinit_error=True)
mytrainer = MyTrainer(config=config)
results = mytrainer.train()

```

This data (it’s in JSON format) can then be used for training another policy. Hope this helps.

Simon

---

<div class="post-metadata">

**Author:** ![Lars\_Simon\_Zehnder](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lars_simon_zehnder/32/1185_2.png) [@Lars\_Simon\_Zehnder](https://discuss.ray.io/u/Lars_Simon_Zehnder)\
**Post date:** [July 29, 2021, 6:03pm UTC](https://discuss.ray.io/t/sample-rule-based-expert-demonstrations-in-rllib/3065/3 "2021-07-29T18:03:18Z")

</div>

Also, maybe [this thread](https://discuss.ray.io/t/how-to-pretrain-a-model-with-behavior-cloning/278) helps you.

---

<div class="post-metadata">

**Author:** ![mickelliu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mickelliu/32/1347_2.png) [@mickelliu](https://discuss.ray.io/u/mickelliu)\
**Post date:** [July 30, 2021, 5:44am UTC](https://discuss.ray.io/t/sample-rule-based-expert-demonstrations-in-rllib/3065/4 "2021-07-30T05:44:39Z")

</div>

Thanks, @Lars_Simon_Zehnder for your quick response.

Yestersday I figured out a way to do it by defining a custom trainable and policy then manually roll out the workers and run them with tune.run(). But it looks that the trainer template is much cleaner to write.

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [August 3, 2021, 6:30pm UTC](https://discuss.ray.io/t/sample-rule-based-expert-demonstrations-in-rllib/3065/5 "2021-08-03T18:30:17Z")

</div>

Hey @mickelliu , you can then also take a look at the BC/MARWIL test cases here:  
`ray.rllib.agents.marwil.tests.test_[bc|marwil].py`

or at this example script:  
`ray.rllib.examples.offline_rl.py`

that show how you hook up your dataset and use it to do offline training.

---

<div class="post-metadata">

**Author:** ![fksvensson](https://avatars.discourse-cdn.com/v4/letter/f/839c29/32.png) [@fksvensson](https://discuss.ray.io/u/fksvensson)\
**Post date:** [January 20, 2023, 2:17pm UTC](https://discuss.ray.io/t/sample-rule-based-expert-demonstrations-in-rllib/3065/6 "2023-01-20T14:17:17Z")

</div>

hello @Lars_Simon_Zehnder!

I am also trying to write a rule based policy for data collection purposes, but `from ray.rllib.agents.trainer_template import build_traine` seems to be deprecated in ray 2.0.0. Do you happen to know what has replaced it?

I am trying to register it as an algoirhtm but there seems too be factors missing in the cutstom policy when i do that

---

<div class="post-metadata">

**Author:** ![Lars\_Simon\_Zehnder](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/lars_simon_zehnder/32/1185_2.png) [@Lars\_Simon\_Zehnder](https://discuss.ray.io/u/Lars_Simon_Zehnder)\
**Post date:** [January 24, 2023, 7:11pm UTC](https://discuss.ray.io/t/sample-rule-based-expert-demonstrations-in-rllib/3065/7 "2023-01-24T19:11:05Z")

</div>

Hi @fksvensson, I had to search a little and roll back to ray 1.6.0 to find the `trainer_template.py` 😃 . In Ray 2.0.0 this has been changed to class inheritance. You can (and you should) directly inherit from the [`Algorithm`](https://github.com/ray-project/ray/blob/master/rllib/algorithms/algorithm.py) class which is a `tune` `Trainable`. In there you can define your own training steps. If you take a look at the [`RandomAgent`](https://github.com/ray-project/ray/blob/master/rllib/algorithms/random_agent/random_agent.py) you see how this is for example done for a rule based agent that is not trained. Notice, it overrides the `step()` function and therein defines the actions for the agent with the environment that is provided in the `config` you define when setting up an RLlib algorithm.

Let us know, if this worked out