# How to implement a meta RL

**URL:** <https://discuss.ray.io/t/how-to-implement-a-meta-rl/2693>\
**Category:** RLlib\
**Created:** [June 29, 2021, 4:47am UTC](https://discuss.ray.io/t/how-to-implement-a-meta-rl/2693 "2021-06-29T04:47:23Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![RickLan](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ricklan/32/901_2.png) [@RickLan](https://discuss.ray.io/u/RickLan)\
**Post date:** [June 29, 2021, 4:47am UTC](https://discuss.ray.io/t/how-to-implement-a-meta-rl/2693/1 "2021-06-29T04:47:23Z")

</div>

I am trying to implement an meta RL. For example a outer RL loop with a RL inner loop. How would one start with RLlib? I am asking for some ideas and guidance. Thanks!

---

<div class="post-metadata">

**Author:** ![sven1977](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sven1977/32/53_2.png) [@sven1977](https://discuss.ray.io/u/sven1977)\
**Post date:** [June 29, 2021, 2:17pm UTC](https://discuss.ray.io/t/how-to-implement-a-meta-rl/2693/2 "2021-06-29T14:17:41Z")

</div>

Hey @RickLan , could you take a look at our MAML implementation (`rllib/agents/maml/...`)? @michaelzhiluo already solved this problem of the different loops in there. 🙂

---

<div class="post-metadata">

**Author:** ![RickLan](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ricklan/32/901_2.png) [@RickLan](https://discuss.ray.io/u/RickLan)\
**Post date:** [June 30, 2021, 12:43am UTC](https://discuss.ray.io/t/how-to-implement-a-meta-rl/2693/3 "2021-06-30T00:43:51Z")

</div>

Thank you for the quick reply. I will take a look there!

---

<div class="post-metadata">

**Author:** ![mannyv](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mannyv/32/606_2.png) [@mannyv](https://discuss.ray.io/u/mannyv)\
**Post date:** [June 30, 2021, 3:41am UTC](https://discuss.ray.io/t/how-to-implement-a-meta-rl/2693/4 "2021-06-30T03:41:06Z")

</div>

Hi @RickLan,

Another place that might also make sense is the hierarchical training example. It essentially boils down to an outer and inner loop.

[https://docs.ray.io/en/latest/rllib-env.html?highlight=multiagent#hierarchical-environments](https://docs.ray.io/en/latest/rllib-env.html?highlight=multiagent#hierarchical-environments)

> <https://github.com/ray-project/ray/blob/master/rllib/examples/hierarchical_training.py>

---

<div class="post-metadata">

**Author:** ![RickLan](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/ricklan/32/901_2.png) [@RickLan](https://discuss.ray.io/u/RickLan)\
**Post date:** [July 7, 2021, 3:49am UTC](https://discuss.ray.io/t/how-to-implement-a-meta-rl/2693/5 "2021-07-07T03:49:12Z")

</div>

Thanks, @mannyv ! Will look as well.