# Implementing Jump Start Reinforcement Learning in RLLib **URL:** \ **Category:** RLlib\ **Created:** [April 30, 2022, 9:41pm UTC](https://discuss.ray.io/t/implementing-jump-start-reinforcement-learning-in-rllib/5980 "2022-04-30T21:41:46Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![manjrekarom](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/manjrekarom/32/2469_2.png) [@manjrekarom](https://discuss.ray.io/u/manjrekarom)\ **Post date:** [April 30, 2022, 9:41pm UTC](https://discuss.ray.io/t/implementing-jump-start-reinforcement-learning-in-rllib/5980/1 "2022-04-30T21:41:46Z")
**How severe does this issue affect your experience of using Ray?** - High: It blocks me to complete my task. Greetings, I am applying the paper [JSRL](https://arxiv.org/abs/2204.02372) to visual navigation (pointnav) task using RLLib. I recently came across RLLib and it seems like a great tool. But I am new to it so could someone briefly explain to me how would I go about implementing this paper? The paper explores fast learning using a prior policy in relatively difficult exploration/spare reward environments. I will describe the paper here in brief: 1. The paper uses two policies (i) a previously learned guide policy (a sub-optimal policy that knows what good states are) often learned using small amounts of (offline) data, and (ii) an exploration policy which learns via RL. 2. The goal is to enable fast learning of exploration policy given the guide policy. These policies are value-based (PPO, A2C, etc.) and thus naive initialization might not work (the paper shows experimental proof for this). 3. Training goes as follows. You first rollout guide policy and then in the same episode rollout using exploration policy for the remaining steps. Initially you rollout more with the guide policy (say after 90% of the timesteps are complete) but this amount will gradually decrease as exploration policy gets better in the course of training. Another question related to my implementation I have is whether there is a utility in rllib or ray to collect and save data for offline use from an Env for PointNav task. Pointnav task observations consist of images that the agent sees (could be rgb+depth), gps and compass reading. Thank you! ---
**Author:** ![avnishn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/avnishn/32/1544_2.png) [@avnishn](https://discuss.ray.io/u/avnishn)\ **Post date:** [May 4, 2022, 8:55pm UTC](https://discuss.ray.io/t/implementing-jump-start-reinforcement-learning-in-rllib/5980/2 "2022-05-04T20:55:39Z")
I’ve been thinking about implementing this in RLlib as well and, in the current way that RLlib works it would be somewhat difficult to do. The issue is that we don’t have a direct interface by which you can specify a guide policy/guide data. Are you planning on using a guide policy or guide data? You’d also have to define multiple samplers as well in this case, where the sampler mixes data from the guide policy. You could also use mix in replay to achieve this as well. ---
**Author:** ![manjrekarom](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/manjrekarom/32/2469_2.png) [@manjrekarom](https://discuss.ray.io/u/manjrekarom)\ **Post date:** [May 5, 2022, 7:29am UTC](https://discuss.ray.io/t/implementing-jump-start-reinforcement-learning-in-rllib/5980/3 "2022-05-05T07:29:20Z")
Yes. I went through the tutorials over last couple of days and it feels somewhat harder to implement in RLLib. Could I formulate this as a multi-agent problem thereby allowing multiple policies (one guide and another exploration policy)? About mixing up of replay buffers, I’ll need a way to properly sample the data in that case for training. For example sample batches by filtering with the policy that generated that data. ---
**Author:** ![vishalrangras](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vishalrangras/32/976_2.png) [@vishalrangras](https://discuss.ray.io/u/vishalrangras)\ **Post date:** [May 23, 2022, 1:50pm UTC](https://discuss.ray.io/t/implementing-jump-start-reinforcement-learning-in-rllib/5980/4 "2022-05-23T13:50:55Z")
Hi @manjrekarom, Not sure if this would help but maybe you can combine these concepts to achieve Jump-start RL. > [@Sample Rule-Based Expert Demonstrations in Rllib](https://discuss.ray.io/t/sample-rule-based-expert-demonstrations-in-rllib/3065): > > Hello there, I made a custom environment (with Gym API) and I was able to use Rllib for training agents in this environment. I wrote a rule-based "expert " that does not utilize a neural network to sample its actions. I wish to sample trajectories from this “expert” then warm-start my RL agents using imitation learning based on the trajectories generated from this expert. To this end, I probably need to build an offline dataset to do imitation learning (specifically using BC and MRWIL algorith… [https://docs.ray.io/en/latest/rllib/rllib-concepts.html#how-to-customize-policies](https://docs.ray.io/en/latest/rllib/rllib-concepts.html#how-to-customize-policies) [https://docs.ray.io/en/latest/rllib/rllib-offline.html](https://docs.ray.io/en/latest/rllib/rllib-offline.html) Hopefully, I will also start working on implementing JSRL in July 2022. It would be great to hear about your experiences and progress on this implementation in the meantime. ---
**Author:** ![vishalrangras](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/vishalrangras/32/976_2.png) [@vishalrangras](https://discuss.ray.io/u/vishalrangras)\ **Post date:** [May 23, 2022, 1:51pm UTC](https://discuss.ray.io/t/implementing-jump-start-reinforcement-learning-in-rllib/5980/5 "2022-05-23T13:51:44Z")
@sven1977 Is the implementation of JSRL in the pipeline for any upcoming release? ---
**Author:** ![manjrekarom](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/manjrekarom/32/2469_2.png) [@manjrekarom](https://discuss.ray.io/u/manjrekarom)\ **Post date:** [May 24, 2022, 1:48am UTC](https://discuss.ray.io/t/implementing-jump-start-reinforcement-learning-in-rllib/5980/6 "2022-05-24T01:48:17Z")
Hi @vishalrangras ! Thanks for your reply. I’ll take a look at it. ---
**Author:** ![mahuangxu](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/mahuangxu/32/2550_2.png) [@mahuangxu](https://discuss.ray.io/u/mahuangxu)\ **Post date:** [May 24, 2022, 2:21am UTC](https://discuss.ray.io/t/implementing-jump-start-reinforcement-learning-in-rllib/5980/7 "2022-05-24T02:21:30Z")
It seems like DQfD/POfD algorithm. I wonder if RLlib will provide these algorithms. ---
**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\ **Post date:** [May 27, 2022, 3:13pm UTC](https://discuss.ray.io/t/implementing-jump-start-reinforcement-learning-in-rllib/5980/8 "2022-05-27T15:13:53Z")
For anyone visiting this thread: There is [an open feature request](https://github.com/ray-project/ray/issues/25058) on our GH for this thanks to @mahuangxu. Feel free to +1 on this or share your view of why this is important to express your need. This way we can better track this and assess a priority. ---
**Author:** ![arturn](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/arturn/32/2096_2.png) [@arturn](https://discuss.ray.io/u/arturn)\ **Post date:** [May 27, 2022, 3:17pm UTC](https://discuss.ray.io/t/implementing-jump-start-reinforcement-learning-in-rllib/5980/9 "2022-05-27T15:17:03Z")
> [@manjrekarom](#): > > Another question related to my implementation I have is whether there is a utility in rllib or ray to collect and save data for offline use from an Env for PointNav task. Pointnav task observations consist of images that the agent sees (could be rgb+depth), gps and compass reading. There is no out-of-the-box super easy way. Especially in a cluster environment. If you work locally, you can easily modify our training iteration functions to include to save batches your way there.