Offline data with self made dataset

RLlib
1

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hi guys I have a few questions regarding rllib and using offline dataset of expert knowledge and combine it into environment I have a few questions.I created a batch builder and I do operate on Custom environment. My data comes outside of the environment.

  1. Why do I need agent_index,unroll_id action_probability in state. Since my expert knowledge is human based I cannot really know those things. It seems in tests when they just add SampleBatches directly into newly create Replay Buffer object they only need S,a,S’,r,dones,info and that’s it
  2. It seems when training on it to average reward is always NaN but I think it is expected.
  3. Once I try to use env and expert knowledge (sampler and dataset) concat_samples cannot merge the experiences on “prev_actions” which is weird as both are processed through the same env
  4. When having a dataset with 450k MDPs which is 68mb I run out of memory because it exceeds 90Gb how did it blow up to such proportions
  5. How can I ensure that everything works as supposed to? Some algorithms like DQN works right off the bat some like APEX DQN does not. It seems that distributed replay buffer is the problem. However once I used the option “no_local_replay_buffer” = False I still cannot directly access object from APEX as it is none

If someone would find few minutes to chat me about those problem would make my life easy as I am indeed trying to use it in production. Thank in advance for helpUsing ray 2.3

2

@l_pelcner

The questions asked here are all over the place :slight_smile:
Let’s please divide them into separate questions and possibly their own separate thread so that other community members can jump in or re-use the content if it’s their questions as well.
To be able to answer your questions I need more context on what you are trying to achieve and what your setup looks like? Why do you need env and dataset at the same time. Please include as much information as possible into your questions so we can guide you properly.