Reliable way to distinguish dummy trajectory view data?

RLlib
1

Hi, I’m working on using trajectory view for one of my project. The inference and training of the network requires data from the past, thus at the beginning of an episode, we will get dummy data instead of actual input in order to keep the input dimensions correct.

My question is: What is a reliable way to see how many of the observations are dummy data? What is the pattern being used to create dummy data? Thank you.

2

Hi @Aceticia ,

usually the dummy data is for the view requirements are created at different points depending on which view requirements you define. For example the initial states are created in your get_initial_state() method of your Policy (here is the function calling this method).

The main function is defined in the policy.py file. Here, the view requirements are iterated one by one and the get_dummy_batch_for_space() method is called. This is also why you pass a space to your ViewRequirement definition. From this space the initial values are created in case of shift < 0. The values created will have a shape equal to (BATCH_SIZE, TIME_SIZE, FEATURE_SIZE), where TIME_SIZE equals includes [shift:0] steps.

Hope this helps.

3

Thank you for the detailed reply. I needed this info because I need to mask the dummy observations. I find that the default values used to create dummy values are 0.0, which I can’t use as a criteria since my environment can actually give an observation of all 0’s. I think changing the default filling number in get_dummy_data to NAN would be a solution. Do you know of any other ways that won’t break the interface?

4

Hi @Aceticia,

You use a custom model right?

5

Yes, it’s a custom attention style model.

6

@Aceticia

Could you add your own custom magic number to the beginning of the obs and then discard it from the observation in your custom model?

You could then use the pattern to find the dummy obs.

#in your environment
obs = [8,6,7,5,3,0,9] + real_obs
...
#in forward
out = self.model(input[:,7:])
7

@mannyv ,
Thank you, this looks like a doable approach.

Another possible approach: I noticed that in the SampleBatch object, there is a key called t. What does this field do? Does it tell you the timestep of current batch?

8

@Aceticia ,

to see what t in the SampleBatch is follow the link to the simple_list_collector.py where the SampleBatch gets filled.

1 Like