Hello, I am working to a a project and I have implemented the following python notebook in this github repo: [image] GitHub - mg64ve/TrajectoryTracking Contribute to mg64ve/TrajectoryTracking development by creating an account on GitHub. The aim of this envi…

@mg64ve You can a avoid the issue even in the current version by doing 2 things. Set the config[“simple_optimizer”]=True Set sgd_minibatch_size > max_seq_len

Hi, That is a lot of code for an agent that is only supposed to follow a signal! Also, a couple of things here confuse me. Such as this equation: [Bildschirmfoto 2021-11-11 um 15.36.40] I suppose the “agent value” is the value function? Could you explain? Also, in your code I find the followin…

Thanks @arturn for your reply. The agent value is an helper function and it used to calculate the reward. The reward is higher if the agent value is close to the trajectory value. That been said, what I expect from RLLIB is that find out the actions in order to maximize the rewards. In my opinion…

@mg64ve you might want to try increasing the max _seq_len. Which version of rllib are you using? There is a bug with rnn_sequencing in the current release. [Bug] [rllib] RNN sequencing is incorrect opened 12:43PM - 02 Nov 21 UTC (UTC) …

Hi @mg64ve , Your observations are defined by the following function: def get_samples(self): c = self.data[self.position] c1 = self.data[self.position-1] c2 = self.data[self.position-2] c3 = self.data[self.position-3] return np.array(((c-c1)/c1,(c-c2)/c2,(c-c…

@mannyv I am using version 1.8.0. With max_seq_len=50 the situation does not improve, with max_seq_len=200 I am getting the following error: 2021-11-11 22:40:32,046 ERROR trial_runner.py:924 -- Trial PPO_TrajectoryTrackingEnv_f8d8b_00000: Error processing event. Traceback (most recent call last): …

Exactly @arturn ! In the following example: [image] I am considering 2 agents that know the future values of the trajectory: [image] The first makes much better rewards than the second. Both they know when the trajectory is going to change. The second uses only 2 actyions [0,2] but the first …

@mg64ve , Did you see the issue I linked? I think you are experiencing it.

Thanks @mannyv . I had a quick look but I need to more time to deep dive into it. Is this only for 1.7.0 or for all versions?

Hi @mannyv , I have tested in both 1.8.0 and 2.0.0.dev0 with the following configuration: [image] and the hint you gave me does not solve the problem. I have same a similar behaviour: [image] [image] It looks it now uses action 1 (stay) but not really when it is better to use. I am using 10…

Just trained 250 epochs and it seems better: [image]

TrajectoryTracking with RLLIB

RLlib

mannyv November 11, 2021, 10:15pm 10

@mg64ve

You can a avoid the issue even in the current version by doing 2 things.

Set the config[“simple_optimizer”]=True
Set sgd_minibatch_size > max_seq_len

Use Policy_Trainer with TensorBoard

Topic		Replies	Views
Unable to replicate original PPO performance RLlib	0	216	May 10, 2024
There was an error changing the trajecy_tory_view_api into continuous action space RLlib	7	624	February 22, 2022
Ray RLLIB PPO does not solve very simple problem Configure Algorithm, Training, Evaluation, Scaling	2	510	November 8, 2023
My RLlib implementation seems to compute random actions RLlib	4	955	February 15, 2022
[Rllib] compute_single_action() with an LSTM-PPO trainer fails RLlib	1	1011	February 3, 2023

TrajectoryTracking with RLLIB

Related topics