How severe does this issue affect your experience of using Ray?
- Medium: It contributes to significant difficulty to complete my task, but I can work around it.
Hello, I am trying to train a Reinforcement Learning policy using PPO on ray RLlib version 2.34.0.
I want to use a mean standard deviation filter for observation normalization. I read that now the connector component should be used for this. I would like to know, after training my model, are the weights of the filter stored directly in the neural network? Or are they stored somewhere else?
This is the PPO configuration dictionary I am using:
ppo_config = {
"env": "quadruped_gym_env",
"env_config": env_configs,
"lr": 1e-4,
"gamma": 0.99,
"num_envs_per_worker": 1,
"rollout_fragment_length": 'auto',
"train_batch_size": n_steps,
"sgd_minibatch_size": 128,
"num_sgd_iter": 10,
"clip_param": 0.2,
"vf_clip_param": 1.0,
"entropy_coeff": 0.0,
"lambda": 0.95,
"num_workers": NUM_ENVS - 1,
"num_gpus": 1,
"framework": "torch",
"normalize_actions": True,
"batch_mode": "truncate_episodes",
"num_cpus_per_worker": 1,
"model": {
"fcnet_hiddens": [256, 256], # equivalent to pi and vf layers
"fcnet_activation": "relu"
},
"device": "gpu",
"sample_timeout_s": 120,
"observation": {
"connectors": [
{"type": "MeanStdFilterConnector", "params": {}}
]
}
}