Hello!
I tried to setup Ape-X with SpaceInvaders Environment, following the tuned example.
However i cannot see any improvement in the learning curve.
(ignore label in screenshot, interval is 100.000 steps)
The reported reward is about 200-300, not improving over 10 mil timesteps.
Im running on ryzen 5800X with RTX 3060, cpu load is about 85% but gpu load is only 14%.
Only change in config is setting replay size to 250k, because only 16gb ram available.
Would be thankfull for any advice.
Here is my config:
apex:
env: SpaceInvadersNoFrameskip-v4
run: APEX
stop:
timesteps_total: 10000000
config:
# Works for both torch and tf.
framework: tf
double_q: false
dueling: false
num_atoms: 1
noisy: false
n_step: 3
lr: .0001
adam_epsilon: .00015
hiddens: [512]
buffer_size: 250000
compress_observations: true # decrease size of observations
exploration_config:
final_epsilon: 0.01
epsilon_timesteps: 200000
prioritized_replay_alpha: 0.5
final_prioritized_replay_beta: 1.0
prioritized_replay_beta_annealing_timesteps: 2000000
num_gpus: 1
# APEX
num_workers: 8
num_envs_per_worker: 8
rollout_fragment_length: 20
train_batch_size: 512
target_network_update_freq: 50000
timesteps_per_iteration: 25000