Assert priority > 0 with Rainbow + APE-X

RLlib
1

I’m getting the below error when training a model via Rainbow distributed with ape-x

2021-01-03 00:37:15,154	ERROR worker.py:1018 -- Possible unhandled error from worker: ray::LocalReplayBuffer.update_priorities() (pid=49, ip=10.12.125.4)
  File "python/ray/_raylet.pyx", line 484, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 438, in ray._raylet.execute_task.function_executor
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/execution/replay_buffer.py", line 340, in update_priorities
    batch_indexes, new_priorities)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/execution/replay_buffer.py", line 202, in update_priorities
    assert priority > 0
AssertionError

Here is my trainer config

  trainer_configs:
    adam_epsilon = 0.00015
    buffer_size = 150000
    double_q = True
    dueling = True
    evaluation_interval = 1
    evaluation_num_episodes = 1
    final_prioritized_replay_beta = 1.0
    learning_starts = 20000
    log_level = 'ERROR'
    lr = 0.0001
    n_step = 3
    noisy = True
    num_atoms = 51
    num_cpus_for_driver = 4
    num_cpus_per_worker = 4
    num_gpus = 0
    num_workers = 6
    prioritized_replay = True
    prioritized_replay_alpha = 0.5
    prioritized_replay_beta = 0.4
    prioritized_replay_beta_annealing_timesteps = 1250550
    prioritized_replay_eps = 0.0
    rollout_fragment_length = 10
    sigma0 = 0.5
    target_network_update_freq = 0
    timesteps_per_iteration = 83370
    train_batch_size = 128
    v_max = 5.0
    v_min = -5.0
    worker_side_prioritization = True
    exploration_config:
      final_epsilon = 0.0
      initial_epsilon = 0.0
      type = 'EpsilonGreedy'
    optimizer:
      debug = False
      max_weight_sync_delay = 100
      num_replay_buffer_shards = 2

Any idea what setting might be causing this?