Strange behavior of Apex framework

How severe does this issue affect your experience of using Ray?

  • Low: It annoys or frustrates me for a moment.


In my project, I’m training policy on the real industrial robot. Using algorithms like SAC or DDPG violates the real-time execution (even with tuned rollout_fragment_length and the training_intensity). To achieve execution with proper frequency (10Hz) I integrated SAC with the Apex framework. Everything works great, however, I’ve noticed that from time to time robot stops moving while the gradients are computed and the opposite, the robot is moving while the gradients are not computed.

In this setup, I’m using only 1 actor that corresponds to the 1 robot. My question is, is there possible to modify apex in a way to remove those above-mentioned stops or to modify SAC for asynchronous gradients computation and data collection?

I’m using ray 2.1.0.