How severe does this issue affect your experience of using Ray? High: It blocks me to complete my task. Dear all, I’m trying out Ray and RLlib. I think it is a very promising and powerful library. I like it very much! I created a custom environment, a custom model and even a custom exploration …

For anyone ending up here, please checkout other training architectures. SimpleQTrainer is just too simple and does not support this, but others do! [image] Stefan-1313: I cannot find such a parameter, but I look for a parameter like num_train_batches_after_sample_collection. This parameter sim…

Hi @Stefan-1313 , If you print the results of agent.evaluate() do they look similar to the training results? You probably also want to run more than one episode in evaluation.

@Stefan-1313 , Another thing to be aware of that is not immediately obvious is that the iterations are measured in new steps sampled. So if you double the workers then you will collect twice as many samples per iteration but you will have half as many updates per iteration so to train for the same n…

Thank you mannyv! Your comment made me look at the config once again. I’ve read this: Training APIs — Ray 1.13.0 It seems that the default values mentioned there are incorrect. When looking at the source code I found the actual default values are different. So I changed my ‘distributed compute co…

@Lars_Simon_Zehnder Thank you! I would very much like to read more about this. I tried to read it but the link seems to be broken? Perhaps you made a mistake :slightly_smiling_face:, mind checking it?

@Stefan-1313 good catch! I corrected the link

@Lars_Simon_Zehnder thank you for the link. I may have forgotten about it. @Stefan-1313 one difference between that description and your configuration is you have set config_simple["batch_mode"] ="complete_episodes". With that setting the rollout worker will collect an entire episode which means t…

Hi Mannyv, Lars already thanked you for your extremely detailed answer here: Read Tune console output from Simple Q - #5 by mannyv But I would like to thank you again. It is extremely clearifying. Funny how I struggly with almost the same concepts as @Lars_Simon_Zehnder :). Regarding the complete…

Hi @mannyv , Although all the other questions are of interest, the most important question I still have right now is: Is it possible to increase the number of policy updates without increasing steps collected? Because increasing the workers increases sample collection (which is great). But it decr…

My Ray programs stops learning when using distributed compute

RLlib

Lars_Simon_Zehnder July 31, 2022, 8:33am 5

@Stefan-1313 , there is some worthful discussion with @mannyv about this in another thread. This might help you understanding a little better.

Topic		Replies	Views
Num_gpu, rollout_workers, learner_workers, evaluation_workers purpose + resource allocation Configure Algorithm, Training, Evaluation, Scaling	8	2288	August 24, 2023
Lack of convergence when increasing the number of workers RLlib	18	737	February 18, 2025
Ray Tune process hangs - thread synchronization issue? Ray Tune	16	945	February 10, 2023
How to use rllib to conduct distributed training on multiple machines at the same time Configure Algorithm, Training, Evaluation, Scaling	5	892	February 20, 2023
Seeking recommendations for implementing Dual Curriculum Design in RLlib RLlib	13	835	April 11, 2023

My Ray programs stops learning when using distributed compute

Related topics