How severe does this issue affect your experience of using Ray?
- Medium: It contributes to significant difficulty to complete my task, but I can work around it.
Hi, I have an env with a picture observation of size (9,500,500) (and a numpy array with size (3,) which I neglect due to low cap).
lets say they are all float then size of picture would be : 9500500*4 byte = 9 Mb
with calculating overhead = 30%
every step is 11.7 Mb,
so with this calculations I can take batch size of 1000 when I have 12Gb ram but I can’t.
I go out of ram when I have batch size of 300 or more.
the point is ram is okey during inference/rollout phase and I reach timestamp = 1000 easily during rollout but when I reach batch time, I see a spike in ram usage. so technically everything is good and as calculated before reaching batch time. my question is what is happening in that time that consumes this huge amount of ram?
for this reasons I use batch_size = 256
I’m using num_worker = 0 to insure that the ram issue has nothing with model (using gpu for forward method).
my model is parallel CNN(for pictures) and LSTM(for numpy array) if it helps.