Hey team, we have a TensorflowTrainer
for distributed training that we plug into tune.Tuner
and the TensorflowTrainer uses datasets={"train": train_dataset}
where train_dataset is a Ray Dataset. When I run concurrent trials with Tune, this results in object spilling. We’ve noticed less object spilling when the model is smaller, which makes it seem like perhaps TF objects are being spilled