Using Ray Tune with Distributed Training Causes Object Spilling

Hey team, we have a TensorflowTrainer for distributed training that we plug into tune.Tuner and the TensorflowTrainer uses datasets={"train": train_dataset} where train_dataset is a Ray Dataset. When I run concurrent trials with Tune, this results in object spilling. We’ve noticed less object spilling when the model is smaller, which makes it seem like perhaps TF objects are being spilled