Dose HuggingFaceTrainer Support FSDP CPUOFFLOAD?

Hi Guys:

I wonder if HugggingfaceTrainer from ray is support fsdp cpuoffload, and how much gpu memory it can save ? I have tried but don’t find much difference and i don’t now if i was enable this feature

Yes, FSDP is supported. You can enable it by using the fsdp argument in transformers.TrainingArguments - Ray AIR API — Ray 2.1.0