Hi Guys:
I wonder if HugggingfaceTrainer from ray is support fsdp cpuoffload, and how much gpu memory it can save ? I have tried but don’t find much difference and i don’t now if i was enable this feature
Yes, FSDP is supported. You can enable it by using the fsdp argument in transformers.TrainingArguments - Ray AIR API — Ray 2.1.0