# Error Encountered While Training Generative AI Model in Aviary

**URL:** <https://discuss.ray.io/t/error-encountered-while-training-generative-ai-model-in-aviary/14699>\
**Category:** Ray Train\
**Created:** [May 15, 2024, 11:16am UTC](https://discuss.ray.io/t/error-encountered-while-training-generative-ai-model-in-aviary/14699 "2024-05-15T11:16:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![chitramishra](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/chitramishra/32/6093_2.png) [@chitramishra](https://discuss.ray.io/u/chitramishra)\
**Post date:** [May 15, 2024, 11:16am UTC](https://discuss.ray.io/t/error-encountered-while-training-generative-ai-model-in-aviary/14699/1 "2024-05-15T11:16:51Z")

</div>

Hi everyone,

I’ve been diving into the world of generative AI within the Aviary framework, and I’ve hit a bit of a roadblock.  
I also check this this - [GPT-J-6B Fine-Tuning with Ray Train and DeepSpeed — Ray 2.43.0](https://docs.ray.io/en/latest/train/examples/deepspeed/gptj_deepspeed_fine_tuning.html) and  
During my [generative ai training course](https://www.igmguru.com/machine-learning-ai/generative-ai-training/) sessions with a generative AI model, I keep encountering a persistent error that’s proving to be quite puzzling. It seems that whenever the model reaches a certain stage of training, it abruptly stops and throws an error message that reads, “Error: Unable to converge due to gradient vanishing problem.”

I’ve tried adjusting various hyperparameters, tinkering with the architecture, and even modifying the dataset preprocessing steps, but nothing seems to resolve this issue. I’ve also checked for any anomalies in the dataset, but everything appears to be in order.

Has anyone else encountered a similar error while working with generative AI models in Aviary?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Sam\_Chan](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sam_chan/32/5011_2.png) [@Sam\_Chan](https://discuss.ray.io/u/Sam_Chan)\
**Post date:** [May 16, 2024, 11:44pm UTC](https://discuss.ray.io/t/error-encountered-while-training-generative-ai-model-in-aviary/14699/2 "2024-05-16T23:44:43Z")

</div>

Hard to debug without knowing more; can you share your Ray setup any repro script relevant (information on what model you’re attempting to train plus what data format you are training in are all relevant)?

---

<div class="post-metadata">

**Author:** ![chitramishra](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/chitramishra/32/6093_2.png) [@chitramishra](https://discuss.ray.io/u/chitramishra)\
**Post date:** [May 17, 2024, 4:35am UTC](https://discuss.ray.io/t/error-encountered-while-training-generative-ai-model-in-aviary/14699/3 "2024-05-17T04:35:28Z")

</div>

Thanks for helping @Sam_Chan