# GPT-J6B Sample Code

**URL:** <https://discuss.ray.io/t/gpt-j6b-sample-code/14369>\
**Category:** Uncategorized\
**Created:** [April 10, 2024, 10:02pm UTC](https://discuss.ray.io/t/gpt-j6b-sample-code/14369 "2024-04-10T22:02:41Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jorgen\_Svane](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jorgen_svane/32/2929_2.png) [@Jorgen\_Svane](https://discuss.ray.io/u/Jorgen_Svane)\
**Post date:** [April 10, 2024, 10:02pm UTC](https://discuss.ray.io/t/gpt-j6b-sample-code/14369/1 "2024-04-10T22:02:41Z")

</div>

Hi.

I’ve been trying to follow the samplecode for finetuning [GPT-J-6B](https://docs.ray.io/en/latest/train/examples/deepspeed/gptj_deepspeed_fine_tuning.html)

I’m running on a single node for now with 32 CPUs and two GPUs and have thus reduced various parameters to prevent OOM.

However, it seems to fail because of a ValueError being raised in the train\_func:

```auto
 File "/tmp/ipykernel_49334/1469602992.py", line 122, in train_func
  File "/home/novelty/miniconda3/envs/ray210/lib/python3.10/site-packages/transformers/trainer.py", line 514, in __init__
    raise ValueError("train_dataset does not implement __len__ , max_steps has to be specified")
ValueError: train_dataset does not implement __len__ , max_steps has to be specified

```

My notebook can be found [here](https://github.com/jlsvane/LLM-finetuning/blob/main/GPT-J-6B%20Fine-Tuning%20with%20Ray%20Train%20and%20DeepSpeed.ipynb) with the full error trace.

When I try to run it, both the system RAM and the GPUs VRAM appear to load up and the CPUs/GPUs are clearly running as well but then it fails.

Is there an issue between torchtrainer and HF datasets?

Any solutions?

BR

Jorgen