# Ray Train example with transformers

**URL:** <https://discuss.ray.io/t/ray-train-example-with-transformers/6128>\
**Category:** Ray Train\
**Created:** [May 13, 2022, 6:06am UTC](https://discuss.ray.io/t/ray-train-example-with-transformers/6128 "2022-05-13T06:06:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![77loopin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/77loopin/32/1382_2.png) [@77loopin](https://discuss.ray.io/u/77loopin)\
**Post date:** [May 13, 2022, 6:06am UTC](https://discuss.ray.io/t/ray-train-example-with-transformers/6128/1 "2022-05-13T06:06:51Z")

</div>

**How severe does this issue affect your experience of using Ray?**

- None: Just asking a question out of curiosity
- Low: It annoys or frustrates me for a moment.
- Medium: It contributes to significant difficulty to complete my task, but I can work around it.
- High: It blocks me to complete my task.

Low

I tested Ray Train transformer example.  
([transformers\_example — Ray v1.9.0](https://docs.ray.io/en/releases-1.9.0/train/examples/transformers/transformers_example.html))

And I executed a training with next command by refering [here](https://github.com/ray-project/ray/tree/master/python/ray/train/examples/transformers).

```auto
#!/bin/bash

export TASK_NAME=mrpc

python example.py \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --max_length 128 \
  --per_device_train_batch_size 4 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TASK_NAME/ \
  --address 'ray://[MY_RAY_CLIENT_URL]' \
  --num_workers 8 \
  --use_gpu

```

It works well but it doesn’t use all gpus. It use only one gpu.  
How can I do?

 ![image](https://us1.discourse-cdn.com/flex020/uploads/ray/original/2X/2/2438bd28915b5f991f777c0aed49d3d5417fddf4.png)

---

<div class="post-metadata">

**Author:** ![matthewdeng](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/matthewdeng/32/1446_2.png) [@matthewdeng](https://discuss.ray.io/u/matthewdeng)\
**Post date:** [May 16, 2022, 3:32am UTC](https://discuss.ray.io/t/ray-train-example-with-transformers/6128/2 "2022-05-16T03:32:42Z")

</div>

Hey @77loopin, thanks for posting this! I took a look into this and it seems like there’s indeed a bug in the script. The accelerator is reading from an environment variable that isn’t being set!

Adding the following line to the start of the training function fixes it:

```auto
os.environ["LOCAL_RANK"] = str(ray.train.local_rank())

```

I made a fix for the example [here](https://github.com/ray-project/ray/pull/24832).

---

<div class="post-metadata">

**Author:** ![77loopin](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/77loopin/32/1382_2.png) [@77loopin](https://discuss.ray.io/u/77loopin)\
**Post date:** [May 16, 2022, 5:45am UTC](https://discuss.ray.io/t/ray-train-example-with-transformers/6128/3 "2022-05-16T05:45:28Z")

</div>

Thanks @matthewdeng  
It was resolved!!