# Ray Train on EKS unable to use Pod Identity to access Storage

**URL:** <https://discuss.ray.io/t/ray-train-on-eks-unable-to-use-pod-identity-to-access-storage/21714>\
**Category:** Ray Train\
**Created:** [February 7, 2025, 11:21pm UTC](https://discuss.ray.io/t/ray-train-on-eks-unable-to-use-pod-identity-to-access-storage/21714 "2025-02-07T23:21:59Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jleben](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jleben/32/7574_2.png) [@jleben](https://discuss.ray.io/u/jleben)\
**Post date:** [February 7, 2025, 11:21pm UTC](https://discuss.ray.io/t/ray-train-on-eks-unable-to-use-pod-identity-to-access-storage/21714/1 "2025-02-07T23:21:59Z")

</div>

I am running KubeRay on AWS EKS and using Pod Identity to assign an IAM role to Ray pods.

Ray Train fails to use the IAM role to gain access to S3 bucket through `pyarrow.fs.S3FileSystem`. Here is an excerpt of the traceback:

```auto
    result = trainer.fit()
  File "/usr/local/lib/python3.10/site-packages/ray/train/base_trainer.py", line 589, in fit
    storage = StorageContext(
  File "/usr/local/lib/python3.10/site-packages/ray/train/_internal/storage.py", line 461, in __init__
    self._create_validation_file()
  File "/usr/local/lib/python3.10/site-packages/ray/train/_internal/storage.py", line 489, in _create_validation_file
    self.storage_filesystem.create_dir(self.experiment_fs_path)
  File "pyarrow/_fs.pyx", line 603, in pyarrow._fs.FileSystem.create_dir
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
OSError: When testing for existence of bucket 'earthdaily-epdev-hawkeye': AWS Error ACCESS_DENIED during HeadBucket operation: No response body.

```

Note that running the following successfully returns the role:

```auto
ray job submit -- aws sts get-caller-identity

{
    "UserId": "AROAXILHR6EDJ4ULEWCOH:eks-jakob1-raycluster-5cdabcd0-7d0b-4103-8a7f-ecb44eeec74d",
    "Account": "498970259718",
    "Arn": "arn:aws:sts::498970259718:assumed-role/jakob-test-ec2-role/eks-jakob1-raycluster-5cdabcd0-7d0b-4103-8a7f-ecb44eeec74d"
}

```

And running this script using boto3 also succeeds on the cluster:

```auto
import boto3
c = boto3.client('s3')
print(c.list_objects_v2(Bucket="earthdaily-epdev-hawkeye"))

```

How can I make `pyarrow.fs.S3FileSystem` use the IAM role?

---

<div class="post-metadata">

**Author:** ![eschneeweiss](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eschneeweiss/32/5706_2.png) [@eschneeweiss](https://discuss.ray.io/u/eschneeweiss)\
**Post date:** [March 3, 2025, 8:35pm UTC](https://discuss.ray.io/t/ray-train-on-eks-unable-to-use-pod-identity-to-access-storage/21714/2 "2025-03-03T20:35:44Z")

</div>

Any luck on this @jleben? I am suffering from exactly the same issue after I updated ray to the latest version.

---

<div class="post-metadata">

**Author:** ![jleben](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/jleben/32/7574_2.png) [@jleben](https://discuss.ray.io/u/jleben)\
**Post date:** [March 4, 2025, 4:34am UTC](https://discuss.ray.io/t/ray-train-on-eks-unable-to-use-pod-identity-to-access-storage/21714/3 "2025-03-04T04:34:15Z")

</div>

I have filed this GitHub issue that goes into much more detail in its description: [[Train] Unable to gain long-term access to S3 storage for training state/checkpoints when running on AWS EKS · Issue #50823 · ray-project/ray · GitHub](https://github.com/ray-project/ray/issues/50823)

It also contains a workaround that I am currently using, which is pretty convoluted and I really wish I didn’t need to use.

---

<div class="post-metadata">

**Author:** ![eschneeweiss](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eschneeweiss/32/5706_2.png) [@eschneeweiss](https://discuss.ray.io/u/eschneeweiss)\
**Post date:** [March 4, 2025, 9:31pm UTC](https://discuss.ray.io/t/ray-train-on-eks-unable-to-use-pod-identity-to-access-storage/21714/4 "2025-03-04T21:31:12Z")

</div>

Thanks for documenting this issue so well! I was really struggling to figure out what happened after I updated my AWS EKS cluster.

I just checked out your workaround and I appreciate it, in fact I will probably use it, but damn is it ugly.

Thanks again for you help!