Ray Train on EKS unable to use Pod Identity to access Storage

1

I am running KubeRay on AWS EKS and using Pod Identity to assign an IAM role to Ray pods.

Ray Train fails to use the IAM role to gain access to S3 bucket through pyarrow.fs.S3FileSystem. Here is an excerpt of the traceback:

    result = trainer.fit()
  File "/usr/local/lib/python3.10/site-packages/ray/train/base_trainer.py", line 589, in fit
    storage = StorageContext(
  File "/usr/local/lib/python3.10/site-packages/ray/train/_internal/storage.py", line 461, in __init__
    self._create_validation_file()
  File "/usr/local/lib/python3.10/site-packages/ray/train/_internal/storage.py", line 489, in _create_validation_file
    self.storage_filesystem.create_dir(self.experiment_fs_path)
  File "pyarrow/_fs.pyx", line 603, in pyarrow._fs.FileSystem.create_dir
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
OSError: When testing for existence of bucket 'earthdaily-epdev-hawkeye': AWS Error ACCESS_DENIED during HeadBucket operation: No response body.

Note that running the following successfully returns the role:

ray job submit -- aws sts get-caller-identity

{
    "UserId": "AROAXILHR6EDJ4ULEWCOH:eks-jakob1-raycluster-5cdabcd0-7d0b-4103-8a7f-ecb44eeec74d",
    "Account": "498970259718",
    "Arn": "arn:aws:sts::498970259718:assumed-role/jakob-test-ec2-role/eks-jakob1-raycluster-5cdabcd0-7d0b-4103-8a7f-ecb44eeec74d"
}

And running this script using boto3 also succeeds on the cluster:

import boto3
c = boto3.client('s3')
print(c.list_objects_v2(Bucket="earthdaily-epdev-hawkeye"))

How can I make pyarrow.fs.S3FileSystem use the IAM role?