@rentom from the error logs it looks like the problem may be in pyarrow. Can you try s3_filesystem.get_file_info(s3_path) to see if this problem comes up with just pyarrow (without ray data)?
You may be able to circumvent this by using fsspec/s3fs (pip install s3fs) and a pyarrow.fs.PyFileSystem:
from pyarrow.fs import PyFileSystem, FSSpecHandler
pa_fs = PyFileSystem(FSSpecHandler(fs))
(fs would be your s3fs.S3FileSystem instance).
Since s3fs uses boto under the hood, it should work out of the box.