Building Ray on Linux ARM (AWS Graviton)

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Hi Ray Community!

I’d like to get Ray running on my AWS compute unit which uses ARM architecture. I’ve already happened across a couple of threads (see sources) saying it’s possible with bazel and manually building the package.

I’ve tried using bazel versions 4.2.2, 4.2.3, and the latest 5.3.2 and each one gives out the same issue:

subprocess.CalledProcessError: Command '['bazel', 'build', '--verbose_failures', '--', '//:ray_pkg', '//cpp:ray_cpp_pkg']' returned non-zero exit status 1.

Sources:
Original thread suggesting using bazel to build ray: Build ARM wheels for Ray · Issue #13780 · ray-project/ray · GitHub
Commit indicates bazel 4.2.2 (at least) should work: [CI] Bump Bazel version to 4.2.2 by shrekris-anyscale · Pull Request #24242 · ray-project/ray · GitHub

Thanks in advance for everyone’s time!

What is the log output beyond the failure? That issue seems to indicate that the build should succeed

1 Like

Hi Matti!
Here’s the full stack trace:

   Execution platform: @local_config_platform//:host

    Use --sandbox_debug to see verbose messages from the sandbox
    In file included from src/ray/util/filesystem.cc:15:0:
    bazel-out/aarch64-opt/bin/_virtual_includes/ray_util/ray/util/filesystem.h:17:10: fatal error: filesystem: No such file or directory
     #include <filesystem>
              ^~~~~~~~~~~~
    compilation terminated.
    INFO: Elapsed time: 507.893s, Critical Path: 20.99s
    INFO: 3142 processes: 1115 internal, 2027 linux-sandbox.
    FAILED: Build did NOT complete successfully
    FAILED: Build did NOT complete successfully
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 710, in <module>
        setuptools.setup(
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
        return distutils.core.setup(**attrs)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
        return run_commands(dist)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
        dist.run_commands()
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
        self.run_command(cmd)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
        super().run_command(command)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
        cmd_obj.run()
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/command/develop.py", line 34, in run
        self.install_for_development()
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/command/develop.py", line 114, in install_for_development
        self.run_command('build_ext')
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
        self.distribution.run_command(command)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
        super().run_command(command)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
        cmd_obj.run()
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 698, in run
        return pip_run(self)
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 599, in pip_run
        build(True, BUILD_JAVA, True)
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 547, in build
        return bazel_invoke(
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 314, in bazel_invoke
        result = invoker([cmd] + cmdline, *args, **kwargs)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/subprocess.py", line 373, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['bazel', 'build', '--verbose_failures', '--', '//:ray_pkg', '//cpp:ray_cpp_pkg']' returned non-zero exit status 1.
    error: subprocess-exited-with-error
    
    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> See above for output.
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
    full command: /home/ec2-user/anaconda3/envs/MLEnv/bin/python -c '
    exec(compile('"'"''"'"''"'"'
    # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
    #
    # - It imports setuptools before invoking setup.py, to enable projects that directly
    #   import from `distutils.core` to work with newer packaging standards.
    # - It provides a clear error message when setuptools is not installed.
    # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
    #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
    #     manifest_maker: standard file '"'"'-c'"'"' not found".
    # - It generates a shim setup.py, for handling setup.cfg-only projects.
    import os, sys, tokenize
    
    try:
        import setuptools
    except ImportError as error:
        print(
            "ERROR: Can not execute `setup.py` since setuptools is not available in "
            "the build environment.",
            file=sys.stderr,
        )
        sys.exit(1)
    
    __file__ = %r
    sys.argv[0] = __file__
    
    if os.path.exists(__file__):
        filename = __file__
        with tokenize.open(__file__) as f:
            setup_py_code = f.read()
    else:
        filename = "<auto-generated setuptools caller>"
        setup_py_code = "from setuptools import setup; setup()"
    
    exec(compile(setup_py_code, filename, "exec"))
    '"'"''"'"''"'"' % ('"'"'/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' develop --no-deps
    cwd: /home/ec2-user/ray/ray-ray-1.12.1/python/
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
(MLEnv) [ec2-user@ip python]$ 

*Edit: This happens when running ./build.sh with ray 1.4.1 (basing it off of the timestamps in the threads linked in original post and the most recent release at that time)

Thanks for your time and contributions to the open source community btw! :smiley:

Here’s a more verbose version of that output:

   Execution platform: @local_config_platform//:host

    Use --sandbox_debug to see verbose messages from the sandbox
    In file included from src/ray/util/filesystem.cc:15:0:
    bazel-out/aarch64-opt/bin/_virtual_includes/ray_util/ray/util/filesystem.h:17:10: fatal error: filesystem: No such file or directory
     #include <filesystem>
              ^~~~~~~~~~~~
    compilation terminated.
    INFO: Elapsed time: 507.893s, Critical Path: 20.99s
    INFO: 3142 processes: 1115 internal, 2027 linux-sandbox.
    FAILED: Build did NOT complete successfully
    FAILED: Build did NOT complete successfully
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 710, in <module>
        setuptools.setup(
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
        return distutils.core.setup(**attrs)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
        return run_commands(dist)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
        dist.run_commands()
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
        self.run_command(cmd)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
        super().run_command(command)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
        cmd_obj.run()
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/command/develop.py", line 34, in run
        self.install_for_development()
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/command/develop.py", line 114, in install_for_development
        self.run_command('build_ext')
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
        self.distribution.run_command(command)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
        super().run_command(command)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
        cmd_obj.run()
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 698, in run
        return pip_run(self)
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 599, in pip_run
        build(True, BUILD_JAVA, True)
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 547, in build
        return bazel_invoke(
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 314, in bazel_invoke
        result = invoker([cmd] + cmdline, *args, **kwargs)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/subprocess.py", line 373, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['bazel', 'build', '--verbose_failures', '--', '//:ray_pkg', '//cpp:ray_cpp_pkg']' returned non-zero exit status 1.
    error: subprocess-exited-with-error
    
    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> See above for output.
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
    full command: /home/ec2-user/anaconda3/envs/MLEnv/bin/python -c '
    exec(compile('"'"''"'"''"'"'
    # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
    #
    # - It imports setuptools before invoking setup.py, to enable projects that directly
    #   import from `distutils.core` to work with newer packaging standards.
    # - It provides a clear error message when setuptools is not installed.
    # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
    #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
    #     manifest_maker: standard file '"'"'-c'"'"' not found".
    # - It generates a shim setup.py, for handling setup.cfg-only projects.
    import os, sys, tokenize
    
    try:
        import setuptools
    except ImportError as error:
        print(
            "ERROR: Can not execute `setup.py` since setuptools is not available in "
            "the build environment.",
            file=sys.stderr,
        )
        sys.exit(1)
    
    __file__ = %r
    sys.argv[0] = __file__
    
    if os.path.exists(__file__):
        filename = __file__
        with tokenize.open(__file__) as f:
            setup_py_code = f.read()
    else:
        filename = "<auto-generated setuptools caller>"
        setup_py_code = "from setuptools import setup; setup()"
    
    exec(compile(setup_py_code, filename, "exec"))
    '"'"''"'"''"'"' % ('"'"'/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' develop --no-deps
    cwd: /home/ec2-user/ray/ray-ray-1.12.1/python/
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
(MLEnv) [ec2-user@ip python]$ 


Here’s a more verbose version of that output:

   Execution platform: @local_config_platform//:host

    Use --sandbox_debug to see verbose messages from the sandbox
    In file included from src/ray/util/filesystem.cc:15:0:
    bazel-out/aarch64-opt/bin/_virtual_includes/ray_util/ray/util/filesystem.h:17:10: fatal error: filesystem: No such file or directory
     #include <filesystem>
              ^~~~~~~~~~~~
    compilation terminated.
    INFO: Elapsed time: 507.893s, Critical Path: 20.99s
    INFO: 3142 processes: 1115 internal, 2027 linux-sandbox.
    FAILED: Build did NOT complete successfully
    FAILED: Build did NOT complete successfully
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 710, in <module>
        setuptools.setup(
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
        return distutils.core.setup(**attrs)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
        return run_commands(dist)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
        dist.run_commands()
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
        self.run_command(cmd)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
        super().run_command(command)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
        cmd_obj.run()
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/command/develop.py", line 34, in run
        self.install_for_development()
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/command/develop.py", line 114, in install_for_development
        self.run_command('build_ext')
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
        self.distribution.run_command(command)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
        super().run_command(command)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
        cmd_obj.run()
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 698, in run
        return pip_run(self)
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 599, in pip_run
        build(True, BUILD_JAVA, True)
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 547, in build
        return bazel_invoke(
      File "/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py", line 314, in bazel_invoke
        result = invoker([cmd] + cmdline, *args, **kwargs)
      File "/home/ec2-user/anaconda3/envs/MLEnv/lib/python3.9/subprocess.py", line 373, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['bazel', 'build', '--verbose_failures', '--', '//:ray_pkg', '//cpp:ray_cpp_pkg']' returned non-zero exit status 1.
    error: subprocess-exited-with-error
    
    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> See above for output.
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
    full command: /home/ec2-user/anaconda3/envs/MLEnv/bin/python -c '
    exec(compile('"'"''"'"''"'"'
    # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
    #
    # - It imports setuptools before invoking setup.py, to enable projects that directly
    #   import from `distutils.core` to work with newer packaging standards.
    # - It provides a clear error message when setuptools is not installed.
    # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
    #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
    #     manifest_maker: standard file '"'"'-c'"'"' not found".
    # - It generates a shim setup.py, for handling setup.cfg-only projects.
    import os, sys, tokenize
    
    try:
        import setuptools
    except ImportError as error:
        print(
            "ERROR: Can not execute `setup.py` since setuptools is not available in "
            "the build environment.",
            file=sys.stderr,
        )
        sys.exit(1)
    
    __file__ = %r
    sys.argv[0] = __file__
    
    if os.path.exists(__file__):
        filename = __file__
        with tokenize.open(__file__) as f:
            setup_py_code = f.read()
    else:
        filename = "<auto-generated setuptools caller>"
        setup_py_code = "from setuptools import setup; setup()"
    
    exec(compile(setup_py_code, filename, "exec"))
    '"'"''"'"''"'"' % ('"'"'/home/ec2-user/ray/ray-ray-1.12.1/python/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' develop --no-deps
    cwd: /home/ec2-user/ray/ray-ray-1.12.1/python/
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
(MLEnv) [ec2-user@ip python]$ 

Can you use the aarch64 wheels available for python 3.8+ on PyPI instead of building from source? We have not yet been able to make aarch64 (arm64) conda packages, but it is on the back-burner, see the feedstock PR

I see similar error but on x86 (Amazon Linux). Wondering if there are any alternatives? Thanks