Bazel/Protobuf build errors: libstdc++ with non-system gcc

Ray Core
1

System: IBM AC922 Power9
OS: RHEL 7.6 (ppc64le)
Ray version: ray-1.5.2

I’m trying to build ray using the build.sh command and it’s failing. Because the system gcc is fairly old, I’m using a newer version (7.3.0) in a non-standard location (LD_LIBRARY_PATH has been updated).

ERROR: /scratch/03762/eriksf/src/ray/BUILD.bazel:129:16: Action src/ray/protobuf/core_worker.grpc.pb.h failed (Exit 1): process-wrapper failed: error executing command
  (cd /scratch/03762/eriksf/cache/bazel/_bazel_eriksf/51ccce92adc269c2bf3d820e7c87c561/sandbox/processwrapper-sandbox/1657/execroot/com_github_ray_project_ray && \
  exec env - \
    TMPDIR=/tmp \
  /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-PLUGIN=bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/grpc_cpp_plugin' '--PLUGIN_out=bazel-out/ppc-opt/bin' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=bazel-out/ppc-opt/bin' src/ray/protobuf/core_worker.proto) process-wrapper failed: error executing command
  (cd /scratch/03762/eriksf/cache/bazel/_bazel_eriksf/51ccce92adc269c2bf3d820e7c87c561/sandbox/processwrapper-sandbox/1657/execroot/com_github_ray_project_ray && \
  exec env - \
    TMPDIR=/tmp \
  /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-PLUGIN=bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/grpc_cpp_plugin' '--PLUGIN_out=bazel-out/ppc-opt/bin' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=bazel-out/ppc-opt/bin' src/ray/protobuf/core_worker.proto)
/home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper)
/home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper)
ERROR: /scratch/03762/eriksf/src/ray/BUILD.bazel:129:16 Action src/ray/protobuf/core_worker.grpc.pb.h failed (Exit 1): process-wrapper failed: error executing command
  (cd /scratch/03762/eriksf/cache/bazel/_bazel_eriksf/51ccce92adc269c2bf3d820e7c87c561/sandbox/processwrapper-sandbox/1657/execroot/com_github_ray_project_ray && \
  exec env - \
    TMPDIR=/tmp \
  /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-PLUGIN=bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/grpc_cpp_plugin' '--PLUGIN_out=bazel-out/ppc-opt/bin' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=bazel-out/ppc-opt/bin' src/ray/protobuf/core_worker.proto) process-wrapper failed: error executing command
  (cd /scratch/03762/eriksf/cache/bazel/_bazel_eriksf/51ccce92adc269c2bf3d820e7c87c561/sandbox/processwrapper-sandbox/1657/execroot/com_github_ray_project_ray && \
  exec env - \
    TMPDIR=/tmp \
  /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-PLUGIN=bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/grpc_cpp_plugin' '--PLUGIN_out=bazel-out/ppc-opt/bin' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=bazel-out/ppc-opt/bin' src/ray/protobuf/core_worker.proto)
INFO: Elapsed time: 84.379s, Critical Path: 55.11s
INFO: 2222 processes: 1253 internal, 969 processwrapper-sandbox.
FAILED: Build did NOT complete successfully
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/scratch/03762/eriksf/conda_local/envs/ray/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/scratch/03762/eriksf/conda_local/envs/ray/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/scratch/03762/eriksf/conda_local/envs/ray/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "./python/setup.py", line 430, in <module>
    api_main(*sys.argv)
  File "./python/setup.py", line 402, in api_main
    result = build(**kwargs)
  File "./python/setup.py", line 308, in build
    env=bazel_env)
  File "./python/setup.py", line 190, in bazel_invoke
    result = invoker([cmd] + cmdline, *args, **kwargs)
  File "/scratch/03762/eriksf/conda_local/envs/ray/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bazel', 'build', '--verbose_failures', '--sandbox_debug', '--', '//:ray_pkg', '//cpp:ray_cpp_pkg']' returned non-zero exit status 1.

It seems that the environment is being wiped out before calling ‘protoc’ so it’s only finding the old version of libstdc++ in /lib64 because it’s not seeing LD_LIBRARY_PATH. Is there any way around this issue? I can’t imagine I’m the only one to build with a non-system version of gcc.

2

Hmm I never used build.sh. What about trying bazel build //:ray_pkg?

3

Same error. build.sh just ends up calling

bazel build //:ray_pkg //cpp:ray_cpp_pkg

by default.

4

cc @Clark_Zinzow do you have any clue?

5

This is a common and gnarly issue, especially with different RHEL versions (for some reason). I don’t think there’s much more I can do other than link a bunch of Bazel issues with possible workarounds:

6

E.g. you might need to run bazel clean --expunge and set the CC environment variable to the new non-system GCC that you’re using, and maybe it will work at that point. Bazel’s support for non-system GCC has historically been… rough.

7

I think my issue is strictly related to protobuf. For the most part, things are building properly with a non-system gcc. In fact, the process-wrapper above is properly linked against the correct version of libstdc++. It’s the “exec env -” that is causing the problem. The link most relevant to my problem is

Tensorflow seems to have its own version of protobuf and it was fixed there. From that same ticket it seems like it was also fixed upstream:

How is ray pulling in protobuf? The patch in protobuf is a couple of years old.