# Bazel/Protobuf build errors: libstdc++ with non-system gcc **URL:** \ **Category:** Ray Core\ **Created:** [August 23, 2021, 11:09pm UTC](https://discuss.ray.io/t/bazel-protobuf-build-errors-libstdc-with-non-system-gcc/3329 "2021-08-23T23:09:41Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![eriksf](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eriksf/32/1440_2.png) [@eriksf](https://discuss.ray.io/u/eriksf)\ **Post date:** [August 23, 2021, 11:09pm UTC](https://discuss.ray.io/t/bazel-protobuf-build-errors-libstdc-with-non-system-gcc/3329/1 "2021-08-23T23:09:41Z")
System: IBM AC922 Power9 OS: RHEL 7.6 (ppc64le) Ray version: ray-1.5.2 I’m trying to build ray using the build.sh command and it’s failing. Because the system gcc is fairly old, I’m using a newer version (7.3.0) in a non-standard location (LD\_LIBRARY\_PATH has been updated). ```auto ERROR: /scratch/03762/eriksf/src/ray/BUILD.bazel:129:16: Action src/ray/protobuf/core_worker.grpc.pb.h failed (Exit 1): process-wrapper failed: error executing command (cd /scratch/03762/eriksf/cache/bazel/_bazel_eriksf/51ccce92adc269c2bf3d820e7c87c561/sandbox/processwrapper-sandbox/1657/execroot/com_github_ray_project_ray && \ exec env - \ TMPDIR=/tmp \ /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-PLUGIN=bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/grpc_cpp_plugin' '--PLUGIN_out=bazel-out/ppc-opt/bin' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=bazel-out/ppc-opt/bin' src/ray/protobuf/core_worker.proto) process-wrapper failed: error executing command (cd /scratch/03762/eriksf/cache/bazel/_bazel_eriksf/51ccce92adc269c2bf3d820e7c87c561/sandbox/processwrapper-sandbox/1657/execroot/com_github_ray_project_ray && \ exec env - \ TMPDIR=/tmp \ /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-PLUGIN=bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/grpc_cpp_plugin' '--PLUGIN_out=bazel-out/ppc-opt/bin' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=bazel-out/ppc-opt/bin' src/ray/protobuf/core_worker.proto) /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper) /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper) ERROR: /scratch/03762/eriksf/src/ray/BUILD.bazel:129:16 Action src/ray/protobuf/core_worker.grpc.pb.h failed (Exit 1): process-wrapper failed: error executing command (cd /scratch/03762/eriksf/cache/bazel/_bazel_eriksf/51ccce92adc269c2bf3d820e7c87c561/sandbox/processwrapper-sandbox/1657/execroot/com_github_ray_project_ray && \ exec env - \ TMPDIR=/tmp \ /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-PLUGIN=bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/grpc_cpp_plugin' '--PLUGIN_out=bazel-out/ppc-opt/bin' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=bazel-out/ppc-opt/bin' src/ray/protobuf/core_worker.proto) process-wrapper failed: error executing command (cd /scratch/03762/eriksf/cache/bazel/_bazel_eriksf/51ccce92adc269c2bf3d820e7c87c561/sandbox/processwrapper-sandbox/1657/execroot/com_github_ray_project_ray && \ exec env - \ TMPDIR=/tmp \ /home/03762/eriksf/.cache/bazel/_bazel_eriksf/install/f47ce7bb28e09eb9d195fb6d0d38084a/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-PLUGIN=bazel-out/host/bin/external/com_github_grpc_grpc/src/compiler/grpc_cpp_plugin' '--PLUGIN_out=bazel-out/ppc-opt/bin' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=.' '--proto_path=bazel-out/ppc-opt/bin' src/ray/protobuf/core_worker.proto) INFO: Elapsed time: 84.379s, Critical Path: 55.11s INFO: 2222 processes: 1253 internal, 969 processwrapper-sandbox. FAILED: Build did NOT complete successfully Traceback (most recent call last): File "", line 1, in File "/scratch/03762/eriksf/conda_local/envs/ray/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/scratch/03762/eriksf/conda_local/envs/ray/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/scratch/03762/eriksf/conda_local/envs/ray/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "./python/setup.py", line 430, in api_main(*sys.argv) File "./python/setup.py", line 402, in api_main result = build(**kwargs) File "./python/setup.py", line 308, in build env=bazel_env) File "./python/setup.py", line 190, in bazel_invoke result = invoker([cmd] + cmdline, *args, **kwargs) File "/scratch/03762/eriksf/conda_local/envs/ray/lib/python3.7/subprocess.py", line 363, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['bazel', 'build', '--verbose_failures', '--sandbox_debug', '--', '//:ray_pkg', '//cpp:ray_cpp_pkg']' returned non-zero exit status 1. ``` It seems that the environment is being wiped out before calling ‘protoc’ so it’s only finding the old version of libstdc++ in /lib64 because it’s not seeing LD\_LIBRARY\_PATH. Is there any way around this issue? I can’t imagine I’m the only one to build with a non-system version of gcc. ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [August 25, 2021, 5:18am UTC](https://discuss.ray.io/t/bazel-protobuf-build-errors-libstdc-with-non-system-gcc/3329/2 "2021-08-25T05:18:13Z")
Hmm I never used build.sh. What about trying bazel build //:ray\_pkg? ---
**Author:** ![eriksf](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eriksf/32/1440_2.png) [@eriksf](https://discuss.ray.io/u/eriksf)\ **Post date:** [August 25, 2021, 3:15pm UTC](https://discuss.ray.io/t/bazel-protobuf-build-errors-libstdc-with-non-system-gcc/3329/3 "2021-08-25T15:15:39Z")
Same error. **build.sh** just ends up calling bazel build //:ray\_pkg //cpp:ray\_cpp\_pkg by default. ---
**Author:** ![sangcho](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/sangcho/32/425_2.png) [@sangcho](https://discuss.ray.io/u/sangcho)\ **Post date:** [August 25, 2021, 11:09pm UTC](https://discuss.ray.io/t/bazel-protobuf-build-errors-libstdc-with-non-system-gcc/3329/4 "2021-08-25T23:09:00Z")
cc @Clark_Zinzow do you have any clue? ---
**Author:** ![Clark\_Zinzow](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/clark_zinzow/32/445_2.png) [@Clark\_Zinzow](https://discuss.ray.io/u/Clark_Zinzow)\ **Post date:** [August 26, 2021, 2:43am UTC](https://discuss.ray.io/t/bazel-protobuf-build-errors-libstdc-with-non-system-gcc/3329/5 "2021-08-26T02:43:05Z")
This is a common and gnarly issue, especially with different RHEL versions (for some reason). I don’t think there’s much more I can do other than link a bunch of Bazel issues with possible workarounds: - [How to change the default compiler for bazel when building tensorflow? · Issue #4077 · bazelbuild/bazel · GitHub](https://github.com/bazelbuild/bazel/issues/4077) - [bazel ignores LD\_LIBRARY\_PATH when building tensorflow · Issue #2515 · bazelbuild/bazel · GitHub](https://github.com/bazelbuild/bazel/issues/2515) - [Bazel keeps using wrong GCC · Issue #8548 · bazelbuild/bazel · GitHub](https://github.com/bazelbuild/bazel/issues/8548) - [Make bazel use a non system GCC on RHEL 6 · Issue #4510 · bazelbuild/bazel · GitHub](https://github.com/bazelbuild/bazel/issues/4510) - [Building Bazel fails as included protoc uses wrong libstd++ · Issue #11852 · bazelbuild/bazel · GitHub](https://github.com/bazelbuild/bazel/issues/11852) ---
**Author:** ![Clark\_Zinzow](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/clark_zinzow/32/445_2.png) [@Clark\_Zinzow](https://discuss.ray.io/u/Clark_Zinzow)\ **Post date:** [August 26, 2021, 2:47am UTC](https://discuss.ray.io/t/bazel-protobuf-build-errors-libstdc-with-non-system-gcc/3329/6 "2021-08-26T02:47:27Z")
E.g. you might need to run `bazel clean --expunge` and set the `CC` environment variable to the new non-system GCC that you’re using, and _maybe_ it will work at that point. Bazel’s support for non-system GCC has historically been… rough. ---
**Author:** ![eriksf](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.ray.io/eriksf/32/1440_2.png) [@eriksf](https://discuss.ray.io/u/eriksf)\ **Post date:** [August 30, 2021, 6:33pm UTC](https://discuss.ray.io/t/bazel-protobuf-build-errors-libstdc-with-non-system-gcc/3329/7 "2021-08-30T18:33:43Z")
I think my issue is strictly related to protobuf. For the most part, things are building properly with a non-system gcc. In fact, the process-wrapper above is properly linked against the correct version of libstdc++. It’s the “exec env -” that is causing the problem. The link most relevant to my problem is - [bazel ignores LD\_LIBRARY\_PATH when building tensorflow · Issue #2515 · bazelbuild/bazel · GitHub](https://github.com/bazelbuild/bazel/issues/2515) Tensorflow seems to have its own version of protobuf and it was fixed there. From that same ticket it seems like it was also fixed upstream: - [Adding default shell env to allow non-default compilers #2933](https://github.com/protocolbuffers/protobuf/pull/2933) How is ray pulling in protobuf? The patch in protobuf is a couple of years old.