I have a simple Java test, which only consists the “Ray.init()” call. When I run this test, I got the following GLIBC error:
java.lang.UnsatisfiedLinkError: Unable to load library '/tmp/ray/1682382173023/libcore_worker_library_java.so':
/lib64/libc.so.6: version `GLIBC_2.25' not found (required by /tmp/ray/1682382173023/libcore_worker_library_java.so)
/lib64/libc.so.6: version `GLIBC_2.25' not found (required by /tmp/ray/1682382173023/libcore_worker_library_java.so)
Native library (tmp/ray/1682382173023/libcore_worker_library_java.so) not found in resource path (/home/ysu/.gradle/caches/6.9.2/workerMain/gradle-worker.jar:......)
at com.sun.jna.NativeLibrary.loadLibrary(NativeLibrary.java:301)
at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:461)
at com.sun.jna.NativeLibrary.getInstance(NativeLibrary.java:403)
at io.ray.runtime.util.JniUtils.loadLibrary(JniUtils.java:72)
at io.ray.runtime.RayNativeRuntime.start(RayNativeRuntime.java:80)
at io.ray.runtime.DefaultRayRuntimeFactory.createRayRuntime(DefaultRayRuntimeFactory.java:35)
at io.ray.api.Ray.init(Ray.java:32)
at io.ray.api.Ray.init(Ray.java:19)
at com.linkedin.rayoffspringexperiment.factories.RayActorTest.testRayActor(RayActorTest.java:18)
Apparently, Ray tries to load GLIBC 2.25, which is not available on my host. As upgrading GLIBC could be disastrous to the OS, I am trying to find a workaround. Since jemalloc could be used as a replacement for GLIBC, I am trying to make Ray pick up jemalloc when loading the library.
I found that simply setting LD_PRELOAD doesn’t do the trick. So I am wondering if anybody could advise how to make Ray pick up jemalloc, instead of GLIBC? Or in general, what is the correct way to pass an env var to Ray in Java?
Update: it is found that the GLIBC issue is caused by the fact that there is a static libcore_worker_library_java.so file in the ray-runtime jar. This so file explicitly requires a set of libs, including GLIBC 2.25, which are not available on our hosts. To solve this issue, we re-built Ray on our own host so that the new libcore_worker_library_java.so is compatible with our environment.
However, immediately after the GLIBC error is resolved, we started to see a new error thrown 2 lines after (i.e. the GLIBC error is thrown at line 72 of RayNativeRuntime.java, and this NoSuchFieldError is thrown at line 74):
java.lang.NoSuchFieldError: isAsync
at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2442)
at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2498)
at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694)
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627)
at java.base/java.lang.Runtime.load0(Runtime.java:768)
at java.base/java.lang.System.load(System.java:1837)
at io.ray.runtime.util.JniUtils.loadLibrary(JniUtils.java:74)
at io.ray.runtime.RayNativeRuntime.start(RayNativeRuntime.java:80)
at io.ray.runtime.DefaultRayRuntimeFactory.createRayRuntime(DefaultRayRuntimeFactory.java:37)
at io.ray.api.Ray.init(Ray.java:32)
at io.ray.api.Ray.init(Ray.java:19)
As we keep seeing such incompatible errors, wondering if we are missing anything fundamentally with setting up Ray Java? Any advice would be appreciated. Thanks.
Update again: the NoSuchFieldError is gone by rebuilding ray-api-2.4.0.jar and ray-runtime-2.4.0.jar on our host and then making our application depend on these internally built libs. This temporarily resolved the issue, but wondering if the Ray team has a plan to add a centos 7 build?
@GuyangSong Yes, the libcore_worker_library_java.so included in the released ray-runtime jar is not compatible with CentOS 7.9. Appreciate it if you could take a look. Thanks.
@ysu let me answer these questions for you
Q1: /lib64/libc.so.6: version `GLIBC_2.25’ not found (required by /tmp/ray/1682382173023/libcore_worker_library_java.so)
A1: Maybe you take a look for version of GLIBC, compile ‘‘libcore_worker_library_java.so’’ without specifying GLIBC version. I guess it’s caused by the compilation environment where GLIBC is 2.25, and you can take a look for version of GLIBC by “ldd --version”.
Building “libcore_worker_library_java.so” without specifying the version of GLiBC, I guess that the GLIBC version of the machine used to compile Ray is 2.25. Maybe you can take a look at version of GLIBC by command “ldd --version”. As you see, you can rebuild ray to solve the question.
@XiaodongLv Thanks a lot for this info. So just to confirm: if I update “ray-api” and “ray-runtime” to any released version >= 2.1.0, the libcore_worker_library_java.so would be compatible with our machine, right?
@XiaodongLv I tried to use the publicly released ray-api and ray-runtime 2.4.0, but unfortunately, the GLIBC issue occurs again. So the libcore_worker_library_java.so in the new version of ray-runtime is still not compatible with CentOS 3.9. Wondering if you could help take a look at the compiling way of that so file? Thanks.
I agree with @GuyangSong , because the glibc version of the machine where ray is released is 2.25, so you build ray on your computer will be successful.
@GuyangSong@XiaodongLv Thank you both for your response. So when can we expect the new version based on the compiling in the manylinux2014 image to be released?