Memory leak reported after "import ray"

1

I meet a memory leak when using ray to do parallel computing. To narrow down the causes, I remove all the codes from the python script except below:

import ray
print("just a blank script")

there are still memory leak errors:

==8279== 54 bytes in 1 blocks are definitely lost in loss record 1,697 of 10,519
==8279==    at 0x4C30EDB: malloc (vg_replace_malloc.c:309)
==8279==    by 0x4F23592: PyObject_Malloc (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4F5EAE9: PyUnicode_New (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4F675F0: _PyUnicodeWriter_PrepareInternal (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4F3806E: PyUnicode_DecodeUTF8Stateful (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4F7A9F7: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4F7AFFC: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4F876AD: _Py_BuildValue_SizeT (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x40BA46D: PyInit_setproctitle (setproctitle.c:174)
==8279==    by 0x5053281: _PyImport_LoadDynamicModuleWithSpec (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x5053584: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4FAF706: PyCFunction_Call (in /usr/lib64/libpython3.6m.so.1.0)
==8279== 
==8279== 56 bytes in 1 blocks are definitely lost in loss record 3,102 of 10,519
==8279==    at 0x4C30EDB: malloc (vg_replace_malloc.c:309)
==8279==    by 0x4F23592: PyObject_Malloc (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4F3BB5C: _PyObject_GC_Malloc (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4F4242D: _PyObject_GC_NewVar (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4F5645E: PyTuple_New (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4FD85E8: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4FD8B18: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4FD872E: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4FD8B18: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4FD872E: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4FD965C: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4FDCAA0: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279== 
==8279== 64 bytes in 1 blocks are definitely lost in loss record 3,891 of 10,519
==8279==    at 0x4C30EDB: malloc (vg_replace_malloc.c:309)
==8279==    by 0x40141AD: dl_open_worker (in /usr/lib64/ld-2.28.so)
==8279==    by 0x5E79AB6: _dl_catch_exception (in /usr/lib64/libc-2.28.so)
==8279==    by 0x401365D: _dl_open (in /usr/lib64/ld-2.28.so)
==8279==    by 0x55B81B9: dlopen_doit (in /usr/lib64/libdl-2.28.so)
==8279==    by 0x5E79AB6: _dl_catch_exception (in /usr/lib64/libc-2.28.so)
==8279==    by 0x5E79B52: _dl_catch_error (in /usr/lib64/libc-2.28.so)
==8279==    by 0x55B8938: _dlerror_run (in /usr/lib64/libdl-2.28.so)
==8279==    by 0x55B8259: dlopen@@GLIBC_2.2.5 (in /usr/lib64/libdl-2.28.so)
==8279==    by 0x502DE19: _PyImport_FindSharedFuncptr (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x5053239: _PyImport_LoadDynamicModuleWithSpec (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x5053584: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279== 
==8279== 144 bytes in 1 blocks are definitely lost in loss record 7,610 of 10,519
==8279==    at 0x4C30EDB: malloc (vg_replace_malloc.c:309)
==8279==    by 0x4F23592: PyObject_Malloc (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4FCF8B2: PyCode_New (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x89FE769: __pyx_pymod_exec__raylet(_object*) (in /root/.local/lib/python3.6/site-packages/ray/_raylet.so)
==8279==    by 0x502C2F2: PyModule_ExecDef (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x502C381: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4FAF706: PyCFunction_Call (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4FF3C41: _PyEval_EvalFrameDefault (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4F3FBE7: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4F795DF: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4FBDA31: ??? (in /usr/lib64/libpython3.6m.so.1.0)
==8279==    by 0x4FEDA69: _PyEval_EvalFrameDefault (in /usr/lib64/libpython3.6m.so.1.0)

I use below command to do memory test:

PYTHONMALLOC=malloc valgrind --leak-check=full -v --track-origins=yes \
    --log-file=$SCRIPT_DIR"/algrind_pytest.log" \
    --show-possibly-lost=no \
    python test.py

My environment:
python3.6.8
valgrind 3.15.0
ray1.12.1/2.3.1

Anyone can help?

I want to know the reason of the memory leak. If it is not a problem, how to remove the error reports.

2

I tried heapy to check the memory usage:
No “import ray”: 4721167 bytes
Add “import ray”: 41986865 bytes
Add “import ray”, then do “del ray” and “gc.collect()”: still 41984385 bytes
Looks much memory is consumed after “import ray”, and delete ray module can not release it.

I’m not sure if my operation is right, anyone can help?

3

@sunnaichuan did you try bring up the dashboard and which task may be leaking memory.

import ray seems a bit odd to have leaked memory.

cc: @rickyyx @Clarence_Ng any ideas?

4

@sunnaichuan you might want to try un-importing the module

5

Hi, Jules, my python script only contains “import ray”. After I remove import syntax，there is no memory leaks.
The answer from chatgpt is:

Valgrind is a memory debugging tool for C/C++ programs, it does not fully support the internal implementation of the Python interpreter.

But I use pybind11 in my project, there always combined C++ and python codes. Everything goes well before I introduce the ray module for parallel computing.

6

Hi, Clarence. I tried “del ray”/" sys.modules.pop(‘ray’)", but no luck. There are alway memory leaks.