How severe does this issue affect your experience of using Ray?
- Medium: It contributes to significant difficulty to complete my task, but I can work around
I am running ray locally for a complex data processing pipeline that involves a lot of reading and writing from SQLite. I know that SQLite is not built for multiprocessing, and get “database is locked” errors once in a while (unpredictably). Before replacing SQLite to something that better supports concurrency, I was wondering whether it should work to use simple python multiprocess lock:
lock = multiprocessing.Lock()
and put with on the blocks accessing SQLite (which take a negligible amount of time, I am not worried about performance degradation).
Or perhaps there are other locking mechanisms for multiprocess ray processes? (depending on the library Ray uses, I guess).