How severe does this issue affect your experience of using Ray?
High: It blocks me to complete my task.
I’m implementing some requirements locally on window 11 with the help of ray and I’m having very strange problems:
I’ve been editing the code in pycharm and it works fine on my windows 11 laptop, but if I execute my py file from the command line, I can’t achieve the same result as in pycharm, it also prompts that local ray is on and there are no errors, but after a few seconds it just prompts that the process is finished. I use python’s own idle with this same exception.
If I remove the .idea folder from the pycharm project folder, then the above problem also occurs, but if I set a breakpoint for the program and execute it step by step afterwards, it is normal again! It’s so strange!
I tried ubuntu with the same problem! It doesn’t matter if it’s python 3.9 or python 3.10!
Just as a sanity check, are you using a venv? I believe pycharm sets one up be default. If you are, are you running from the venv when using the command line or just using the python3.9 command from the project’s directory?
Thanks for the reply edit! I’ve tried and still haven’t been able to fix it, I’m basically sure it’s thread related now, but I don’t know the deeper cause.
Hmm the error message indicates (cannot pickle thread.lock) that you capture the lock object to the remote task. This happens when you do something like this
a = threading.lock()
@ray.remote
def f():
print(a) # lock object is implicitly captured, and you cannot pickle lock
Are you sure import selenium package inside a remote task doesn’t work? For me this seems to be the only possible cause… Is it possible to reproduce the issue with your script?
import ray
ray.init()
@ray.remote
class test:
from selenium import webdriver
br = webdriver.Chrome()
w = [test.remote() for _ in range(3)]
Thanks for the code, it gave me an accurate idea of the cause, I tried it and when I hit run I could get the first chrome browser to pop up, but after that it reported an error when I was using [test.remote() for _ in range(3)] to generate 3 objects in a row, the error image was as follows：
import ray
ray.init()
@ray.remote
class test:
from selenium import webdriver
def __int__(self):
br = self.webdriver.Chrome()
w = [test.remote() for _ in range(3)]
If I follow the documentation and put the lock object into the method, when I hit run, no more chrome browser will pop up and the following error will be reported (there is no active effort to use the keyboard to stop the program):
import ray
ray.init()
@ray.remote
class test:
def __init__(self):
from selenium import webdriver
br = webdriver.Chrome()
w = [test.remote() for _ in range(3)]
When ray starts an actor, it pickles (serializes) the actor class and deserializes it when it is initialized in other processes.
When Python pickles objects, and if the object contains the module (e.g., Chrome for example), it stores the reference to the module. When deserialization happens, it imports the module to the process.
So if you import webdriver, it is imported again when the actor (class test) is created. My guess is the webdriver module has some global lock, which cannot be pickled.
The workaround is to import it within __init__. In this case, you can defer the module import until the actor is created, so lock object is not included when you pickle the test class.
Alternatively, I feel like this could also work.
import ray
import selenium
ray.init()
@ray.remote
class test:
def __init__(self):
from selenium import webdriver
br = selenium.webdriver.Chrome()
w = [test.remote() for _ in range(3)]
