I’m trying to start a Workflow from a Python thread (using import threading), but it appears Ray is not set up for this. Ray calls asyncio which is unable to find the thread it created.
Ray is trying to create new threads from a thread made by PyClowder. Since it’s creating them from a ‘non-main thread,’ “automatic assignment of a new event loop does not occur. This explains my problem exactly: python 3.x - Python3 asyncio There is no current event loop in thread when spawn a new thread - Stack Overflow
My question: is it possible to edit how Ray creates async threads such that it manually sets the event loop with asyncio.set_event_loop(asyncio.new_event_loop()) ?
Thanks for any info -Kastan
Stack trace:
2022-01-31 10:42:23,320 [Thread-5 ] DEBUG : pyclowder.connectors - ['kastanvday@gmail.com']
2022-01-31 10:42:23,321 [Thread-5 ] INFO : pyclowder.connectors - [61a5853c87b65e03e7925706] : StatusMessage.start: Started processing.
2022-01-31 10:42:23,323 [Thread-5 ] DEBUG : pyclowder.extractors - default check message : {'notifies': ['kastanvday@gmail.com'], 'source': {'id': {'resourceType': "'file", 'id': '61a5853c87b65e03e7925706'}, 'mimeType': 'text/plain', 'extra': {'filename': '10chars.txt'}}, 'jobid': '61f82d8e193c9580af4aa3bf', 'msgid': '61f82d8e193c9580af4aa3c0', 'flags': '', 'intermediateId': '61a5853c87b65e03e7925706', 'host': 'http://host.docker.internal:9000', 'datasetId': '61a41c945e5855829e5923d8', 'id': '61a5853c87b65e03e7925706', 'fileSize': '11', 'filename': '10chars.txt', 'target': '{}', 'secretKey': '70c420d3-918a-45d9-a4e3-987f726c6694', 'activity': 'submitted', 'routing_key': 'extractors.ncsa.rayWorkflow', 'parameters': {}, 'action': 'manual-submission', 'retry_count': 1}
2022-01-31 10:42:23,370 [Thread-5 ] INFO : pyclowder.connectors - [61a5853c87b65e03e7925706] : StatusMessage.processing: Downloading file.
2022-01-31 10:42:23,461 [Thread-5 ] INFO : pyclowder.connectors - [61a5853c87b65e03e7925706] : StatusMessage.processing: Loading contents of file...
10:42:23.462680
2022-01-31 10:42:23,479 INFO execution.py:49 -- Workflow job created. [id="rayWorkflow-10:42:23.462680", storage_url="file:///tmp/data"]. Type: FUNCTION.
2022-01-31 10:42:23,481 [Thread-5 ] ERROR : pyclowder.connectors - [61a5853c87b65e03e7925706] There is no current event loop in thread 'Thread-5'.
Traceback (most recent call last):
File "/home/ray/anaconda3/lib/python3.7/site-packages/pyclowder/connectors.py", line 426, in _process_message
self.process_message(self, source_host, secret_key, resource, body)
File "rayWorkflow.py", line 68, in process_message
output = self.workflow.run(f"rayWorkflow-{current_time}")
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/common.py", line 363, in run
return ray.get(self.run_async(workflow_id, metadata))
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/common.py", line 405, in run_async
return run(self, workflow_id, metadata)
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/execution.py", line 56, in run
ws.save_workflow_user_metadata(metadata)
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/workflow_storage.py", line 416, in save_workflow_user_metadata
self._put(self._key_workflow_user_metadata(), metadata, True))
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/workflow_storage.py", line 55, in asyncio_run
loop = asyncio.get_event_loop()
File "/home/ray/anaconda3/lib/python3.7/asyncio/events.py", line 644, in get_event_loop
% threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'Thread-5'.
2022-01-31 10:42:23,483 [Thread-5 ] INFO : pyclowder.connectors - [61a5853c87b65e03e7925706] : StatusMessage.retry: (#2) There is no current event loop in thread 'Thread-5'.