Running Workflows from Python thread (RuntimeError: There is no current event loop in thread)

I’m trying to start a Workflow from a Python thread (using import threading), but it appears Ray is not set up for this. Ray calls asyncio which is unable to find the thread it created.

Ray is trying to create new threads from a thread made by PyClowder. Since it’s creating them from a ‘non-main thread,’ “automatic assignment of a new event loop does not occur. This explains my problem exactly: python 3.x - Python3 asyncio There is no current event loop in thread when spawn a new thread - Stack Overflow

My question: is it possible to edit how Ray creates async threads such that it manually sets the event loop with asyncio.set_event_loop(asyncio.new_event_loop()) ?

Thanks for any info -Kastan

Stack trace:

2022-01-31 10:42:23,320 [Thread-5       ] DEBUG   : pyclowder.connectors - ['kastanvday@gmail.com']
2022-01-31 10:42:23,321 [Thread-5       ] INFO    : pyclowder.connectors - [61a5853c87b65e03e7925706] : StatusMessage.start: Started processing.
2022-01-31 10:42:23,323 [Thread-5       ] DEBUG   : pyclowder.extractors - default check message : {'notifies': ['kastanvday@gmail.com'], 'source': {'id': {'resourceType': "'file", 'id': '61a5853c87b65e03e7925706'}, 'mimeType': 'text/plain', 'extra': {'filename': '10chars.txt'}}, 'jobid': '61f82d8e193c9580af4aa3bf', 'msgid': '61f82d8e193c9580af4aa3c0', 'flags': '', 'intermediateId': '61a5853c87b65e03e7925706', 'host': 'http://host.docker.internal:9000', 'datasetId': '61a41c945e5855829e5923d8', 'id': '61a5853c87b65e03e7925706', 'fileSize': '11', 'filename': '10chars.txt', 'target': '{}', 'secretKey': '70c420d3-918a-45d9-a4e3-987f726c6694', 'activity': 'submitted', 'routing_key': 'extractors.ncsa.rayWorkflow', 'parameters': {}, 'action': 'manual-submission', 'retry_count': 1}
2022-01-31 10:42:23,370 [Thread-5       ] INFO    : pyclowder.connectors - [61a5853c87b65e03e7925706] : StatusMessage.processing: Downloading file.
2022-01-31 10:42:23,461 [Thread-5       ] INFO    : pyclowder.connectors - [61a5853c87b65e03e7925706] : StatusMessage.processing: Loading contents of file...
10:42:23.462680
2022-01-31 10:42:23,479	INFO execution.py:49 -- Workflow job created. [id="rayWorkflow-10:42:23.462680", storage_url="file:///tmp/data"]. Type: FUNCTION.
2022-01-31 10:42:23,481 [Thread-5       ] ERROR   : pyclowder.connectors - [61a5853c87b65e03e7925706] There is no current event loop in thread 'Thread-5'.
Traceback (most recent call last):
  File "/home/ray/anaconda3/lib/python3.7/site-packages/pyclowder/connectors.py", line 426, in _process_message
    self.process_message(self, source_host, secret_key, resource, body)
  File "rayWorkflow.py", line 68, in process_message
    output = self.workflow.run(f"rayWorkflow-{current_time}")
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/common.py", line 363, in run
    return ray.get(self.run_async(workflow_id, metadata))
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/common.py", line 405, in run_async
    return run(self, workflow_id, metadata)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/execution.py", line 56, in run
    ws.save_workflow_user_metadata(metadata)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/workflow_storage.py", line 416, in save_workflow_user_metadata
    self._put(self._key_workflow_user_metadata(), metadata, True))
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/workflow/workflow_storage.py", line 55, in asyncio_run
    loop = asyncio.get_event_loop()
  File "/home/ray/anaconda3/lib/python3.7/asyncio/events.py", line 644, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'Thread-5'.
2022-01-31 10:42:23,483 [Thread-5       ] INFO    : pyclowder.connectors - [61a5853c87b65e03e7925706] : StatusMessage.retry: (#2) There is no current event loop in thread 'Thread-5'.