Getting output from remote function

How severe does this issue affect your experience of using Ray?

  • High: It blocks me to complete my task.

Using Jupyter notebooks I have a Python function that works fine on it’s own, but when I use an @ray.remote decorator and try to use function.remote(parameters) it provides no console output and does not update a file I’m appending to.

The following example is similar to the rather large notebook that I am trying to get working. This example does not provide console output but DOES create the expected file, unlike my larger notebook:

import ray
import datetime

ray.init(ignore_reinit_error=True)

def logger(text):
    """ Write text to a log """
    now = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    with open("scheduler_log.txt", "a+") as fh:
        fh.write(f"{now}\n")
        fh.write(f"{text}\n\n")    
    return text
    
@ray.remote
def test(x):
    print(x)
    logger(x)
    
logger("Starting")

jobs=[]
for x in ["A","B","C","D"]:
    jobs.append(test.remote(x))

When a function is run remotely, is there a chance that it is run in a different ray instance and thus not create the file where I expect it to be? How about console output… is there a method to ensure that console output appears?

The function that I am trying to run remotely works fine stand alone, so I’m trying to figure out why it doesn’t appear to work as a ray.remote function. If there is an error, it does not appear on my console. In my function I try to log output to a file, but the file is never updated on my filesystem.

How can I debug my ray.remote function?

Apologies if this sounds a bit confusing as I’m pretty new to using Ray.

If you are trying to get exceptions from the task, you will need to call ray.get to receive the exception.

print statements in the task should be automatically routed to the driver.

If the task writes to a file, and the task executes on a remote node, you will not see the file being changed on your local node because the file is on a different node.