I’d need to capture the logs for teh application, not the per worker logs, and we grab the result from stout, probelm is we get a lag and not all the logs get printed out. Is there a way to get all the logs?
Hi @delioda79, by default all of the logs should be streamed to the stdout of the driver (the main python script you run). There may be a slight delay before these logs show up because they are streamed from all of the nodes in the cluster. Could you elaborate on what you mean by there is a lag? Is this because the program is hanging?
Thanks, it was just something totally stupid. We had forgotten to set PYTHONUNBUFFERED=1 in our environment