Hello!! Sorry for the late reply. I was out of office for a few days but I am back and catching up on threads.
I think this might be because you are on local_mode = True. I took a look at some older threads like this one: What is local mode?
It will run things in series instead of in parallel.
The mode LOCAL_MODE should be used if this Worker is a driver and if
you want to run the driver in a manner equivalent to serial Python for
debugging purposes. It will not send remote function calls to the
scheduler and will instead execute them in a blocking fashion.
Have you tried turning off local mode and seeing if that’ll help with the parallelism?