Add manual actions to improve learning

Is there a way to add manual actions to impala for example from which reinforcement learning should continue to learn? This can skip many hours of training and use less computing power for people who cannot afford Tesla GPUs not to mention the crazy energy prices.
I don’t want a complete offline strategy, but a reinforcement of an offline strategy.
Is this possible?
Thanks!