No, you shouldn’t have to modify your spaces, RLlib will handle that. For compute_single_action, make sure you use algo.compute_single_action and not the policy. method, since the former will automatically handle the filters. See: How to correctly apply observation normalization?