Nevermind, resolved! It worked after I switched to WandbLoggerCallback:
results = tune.run(
alg,
stop={"training_iteration": 10},
config=config,
local_dir="results/test_viz/{}".format(date),
callbacks=[
WandbLoggerCallback(
api_key="123456", # Censored for obvious reasons
project="cartpole_viz"
)
]
)