In the end, no one was negatively impacted from this as everyone who joined in agreed to it. Nor when Draxen joined, didn't tell me to stop when he came back.
I would like to also state that: I am not doing this when the server is full of players knowingly that it will cause players to be affected by it. I only do this now and again if users agree to it and the server is near empty and that no action is going on.