... What you DO need to rotate is your round in the pipe of your carry piece. Each time you cycle that round into the chamber, the bullet sets back ever so slightly. A handful of times is no big deal. ...
In addition to setback, the other thing that can happen is fracturing of the primer cake (leading to rounds that will not fire). I've had this enough (personal experience) I've gone to the practice of never using a rechambered round for anything other than the range. If it's jacked from the chamber, it's removed from carry. YMMV.