It can have an impact, but beyond the physical impact, it can have an effect on the pitches themselves (e.g. the sinker doesn't sink the way it normally does.)
I know it can. I'm just saying, it really doesn't have a huge impact on a game like baseball like it does in other sports. Teams don't seem to send pitchers ahead of the team like they used to, at least not during the regular season. It's not something I see worth getting worked up over.