I'm not sure, but isn't this some sort of confirmation bias thing? Like, no one ever remembers when a pitcher is yanked right before he would have thrown a meatball that homered to left-center. Because he didn't throw the pitch if he got yanked.
Also, the times when a starter is left in too long form definite events (blowing a lead) that tend to stick in our head, while a 5.2-inning start might be more easily forgotten?
Not sure if I actually believe this, just saying.
I think that is very true. But the point is that people rarely question the removal of a pitcher unless he is cruising and seemingly in complete command. It is easier to tell when a pitcher is starting to leave things up in the zone and starting to get hit solidly by the opposition the 3rd time through the lineup. So when a batter is left in a batter or two longer than most would like to see, you're not surprised when things go bad. You can be excused for not anticipating when the pitcher is suddenly going to lose it.