People are mad at Hairston for doing what Harper just did. Which was swat at Ramos's glove.
Not the swat at the ball (which Hairston missed).
Hairston clearly swatted at the ball in Ramos' glove.
Harper put his arms up and made contact, but it could have been percieved as fending off contact, sort of a natural self defense maneuver. Though in the end the effect was the same.