Baseball's fan base is older than the others. The sport is losing generations of new fans partly because very few kids on the east coast have watched a World Series game past the 3rd inning in 15 years. The games start at 8:30 and end after midnight most times.
To go along with the above point, baseball is not the trendy/hot sport. African Americans have almost completely stopped playing and so pop-culture-wise, the sport lacks all cache. This could be due to pace of play, or lack of perceived athleticism needed to play the sport.
The rising popularity of lacrosse. Where I live in Howard County, the kids play lacrosse, not baseball.
Gambling! The rise of football was fueled by how great a sport it is to gamble on. Fantasy football is way more popular than fantasy baseball.
Unwritten rules are lame.