Baltimore did not deserve the Browns. Bill Veeck wanted to move his team to Kansas City, and hoped that radio fans from west of Saint Louis would take up the KC Browns. Unfortunately, owners hated Veeck, so they blocked him.
In 1954, Baltimore's metro area was about 2/3 the size of the DC area, and maybe less. However, Washington was a city of government workers, of GS-this or that's who read Bill Gold's "The District Line" in the Post to see which agency had openings. The Nats were owned by the Griffith family, which had no income except from the ball team. Baltimore had industry, and industry owners. A beer family, the Hoffbergers, were allowed to buy the Browns. Baltimore's industry died away, while Washington kept growing because the Federal government does not suffer from recessions. In addition, the government drew computer professionals, so the boss of the computer division at GE found that there were more "software engineers" in the DC area than anywhere else in the US. He moved his division to Bethesda and then to Rockville, and then to Gaithersburg. Other companies saw the same thing and moved to I-270 and to Northern Virginia. Nothing moved to Baltimore.
The Griffiths hoped that fans would drive the Baltimore-Washington Parkway to Senators-Orioles games, and there was a rivalry from 1954 onward. Washington kids learned about "the Bush League Baltimore Orioles". Best I remember, the Orioles drew from Baltimore City and County, but not from Anne Arundel, Howard, or Frederick counties. The big Maryland counties, Montgomery and PG, were full of Nats fans.
Offhand, I don't think Baltimore drafted especially well, given that they picked near the top year after year as they tanked.
Baltimore woould be a good AAA town for the Nationals.
