I'd say Manny's comment was directed to Bowden as much as it was directed to Dukes himself. In other words, "Jim, get him the eff out of here."
Yep, I think it's clear from his little outburst that he and Jim have quite different ideas of how good this team is and how to make it better.