Sorry but you are completely off on this. Where I work we have very strict rules to prevent the appearance of impropriety, which is standard operating procedure in many industries and definitely for government.
In who's eyes? If I had to worry about what every Tom, Dick and Harry thought about how I conducted legitimate activities, I'd blow my brains out.
And if they did nothing wrong, they have nothing to fear...