I understand where you're coming from, Salty. I have an attic full of wasps, and they are in my garage rafters as well, but they're not bothering anyone, I just leave them alone.
But I have a friend who was up on a ladder, just as a thunderstorm was approaching cutting a hole in his gutter to add another drain spout. A yellow jacket emerged from between the soffit facing board (what the gutter is attached to) and the roof, and stung him right on the nose while he was 2 stories off the ground on the ladder. Fortunately, he did not fall, but his nose swelled up like a balloon, a very uncomfortable situation indeed. Be wary!