[masraum]

It doesn't sound rude to me. It sounds to me like you're being considerate of them and their guests.
"I don't want to create an unpleasant environment for you and your guests so I'll wait" <- to me that is basically what you said.
If anyone took offense at what you said, I'd be pretty unconcerned. But then I'm not sensitive and, I think, relatively no-nonsense.

I do know that there are lots of sensitive folks in the world that get offended by all sorts of stuff that I would consider completely benign or at least neutral. I would prefer not to offend, and I try not to most of the time. But I don't think that you can never offend anyone.