Since I do commercial UAS for a living, I have spent some time trying to understand the rules. which are a bit obtuse.
Start with
United States v. Causby - Wikipedia, the free encyclopedia
The general drift of regs are that a property owner owns and controls the airspace to the altitude of 500 feet agl about their property.
From another website: "The reason for this is simply aircraft flying overhead. The FAA requires any aircraft to fly NO LESS than 500 feet above the ground. Therefore "if a land owner is to have full enjoyment of his land, he must have exclusive control of the immediate reaches of the enveloping atmosphere.....Thus a landowner owns at least as much of the space above the ground as he can occupy or use in connection with the land", and the invasion of that space "are in the same category as an invasion of the ground."
Here is what I do. If I am flying (I do it all legally) and I think there may be a question in terms of intrusion or the perception of intrusion, I make sure I let people know we are flying. I make sure everyone KNOWS we are flying.
I follow the FAA rules and common sense. If anyone who flys for me parks a quad next to a pool and ogles the talent they are fired.
If someone shoots a UAS of mine parked near a pool, I just lost a UAS.
Think about it. If I saw my neighbor watching/leering at my daughter though binoculars from his yard, he and I are going to have a chat.