Before I read the other comments; my first thoughts also were the neighbors (not the TV show). I had site office, we call them a portacom, next to the boundary fence and some moron woman complained it may blow over in a storm...
Meetings and 20 emails later I lost the fight with the city over it and moved it. So I recommend turn up at the neighbors with a whole lot of alcohol or cash and explain how sorry you are etc before it hits the fan.