Paper wasps, hornets and yellowjackets are highly aggressive insects that may or may not be better left alone. In late summer a yellowjacket colony can grow to 5000 or more insects and an unprepared encounter with these guys can be deadly. Identify which species you have so you know exactly what you are dealing with. If you must eradicate it, do it late at night in a full wasp suit. More info on recommended treatments can be found here:
Urban Entomology [Ebeling Chap. 9 part 2] Pests Attacking Man and His Pets
Years ago we had a large underground yellowjacket nest in our south 40 and the bugs were making it impossible to enjoy being poolside in August. A late night assault of 1qt kerosene down the hole was effective.