The vacuum advance is not supposed to "see" any vacuum except at (VERY!) small throttle openings. When the throttle is closed, no vacuum advance. When the throttle is 1/4 of the way open or more, no vacuum advance. It is only supposed to act when the throttle is open a very small amount.
This is why the correct port for vacuum advance is on the throttle body, just barely "upstream" of the place where the throttle plate sits when the throttle is closed. When the throttle plate opens, the edge of the plate is very close to the advance port on the throttle body. The venturi effect creates a relatively-high vacuum at this spot (the air is going through a small opening so it has to go quickly, so the pressure decreases). This pulls on the diaphragm in the distributor dashpot, pulling the arm to advance the timing.
Having the advance connected to manifold vacuum means that it is seeing a higher vacuum at idle than any other time. So the timing is likely advanced more than it should be at idle--which will result in (among other things) a higher RPM idle than you want.
When you do not have two fittings on your throttle body, you leave the advance fitting open to the air. If it makes you nervous to have a bare fitting, then hook up a hose and tuck the other end out of sight underneath the manifold. Don't cap the fitting, or it will interfere with the vacuum retard somewhat.
Make sure the diaphragm in the dashpot is not leaking.
Remember to set your timing at 3500 RPM with the distributor vacuum hose(s) disconnected. You can fiddle with it a few degrees one way or the other if you like, as well.
The functioning of the stock PCV valve is discussed on Brad Anders' website.
http://members.rennlist.com/pbanders . He and Ray disagree somewhat on how the PCV is supposed to function; I agree with Brad more than with Ray. I'm not sure how you check to see if your PCV valve is only opening under light loads, though.
Try a very very small orifice in the PCV line restriction, if you don't want to try going with the stock functionality. That should also bring the idle RPM down. Plugging the line entirely will likely drop it further, though that's not that great an idea for anything other than testing purposes...
--DD