If it helps anyone, I decided to pull it apart and check it out, as I knew I couldn't resist. I first bought new belts and a new idler pulley and tensioner pully. I had suspected it to be the idler pulley, but thought I would replace both, so they would wear together evenly.
As it turns out, the tensioner pulley was shot. Belts were fine. Replaced the pulleys - problem solved.
BTW - my car had the hydraulic style tensioner pulley - something I checked ahead of time before buying the part. Some e46 models came equipped with the mechanical style pulley, so definitely check which one you have before replacing.
If anyone has this problem or has any questions on replacement procedure, let me know.