You're right, it isn't fair.
Remember this though. At times, it may be difficult to weather at the time, but in the end, you teaching your child this very important lesson in dignity will serve him for a lifetime and lessons learned from his mother will most likely "backfire" for lack of better word. Being a strong role model is a very effective method for raising healthy children.
These situations with people like that (seems to usually be the woman for some reason
)always make me sad because of who really suffers. Children are always first...before ego, before jealousy, before anger, before...ME...... ya know?
Edit: Oh, BTW, you're not evil by any means. Sounds like you've been more than patient.