I have the same problem with shirts.
When I was still working on the tools, I went to a good clothing store and was the guy took my neck and sleeve measurements and said "Oh, your one of those..."
I am pretty sure it wasn't a compliment so I left without buying anything.
As mentioned different brands fit better. I seem to fit clothes from an English designer, Ben Sherman.