I say compromise or dig into your wallet.

I once went into the men's store complaining that the fit on my pants, shirts and suit jackets was all mismatched.

The sales assistant said it didn't matter if everything was loose, if everything was tight, or if everything was in the middle fit wise.

What matters is consistency, that the pieces all fit with the same fit theme in mind.

I am hard to fit as well. One thing I had to overcome was the nice shirt that didn't fit me right.

It don't look nice if it don't fit right - a hard thing to overcome for the frugal minded.

EDIT:
I suggest resisting the temptation to fall for the buy 2, get one free.

I bought a bunch of nice stuff 3 years ago, now it's all too big for me as I've been working out nearly everyday for three years.

i buy a wrinkle free shirt from Nordstrom. changed my life. i tossed out the iron.

i am relatively short but with a lager collar size. I find that Van Heusen shirts offer the correct combination for me, and probably as they do various collar sizes for each shirt size. for most others.
And they have plenty of outlet stores, where the prices are really keen for quality shirts.

I recently bought 2 dress shirts from them.

Very happy with the quality and price, and they are the best fitting shirts I've ever had.

Plus i don't have to leave the house to get them.

Awesome! That matches the other feedback that I've read online. I just have to stop procrastinating and do it.

It's no risk, too. They are very accommodating, if you don't like the fit they'll redo it for free, or give you your money back if you don't like the shirt.

I had no issues, though.

Not too far different in size, big shoulders and longer arms, but I found “L” shirts which fit fine especially the gorilla arms. Right off the rack..

I've been buying shirts from Land's End for years - you can follow their sizing guide and get a semicustom fit