Seeing someone with a phone stuck to their head annoys me too but even with a hands free people can become distracted when talking to someone on the phone.
When you think about it what's the difference between holding a phone to your head or talking on an hands free?
Besides the obvious difference of actually having to hold the phone not much especially if you drive an automatic car.
I used to use a Blue Tooth hands free in my car but since going to an iPhone I've "retired" it.
Not that it didn't work with my iPhone but it's just a PITA answering a phone while driving. Also having to "connect" to the hands free every time you get in is a hassle.
So I just put my iPhone in a small pocket in the console and ignore it when I'm driving.
If it's important they'll leave a message...