I'm sure there are many traditions I don't fully understand, but the whole "hat off to show respect" is close to the top of the list.
Why is removing your hat noble/polite when shaking hands? Whether it's to enter someone's house, or "congratulations on an enjoyable game" while walking off the 18th green.
Here's the scenario +90 % of the time. Hat is removed and held with the left hand as you approach. The person runs their right hand through their sweaty, oily hair (often more than once) to fluff out the "hat hair", and then extends the same hand for the shake. Living in the south most of my life has taught me than almost anyone wearing a hat outside for more than 30 minutes will look much better hat on than hat off....at least until a shower and comb are available. Maybe back when no one bathed more than once a week, the sweaty/oily hair was the expected norm.
I really don't find anything polite about being offered a sweaty/oily hand. Only marginally better than extending a hand after using the bathroom.
Just keep the damn hat on. Makes so much more sense to me.