Yes, we lose something when we start to look down on physical labor - regardless of how well it pays, the needs we have and the technical skills it demands. I did some construction in college and still love to build stuff and volunteer.
I am in the marine industry (on the landlubber side) my old boss had been a boat builder, research diver, charter boat captain and inter-island pilot. He used to look around the office and say "this is NOT men's work". We would all look at our shoes and agree...