I guess it's not so much the cold as it is the humidity with the cold. When it's 'foggy' down at the beach it can get really cold.
I went to REI and picked up some long bottom under layer pants. I think they'll work just fine.
Gives me enough of an unrefined look to not be approachable.