There's a zillion ways they can get your email address. Do you have your email configured as your FTP passwd in your browser? Could also be a dictionary attack.
Only 20 a day? I get over 10k spams a month. I implemented a challenge-response system, which cut down the spam dramatically. When a new user sends me an email, they get an email back which to which they must respond within 24 hours, or the email isn't delivered to my inbox. For the most part, this works well, but there are still legitimate users with the IQ just above the common houseplant that can't figure it out, and will write a follow-up email to the challenge email, or send me another fresh email asking why I haven't responded...
