You are and you aren't.
The anti-drug brainwashing makes the scene worse than it is. True, certain drugs will entice you to dance with the devil for life, but most are harmless during the college experimentation days. Acid and 'shrooms can really open your mind and let you see things that you normally can't see.
At the same time, staying away is perfectly fine too.
You really aren't going to be looked down upon for staying away. Not like the labels you can get if you tell someone you tried acid.. You doper! You loser! Yeah, right... Whatever.