The EXACT same thing can be said by those people (or anyone really) about you based on the evil death metal music that you listen to. It's not really that far fetched that someone might think you're evil or harmful, or a downright bad person without knowing anything about you other than the fact that you've got an illegible band logo and painting of a naked nun impaled on a stake on the front of the shirt you're wearing when they see you in a grocery store. We know you're not, but anyone that doesn't know you could easily make that wrong assumption.
I agree with LF. Let the idiots be idiots and don't waste any energy on them one way or the other. It ain't worth your time.