I'm definitely in the "moderation" camp, particularly because I'm now basically off listening to music on earphones/headphones. (And I'm a hobby musician!).
I was diagnosed with Ménière's Syndrome sometime back, but before I knew, I had terrible episodes of vertigo -- attacks that make movement impossible for hours, and make you vomit should you decide to go ahead and give it a shot anymore -- and my hearing on my right ear, in the lower frequencies, had already gone really bad: in the range of 40dB, which went as high as 60dB in about a month or so (around 60dB is severe hearing loss, although I was affected only in the lower freqs).
While the reasons for why I got Ménière's is not clear (it's idiopathic for now), my loud drumming (sans ear protection) and listening to music through ordinary earphones in busy buses didn't help either.
My hearing has basically come back to normal now in both ears (not very usual, this, I'm lucky), but the only time I put on a headpiece now is when I'm making a call in Skype.
I thought at one point that there should be regulation requiring that every store where musical instruments are sold (drums, especially), should also sell ear protection. I got custom-fitted earplugs some time back, and I carry them with me.
I don't intend to stop drumming, going to live shows or playing -- these are things I love and would like to always get to do -- but I have indeed learned the importance of basic protection.
Get some earplugs; I cannot stress enough how much they're worth it.