I don't believe people who say stuff like this. Suppose you really liked some doom metal record, thought it was epic, great riffs, etc. and then you later found out that the lyrics were all about having buttsex orgies in animal costumes, and you looked through the cd booklet (or whatever) and saw just pictures of people in animal costumes having buttsex with each other. I think this would have some kind of effect on you. I don't think listening to that music would be quite the same experience for you after that.
But even if your experience of it remained the same, I think that would only be the case because you actively forced the lyrical subject matter out of your mind, because doom metal records cannot be about having buttsex in animal costumes. It's simply not a thing! Those things don't even remotely go together. You couldn't take a doom metal record seriously if you thought of it that way. And that's why, by the way, doom metal bands don't go in for subject matter like animal costume buttsex orgies.
If lyrics didn't matter, then metal bands would just put out records with random lyrics about nothing at all, or about completely random subject matter, or they'd just make grunting noises that didn't correspond to real words at all. That would be easier, wouldn't it? Nobody can tell what they're saying, so who cares? But in fact there are lyrical norms in metal. Your position has to be that virtually the whole history of extreme/underground metal is built on a confusion.
I don't believe that there is such a thing as listening to music as pure sound if you are an adult human being, even if it's purely instrumental music. So I think people who say they don't care about lyrics at all are usually full of shit. You may not care about the gritty details of the lyrics of music you listen to, but I do believe that when you listen to it, you are not approaching it as though it's pure sound, but rather as sound associated with certain concepts and themes. Even that grants too much, actually, because I think that grown-ass human beings actually experience various musical sounds as fundamentally inseparable from the themes and concepts they're associated with. That is to say, they don't hear them as a coupling of sound and theme/concept. Rather, they hear the themes/concepts in the sound that's playing, and they approach music with expectations beyond even just that. So I think the "I don't care at all about lyrics" people are either lying or just confused.