While it's a wonderfully romantic notion that music bypasses the high-level analytical circuits, it's also empirically untrue. Before music reaches your "heart and soul", the organ it passes through is your ears. And if it sounds muddled to your ears, it's not likely to have the same impact on your heart and soul.
Well, "what you hear" (your ears) and "what you feel" (your heart and soul) are actually
all created by a single thing: your brain. What we "hear" is something quite different than the physical soundwaves propagating through air. We only "hear" after our biased and simplifying brains have translated those soundwaves into something we can understand. This mismatch between the physical world and what we perceive is exactly what lossy compression algorithms like mp3 take advantage of in order to work.
So yes, obviously "what you hear" comes before "what you feel", but for music that you're familiar with, I think the "what you hear" part becomes less and less important. Essentially your brain just needs a trigger to internally reproduce memories and feelings that came from listening to the music before, and for that, "what you hear" can be
extremely muddled, and "what you feel" won't really notice. But yeah, for new music, where no trigger to your soul yet exists, it's probably best for "what you hear" to be as close as possible to what the artist intended. And certainly every time your re-listen to something, your feelings/memories are updated, so actually listening to a song will have a greater/different effect than just reading the lyrics and replaying it entirely in your head.
Of course for something to actually sound "muddled", that's a problem with the original recording/mix, or the playback mechanism is broken. No real-world encoding will actually make something sound "muddled".
We've all gone to see one of our favorite bands when the sound was off. It just doesn't have the same impact as when the sound was dead on.
Citation needed.

First, live sound is always "off". Even with the most-perfect sound engineer and venue, the difference between live sound and CD is going to be like 1000 times the difference between CD and poorly-encoded mp3. But because the melodies and rhythms are still perceptible (hopefully!), we don't notice how different it really is due to those brain heuristics.
I've certainly been to shows where the sound is more "off" than normal, and I've been to shows that haven't been as good as others by the same band playing the same song, but I don't have enough data to say that there's a genuine correlation between the two. There probably is, but even at shows where I think the sound sucks and it's hurting my enjoyment, I still see other people having the time of their lives, so maybe I just wasn't in the right mood, and I'm just using "the sound" as an explanation without basis? I think at the next show I'm at where I think the sound is bad, I need to pass out a questionnaire to all audience members asking them to score both the sound quality and their enjoyment level. Don't know why I've never done that before!
So let's say you were listening to Spotify on your phone. During the last app upgrade the sound quality setting had been reset from 320 down to 96. Can I assume you wouldn't bother adjusting that setting back to 320 because your fuzzy organs are so wonderfully imprecise?
No, I'd adjust it back to 320, but only because I, like everyone, have a "sound lover" living inside me simultaneously with the "music lover", and the stupid "sound lover" will always pipe up and whine when he notices something different. But if I was not
allowed to change it back to 320, and all music-playback was changed to 96 and I just had to live with it, I have no doubt that after a brief adjustment period, my heart and soul would feel no different than they did when listening to 320 music. To believe otherwise would be to believe that people got more happiness from music in the late 90s (when CDs become dominant) than they did in the late 80s (when cassettes were dominant). And I just don't remember that. Sure, I remember people saying "hey, this sounds great!" but was there anyone who ever said "you know, I didn't really care for music before this, but now that I can hear it with a 16-bit dynamic range, it's totally changed my mind and I think I'm going to start getting into it"? Are teenagers today jerking off to HD porn actually enjoying it any more than I enjoyed my 256-color Christina Applegate fakes back in the day? Hard to imagine!
Earlier you said "sound lovers" and "music lovers" are two groups who have almost no overlap. Yet here you're acknowledging that seeking out better speakers/headphones can turn music from "a weak pop-up to short into a soaring home run" for even the "most casual fan". These two position seems diametrically opposed.
No, they aren't two separate groups, they're two separate personalities, both of which exist in everyone, to different degrees. Almost everyone is born with the "music lover" dominating over "the sound lover", and most people maintain that balance. But some, over time (the audiophiles) give themselves over almost completely to the "sound lover".
So even for people where it's the "music lover" that dominates, there is still a "sound lover", and that's the side that easily notices the difference in speakers/headphones. If that side is powerful enough, they'll convince their host human to upgrade, but the "music lover" will be like "Uh, thanks for trying I guess, but that didn't actually make anything better for me."
Now hopefully that was enough psychobabble BS rambling to make everyone give up on this thread!