Yeah, it's the non-linearities that are missing. Power tube saturation, speaker excursion, air elasticity, etc.
They could be "stored" in a impulse response, but cannot be restored by the convolution algorithm, since convolution is linear only. Hence, they corrupt the IR.
So, the lower the...