Hehe, yeah, I agree. Just check out how short albums like Opeth's Damnation or Antimatter's Planetary Confinement are, and I wouldn't want them any longer than they were at all.
The thing is, I think I usually do favour the more indepth, lengthier albums - Opeth, Virgin Black, In the Woods... - you know, stuff like that. But, their music suites the epic length of their respective albums and in the end, it really doesn't matter. Music is music, and so long as it's effective and moves me I will embrace it wholeheartedly.