Could be two things:
1. The speaker IS crap.
2. If you play more often with headphones, you've probably developed a preference for that, even if subconsciously. Lately, I play/practice with headphones 95% of the time, and there's definitely an adjustment period when I play with a "real" amp. I think the reason might be that with headphones, the sound is right there in your ears, and there is virtually no delay between the time you play a note and when you hear it. When you play with a "real" amp, your ears are exponentially farther from the sound source, than if you were wearing headphones. The delay is almost imperceptible, and most people would say this is bullshit, but I think it does make a difference.
Also, if I play with a different amp than the one I normally use, I get thrown off and think - that doesn't sound like me, and I don't play as well.