Not necessarily. Some perfect albums IMO:
Satyricon - Nemesis Divina
David Bowie - Ziggy Stardust and the Spiders from Mars
Alice Cooper - Billion Dollar Babies
None of those are even in my top 5 favorite albums. It may not make any sense, but bear with me here - a perfect album has these qualities:
1. Killer production
2. Seamless flow from track to track
3. Outstanding musicianship
4. Great lyrics
5. Great songwriting
One's all time favorite albums need have only 1 quality: the songs that you love, for tangible or intangible reasons, more than any others.