Okay, to me selling out is changing your musical style to follow a trend. Basically, its when a band tries more to please the masses than to please their own fans.
That is why Load is a sellout for instance. Who was the target audience of that album? The alternative rock fans of 1996, or the Metallica fans who had stuck by the band all those years?
Whether or not the band also wanted to experiment more, they sold out because the musical change was to something more popular at the time and it was a complete turn against what the fans wanted. The music, production, image, hairstyles, album cover, new logo, album title... everything about it beared no resemblance to the Metallica that the fans had grown to love.
The black album, however, I think was an example of progression rather than selling out. Each album they did from the start was slightly more melodic and had less S.F. thrash attitude than its previous album, and the Black album just happened to be the one where they hit the big time because the more melodic side to their music incorporated into that album appealed to the Skid Row/Guns N Roses fans at the time, while still pleasing most of their thrash fans.
ALSO... I think a sellout album isn't necessarily a bad album.
Here is an example... Ifyou look at Kiss' career, as Wrathy said they are huge sellouts... I mean Animalize and Asylum are sellout albums because they hopped on the lipstick/hair/LA glam badwagon, but they are great albums still. And Carnival Of Souls jumped on the mid-90s grungy heavy rock bandwagon of Alice In Chains, Ugly Kid Joe, Soundgarden, and bands like that... but that also rocks! Then if you look at Psycho Circus, where they tried to recapture the "classic" Kiss sound, its shit except for about 2 songs!
Another thing, I don't always think that the "poser" bands playing commercial mallcore are usually sellouts either. For nstance Korn and Slipknot aren't sellouts, because they never claimed to be a true metal band then decided to incorporate rapping because it would sell more records. (It's the 12 year old skater fans and idiotic music industry dingbats who call those bands metal because they hear heavy guitars and don't know what metal is). They just happen to suck and I just happen to hate their music. (BUT: I spose the test whether or not those bands are sellouts will come when the mallcore trend dies, and we will see if they keep playing that music or if they jump on the next bandwagon)
Same with deal with bands like Poison. Sure they were VERY commercial and far from "true metal", but Poison have proven that they are FAR from being sellouts, as they've pretty much stayed glam even through the 90s and 2000s now, in a time when glam metal is the laughing stock of the music industry. Poison though, unlike Korn and Slipknot, happen to rock and be good musicians and entertainers.
Warrant, on the other hand, were a much better band than Poison, but they were sellouts because when glam died, they went alternative.
So yeah.... basically to me, to sell out is to abandon your fans and your roots to follow the trends that will bring you more money, and the bands that don't sell out are the ones who stick by what their fans and their music whether it is the popular thing at the time or not... And whether or not a band sells out doesn't make them a good or bad band because Kiss are my favourite band of all time despite their bandwagon jumping.