mainstream is something that is brought into the culture and fabric of an everyday person.
Bands like Radiohead, Coldplay, The Mars Volta, Tool are definitly mainstream.
Bands like Lamb Of God, Mastodon, In Flames are borderline. Meaning some people know them that might not be formilure with the underground metal scene.
Then bands like Enslaved, Pain Of Salvation, Iced Earth ect.. are ones that are known quite alot in the metal scene, but are virtually unknown to people outside of it. atleast how im looking at it (note not all bands iv mentioned are good)