Addressing the original post, experimental metal is great if done right. Metalcore for the most part is not composed music that is thematically based, but something heavy that you can mosh or party to. Which is why it is generally looked down upon by metalheads. Same with the punk scene, where pop punk bands gained popularity by having feel good choruses that the majority could party and have fun with it going on in the background. This would explain why for the most part, the mainstream is filled with bands that are not as stylistically creative or as instrumentally impressive as metalheads who want to compose an album with the sole interest of creating music in mind. Anybody who has met a "hardcore kid" should be able to notice his tendency to simply want to headbang to a simple rhythm, fast breakdown, etc.
This of course implies a superiority complex about bands like Blut aus Nord, who create compositionally impressive material yet still remain in the underground. Or perhaps a band like Dead Congregation, while still staying true to the traditional death metal sound create unique and crushing death metal. Compared with bands that focus on directionless technicality or catchy choruses, underground bands put more thought into creating a cohesive album focused on unique styles, tonalities, and perhaps even creative concepts. Unsigned bands also dont have to worry about the pressures of record labels who impose on a band's creativity. As a result, the underground and mainstream will always maintain disconnected from one another in terms of trends or popularity. Who knows what the next trend in "metal" will be, but needless to say it will result in a wave of bands who attempt to fit the mold of the trend while creating watered down albums with subpar content.
The future of metal is going to be similar to what we have seen originate in the 90s, and perpetuated in the last decade; Which is the creation and expansion of subgenres that originated from the original metal scaffold. I wont go crazy with genre labeling, but the majority of metal can fit into a few basic categories, including heavy/traditional metal, doom, death metal, and black metal. For example, nu-metal has roots in the heavy metal scene, metalcore in death metal, etc.
The underground scene, as it still remains, will have bands that stick more to the traditional confines of the genre they are playing (death metal, black metal). Others will be combining their influences to create less traditional forms of music, be it successful or a failure (this judgment call is inevitably in the ears of the listener). From what I have seen from the last few years, I am rather optimistic about the future of the metal underground. As long as people continue to listen to metal, there will always be an interest among the fans to create new music from their perspective. Of course we will still be flooded with subpar musicianship, amateur compositions, and failed experimentation and emulation, but there will always be the exceptions from the norm that outshine the majority.