Avan-Garde Metal?

I heard Arcturus 'the sham mirrors' and do not really get the hype, the first song was really good and then a 1,2 other songs were cool but that's it, garm sounds great and musically the album is not all the impressive. I never really though wow this album is just beyond everything. For something labelled Avant-Garde I expected a lot more.

The only two Arcturus that even potentially qualify for being avant garde are La Masquerade Infernale and maybe Disguised Masters (though that album was very hit and miss). For the rest they are a good band but not really as outthere as some people will have you believe. The Sham Mirrors is probably their most straightforward album.
Progressive metal is a clear-cut style of metal; it is characterized by Dream Theater/prog rock-like sound (usually) and contains clean vocals, epic song structures with meandering (not insulting here) instrumental sections, lots of tempo/time changes, etc. In a sense, prog metal is not truly progressive...the irony!
I think that, basically is the key element that genrelizes something as progressive.

Does Avant-Garde = Progressive + Extremely Experimental Metal?
while hammers kicks ass. i wouldnt call them avant garde. they dont really expand past normal song structures like a traditional avant garde song would. theyre just unique and awesome.

Yeah I don't really see them as avante-garde either. They're just a stretched version of Trad. Metal.
I see Avant Garde as simple meshing in many different genres and styles, some that are polar opposites in sound, together and constructing actual music out of it. It isn't really just taking a genre and pulling it but a extravaganza of genres into a coherent song. Mr. Bungle is the first thing that comes to my mind when talking about Avant Garde.
What do you think of Sideshow Symphonies ?

Somewhat boring album. Vortex is the only reason I ever listen to it. It's got some interesting music but sounds much weaker in comparison to The Sham Mirrors, which executes a spacey sound that's much more straight-forward. I mainly blame the guitar-unfriendly production for dragging Sideshow Symphonies down.
Have you heard Sleepytime Gorilla Museum?

Yeah I have, I quite liked what I heard but I haven't sought out any of their albums yet.

Another band is Estradasphere, from what I've heard of them they are quite avant-garde, but more from a rock perspective than metal.
I'd consider it to be anything that favours pushing boundaries and prioritising experimentation over musicality. Fantômas "Delirium Cordia" is a perfect example.
The avant-garde will always be whichever bands are leading musical experimentation at any perticular time, there could be a metal avant garde which is composed of bands doing radical things while remaining within the style but as W&W said those bands are still going to be pretty traditional compared with artists who are trying to challenge the basis on which music is made and aren't really looking for an audience of any kind (beyond whatever post-post-modern art community there is right now).