What a band sounds like when they "sell out" is pretty difficult to predict, especially with a band like Opeth, and also considering the definition of the phrase.
I think you first need to ask the question: What does it mean to sell out?
For me, selling out means that a musician sacrifices their creative freedom and desires for the sake of a different, undesirable creative direction that will seemingly lead to more popularity and profit.
So, let's say "death metal" becomes the next BIG thing in the music world, and there's a huge influx of fans and profitability to be had by jumping on the death metal bandwagon. If Carrie Underwood were to release a death metal album solely for the purpose of gaining profit, then her move could be considered as selling out.
If Opeth were to write an entire album of power/country ballads, because they were told that they could become the next big thing in Nashville and would sell millions of albums, then that could be considered selling out.
However, I should also state that I personally believe that a musician is not "selling out" simply because of selling more albums. So in my Carrie Underwood example, if she truly loves death metal and wants to release a death metal album, regardless of it's new famed popularity, then I don't believe she is selling out. She might be selling more records, but she wouldn't be comprising her creative integrity. Same goes for my Opeth example, if in fact they were passionate about creating a country/power ballad album.
Knowing that Mikael generally has a dislike for hip-hop and rap music, I think it would be fair to say that if Mike were to write a legitimate hip-hop album with the intention of gaining more popularity and profits (while also despising the music he's creating), then he'd be selling out.