It really depends on what you mean by cultural genocide, so the answer could be "there's nothing wrong with cultural genocide" or "there's everything wrong with cultural genocide".
The important thing to remember is that you can't kill a human's culture--we're too resilient. You can only change it. African-Americans lost their African culture, but have develoepd arguably one of the most complex, interesting, and rich cultures I can think of to take its place. That's just what humans do.
And the end of certain cultural practises isn't even necessarily bad. Cultures change on their own and are even in part defined by their changes, even when those changes are forced by outside forces. Integral to the American culture is the theme of redemption from forcibly ending slavery culture, for example.
In the end, what's worse than cultural genocide is an adherence to cultural purity. This was the central facet of Naziism and it's entirely inhuman. What yuo're essentially doing is taking a moment in time, crystallizing it, saying that your culture always has to be exactly like this, and attacking everyone who doesn't abide by it. It's unnatural and anti-human to prevent cultures from growing and changing and, because the end or mutation of some practises is the CENTRAL way cultures grow and change, to even go to extraordinary lengths to prevent the end/change of some practises is horrendous. The French separatists in Quebec are an example of this .
A culture that doesn't accept change stagnates and doesn't grow, learn, and become better as time passes (as human cultures naturally do). What if the United States had made a law at its founding that everybody within its borders had to speak English or be expelled? Or that tri-cornered hats were an immutable part of American culture and everybody had to wear them forever and ever amen? How much of our current culture would we have lost? (although we wouldn't have the trucker-hat problem)
The Germans tried to protect their Wagnerian culture with laws that decreed new forms of art and music "Jewish" and poisonous to the Aryan ideal. Abstract the "Jewish" excuse from there and you still have a culture trying to destroy innovation and prevent itself from naturally changing and growing.
Okay, there's certainly times it goes too far. Saddam Hussein's campaign against the Kurds wasn't just physical. He forced Kurds to name their children Arab, not Kurdish names; he even plowed under tombstones in Kurdish cemetaries that did not have Arabic names. This was all in an effort to ethnically cleanse the Kurdish people and replace them with Arabs.
That's an extreme example, but it's a genuine instance in which cultural genocide is wrong. The danger in our world is the Nazi-ish desire to cry "cultural genocide!" and make laws to protect aspects of your culture that are changing or even being changed from outside naturally.
(and to answer the question, sort of what Avi said: physical genocide is much more serious, because people can always recover from cultural genocide and incorporate the harsh attempts to destroy their culture into their new culture. those Kurds cleansed by Saddam lost their Kurdish names, but that just became a part of their Kurdish culture. culture's too tough to wipe out completely just by outlawing some of its silly practises, like wearing an Afghan knife or bringing something blue to a wedding--it just changes a little)
I mean if half a people are annihilated, the other half might grow stronger from it generations later. When I see Aboriginals already fucked up at 9am in the streets of my town, I can't help but think of those boarding schools whose primary purpose was to completely uproot Aboriginals, effectively depriving them of their identity, their culture, etc. There's no way of healing that now.
