I believe you're criticizing the way in which racism is often invoked as an institutional or sytemic problem, rather than a problem of individual "racists." If the latter were the case, then a black person calling a white person "honky" could be classified as racist.
In fact, the idea of racism as systemic precedes poststructuralist thought; it goes all the way back to the social sciences of the 1950s and '60s, and manifests even earlier in the work of certain black thinkers. The upshot of this theoretical approach is that it takes historical effects into account. Now, before people get up in arms about the invocation of history (again, I know), let's clarify a few things:
I'm not blaming white people today for slavery in the nineteenth century, nor am I blaming anybody for lynchings in the early-to-mid-twentieth century. I'm not assigning personal blame to anyone. The very nature of a systemic approach diminishes the importance of individual responsibility. If I'm saying that a black person saying "honky" isn't being racist, then I also can't say that white people today are responsible for slavery. I'm not making any individualist claims.
What history does demonstrate is that racial dynamics have evolved in a very particular way, and this is where we notice the effects of racial inequality.
When the vast majority of black people pass white people on the street, they don't think "Those honkies are going to mug me because they're white"; when the majority of white people walk past a police officer outside a coffee shop, they don't think "that pig is going to give me trouble because I'm white." A black person can absolutely have prejudiced thoughts toward white people, but they will take the form of a relationship that places the white person in power: "I hate those fucking rich white women."
Hell, for the sake of anecdotes, I was once called a snake by a black person on the street. He said to me: "I fucking hate white people. Snakes!" I acknowledge that there is prejudice here; but as soon as you ask yourself why black people say that today the answer becomes quite clear. There isn't evidence for natural animosity between humans of different skin color; this kind of behavior is conditioned.
The reason why certain academics insist on reserving racism strictly for whites (or, more appropriately, Western culture at large) is that it signifies, before all else, a state of conditions in a society. Value gets added later; that is, after we determine the racial dynamics of a society, then people begin to attribute negative or positive qualities to those dynamics. It is at this point that the word racism then attains a use than can describe an individual's behavior - because if an individual commits a specific kind of act that falls under the rubric of racial conditions/dynamics, then we attribute those qualities to that person.
There are a specific set of conditions that have led to the racial dynamics in this country; and while blacks can feel animosity toward whites, social scientists often resist the word "racism" because it doesn't correspond to the general history of those conditions.
In my personal opinion, it's the most scientific and useful means to talk about the situation because it attempts to measure material conditions, not the indeterminable intentions of an individual person.
