I could understand that if I was writing about a film for a philosophy class or some other non film analysis related field. However, this was for a comm studies class, a field that makes examining media a huge part of its focus. The professor even used the terms 'diegetic' and 'nondiegetic' to talk about sound within films, both words derive directly from the root 'diegesis'. In my mind I shouldn't have to define a term if it is one the professor already makes reference to. Moreover, self-reflexive is term which should be apparent to anyone who has read anything English or Film Studies related. I may have not explained the term itself, but I did give a fully developed example of its use within the context of a particular film, followed by the sentence beginning with "This self-reflexivity...". If the term is still vague and unknown to the professor or any reader at this point they should either put an end to their career or quit school immediately.
The only place I could see defining one's terms being absolutely necessary is if an entirely new term is being postulated and utilized throughout the same paper. Otherwise, well used academic terminology should not be the grounds for a complaint of a paper in an academic setting.
I am still considering taking it up with him, but I'll probably avoid it just because the resultant mark is still agreeable to my standards, if it were any lower I would most certainly pursue this further.