Answer

He feels like he is accused of this. It’s as if someone can only see something from one viewpoint and is narrow minded. It is almost saying that is what others are like who use the restrictive term ‘halfe-caste’.