It bothers them probably, but deep down they probably respect you more for having integrity and standing your ground (in a civil way).