I feel like this could become a bit like the 51% bitcoin problem only much easier:  if we have even as many as 100 relay operators used by most people it wouldn't take much for all of them to agree to ban someone by sharing the same ban list  - which already happens with spam lists - even if not intentionally its likely that censored people will just get added to a 'censor list' without much thought.