I don't see this as self-censorship. If you value spam and others don't they are not your WoT. Your follows are your business. If someone unfollows you, it does nothing to your speech other than to signal you are not saying/doing things they like. If you speak only with the goal of pleasing everyone you are doomed to fail.