Public OutBox and Public Inbox are for your followers. If they can't use those relays, there is no point on keeping them there. 

You can use the Private Home, Local Relays or the general list to keep things that only you can access.