Phones are too easy to steal and insecure. 

Having your phone associated to your npub would actually make them way safer. But npub would need to dictate sim, not the other way around.