Run your own server and avoid adding your phone number/email to your matrix account.