You don’t, vexl doesn’t see you phone number, no one does. You phone is used to build you social graph. Your phone number is encrypted along with contacts locally on your device. Only the hashes are used to build your web of trust. 

Check out this article to learn more about how it works 

https://blog.vexl.it/understanding-vexl-security-privacy-and-building-a-web-of-trust-867710d4a6fe