it's mostly an opsec leak, people can see where you are (sometimes quite precise) by geolocating your IP
it also allows correlation with other services you might be using, especially if you run other P2P software on the same IP
for defense in depth reasons i absolutely use a VPN for all nostr related things