It involves a lot more than just that, but yeah no one will keep your xpub private for you like you would; if you share it with anyone, assume at some point everyone will know it (even if they may not know it's yours, it's still a public account history of sorts, and there's value to be taken from that information)