It involves a lot more than just that, but yeah no one will keep your xpub private for you like you would; if you share it with anyone, assume at some point everyone will know it (even if they may not know it's yours, it's still a public account history of sorts, and there's value to be taken from that information) 

 Imagine if the IRS required you to give them your xpub. That's what I meant more than accidentally leaking it.