Maybe they can't but that doesn't mean you can't rug yourself with a compromised device or network connection. 

 True! Always the risk with hot wallets