If someone guesses your private key without violating your physical property he will be able to move any UTXOs controlled by those keys. If he does and you claim your bitcoin has been stolen you would have to argue why that person’s action is illegitimate. In other words, why are you the only person that is able to arrange characters in a particular way (generate a particular private key)? Obviously, this scenario is virtually impossible assuming the entropy used to generate the keys is adequate, but it’s an interesting thought experiment. 

 This is a good thought experiment, I like where you went with it.

I don't have a good answer yet.