First, they need physical access
Second they need to know your pin of the yubikey
Still wouldn’t put my private key on there but no need to panic.
Just remove the keys now and use a proper HWW