It seems like the trade off is to either trust your wallet software to not generate leaking nonce or to trust your hardware wallet to not leak via nonce. IMO trusting the hardware wallet is the better option as that is the device that you are trusting to not be compromised already