Use a very simple pass phrase that you memorize and store in a password manager. You can call it a single point of failure, but that's hardly an issue if the chance of failure is close to zero for the wallet owner.

On the other hand no pass phrase leads to a very significant single point of failure in your seed phrase. 

 Yes I agree. 

Plus, something happening to all 3 is very unlikely. So if something happens to the HW, there’s the seed + pass backup, and if something happens to the seed, there’s the HW to help you recover the seed. 

 Yeah I forgot about the hardware itself.

For some HW devices it's possible to extract the seed from it, so without a pass phrase you have multiple ways of being owned (HW compromised, or paper / metal backup stolen).