I'm using Bitcoin since 2013 and since many years I manage my daily life with Bitcoin.
That you lost your key is horrible but you can also loose your access to your hardware wallet. I know ppl with that problem. Also the whole supply chain attack is really something to think about. In total there are more risks with a hardware wallet.