Secure elements and hardware wallets

Why are secure elements used in Hardware Wallets?

Plain and simple for UX.

If your encryption depends on a numerical PIN of a few digits you are screwed.

The secure element adds a layer of encryption so that in case you can extract the seed from the chip you can not attack it by brute force.

If you use a sufficiently strong passphrase or if hardware wallets allow you to enter a password for encryption, there would be no need for secure elements, it's all about UX.

On the other hand, in cryptography you should never trust. Secure elements are a black box, and by contract you can't even publish their vulnerabilities. What's the point of using encryption if you're trusting a third party? that's useless. 

It is true that there are solutions like coldcard where two secure elements are used so you don't have to trust them, but I still think it's stupid, I don't care about the UX, I want my bitcoins to be safe.

When you understand this, you realize how stupid the hardware wallet industry is.