oh that sounds pretty complex, keep in mind the hardware wallet to be secure it must have a secure element (besides other hardware complex stuff)  

 I heard something about devices which didn´t safe the passphrase.
So maybe that devices don´t need a secure element. 
But i´m not sure about that