If you are using a 6 digit or higher PIN then you rely on the secure element to protect you from brute force attacks. If such a component is exploited then it could be possible to brute force quickly/without a throttle. Using a strong passphrase (Diceware 7 random words, or random case letters, symbols and digits for 18 characters or better) makes this impossible.