Seems like vote secrecy is the big issue.  I’m assuming you’d get a private key when you registered to vote.  So one issue is guaranteeing that the agency that issued you the private key didn’t keep a record of it and in doing so, the ability to generate your public key and uncover your vote.  Additionally, in situations where your private key was compromised and someone else voted with it as well as you (i.e., a double spend of your vote) you would have to reveal your public key to rectify the issue which would again void the secrecy of your vote.