If signing was done with your public key then anyone could sign your notes for you.