That only works for unlocked tokens. You can zap someone an unlocked but encrypted token and that should work without talking to the mint. However, then it becomes less publicly verifiable.