The sender can find out the real address on send, because they are the ones who create the output. But they can't see any other sends to the silent payment address from any other sender. And no, nobody else can figure out any other payments to the recipient without the recipient's private key.