Yes I think using nostr relays to setup rendezvous is a great idea, with hole punching something like this https://itnext.io/p2p-nat-traversal-how-to-punch-a-hole-9abc8ffa758e

What lower level protocol are you thinking of using? UDP messages? TCP with message framing added back? WebSockets? If we are starting a new protocol, can we go binary?