Downsides might be congestion and i think nostr:npub180cvv07tjdrrgpa0j7j7tmnyl2yr6yr7l8j4s3evf6u64th6gkwsyjh6w6 would likely have a "fit to purpose" argument, where using nostr for this and ( possibly expanding the protocol parameters in the future if this gets popular) overextends the censorship routing and message delivery focus of the relay structure (the way the lightning project and its advocates did to Bitcoin)
But i don't want to speak for fiatjaf.

It's great to innovate but it's also great to be vigilant that certain things are building their own path to eventually forking.