The web client shouldn’t be fetching the invoice, it just passes the whole offer on to the wallet and asks it to pay. When the wallet completes it does a callback to the initiating app.
Admittedly I wasn’t really thinking about web apps in the above design though. Nothing specific shouldn’t work except that the spec explicitly says wallets can’t open a callback URI that has a scheme of http(s). I was worried about it being a way for a payee to discover the payers IP, but I think maybe it doesn’t matter.