Ok, now I got it to work!
After I delete an application, but then send a remote signing request, I still get the amber notification, but when I click accept or reject, it says no relays found.