Not sure if this is your issue, but I think a lot of implementations of silent payments in general is way too slow to sync. So what happens is if you have not synced for awhile it is ridiculous how long you have to wait to catch up. This has been my experience at least. 

 Yer it’s slow but I can’t even sync with one block. Or set the blockheight at only 2 or 3 blocks behind.