another issue is probably your ram.  As UTXOs are swapped out to the disk that paging is slow.  The bigger the UTXO set the more paging you get.