What did you do? I have an open PR to Alby to cache shared key generation, which helps a lot, but it sounds like you might have more going on.