Look into setting up Wireguard between your node and the wallet. This might shed more light: https://v2.minibolt.info/bonus-guides/system/wireguard-vpn