if you visit a web page and then turn off VPN and refresh these two pieces of info are trivially linked

you've revealed a lot of information that forms a fingerprint even without an IP address. Now the site provider can trivially link these two IP addresses.

what's more is that they're probably using a CDN like AWS CloudFront who also see this info and they've probably seen your browser fingerprint and VPN IP on other sites they host, so they know who you are as an entity and can just trace back your non-VPN IP address to your ISP that knows all your identity info.

best strategy here is to retry the site with another device, or wait a few days and try without VPN from the start