Totally. If someone changes their relay list, outbox doesn't know it existed. In a perfect world, NIP-65 would also store timestamps for when a relay was added. When removing the relay, clients user events from that timestamp. Or otherwise, relays are not "removed" from NIP-65 lists, but marked inactive and marked with a timestamp indicating when the user left.
pre-coffee notes are imperfect.
post-coffee
*clients would sync events from that timestamp to other relays.
*but marked inactive with a timestamp indicating when the user left.