Besides the metadata leaks that come with being a federated protocol, Matrix keeps content indefinitely and uses "throwing away the key" as a bandage solution.
Privacy groups often discuss the dangers of services having a large amount of information about you, yet even Google and Facebook are better at removing content.
I can't even deletemy own account from my own Matrix instance.
AIUI, Matrix is not quantum-resistant, so if the encryption were to be broken, all retained information could potentially be accessed.
A common argument against this complaint is the inevitability of analog holes¹. We should not let the existence of analog holes compromise privacy.
[1] https://en.wikipedia.org/wiki/Analog_hole