Technically nothing prevents them from storing metadata on servers - only contents of messages are encrypted, same with Matrix. So, you have to trust Signal servers.
They also know your phone number 😅. 
Signal is so much celebrated because it has great UX, large userbase, battle-tested encryption (WhatsApp is based on the Signal protocol) and it was the first.