Turn that around. How much readers wouldn't dare interact with an account with a large following? Maybe afraid to say something 'wrong'. Or think their interactions are not worth it or will be snowed under in the replies? Would they participate more if these stats were hidden? Maybe... I don't see growing a follower count as 'progress'. I personally never look at it and don't care.
This is indeed a good argument for the other side. However, I still prefer to have this metric visible. I do care about follower count, since I plan to go write long form content soon, and it's nice to know that you write for a big (er) audience. There is a good chance that the interaction will be very low in the beginning. In that case a growing follower count will be a motivator. Happily this decision will be made per client, so we will probably both get what we want!