Even then, if the source was viewable but nobody looked, what difference would it make if it was closed source?

No-one is reviewing the whole Linux kernel + drivers + firmwares + microcode + etc. etc.

"Reflections on trusting trust - Ken Thompson" https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf 

 That it is provable if something is wrong or allegations are made. Simple as that.