The WoT relays of today calculate your WoT using follows, as you correctly point out. It’s very useful, but it’s imperfect for exactly the reasons you point out.

The path forward will be to calculate WoT using a more diverse range of raw data. Follows, mutes, reports, zaps, reactions, etc etc. Which may sound too complicated to work. But this is how the grapevine works. In principle, ANY source of data can be converted into a format that allows it to be consumed by the grapevine algorithm. Different data sources can be combined in different ways to come up with different GrapeRank scores which can be used for a variety of purposes. One score will be used to decide who can post to your WoT relay. Another score will be used to determine who you trust to curate a list of trusted ecash mints. And so on.