Right and that's the thing - if you assign scores, a low score on its own isn't not a bad thing, but when people see that they have a low score, they think somehow that makes them less trustworthy. 

One of the points I'm trying to make is that we shouldn't use anything that gives people a negative perception of themselves or others. 

A simple "Followed by X and Y" (people you follow) is better than assigning scores. From there, you can decide if that person is worth following or is trustworthy.