The short guys I know don't have too many issues in that regard (though I still mock them freely). If anything, I think the height just gets you more attention, like how a woman with huge tits will stand out in a crowd. 

 reason it's brought up is most minorities are shorter than white people. 

 Paradoxically midgets also stand out in a crowd. 

 Uoh flat chest