It's 100% that. I have the same problem but with trying to find beach's to surf where there aren't a bunch of fuck heads clogging up the water with no etiquette or decency. 

It is 100% about difficulty to reach/find the location.  

 I’d take two sides on that. On the one hand, the more remote, the more effort required, the more likely to attract a certain type of discerning traveler. I’ve certainly found places in French Polynesia and Thailand that fit the bill.  On the other hand, Trolltunga (Norway) is remote AF and insta-crowds ruined it, regardless.