Right to travel does not equate to right to be accepted by the locals. Down to them refusing to sell you land and refusing to sell you building materials and food if one of their number does sell you land. In the absence of laws regulating discrimination that would happen some places.

Sincerely a local from a part of the world where if you moved here tomorrow your descendants might be considered locals in 100 years. Maybe.