I don't pay it much attention anymore. The people in my neighborhood know me. The kids at school and their parents don't seem to treat us any differently, but it's a very small school.
If we're out somewhere else, kids do stare and some want to come and say hello or whatever - their minds are probably just trying to come to grips with seeing someone non-Japanese in real life.
I felt way more stared at when I lived here 15 years ago, but perhaps I just don't care anymore? When I'm walking my dog at 6 in the morning and a car goes by I sometimes wonder if their brains are exploding. What is a random white dude doing walking a dog in the middle-of-nowhere Japan?
One of the absolute best parts of Japan is people just leave you the fuck alone to live your life. No one is offering you unsolicited advice. Instead they give you vegetables 😄