Not quite. It is like casual alcoholic saying, "oh I don't get drunk" until his liver starts complaining and either motivates him to get straight, or he is a toast.
With milk, you "tolerate" it while only wearing man-boobs, and same on women but less visible due to unhealthy aesthetic standards for millenniums, until you get cancer, blame tobacco or your wonderful genes that tolerates milk, and usually toast yourself with chemo.
Well considering my age and my health this is not my situation. I always had milk and I still drink a big full cap of milk every day with some coffee. I'm under zero medication, never had a surgery, my teeth are perfect without a dentist and my daily schedule is 14 hours of work. My grampa retired at my age plus one. Btw I don't have a single white hair on my head and I eat a lot of meat and salad.
I am like you with age and all the health, except I don't eat milk, eat a little selected vegetables and fruits, eat mostly meat of animals with horns, wild fish, and eggs. But most important exceptions are:
I hardly ever work (neither physical nor brain)
I read much
I learn a bit
Listen to music much
Be with city and nature sounds outside much
I don't exercise other than run 5 km like twice or once a week
I mostly lie down, don't sit unless I drive or eat
I don't drive much
I eat once a day, no snacks
I get much sun on full body daily
I smoke good cigars (not cigarillos) daily
I drink much black coffee and green tea
Looks good. I lived in nature for the last 15 years and outside a little city the previous life. I keep young not watching TV or newspapers. I studied a lot but have more fun building and fixing stuff. Never smoked in my life. Beer and wine just because I like to cook and some specific food require a specific drink. I don't.crave alcohol. I've never been addicted to anything.
We have more in common than we are different.
I am glad for you still. Surely reasonable amounts of milk alone should not kill you. Unlike alcohol 😉