You say “few drugs” but I think natural drugs are ok, it’s when they’re synthesised, super processed and chemically enhanced that they become implicitly avoidable. 
Just a thought from a non-user libertarian