You say “few drugs” but I think natural drugs are ok, it’s when they’re synthesised, super processed and chemically enhanced that they become implicitly avoidable. 
Just a thought from a non-user libertarian 

 Many natural substances are very harmful 

 Natural cyanide is delicious tho 

 It really is tho 😋