I'd definitely agree. 
History of asylums is filled with the most heinous transgressions of human rights. 

I think I was reflecting (probably too simplistically) on the institutional mandates...

To care for (even though that was poorly or inhumanely executed) vs 
To punish. 

But I definitely take your point