I'm not sure there's anything inherently "wrong" with them. More that the vast majority of people overconsume them, which upsets the balance of fatty acids within the body (too much omega 6 displacing omega 3), which in turn can create inflammation.

Another issue is that they have a low smoke point, which means if you try to cook with them, unlike the animal fats, they'll oxidise at a much lower temperature, (possibly forming AGEs), and again, help to drive inflammation in the body.

I would also question their ability to supply the same quantity and quality of the fat soluable soluble vitamins that the animal fats can supply. e.g. A, D, E & K. 

Personally, I think a little is OK. 
e.g. eating olives & olive oil, tahini, etc... (assuming it's not rancid, over heated, etc).

But just as our ancestors did for generations, we're likely better off getting the bulk of our calories (and fat soluble vitamins) from nutrient dense animal fats. 

[An interesting observation is to note the rise in cancer cases, over the same time period that the propaganda has been most aggressive, in terms of the demonisation of animal fats, in conjunction with the extent to which most cancer patients tend to flag as deficient in the fat soluable vitamins....]