For me it's been lack of reliability in clients. Something is always not working, even though devs are at it 24/7 trying to make it work.  It just seems like relays introduce a level of complexity that makes everything more painful.