I tried arguing with Google for a while however their 7 day deadline to admit publicly that I'm uploading the contact list is running out tomorrow so to not risk the app being removed again I was forced to publish a version without address book integration.

It seems like you can only have an app that requests contacts permission on the PlayStore if you boldly declare that you upload them.