The problem is not enjoying that a woman is beautiful. 
The problems are:

1. The idea that you are entitled to fantasize about a beautiful woman

2. The idea that women exist for the enjoyment of men