It stops them on the notifications tab, but clicking that button *should* turn on the “only notifications from friends” option in notification settings. It currently does not so you have to turn them on separately. We will fix that