Recent testing of OpenAI's ChatGPT model revealed an unexpected feature: it can unintentionally imitate users' voices. According to the system card, the AI model produced a voice that sounded similar to the user's in rare instances during Advanced Voice Mode conversations. This is not a new capability for GPT-4o, which can synthesize almost any type of sound found in its training data. OpenAI has safeguards in place to prevent this from happening and ensures authorized voice samples are used.

Source: https://arstechnica.com/information-technology/2024/08/chatgpt-unexpectedly-began-speaking-in-a-users-cloned-voice-during-testing/