The social media network X has sparked significant anger after people utilized its AI chatbot Grok to manipulate pictures of women by taking off their garments.
Several instances have been seen where the tool has been employed to digitally undress women to show them in swimwear without consent, as well as to place them in sexually explicit contexts.
Developer Response and Legal Position
XAI, the firm that created Grok, did not respond to requests for comment, except for an auto-reply message stating "legacy media lies".
A Home Office official stated that new laws is being proposed to ban nudification tools. Under a new law, anyone who provided such tools would "be subject to a prison sentence and significant penalties".
"X or Grok is capable of halting these types of abuse if they desired to," said a law professor, adding they "appear to enjoy a lack of consequences."
Regulator Position and Platform Capabilities
The communications watchdog Ofcom said tech firms must "evaluate the potential" of people viewing unlawful material on their platforms, but did not specify whether it was at this time examining X or Grok in regarding AI-generated pictures.
Grok is a free artificial intelligence—with some additional functions—which answers X users' requests when they tag it in a post.
It is frequently used to give reactions or further background to other posters' comments, but people on X are also can alter an posted picture through its AI image editing tool.
History of Criticism and Policy
The system has been condemned for allowing users to produce pictures and videos with nudity and sexually suggestive material, and it was earlier alleged to have been involved in generating inappropriate material of public figures.
"The platform has been enabling the generation and distribution of these pictures for a long time without taking any action and we have not seen any action by oversight bodies," Professor McGlynn added.
XAI's own terms of service bans "portraying images of individuals in a graphic way".
In a comment, Ofcom said it was unlawful to "make or distribute private pictures or illegal content" and confirmed this applied to sexual deepfakes generated with AI.
It said networks such as X were obligated to take "appropriate actions" to "reduce the danger" of users coming across unlawful content on their services and to take down it swiftly when they are notified of it.
