San Francisco City Attorney David Chiu has formally demanded that Apple and Google remove "nudify" applications from their respective app stores, citing the proliferation of AI-generated non-consensual sexual imagery. In a letter sent to both tech giants, Chiu argued that these platforms facilitate the distribution of deepfake content, which disproportionately targets women and minors, and requested that the companies implement stricter enforcement policies to prevent such apps from reaching consumers.
San Francisco Legal Action Against App Store Policies
On October 23, 2024, San Francisco City Attorney David Chiu sent formal letters to Apple CEO Tim Cook and Google CEO Sundar Pichai. The communication explicitly identifies several applications available on the Apple App Store and Google Play Store that use artificial intelligence to strip clothing from images of individuals without their consent.
According to the San Francisco City Attorney’s Office, these applications violate the companies’ own developer guidelines regarding the promotion of non-consensual sexual content. Chiu’s office noted that while both companies have policies prohibiting the distribution of sexually explicit material, the presence of these "nudify" tools suggests a failure in the review process or a lack of proactive enforcement against AI-driven harms.
The Scope of AI-Generated Non-Consensual Imagery
The legal pressure from San Francisco follows a broader trend of legislative and regulatory bodies attempting to address the rise of deepfake technology. These AI applications function by processing user-uploaded photographs to generate hyper-realistic, nude versions of the subjects.
Research from The Brookings Institution highlights that the accessibility of these tools has lowered the barrier for digital harassment. Unlike traditional image manipulation, which required professional software skills, current "nudify" apps are designed for mass-market use, allowing individuals to generate harmful content with minimal technical expertise.
Comparison of Platform Responses
Both Apple and Google have faced criticism regarding their roles as gatekeepers for the digital ecosystem.
| Feature | Apple App Store | Google Play Store |
|---|---|---|
| Content Policy | Prohibits sexually explicit content | Prohibits non-consensual sexual content |
| Review Process | Human and automated moderation | Automated scanning with human oversight |
| Current Status | Reviewing legal demands | Reviewing legal demands |
While both companies maintain strict developer agreements, the TechCrunch report notes that the persistence of these apps on their platforms underscores the difficulty of moderating AI-generated content at scale. Apple and Google have historically relied on automated systems to flag prohibited content, but critics argue these systems are frequently bypassed by developers who mask the true functionality of their applications during the submission process.
Implications for Future Digital Regulation
The move by the San Francisco City Attorney serves as a warning that local governments are increasingly willing to hold platform operators accountable for the content hosted within their digital storefronts. By framing the presence of these apps as a violation of consumer safety, the city is shifting the focus from individual developers to the platforms that provide the infrastructure for distribution.
As of late October 2024, both Apple and Google have stated they are reviewing the concerns raised by the city. The outcome of this request may set a precedent for how tech platforms manage AI-powered tools that facilitate harassment and privacy violations in the future.
Related reading