Published on April 19, 2025 1:54 PM GMT
1. The Risks (and Motivation)
On the 25th March 2025, OpenAI quietly unveiled their latest image generation capabilities, built into GPT-4o. This model was made available to all users with their $20/month ‘Plus’ membership, meaning over 11 million users have access to this model.
The latest model is significantly more capable than the previous, notably in its ability to produce photorealistic images, and images including text (e.g. receipts and road signs).
This is a sobering advancement in the state-of-the-art, and I now believe that we are no further than 12 months from the release of AI models which can produce images that cannot be detected by humans or computers.
This technology will have enormous ramifications across the entire planet. Some examples of the damaging consequences of this technology are listed below:
1.1 Legal System Risks
While I have been unable to find a figure specifically for images, a study by Nottingham Trent University in 2017 found that CCTV proved useful in 65% of criminal cases. The possibility to automatically generate images of crimes will lead to one of two outcomes. The first is that the legal system (absurdly) continues to allow photographic evidence in the courtroom, and hundreds, thousands, or more, individuals are wrongfully convicted. The second is that the legal system no longer deems photographic evidence as valid, and individuals are found not guilty despite photographic evidence existing of their crimes.
1.2 Political and Social Manipulation
Society will see sweeping change as a result of this technology. While people who are technology-literate will quickly begin to disregard all photographic evidence as unreliable, those who are not informed (specifically older generations), will become victim to constant disinformation campaigns. These will affect the electoral system in every country, and will be extremely damaging to many individuals in the public eye.
1.3 Privacy and Consent Violations
As we have already seen, pornographic and compromising content involving anybody in the public eye will become increasingly common. This may not be restricted to public figures, as low-cost technology of this kind may enable this content to be generated by anybody, of anybody, with only a few images of the victim.
2. Approaches to Risk Mitigation
It is clear, given the risks outlined previously, that systems need to be developed to identify AI-generated images.
2.1 Steganographic Watermarking
Steganographic watermarks, hidden inside of images and invisible to the human eye, are currently the industry’s only attempt at a real solution. Google’s DeepMind has created the current state-of-the-art, called SynthID.
“It doesn’t compromise image or video quality, and allows the watermark to remain detectable — even after modifications like cropping, adding filters, changing colors, changing frame rates and saving with various lossy compression schemes”. – Google DeepMind
This kind of watermarking, at least for now, may be effective, but not every AI provider is doing it. OpenAI, for example, is still only watermarking their images with metadata which can easily be erased.
In the long-term, it is unclear to what extent this watermarking will continue to be effective. One can easily envision adversarial technologies being created for the express purpose of removing these watermarks. Google has stated that SynthID is resilient to cropping, compression, etc, but they have not publicly stated its resilience to malicious attack. In past experiments AI watermarking has proven to be flimsy at best.
2.2 The Alternative: An AI Image Database
So, what is the alternative? I am proposing an alternative strategy towards AI image identification which does not require a cat-and-mouse chase between researchers and adversaries.
A provider of an AI image generating tool could create a central portal, to which users can upload any image they deem suspicious. This portal would then perform a reverse-image-search on a database containing all images that the platform has ever generated. If a match is found, the image is confirmed to be AI generated.
This approach is preferable for a number of reasons:
- No cat-and-mouse chase: There will be no option for hackers to reverse-engineer and remove identifying features in post, as identifying features are not used. The image itself is the identifying feature. The beaten track: As can be seen with technologies like Google Lens, reverse-image-search technology is already very advanced. Even with heavy image alteration, algorithms have been devised which can still identify similarities with a source image.
This idea could be further improved if it were to be shared between providers. If OpenAI, Google, Anthropic, etc were to collaborate in the name of Safe AI, a central portal could be devised which checks all of their databases at once for matching images.
If organisations are not willing to collaborate, then forward-thinking governments could enforce this collaboration by devising their own central portal, and mandating cooperation with its developers. The EU is one such government which has already developed a positive track record for AI regulation, and represents a population large enough to be taken seriously by these companies.
3. What About Open-Source?
Clearly this solution would not work for open-source models, as these would not be linked to a single image lookup database. There are two potential courses of action for governments to mitigate open-source risk:
3.1 Mandating Training with Watermarks
It may be possible to train an image generation model to produce watermarks on all images natively (e.g. by training the model exclusively on images that have been watermarked). In this case, governments may be able to mandate that open-source models are permitted, but only if they have been trained in this way.
If watermarking can only be performed as a separate function, then it will be pointless for governments to mandate watermarking capabilities of open-source models, as bad actors will be able to simply remove the watermarking functionality on their own machines.
3.2 Outright Bans
This is an extreme precaution, but desperate times call for desperate measures. If the previous approach was unsuccessful, the only way to mitigate the risk of these models would be to ban open-source variants of AI image-generating models that are capable of generating photorealistic images.
Some may argue that it is unconstitutional or unrealistic to ban AI models, as they are essentially huge repositories of matrices, and therefore ‘just maths’. However, there is already a large variety of digital (“just maths”) files which are deemed illegal to own, including (but not limited to): Classified government documents, illegal numbers, and illegal pornography. Therefore, there is clearly already a legal precedent to enable this kind of regulation.
4. Conclusion
I believe that the vast majority of the public have not considered the vast and far-reaching implications of this technology, and neither have the government. I have outlined above some mitigations that would (thinking optimistically) greatly mitigate these effects, however I’m not optimistic that any action will be taken on this front until it is too late.
Discuss
