DEEPFAKE DANGER: OpenAI Warns AI-Generated Images and Videos May Soon Be Impossible to Detect as Federal Courts Confront Evidence Crisis
The rapid advancement of artificial intelligence could soon make it virtually impossible to determine whether photographs and videos are genuine or computer-generated, representatives of OpenAI and Meta warned federal judges Thursday, raising serious questions about the future reliability of visual evidence in American courtrooms.
The warnings were delivered during a meeting in Boston of the U.S. Judicial Conference’s Advisory Committee on Evidence Rules, which is examining whether existing courtroom procedures are sufficient to address increasingly sophisticated AI-generated fabrications, commonly known as deepfakes.
The committee has been studying the issue since 2023, amid growing concern that artificial intelligence could allow litigants, witnesses or other individuals to introduce fabricated photographs, recordings and videos that appear authentic. The technology is advancing so quickly that even experts may struggle to distinguish genuine material from convincing digital imitations.
Shira Anderson, policy counsel for OpenAI, told the judges that advances in image-generation technology could eventually eliminate the possibility of reliably identifying AI-created content through scientific analysis alone.
“AI images are only getting better, and I do not think that there will be a scientific way going forward to detect them,” Anderson said.
Meta offered a similarly troubling assessment, warning that the newest artificial intelligence systems can already produce visual material that ordinary viewers cannot distinguish from authentic photographs or recordings.
Devin Norder, Meta’s director of product management, said the sophistication of the technology has reached a point where human observation can no longer be counted on to establish whether a piece of media is genuine.
“Media generated from frontier models, at least in the last few generations, are indistinguishable by humans from authentic or AI-generated media,” Norder said.
The ease with which such material can now be manufactured makes the problem even more significant. Creating realistic fake images and videos once required specialized technical expertise and substantial resources, but modern AI systems have dramatically reduced those barriers.
“And the cost and the time it takes to create this media really are quite trivial at this point,” Norder added.
Both OpenAI and Meta have developed technologies intended to help identify or authenticate digital material. Such systems can include embedded information indicating how an image was produced, where it originated or whether artificial intelligence was involved in its creation.
Governments have also begun responding to the problem. California and the European Union have adopted requirements involving the labeling or watermarking of certain AI-generated content, reflecting growing concern that fabricated digital media could be used to mislead the public or interfere with legal proceedings.
However, Anderson cautioned that authentication systems have significant limitations, particularly because different technology companies use different methods that may not be compatible with one another.
Even as detection and verification tools improve, she suggested, no available approach is likely to provide an absolute guarantee that fabricated content will always be identified.
“My guess is that this absolutely will continue to get better. I do not think it’ll ever be bulletproof,” Anderson said.
The warnings have created a difficult question for federal judges: Should the courts establish special evidentiary rules for artificial intelligence before deepfakes become a widespread problem, or should they continue relying on existing procedures for challenging questionable evidence?
Despite the alarming predictions from technology companies, the committee has found relatively few documented instances in which deepfake allegations have become an issue in federal court proceedings.
A survey released in March by the Federal Judicial Center, the research and education arm of the federal judiciary, questioned 931 judges about their experiences with potentially fabricated AI evidence.
Only 15 respondents reported having presided over a case in which a party challenged evidence on the grounds that it was a deepfake.
U.S. District Judge Jesse Furman of Manhattan, who chairs the advisory committee, pointed to the difference between the widespread concern surrounding the technology and the limited number of cases in which the issue has actually arisen.
“There’s a lot of academic commentary and other commentary predicting that courts will be dealing with deepfakes, but we haven’t actually seen a whole lot of that yet,” Furman said.
The relatively small number of reported cases has contributed to disagreement among committee members about whether a new rule specifically addressing AI-generated evidence is necessary.
Some participants believe the technology presents sufficiently unusual challenges to justify additional protections, particularly because highly realistic fabrications could be difficult for judges, jurors and attorneys to recognize.
Others maintain that courts already possess procedures for evaluating authenticity, examining witnesses, challenging unreliable evidence and imposing penalties when parties attempt to deceive the judicial system.
U.S. District Judge Valerie Caproni of Manhattan argued that the fundamental problem of fabricated evidence is hardly new and that existing legal safeguards may be capable of addressing AI-generated material without creating an entirely separate set of rules.
“We’ve dealt with bad actors presenting fake evidence since Cain and Abel. That’s a longstanding problem,” Caproni said.
The issue extends far beyond courtrooms. Artificial intelligence is also providing criminals with increasingly powerful tools for impersonation, deception and financial fraud, allowing scammers to create convincing simulations of real people using photographs, videos and recordings obtained online.
Wynton Hall, author of the book Code Red: The Left, the Right, China, and the Race to Control AI, has warned that fraudsters can exploit information available on social media to build detailed profiles of potential victims before using AI to impersonate people they know and trust.
During an appearance on Newsmax, Hall described how criminals can combine personal information with sophisticated voice and image-generation technology to create highly persuasive scams.
“They can now mine your social media and then they build a detailed dossier on your life, and then they use the AI to mimic a loved one so convincingly that your instincts will tell you that it’s real,” the author said.
The growing availability of these capabilities means that criminals no longer need substantial financial resources or advanced programming skills to deploy them.
“These tools are now becoming democratized so that they’re cheap, they’re easy to use, and that means they’re accessible at a mass scale,” Hall added.
Federal law enforcement statistics illustrate the financial consequences of AI-assisted fraud. According to the FBI, scams involving artificial intelligence generated more than 22,000 complaints during 2025, with reported losses approaching $893 million.
The committee has also considered a possible amendment to Federal Rule of Evidence 901, which governs how parties establish that evidence is authentic before it can be admitted in court.
Under a proposal discussed by the judiciary, a party challenging a photograph, video or recording as an AI-generated fabrication would first need to present a credible basis for suspecting that the material was falsified. If that threshold were met, the party seeking to introduce the evidence could face a greater burden of demonstrating its authenticity.
The proposal remains under consideration, and the committee has not reached a final decision on whether an AI-specific rule should be adopted.
The Federal Judicial Center’s research also illustrates the difficulty facing judges. Although only 15 of the 931 surveyed judges reported encountering challenges involving alleged deepfakes, the survey was designed to assess both their experiences and their views on whether existing evidentiary standards are adequate.
The judiciary has acknowledged that the speed of AI development creates an additional complication: A rule designed to address the technology as it exists today could become outdated before the lengthy federal rulemaking process is completed.
For now, federal courts continue to rely on existing authentication requirements while the advisory committee examines whether additional protections are necessary. The committee has assigned further study of the issue as judges weigh the risk of increasingly convincing fabricated evidence against the limited number of documented courtroom incidents.
